mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-09-03 04:00:18 +00:00
Compare commits
34
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ce4e68b465 | ||
|
|
25d2ea5fe9 | ||
|
|
3c68cd12ad | ||
|
|
798a7455e4 | ||
|
|
bd584b7402 | ||
|
|
a4500f5b20 | ||
|
|
bdd12e8ac6 | ||
|
|
48204d4e02 | ||
|
|
1bc873cbbd | ||
|
|
4dd88375e1 | ||
|
|
fc899c6966 | ||
|
|
98bd632585 | ||
|
|
30a91a3a78 | ||
|
|
23126106a3 | ||
|
|
4b001b4c9d | ||
|
|
6c9eb2296d | ||
|
|
1ad17beaf6 | ||
|
|
110f6deb1e | ||
|
|
b8299b1427 | ||
|
|
bbd423c50a | ||
|
|
6b36379f15 | ||
|
|
af829f5f20 | ||
|
|
930e7f9b71 | ||
|
|
e335971dcd | ||
|
|
78682076d5 | ||
|
|
1227947be5 | ||
|
|
b4a14d87f5 | ||
|
|
3bf89e523f | ||
|
|
2b5b62bb8d | ||
|
|
3a0f3f672a | ||
|
|
bd1ba24b24 | ||
|
|
e8ff36f088 | ||
|
|
ec9e63e16f | ||
|
|
274d5d1195 |
BIN
Binary file not shown.
@@ -11,6 +11,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- **Salesforce ingestor** (#1240) by @Sameer6305
|
||||
- New `SalesforceConnector` / `SalesforceData` / `SalesforceIngestor` (`semantica.ingest`, lazy export), following the same Connector + Data + Ingestor pattern already used for Snowflake/Databricks/SAP
|
||||
- Auth covers both landscapes Salesforce actually uses: username + password + security token (SOAP login), session_id + instance_url (reusing an existing session), and username + consumer_key + private key (JWT Bearer); production and sandbox are selected via `domain`, and credentials can come from environment variables. Credential material is never intentionally written to logs, exceptions, or `repr()`
|
||||
- `ingest_sobject()`, `ingest_query()`, `list_sobjects()`, `get_sobject_schema()`, `export_as_documents()` against standard sObjects, custom objects (`__c`), custom metadata (`__mdt`), platform events (`__e`), namespaced objects, and relationship-field traversal (e.g. `Owner.Name`); pagination follows `nextRecordsUrl`/`query_more()` and stops once a caller's `limit` is satisfied
|
||||
- New `pip install semantica[db-salesforce]` extra (`simple-salesforce>=1.12.0`)
|
||||
- New `tests/test_salesforce_ingestor.py`
|
||||
- Docs: `docs/integrations/salesforce.md`
|
||||
|
||||
- **`ErasureCoordinator` completes the erasure workflow `purge_node()` only starts — the graph node was removed while the same content survived verbatim in `AgentMemory` and as an embedding** (closes #1018) by @pravit-amp
|
||||
- New `semantica/context/erasure.py`, exporting `ErasureCoordinator` and `ErasureReceipt` from `semantica.context`. `purge_node()`/`purge_edge()` (#957) are graph-scope by design and their changelog entry documents this gap explicitly; the changelog also names GDPR Article 17 as the motivation, and an Article 17 erasure that removes the node while the content stays retrievable by similarity search is not an erasure — it is worse than not offering one, because `purge_node()` returns `True` and writes a tombstone attesting the content is gone
|
||||
- The coordinator **composes** the existing public APIs — nothing in `context_graph.py` or `agent_memory.py` changes behaviorally, and `ContextGraph` keeps its documented graph-scope contract rather than acquiring references to `AgentMemory`/`vector_store` that would invert the dependency
|
||||
@@ -151,6 +159,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **Also fixed, on the JSON-LD paths**: the first fix covered the Turtle, N-Triples and RDF/XML serializers, and left both JSON-LD writers interpolating the entity's own text into `f"semantica:entity/{text}"` and the endpoints into `f"semantica:rel/{source}_{target}"`. Three consequences, all live in 0.6.5: an entity whose text contained a space produced an invalid IRI, and a JSON-LD parser dropped that node in full rather than reporting it, so the entity disappeared from the export; every relationship carrying `source`/`target` rather than `source_id`/`target_id` minted the identical `semantica:rel/_`, collapsing all of them onto one node whose types and endpoints merged; and the JSON-LD `@id` disagreed with the Turtle IRI for the same entity, so the two serializations of one knowledge graph were two different graphs. Both JSON-LD writers now use `mint_entity_iri`/`mint_relationship_iri`, and `JSONExporter.export_entities`/`export_relationships` declare the `semantica` prefix their `@context` was already writing `semantica:entities` against — without it a processor reads that as an IRI in the scheme `semantica`, which is the original #1101 defect on a third path
|
||||
- `tests/export/test_jsonld_iri_minting.py` parses each export with a real JSON-LD processor and asserts the entity survives, the relationships stay distinct, no term expands into the `semantica` scheme, and the JSON-LD `@id` equals the Turtle IRI
|
||||
- 236 export and ontology tests pass
|
||||
- **`semantica.evals` runner gains per-metric objectives** (#1091)
|
||||
- `evaluate()` now accepts `config={"<evaluator>": {"objective": {"direction": "maximize"|"minimize", "threshold": X}}}` to override the evaluator's default pass verdict with a threshold; `{"objective": {"expect": bool}}` expresses a Boolean expectation
|
||||
- `minimize` requires a `threshold` — omitting it or setting it to `None` raises `ValueError`; `maximize` without a threshold is a no-op (the evaluator's own verdict stands); `expect` cannot be combined with `direction`/`threshold`; invalid config raises `ValueError` before any evaluator runs
|
||||
- Error metrics are never affected by objectives (error wins over fail)
|
||||
- Backward compatible: no `objective` key → existing behavior unchanged
|
||||
- New tests in `tests/evals/test_runner.py::TestObjective`
|
||||
- **`semantica.evals` is now a fully implemented evaluation module** (was a "Coming Soon" stub in the package layout)
|
||||
- `evaluate(cases, evaluators, config=None, target_fn=None)` runner with per-case `pass`/`fail`/`error` status and an aggregate `pass_rate`, using a registry of named evaluators (`list_evaluators()`)
|
||||
- 10 built-in evaluators: `exact_match`, `regex_match`, `numeric_range`, `temporal_range`, `length_range`, `keyword_check`, `levenshtein` (edit-distance similarity), `rouge` (in-house token F1, no new dependencies), `llm_as_judge` (lazy: caller-supplied `judge_fn`), and `decision_scores` (composite over `semantica.context.Decision`)
|
||||
- `decision_scores` validates field-level (expected outcome, confidence bounds, non-empty maker/reasoning/scenario) and governance-level (provenance record presence; opt-in `PolicyEngine.check_compliance`) checks, coercing dict inputs via `Decision(**actual)` and never crashing on malformed input; an interface slot for causal-chain/embedding checks is reserved and raises `NotImplementedError` (V2)
|
||||
- `__version__` is `0.1.0`, and the module ships a usage guide at `semantica/evals/usage.md` with worked import/run/interpret examples
|
||||
- `semantica.evals` is reachable through the root package lazy module proxy (`semantica.evals`)
|
||||
- 99 unit tests in `tests/evals/` covering every evaluator, registry errors, runner aggregation, decision coercion, and per-metric objectives; `python -m pytest tests/evals -q` → 99 passed
|
||||
|
||||
- **First-class CrewAI integration** (#988, closes #962) by @Shindevrp
|
||||
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
|
||||
**Context Management · Knowledge Modeling · Deterministic Reasoning · Ontology Management · Decision Intelligence · End-to-End Traceability**
|
||||
|
||||
**Open Source · Self-Hostable · Auditable · Governed · Zero Vendor Lock-In**
|
||||
**Open Source · Governed · Zero Vendor Lock-In**
|
||||
|
||||
**Polyglot Graph Storage · RDF & LPG Support · W3C Standards · Interoperable**
|
||||
|
||||
@@ -62,12 +62,12 @@ Most AI agents run on embeddings, not meaning: similarity scores with no structu
|
||||
|
||||
**Who it's for:**
|
||||
|
||||
- **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index
|
||||
- **Data platform teams on Databricks or Snowflake** who need to turn tables already sitting in Unity Catalog or a Snowflake warehouse into a governed, lineage-tracked knowledge graph, without exporting that data to a third-party SaaS first
|
||||
- **Compliance, risk, and audit teams** who need a straight answer to "why did the AI do that?" in a format a regulator will actually accept
|
||||
- **Regulated enterprises** (finance, healthcare, legal, government, defense) that can't ship a black box, and can't send their data to someone else's SaaS to get one
|
||||
- **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context, not just a vector index
|
||||
- **Data platform teams on Databricks or Snowflake** turning tables already in Unity Catalog or a warehouse into a governed, lineage-tracked knowledge graph, without exporting to a third-party SaaS
|
||||
- **Compliance, risk, and audit teams** who need a straight answer to "why did the AI do that?" in a format a regulator accepts
|
||||
- **Regulated enterprises** (finance, healthcare, legal, government, defense) that can't ship a black box or send their data to someone else's SaaS to get one
|
||||
- **Platform and infra engineers** who want the KG, reasoning, and provenance stack self-hosted and swappable, not locked to one vendor's backend
|
||||
- **Data and knowledge engineers** building a KG from messy, multi-source data: entities and relationships get extracted, conflicting or contradictory facts are flagged instead of silently overwritten, and duplicates are merged before they turn into noise
|
||||
- **Data and knowledge engineers** building a KG from messy, multi-source data, where conflicting facts get flagged and duplicates get merged, not silently overwritten
|
||||
|
||||
**[Quick Start](#quick-start)** · **[Architecture](#architecture)** · **[What You Get](#what-semantica-gives-you)** · **[Why Semantica](#why-semantica)** · **[Decision Intelligence](#decision-intelligence)** · **[Context Graphs](#context-graphs)** · **[Recipe: Audit Trail](#recipe-audit-trail-for-a-regulated-decision)** · **[Module Reference](#module-reference)** · **[Integrations](#integrations)** · **[CLI](#cli)** · **[Performance](#performance)** · **[Install](#installation)**
|
||||
|
||||
@@ -81,7 +81,7 @@ Most AI agents run on embeddings, not meaning: similarity scores with no structu
|
||||
- **Full Auditability:** W3C PROV-O provenance on every fact, with audit trails exportable to JSON, CSV, or RDF
|
||||
- **Deterministic Reasoning:** Forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes
|
||||
- **Knowledge Pipeline:** Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and knowledge graph construction, with semantic deduplication and provenance-preserving merges throughout
|
||||
- **Enterprise Data Platforms:** Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection) and Snowflake (warehouse/database/schema, key-pair and OAuth auth), so tables already living in your lakehouse or warehouse become graph nodes with provenance, not another export/import hop
|
||||
- **Enterprise Data Platforms:** Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection), Snowflake (warehouse/database/schema, key-pair and OAuth auth), and SAP OData (Business Partners, Sales Orders, OAuth2/Basic auth), so data already living in your lakehouse or warehouse becomes graph nodes with provenance, not another export/import hop
|
||||
- **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
|
||||
- **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
|
||||
- **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
|
||||
@@ -139,10 +139,6 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
|
||||
|
||||
```bash
|
||||
semantica doctor
|
||||
# Python 3.11.9 pass
|
||||
# semantica 0.6.7 pass
|
||||
# faiss vector store pass
|
||||
# Config file pass ~/.semantica/config.yaml
|
||||
```
|
||||
|
||||
**Running in a script or CI?** Progress bars are written only when stdout is an interactive terminal (or a Jupyter notebook), so piping and redirecting stay clean by default. Override with `SEMANTICA_DISABLE_PROGRESS=1` to silence progress everywhere, or `SEMANTICA_FORCE_PROGRESS=1` to keep it when stdout is redirected. `SEMANTICA_DISABLE_PROGRESS` takes precedence.
|
||||
@@ -167,7 +163,7 @@ Sources → Ingest → Parse → Normalize → Split → Extract → Conflict De
|
||||
→ Vector Store + Polyglot Graph Store (RDF & LPG) → Export / Visualize / REST · MCP · CLI
|
||||
```
|
||||
|
||||
- **Ingest:** files, web, databases, enterprise data platforms (Databricks, Snowflake), cloud (Google Drive, Elasticsearch), streams (Kafka, Kinesis), Git, email, MCP
|
||||
- **Ingest:** files, web, databases, enterprise data platforms (Databricks, Snowflake, SAP), cloud (Google Drive, Elasticsearch), streams (Kafka, Kinesis), Git, email, MCP
|
||||
- **Parse → Normalize → Split:** document parsing, text/entity/date normalization, GraphRAG-native entity-aware chunking
|
||||
- **Extract → Conflict Detection → Deduplication:** NER, relations, events, triplets; conflicting facts flagged and resolved before they merge
|
||||
- **Knowledge Graph:** `GraphBuilder` constructs the graph; bi-temporal facts and full graph analytics (centrality, communities, link prediction) run on top of it
|
||||
@@ -320,7 +316,7 @@ Every module below is independently importable, with working code samples verifi
|
||||
|
||||
| Module | What it does |
|
||||
| --- | --- |
|
||||
| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Databricks, Snowflake, MCP |
|
||||
| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Databricks, Snowflake, SAP, MCP |
|
||||
| [`semantica.semantic_extract`](#semanticasemantic_extract-ner-relations-events-triplets) | NER, relation extraction, event detection, triplet generation |
|
||||
| [`semantica.kg`](#semanticakg-knowledge-graph-construction--analysis) | Graph construction, centrality, communities, link prediction |
|
||||
| [`semantica.reasoning`](#semanticareasoning-forward-chaining-rete-datalog-sparql) | Forward chaining, Rete, Datalog, SPARQL, fully explainable |
|
||||
@@ -349,7 +345,7 @@ Expand any module below for its runnable example.
|
||||
<summary><b><code>semantica.ingest</code></b>: Multi-Source Ingestion</summary>
|
||||
<a id="semanticaingest-multi-source-ingestion"></a>
|
||||
|
||||
Ingest from files, web, databases, APIs, streams, email, Git repos, Parquet, Databricks, Snowflake, or MCP servers, all through a unified interface.
|
||||
Ingest from files, web, databases, APIs, streams, email, Git repos, Parquet, Databricks, Snowflake, SAP, or MCP servers, all through a unified interface.
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor, WebIngestor, ParquetIngestor, DBIngestor
|
||||
@@ -400,7 +396,7 @@ orders = snowflake.ingest_table("ORDERS", limit=10_000)
|
||||
|
||||
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
|
||||
|
||||
**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Databricks (Unity Catalog + Delta Lake) · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`)
|
||||
**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Databricks (Unity Catalog + Delta Lake) · Snowflake · SAP (OData v2/v4) · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`)
|
||||
|
||||
DuckDB, Elasticsearch, Google Drive, HuggingFace, MongoDB, and Pandas ingestion also ship (`DuckDBIngestor`, `ElasticIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MongoIngestor`, `PandasIngestor`) but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly: `from semantica.ingest.duckdb_ingestor import DuckDBIngestor`.
|
||||
|
||||
@@ -1145,7 +1141,7 @@ if report.valid:
|
||||
| **Vector Store** | FAISS · Pinecone · Weaviate · Qdrant · Milvus · PgVector · hybrid + filtered search |
|
||||
| **Graph Databases (LPG)** | Neo4j · FalkorDB · Apache AGE · AWS Neptune |
|
||||
| **Triple Stores (RDF)** | Oxigraph (embedded) · Blazegraph · Apache Jena · Eclipse RDF4J · unified `TripletStore` interface · SPARQL query & bulk load |
|
||||
| **Enterprise Data Platforms** | Databricks (`DatabricksIngestor`: Unity Catalog + Delta Lake, PAT/OAuth M2M, table/query ingestion, catalog/schema/table/lineage introspection) · Snowflake (`SnowflakeIngestor`: warehouse/database/schema, password/key-pair/OAuth auth) |
|
||||
| **Enterprise Data Platforms** | Databricks (`DatabricksIngestor`: Unity Catalog + Delta Lake, PAT/OAuth M2M, table/query ingestion, catalog/schema/table/lineage introspection) · Snowflake (`SnowflakeIngestor`: warehouse/database/schema, password/key-pair/OAuth auth) · SAP (`SAPIngestor`: OData v2/v4, OAuth2/Basic auth, Business Partners/Sales Orders) |
|
||||
| **LLM Providers** | **All already supported today:** OpenAI (GPT-4o, o1, o3) · Anthropic (Claude) · Google Gemini · Mistral · Meta Llama · Groq · Cohere · Azure OpenAI · AWS Bedrock · Ollama · DeepSeek · Perplexity · Together AI · Fireworks AI · Replicate · HuggingFace · via `semantica.llms` and LiteLLM |
|
||||
|
||||
---
|
||||
@@ -1517,6 +1513,7 @@ pip install semantica[vectorstore-qdrant] # Qdrant vector store
|
||||
pip install semantica[vectorstore-pinecone] # Pinecone vector store
|
||||
pip install semantica[db-snowflake] # Snowflake
|
||||
pip install semantica[db-databricks] # Databricks (SDK + SQL connector)
|
||||
pip install semantica[ingest-sap] # SAP OData
|
||||
pip install semantica[ingest-parquet] # Parquet / PyArrow
|
||||
pip install semantica[ingest-arrow] # Apache Arrow, Feather, IPC
|
||||
pip install semantica[viz] # HTML interactive visualization
|
||||
|
||||
@@ -226,7 +226,7 @@ Pick your goal to see the minimum imports and a working skeleton.
|
||||
</Tab>
|
||||
|
||||
<Tab title="MCP — Claude / Cursor">
|
||||
Use Semantica from Claude Desktop, Cursor, VS Code, or any MCP-aware tool — no Python code required after setup. 12 tools available instantly.
|
||||
Use Semantica from Claude Desktop, Cursor, VS Code, or any MCP-aware tool — no Python code required after setup. 15 tools available instantly.
|
||||
|
||||
**Step 1 — Install:**
|
||||
```bash
|
||||
|
||||
+2
-2
@@ -53,7 +53,7 @@ python -c "import semantica; print(semantica.__version__)"
|
||||
- **semantica-server** — Starts the REST API server. Binds to `0.0.0.0:8000`. Use this when another service or application needs programmatic access to Semantica over HTTP.
|
||||
- **semantica-worker** — Background task processor. Run alongside `semantica-server` when you need async pipeline execution outside the request cycle. Start the server first, then start one or more workers pointing at the same backend.
|
||||
- **semantica-explorer** — Launches the browser dashboard. Requires `pip install semantica[explorer]`. Use this to explore a saved knowledge graph interactively. See [Explorer Setup](explorer-setup).
|
||||
- **semantica-mcp** — Runs the MCP server over stdio. Configure it in your MCP client's settings file to expose all 12 tools and 3 resources to Claude Desktop, Cursor, Windsurf, or any MCP-aware client. See [MCP Server](reference/mcp_server).
|
||||
- **semantica-mcp** — Runs the MCP server over stdio. Configure it in your MCP client's settings file to expose all 15 tools and 3 resources to Claude Desktop, Cursor, Windsurf, or any MCP-aware client. See [MCP Server](reference/mcp_server).
|
||||
|
||||
|
||||
## Usage Examples
|
||||
@@ -229,6 +229,6 @@ Install the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/
|
||||
## Next Steps
|
||||
|
||||
- [Explorer Setup](explorer-setup) — Build a graph, save it, and launch the browser dashboard.
|
||||
- [MCP Server](reference/mcp_server) — All 12 tools and 3 resources exposed over the MCP protocol.
|
||||
- [MCP Server](reference/mcp_server) — All 15 tools and 3 resources exposed over the MCP protocol.
|
||||
- [Installation](installation) — Virtual environments, optional extras, and platform-specific notes.
|
||||
- [Quickstart](quickstart) — End-to-end pipeline walkthrough with working code.
|
||||
|
||||
+6
-6
@@ -16,7 +16,7 @@ icon: "circle-question"
|
||||
| Python version? | 3.8+ (3.11+ recommended) |
|
||||
| API key required? | Optional: pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| Production-ready? | Yes: 1,000+ tests, security fixes shipped in every release (see [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md)) |
|
||||
| Latest version? | **v0.6.7** (August 2026) |
|
||||
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
|
||||
@@ -70,9 +70,9 @@ Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities r
|
||||
|
||||
<Accordion title="What's the latest version?" icon="star">
|
||||
|
||||
**v0.5.0**: released May 2026.
|
||||
**v0.6.7**: released August 2026.
|
||||
|
||||
Highlights: Ontology Hub, Distance Intelligence, Parquet/XML ingestion, 12 security fixes, Graph Explorer redesign, NER gateway fix.
|
||||
Highlights: first-class LangChain integration, SAP OData ingestor, human-editable Markdown round-trip persistence for `ContextGraph`, a structured Action layer for the reasoning engine, and a public `run_shacl_validation` entry point. The 0.6.x line also added first-class CrewAI support and the Semantica RDF vocabulary with deterministic IRIs. See the [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md) for the full history.
|
||||
|
||||
```bash
|
||||
pip install --upgrade semantica
|
||||
@@ -173,7 +173,7 @@ This includes PyTorch with CUDA, FAISS GPU, and CuPy.
|
||||
<Accordion title="How does Semantica handle large datasets?" icon="layer-group">
|
||||
|
||||
- **Batching**: process documents in configurable chunks to control memory usage
|
||||
- **Parallel processing**: `Pipeline(workers=N)` runs extraction steps concurrently
|
||||
- **Parallel processing**: `PipelineBuilder().set_parallelism(N)` runs independent pipeline steps concurrently
|
||||
- **Delta processing**: update graphs incrementally without full recompute on new data
|
||||
- **Persistent backends**: swap in-memory NetworkX for Neo4j, FalkorDB, or Apache AGE for large-scale production graphs
|
||||
|
||||
@@ -269,13 +269,13 @@ Groq, OpenAI, Anthropic, Google Gemini, Ollama (fully local), DeepSeek, Novita A
|
||||
|
||||
<Accordion title="Is Semantica production-ready?" icon="shield-check">
|
||||
|
||||
Yes. v0.5.0 ships with:
|
||||
Yes. Every release ships with:
|
||||
|
||||
- 1,000+ passing tests across Python 3.8–3.12
|
||||
- `PipelineValidator` and `FailureHandler` with exponential backoff and configurable retry policies
|
||||
- W3C PROV-O provenance tracking across all modules
|
||||
- Change management with SHA-256 checksums and full audit trails
|
||||
- 12 security vulnerability fixes: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, path traversal, and more
|
||||
- Ongoing security hardening: eval injection, pickle deserialization, SQL injection, XXE, SSRF, ReDoS, and path traversal fixes have all landed across recent releases (see the [CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md) security sections)
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
@@ -183,7 +183,7 @@ icon: "rocket"
|
||||
}
|
||||
```
|
||||
|
||||
12 tools available instantly: extract entities, query graph, record decisions, run reasoning, export results.
|
||||
15 tools available instantly: extract entities, query graph, record decisions, run reasoning, export results.
|
||||
|
||||
**Next:** [MCP Server reference →](reference/mcp_server)
|
||||
</Tab>
|
||||
|
||||
@@ -11,7 +11,7 @@ MCP stands for the Model Context Protocol. It is an open standard that allows ex
|
||||
The Semantica MCP server exposes your knowledge graph as 12 callable tools. By connecting it, any compatible AI client can traverse the graph live, record decisions, run analytics, and export results during a conversation — without you having to write custom tool wrappers.
|
||||
|
||||
<Info>
|
||||
The Semantica MCP server exposes 12 tools and 3 read-only resources. All tools accept and return JSON. No configuration beyond an optional environment variable for graph persistence is required.
|
||||
The Semantica MCP server exposes 15 tools and 3 read-only resources. All tools accept and return JSON. No configuration beyond an optional environment variable for graph persistence is required.
|
||||
</Info>
|
||||
|
||||
## Architecture & Communication
|
||||
@@ -132,7 +132,7 @@ docker run --rm -i \
|
||||
ghcr.io/semantica-agi/semantica-mcp:latest
|
||||
```
|
||||
|
||||
## What the Agent Can Do: The 12 Tools
|
||||
## What the Agent Can Do: The 15 Tools
|
||||
|
||||
Once connected, the LLM can call any of these tools during a conversation. The agent chains them automatically — you do not orchestrate the sequence, you just describe what you want.
|
||||
|
||||
@@ -140,6 +140,8 @@ Once connected, the LLM can call any of these tools during a conversation. The a
|
||||
|
||||
**Knowledge graph manipulation** — `add_entity` adds a node, `add_relationship` adds a directed edge. After extraction, the agent calls these to persist what it found into the live graph.
|
||||
|
||||
**Live graph queries and edits** — `query_graph` reads the graph without exporting it: fetch one node, walk its neighbours up to five hops, or keyword-search nodes. `update_node` merges properties onto an existing node (for example marking a task node `done`), and `delete_node` archives a node it no longer tracks. When `SEMANTICA_KG_PATH` is set, `update_node` and `delete_node` write their changes back to that file so they survive a restart.
|
||||
|
||||
**Decision intelligence** — `record_decision` writes a decision as a provenance node with confidence score, reasoning, and decision maker identity. `query_decisions` retrieves past decisions by query or category. `find_precedents` finds the most similar past decisions by semantic similarity. `get_causal_chain` traces decision causality upstream or downstream.
|
||||
|
||||
**Reasoning** — `run_reasoning` applies forward-chaining IF/THEN rules over a set of facts and returns derived conclusions.
|
||||
|
||||
+2
-2
@@ -369,7 +369,7 @@ Semantica was designed for domains where every decision must be explainable and
|
||||
| `semantica.reasoning` | Forward chaining, Rete, deductive, abductive, SPARQL, Datalog |
|
||||
| `semantica.ontology` | SHACL, SKOS, alignments, diff/migration, auto-generation, OWL/RDF |
|
||||
| `semantica.explorer` | FastAPI Knowledge Explorer, Ontology Hub, Distance Intelligence, SHACL Studio |
|
||||
| `semantica.mcp_server` | MCP stdio server: 12 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline |
|
||||
| `semantica.mcp_server` | MCP stdio server: 15 tools for Claude Desktop, VS Code, Cursor, Windsurf, Cline |
|
||||
| `semantica.vector_store` | FAISS, Pinecone, Weaviate, Qdrant, Milvus, PgVector |
|
||||
| `semantica.graph_store` | Neo4j, FalkorDB, Apache AGE, Amazon Neptune |
|
||||
| `semantica.triplet_store` | In-memory and persistent RDF triple store with SPARQL |
|
||||
@@ -404,7 +404,7 @@ Semantica was designed for domains where every decision must be explainable and
|
||||
- 1,000+ passing tests with full regression coverage
|
||||
- `PipelineValidator` catches configuration errors at startup
|
||||
- `FailureHandler` with exponential backoff and dead-letter queues
|
||||
- 12 security vulnerabilities fixed in v0.5.0
|
||||
- Ongoing security hardening: fixes shipped in every release ([CHANGELOG](https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md))
|
||||
|
||||
**Modular by Design** — Import only what you need.
|
||||
- Use `NERExtractor` without a graph store
|
||||
|
||||
+1
-1
@@ -438,7 +438,7 @@ Exposes Semantica as an MCP stdio server for IDE and agent integrations.
|
||||
python -m semantica.mcp_server
|
||||
```
|
||||
|
||||
**Integrations:** Claude Desktop, VS Code, Cursor, Windsurf, Cline: 12 MCP tools exposed
|
||||
**Integrations:** Claude Desktop, VS Code, Cursor, Windsurf, Cline: 15 MCP tools exposed
|
||||
|
||||
### Seed
|
||||
|
||||
|
||||
+69
-43
@@ -5,7 +5,7 @@ icon: "rocket"
|
||||
---
|
||||
|
||||
<Info>
|
||||
**v0.5.0** — Ontology Hub, Distance Intelligence, Parquet & XML ingestion, 12 security fixes. <a href="https://github.com/semantica-agi/semantica/releases" style={{color:"#10B981",fontWeight:600,textDecoration:"none"}}>What's new →</a>
|
||||
**v0.6.7** — first-class LangChain integration, SAP OData ingestor, human-editable Markdown persistence for `ContextGraph`, and a structured Action layer for the reasoning engine. <a href="https://github.com/semantica-agi/semantica/releases" style={{color:"#10B981",fontWeight:600,textDecoration:"none"}}>What's new →</a>
|
||||
</Info>
|
||||
|
||||
This guide walks you through the end-to-end pipeline for building your first knowledge graph. Start here after installation. An LLM API key is optional: pattern-based extraction works out of the box.
|
||||
@@ -35,7 +35,7 @@ Verify:
|
||||
|
||||
```bash
|
||||
python -c "import semantica; print(semantica.__version__)"
|
||||
# 0.5.0
|
||||
# 0.6.7
|
||||
```
|
||||
|
||||
|
||||
@@ -62,18 +62,19 @@ sources = ingestor.ingest("data/report.pdf")
|
||||
```python Web
|
||||
from semantica.ingest import WebIngestor
|
||||
|
||||
ingestor = WebIngestor(max_depth=2)
|
||||
sources = ingestor.ingest("https://example.com/article")
|
||||
ingestor = WebIngestor()
|
||||
page = ingestor.ingest_url("https://example.com/article")
|
||||
# WebContent: page.text, page.title, page.html, page.links, page.metadata
|
||||
```
|
||||
|
||||
```python Parquet / XML (v0.5.0)
|
||||
```python Parquet / XML
|
||||
from semantica.ingest import ParquetIngestor, XMLIngestor
|
||||
|
||||
# Single file or Hive-partitioned directory
|
||||
sources = ParquetIngestor().ingest("data/events.parquet")
|
||||
|
||||
# XML with XSD schema validation
|
||||
sources = XMLIngestor(validate_xsd="schema.xsd").ingest("data/records/")
|
||||
# XML; pass an XSD to validate against during ingestion
|
||||
sources = XMLIngestor().ingest("data/records/", schema_path="schema.xsd")
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
@@ -88,22 +89,24 @@ Extract structured text and layout from raw documents.
|
||||
from semantica.parse import DocumentParser
|
||||
|
||||
parser = DocumentParser()
|
||||
parsed = parser.parse(sources[0])
|
||||
parsed = parser.parse(sources[0].path) # parse() takes a path string
|
||||
|
||||
print(parsed.text[:200]) # extracted text
|
||||
print(parsed.metadata) # title, author, date, source
|
||||
print(parsed["text"][:200]) # extracted text
|
||||
print(parsed["metadata"]) # file_path, encoding, size, and format-specific keys
|
||||
```
|
||||
|
||||
`parse()` returns a `dict` with `text`, `full_text`, and `metadata` keys.
|
||||
|
||||
<Tip>
|
||||
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser`: it applies advanced layout analysis and returns structured table data alongside text.
|
||||
For PDFs with tables, charts, or multi-column layouts, use `DoclingParser` (`pip install semantica[parse-docling]`): it applies advanced layout analysis and returns structured table data alongside text.
|
||||
</Tip>
|
||||
|
||||
```python
|
||||
from semantica.parse import DoclingParser
|
||||
|
||||
parser = DoclingParser()
|
||||
parsed = parser.parse(sources[0])
|
||||
print(parsed.tables) # structured table objects
|
||||
parsed = parser.parse(sources[0].path)
|
||||
print(parsed["tables"]) # structured table data
|
||||
```
|
||||
|
||||
</Step>
|
||||
@@ -117,26 +120,28 @@ Identify named entities and extract typed relationships between them.
|
||||
```python Pattern-based (fast, no API key)
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
|
||||
ner = NERExtractor(method="pattern")
|
||||
entities = ner.extract(parsed)
|
||||
# Returns: [{"text": "Apple Inc.", "type": "ORGANIZATION", "confidence": 0.98}, ...]
|
||||
text = parsed["text"]
|
||||
|
||||
rel = RelationExtractor(method="rule")
|
||||
relationships = rel.extract(parsed, entities=entities)
|
||||
# Returns: [{"subject": "Steve Jobs", "predicate": "founded", "object": "Apple Inc."}, ...]
|
||||
ner = NERExtractor(method="pattern")
|
||||
entities = ner.extract(text)
|
||||
# Returns: [Entity(text="Apple Inc.", label="ORG", start_char=0, end_char=10, confidence=0.7), ...]
|
||||
|
||||
rel = RelationExtractor(method="pattern")
|
||||
relationships = rel.extract(text, entities=entities)
|
||||
# Returns: [Relation(subject=Entity(...), predicate="founded_by", object=Entity(...), confidence=0.7), ...]
|
||||
```
|
||||
|
||||
```python LLM-powered (higher accuracy)
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.llms import Groq
|
||||
|
||||
llm = Groq(model="llama-3.3-70b-versatile")
|
||||
# Reads GROQ_API_KEY from the environment; provider/llm_model select the backend
|
||||
text = parsed["text"]
|
||||
|
||||
ner = NERExtractor(method="llm", llm_provider=llm)
|
||||
entities = ner.extract(parsed)
|
||||
ner = NERExtractor(method="llm", provider="groq", llm_model="llama-3.3-70b-versatile")
|
||||
entities = ner.extract(text)
|
||||
|
||||
rel = RelationExtractor(method="llm", llm_provider=llm)
|
||||
relationships = rel.extract(parsed, entities=entities)
|
||||
rel = RelationExtractor(method="llm", provider="groq", llm_model="llama-3.3-70b-versatile")
|
||||
relationships = rel.extract(text, entities=entities)
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
@@ -198,16 +203,17 @@ exporter.export(graph, file_path="graph.nt", format="nt")
|
||||
from semantica.export import ParquetExporter
|
||||
|
||||
exporter = ParquetExporter()
|
||||
exporter.export(graph, file_path="output/graph.parquet")
|
||||
# Writes nodes.parquet + edges.parquet: ready for Spark, BigQuery, Databricks
|
||||
exporter.export(graph, file_path="output/graph")
|
||||
# Dict input writes one file per key: output/graph_entities.parquet and
|
||||
# output/graph_relationships.parquet: ready for Spark, BigQuery, Databricks
|
||||
```
|
||||
|
||||
```python ArangoDB
|
||||
from semantica.export import ArangoAQLExporter
|
||||
|
||||
exporter = ArangoAQLExporter()
|
||||
aql = exporter.export(graph)
|
||||
# Returns ready-to-run AQL INSERT statements
|
||||
exporter.export(graph, file_path="graph.aql")
|
||||
# Writes ready-to-run AQL INSERT statements to graph.aql
|
||||
```
|
||||
|
||||
</CodeGroup>
|
||||
@@ -272,14 +278,21 @@ relationships = rel.extract(text, entities=entities)
|
||||
<Accordion title="Multi-source incremental graph build" icon="layer-group">
|
||||
|
||||
```python
|
||||
from semantica.ingest import FileIngestor
|
||||
from semantica.parse import DocumentParser
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor
|
||||
from semantica.kg import GraphBuilder
|
||||
|
||||
builder = GraphBuilder(merge_entities=True)
|
||||
all_entities, all_rels = [], []
|
||||
parser = DocumentParser()
|
||||
ner = NERExtractor(method="pattern")
|
||||
rel = RelationExtractor(method="pattern")
|
||||
builder = GraphBuilder(merge_entities=True)
|
||||
|
||||
for doc in parsed_docs:
|
||||
entities = ner.extract(doc)
|
||||
rels = rel.extract(doc, entities=entities)
|
||||
all_entities, all_rels = [], []
|
||||
for source in FileIngestor().ingest("data/reports/"):
|
||||
text = parser.parse(source.path)["text"]
|
||||
entities = ner.extract(text)
|
||||
rels = rel.extract(text, entities=entities)
|
||||
all_entities.extend(entities)
|
||||
all_rels.extend(rels)
|
||||
|
||||
@@ -359,7 +372,8 @@ graph = builder.build({"entities": entities, "relationships": relationships})
|
||||
# Retrieve full lineage for any entity
|
||||
sources = prov.get_all_sources("Apple Inc.")
|
||||
print(sources[0])
|
||||
# {"source": "data/report.pdf", "location": None, "timestamp": "...", "confidence": 0.98}
|
||||
# {"source": "data/report.pdf", "location": None, "timestamp": "...",
|
||||
# "confidence": 1.0, "metadata": {"confidence": 0.98}}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -373,30 +387,42 @@ print(sources[0])
|
||||
|
||||
<Accordion title="No entities extracted" icon="magnifying-glass">
|
||||
|
||||
The document likely contains scanned images rather than machine-readable text. Enable OCR:
|
||||
The document likely contains scanned images rather than machine-readable text. `DocumentParser` warns when a PDF has no text layer; switch to `DoclingParser` with OCR enabled:
|
||||
|
||||
```python
|
||||
from semantica.parse import DocumentParser
|
||||
from semantica.parse import DoclingParser # pip install semantica[parse-docling]
|
||||
|
||||
parser = DocumentParser(ocr=True) # enables Tesseract OCR
|
||||
parsed = parser.parse(sources[0])
|
||||
parser = DoclingParser(enable_ocr=True)
|
||||
parsed = parser.parse(sources[0].path)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Slow processing on large corpora" icon="gauge">
|
||||
|
||||
Enable parallel processing and GPU acceleration:
|
||||
Enable GPU acceleration and run pipeline steps in parallel:
|
||||
|
||||
```bash
|
||||
pip install semantica[gpu]
|
||||
```
|
||||
|
||||
```python
|
||||
from semantica.pipeline import Pipeline
|
||||
from semantica.pipeline import PipelineBuilder, ExecutionEngine
|
||||
|
||||
pipeline = Pipeline(workers=8, batch_size=32)
|
||||
pipeline.run(sources)
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("ingest", step_type="ingest", source="data/reports/", recursive=True)
|
||||
builder.add_step("extract", step_type="ner_extract")
|
||||
builder.add_step("build", step_type="kg_build", merge_entities=True)
|
||||
|
||||
pipeline = (
|
||||
builder
|
||||
.connect_steps("ingest", "extract")
|
||||
.connect_steps("extract", "build")
|
||||
.set_parallelism(8)
|
||||
.build(name="reports_pipeline")
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -6,7 +6,7 @@ icon: "plug"
|
||||
|
||||
**`semantica.mcp_server`** exposes Semantica's knowledge graph, decision intelligence, semantic extraction, and reasoning capabilities as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) **server over stdio**:
|
||||
|
||||
- 12 MCP tools exposed: extract entities, query graph, record decisions, run reasoning, export results
|
||||
- 15 MCP tools exposed: extract entities, query graph, record decisions, run reasoning, export results
|
||||
- No Python code required after launch: configure once, use from any MCP-aware client
|
||||
- Compatible with Claude Desktop, Windsurf, Cline, Continue, VS Code, Roo Code, Cursor
|
||||
|
||||
@@ -40,7 +40,7 @@ python -m semantica.mcp_server
|
||||
|
||||
## What You Get
|
||||
|
||||
- **12 MCP Tools** — Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export graph.
|
||||
- **15 MCP Tools** — Extract entities, extract relations, record decisions, query decisions, find precedents, trace causal chains, add entities, add relationships, run analytics, summarise graph, run reasoning, export graph, query the live graph, update nodes, archive nodes.
|
||||
- **3 Readable Resources** — Live graph JSON (`semantica://graph/summary`), decision list, and schema/version info: readable by any MCP client.
|
||||
- **Zero Infrastructure** — Runs over stdio: no server, no port, no Docker required. One config block to activate in any MCP client.
|
||||
- **Persistent Graphs** — Point `SEMANTICA_KG_PATH` at a saved graph file to reload it automatically on every server startup.
|
||||
@@ -159,7 +159,7 @@ The MCP server is included in the base install: no extras required.
|
||||
|
||||
## Tools
|
||||
|
||||
The MCP server exposes 12 tools that any connected AI assistant can call:
|
||||
The MCP server exposes 15 tools that any connected AI assistant can call:
|
||||
|
||||
| Tool | Category | Description |
|
||||
| :---- | :-------- | :----------- |
|
||||
@@ -173,6 +173,9 @@ The MCP server exposes 12 tools that any connected AI assistant can call:
|
||||
| `add_relationship` | Graph Operations | Add a directed edge between two nodes |
|
||||
| `get_graph_summary` | Graph Operations | Node count, decision count, graph status |
|
||||
| `get_graph_analytics` | Graph Operations | PageRank centrality and community detection |
|
||||
| `query_graph` | Graph Operations | Fetch a node, traverse its neighbours, or keyword-search nodes |
|
||||
| `update_node` | Graph Operations | Merge properties onto a node and persist to `SEMANTICA_KG_PATH` |
|
||||
| `delete_node` | Graph Operations | Soft-delete (archive) a node and persist to `SEMANTICA_KG_PATH` |
|
||||
| `run_reasoning` | Reasoning | Forward-chain IF/THEN rules over facts |
|
||||
| `export_graph` | Reasoning & Export | Serialise the graph (`turtle`/`ttl`: RDF Turtle aliases, `nt`, `xml`, `json-ld`, `json`) |
|
||||
|
||||
@@ -386,6 +389,51 @@ Takes no input parameters.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="query_graph" icon="magnifying-glass">
|
||||
|
||||
Read the live graph in one of three modes, set by `mode`:
|
||||
|
||||
- `node` — return a single node by `node_id`.
|
||||
- `neighbors` (default) — traverse outward and inward from `node_id` up to `depth` hops (clamped to 1-5, default 1). Optional `relationship_types` filters edge types; optional `limit` caps results.
|
||||
- `search` — keyword match `query` against each node's id and content. Optional `node_type` restricts the scan; `limit` defaults to 50.
|
||||
|
||||
**Input:**
|
||||
|
||||
```json
|
||||
{ "mode": "neighbors", "node_id": "apple_inc", "depth": 2 }
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="update_node" icon="pen">
|
||||
|
||||
Merge a set of properties onto an existing node. The change is applied in memory and, when `SEMANTICA_KG_PATH` is set, written back to that file so it survives a restart. Returns `persisted: false` when no path is configured.
|
||||
|
||||
**Input:**
|
||||
|
||||
```json
|
||||
{
|
||||
"node_id": "task_42",
|
||||
"properties": { "status": "done", "note": "shipped in v0.6.7" }
|
||||
}
|
||||
```
|
||||
|
||||
`node_id` and a non-empty `properties` object are required. Updating a missing node returns an error.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="delete_node" icon="box-archive">
|
||||
|
||||
Soft-delete a node: it stays in the graph for history but is marked `status: "archived"`. Persists to `SEMANTICA_KG_PATH` when configured.
|
||||
|
||||
**Input:**
|
||||
|
||||
```json
|
||||
{ "node_id": "task_42" }
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
</AccordionGroup>
|
||||
|
||||
### Reasoning
|
||||
|
||||
@@ -0,0 +1,338 @@
|
||||
# Objective Layer for semantica.evals Runner — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add per-metric objective support (direction + threshold, or Boolean expectation) to the `evaluate()` runner, overriding evaluator default pass verdicts, backward-compatible when no objective is configured.
|
||||
|
||||
**Architecture:** The runner already iterates evaluators and computes per-case status. Objectives are read from `config["<name>"]["objective"]`, validated up front, and applied to each returned metric's `passed` field (and `details`) before aggregation. Error metrics always win over objectives.
|
||||
|
||||
**Tech Stack:** Python 3.8+, stdlib only (typing, dataclasses). pytest for tests.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Python >= 3.8: use `typing.Dict/List/Optional/Union`, never builtin generics or `|`.
|
||||
- Zero new dependencies.
|
||||
- Do not change the `EvalMetric` shape, the `evaluate()` signature, or the evaluator function signature.
|
||||
- Existing behavior with no `objective` configured must be byte-for-byte unchanged (all 62 existing tests keep passing).
|
||||
- Error metrics (`meta` contains `"error"`) always classify the case as `error`, regardless of objective.
|
||||
- Config errors are programmer errors: raise `ValueError` from `evaluate()` before any evaluator runs (fail-fast).
|
||||
- Tests go in `tests/evals/`, pytest class style, no new files outside the listed paths.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Objective parsing, validation, and re-decision in the runner
|
||||
|
||||
**Files:**
|
||||
- Modify: `semantica/evals/runner.py`
|
||||
- Test: `tests/evals/test_runner.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `EvalMetric` from `.types` (fields: `score`, `passed`, `meta`); `evaluate(cases, evaluators, config=None, target_fn=None)` existing signature.
|
||||
- Produces: private helpers `_parse_objective(name, eval_config) -> Optional[Dict]` (returns `None` when no objective configured, raises `ValueError` on invalid config) and `_apply_objective(metric, objective) -> bool` (returns the re-decided `passed`). Public `evaluate()` behavior extended as specified.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Append a new test class to `tests/evals/test_runner.py`:
|
||||
|
||||
```python
|
||||
class TestObjective:
|
||||
def test_maximize_with_threshold_pass(self):
|
||||
# levenshtein similarity 1.0 for identical, objective demands >= 0.5
|
||||
result = evaluate(
|
||||
[("apple", "apple")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.5}}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is True
|
||||
|
||||
def test_maximize_with_threshold_fail(self):
|
||||
result = evaluate(
|
||||
[("apple", "aple")], # similarity < 1.0
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.99}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is False
|
||||
assert "levenshtein" in result.cases[0].details
|
||||
|
||||
def test_minimize_with_threshold_pass(self):
|
||||
# edit distance normalized ~0.2; objective: distance <= 0.5
|
||||
result = evaluate(
|
||||
[("night", "nacht")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is True
|
||||
|
||||
def test_minimize_with_threshold_fail(self):
|
||||
result = evaluate(
|
||||
[("night", "nacht")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.1}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
def test_expect_true_on_boolean_metric(self):
|
||||
result = evaluate(
|
||||
[("ok", "ok")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"expect": True}}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
|
||||
def test_expect_false_overrides_passing_metric(self):
|
||||
# exact_match passes (score 1.0) but expectation is false -> fail
|
||||
result = evaluate(
|
||||
[("ok", "ok")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"expect": False}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
assert result.cases[0].metrics["exact_match"].passed is False
|
||||
assert "exact_match" in result.cases[0].details
|
||||
|
||||
def test_maximize_without_threshold_is_noop(self):
|
||||
# identical behavior to no objective: evaluator's own verdict stands
|
||||
result = evaluate(
|
||||
[("ok", "no")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"direction": "maximize"}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
def test_minimize_without_threshold_raises(self):
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize"}}},
|
||||
)
|
||||
|
||||
def test_bad_direction_raises(self):
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "sideways", "threshold": 0.5}}},
|
||||
)
|
||||
|
||||
def test_expect_with_direction_raises(self):
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"expect": True, "direction": "maximize"}}},
|
||||
)
|
||||
|
||||
def test_error_metric_wins_over_objective(self):
|
||||
result = evaluate(
|
||||
[("[invalid", "x")],
|
||||
evaluators=["regex_match"],
|
||||
config={"regex_match": {"objective": {"direction": "maximize", "threshold": 0.0}}},
|
||||
)
|
||||
assert result.cases[0].status == "error"
|
||||
assert result.errors == 1
|
||||
assert result.failed == 0
|
||||
|
||||
def test_no_objective_unchanged(self):
|
||||
result = evaluate([("ok", "no")], evaluators=["exact_match"])
|
||||
assert result.cases[0].status == "fail"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `python3 -m pytest tests/evals/test_runner.py -q`
|
||||
Expected: the new `TestObjective` tests fail (objective config ignored → `exact_match` passes under `expect:false` etc.); the pre-existing tests in the file still pass.
|
||||
|
||||
- [ ] **Step 3: Implement objective parsing, validation, and re-decision**
|
||||
|
||||
In `semantica/evals/runner.py`, add two helpers before `evaluate` and wire them into the evaluator loop.
|
||||
|
||||
```python
|
||||
def _parse_objective(name, eval_config):
|
||||
"""Return the validated objective dict, or None when not configured.
|
||||
|
||||
Raises ValueError for invalid configurations (programmer error).
|
||||
"""
|
||||
objective = (eval_config or {}).get("objective")
|
||||
if objective is None:
|
||||
return None
|
||||
direction = objective.get("direction")
|
||||
threshold = objective.get("threshold")
|
||||
expect = objective.get("expect")
|
||||
|
||||
if expect is not None:
|
||||
if direction is not None or threshold is not None:
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'expect' cannot be combined with "
|
||||
"'direction' or 'threshold'"
|
||||
)
|
||||
return {"expect": bool(expect)}
|
||||
if direction == "minimize":
|
||||
if threshold is None:
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'minimize' requires a 'threshold'"
|
||||
)
|
||||
return {"direction": "minimize", "threshold": float(threshold)}
|
||||
if direction == "maximize":
|
||||
if threshold is None:
|
||||
# no bar to re-decide against; treat as absent (evaluator default stands)
|
||||
return None
|
||||
return {"direction": "maximize", "threshold": float(threshold)}
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'direction' must be 'maximize' or 'minimize' "
|
||||
f"(got {direction!r})"
|
||||
)
|
||||
|
||||
|
||||
def _apply_objective(metric, objective):
|
||||
"""Return the objective-adjusted pass verdict for a non-error metric."""
|
||||
if "expect" in objective:
|
||||
return bool(metric.score) == objective["expect"]
|
||||
if objective["direction"] == "minimize":
|
||||
return metric.score <= objective["threshold"]
|
||||
return metric.score >= objective["threshold"]
|
||||
```
|
||||
|
||||
Then modify the evaluator loop in `evaluate()` so the parsed objective is computed once per case (outside the evaluator loop, since it only depends on merged config), and applied inside the loop:
|
||||
|
||||
```python
|
||||
objective_by_name = {
|
||||
name: _parse_objective(name, merged.get(name) or {})
|
||||
for name in evaluators
|
||||
}
|
||||
metrics: Dict[str, EvalMetric] = {}
|
||||
details: Dict[str, Any] = {}
|
||||
failed, errored = False, False
|
||||
for name in evaluators:
|
||||
eval_config = merged.get(name) or {}
|
||||
try:
|
||||
metric = get_evaluator(name)(actual, expected, config=eval_config)
|
||||
objective = objective_by_name.get(name)
|
||||
if objective is not None and "error" not in metric.meta:
|
||||
metric = EvalMetric(metric.score, _apply_objective(metric, objective), metric.meta)
|
||||
metrics[name] = metric
|
||||
if "error" in metric.meta:
|
||||
errored = True
|
||||
details[name] = metric.meta
|
||||
elif not metric.passed:
|
||||
failed = True
|
||||
details[name] = metric.meta
|
||||
except Exception as exc: # noqa: BLE001
|
||||
errored = True
|
||||
metrics[name] = EvalMetric(0.0, False, {"error": str(exc)})
|
||||
details[name] = {"error": str(exc)}
|
||||
```
|
||||
|
||||
Note: `objective_by_name` is computed once per case (it depends only on merged config), so invalid config raises `ValueError` at the first case — satisfying the fail-fast requirement. `EvalMetric` is a frozen dataclass, so the re-verdict constructs a new instance preserving score/meta.
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `python3 -m pytest tests/evals/test_runner.py -q`
|
||||
Expected: all `TestObjective` tests pass; pre-existing tests still pass.
|
||||
|
||||
- [ ] **Step 5: Run the full evals suite**
|
||||
|
||||
Run: `python3 -m pytest tests/evals -q`
|
||||
Expected: 62 existing + new tests all pass (no regressions).
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add semantica/evals/runner.py tests/evals/test_runner.py
|
||||
git commit -m "feat(evals): add per-metric objective support to runner"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Documentation — usage.md and CHANGELOG
|
||||
|
||||
**Files:**
|
||||
- Modify: `semantica/evals/usage.md`
|
||||
- Modify: `CHANGELOG.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the objective config surface implemented in Task 1 (exact keys: `objective.direction`, `objective.threshold`, `objective.expect`; validation rules).
|
||||
- Produces: docs only.
|
||||
|
||||
- [ ] **Step 1: Add objective section to usage.md**
|
||||
|
||||
Append a section after the existing "Run the runner over decision records" section:
|
||||
|
||||
```markdown
|
||||
## Set per-evaluator objectives
|
||||
|
||||
By default each evaluator decides its own pass/fail. To override that
|
||||
verdict at the run level, configure an **objective** per evaluator name:
|
||||
|
||||
```python
|
||||
from semantica.evals import evaluate
|
||||
|
||||
# Require a minimum similarity (default direction is maximize):
|
||||
evaluate(
|
||||
[("apple", "aple")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.7}}},
|
||||
)
|
||||
|
||||
# Lower is better — override the direction:
|
||||
evaluate(
|
||||
[("night", "nacht")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}},
|
||||
)
|
||||
|
||||
# Boolean expectation on a 0/1 metric:
|
||||
evaluate(
|
||||
[("ok", "ok")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"expect": False}}},
|
||||
)
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `maximize` + `threshold`: pass iff `score >= threshold`. `maximize` without
|
||||
a threshold is a no-op (the evaluator's own verdict stands).
|
||||
- `minimize` + `threshold`: pass iff `score <= threshold`. `minimize`
|
||||
**requires** a threshold — omitting it raises `ValueError`.
|
||||
- `expect` (`true`/`false`): pass iff `bool(score)` matches; cannot be
|
||||
combined with `direction`/`threshold`.
|
||||
- A metric whose `meta` contains `"error"` is always an error, never affected
|
||||
by an objective.
|
||||
- Invalid objective config raises `ValueError` before any evaluator runs.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add CHANGELOG entry**
|
||||
|
||||
Under `## [Unreleased]` → `### Added`, insert a new bullet at the top (before the `semantica.evals` module entry), following existing style:
|
||||
|
||||
```markdown
|
||||
- **`semantica.evals` runner gains per-metric objectives** (#1091)
|
||||
- `evaluate()` now accepts `config={"<evaluator>": {"objective": {"direction": "maximize"|"minimize", "threshold": X}}}` to override the evaluator's default pass verdict with a threshold; `{"objective": {"expect": bool}}` expresses a Boolean expectation
|
||||
- `minimize` requires a `threshold`; `maximize` without one is a no-op; `expect` cannot be combined with `direction`/`threshold`; invalid config raises `ValueError` before any evaluator runs
|
||||
- Error metrics are never affected by objectives (error wins over fail)
|
||||
- Backward compatible: no `objective` key → existing behavior unchanged
|
||||
- New tests in `tests/evals/test_runner.py::TestObjective`
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify docs examples run**
|
||||
|
||||
Run the three examples from Step 1 as a Python script (import `evaluate`, run each snippet) to confirm they don't raise unexpectedly. No test output assertion needed beyond "no exception" and sensible status values.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add semantica/evals/usage.md CHANGELOG.md
|
||||
git commit -m "docs(evals): document per-metric objectives"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- **Spec coverage:** §3.1 (config surface) → Task 1 helpers + Task 2 docs; §3.2 (semantics: maximize/minimize/expect) → Task 1 `_apply_objective`; §3.3 (error wins) → Task 1 error branch + `test_error_metric_wins_over_objective`; §3.4 rules 1-3 (validation) → Task 1 `_parse_objective` + 4 validation tests; §3.4 rule 4 → error branch; §3.5 (aggregation unchanged, details on final verdict) → Task 1 loop + `test_expect_false_overrides_passing_metric` asserts `details`; §4 (fail-fast ValueError) → `_parse_objective` at case top; §5 (tests) → Task 1 test class; §6 (compat) → `test_no_objective_unchanged` + full-suite green.
|
||||
- **Type consistency:** `_parse_objective(name, eval_config) -> Optional[Dict]`, `_apply_objective(metric, objective) -> bool`; `EvalMetric(score, passed, meta)` positional construction preserved everywhere.
|
||||
- **Backward compat:** objective parsed to `None` for absent config → loop behavior identical to before.
|
||||
@@ -0,0 +1,115 @@
|
||||
# Design: Objective layer for `semantica.evals` runner
|
||||
|
||||
**Date:** 2026-08-19
|
||||
**Issue:** semantica-agi/semantica#1091 (assigned to pkupt)
|
||||
**Base:** PR #1090 (`semantica.evals` module)
|
||||
|
||||
## 1. Problem
|
||||
|
||||
`semantica.evals` runs named evaluators and aggregates per-case pass/fail, but the pass judgement is hard-coded inside each evaluator — a higher score always means "better". There is no way to express an evaluation objective at the run level:
|
||||
|
||||
- apply a threshold the evaluator does not encode (e.g. "F1 must be ≥ 0.7");
|
||||
- reverse the direction (e.g. "lower edit distance is better");
|
||||
- express a Boolean expectation (e.g. "this metric should be `false`").
|
||||
|
||||
This blocks the domain-specific benchmark harnesses `docs/community-projects.md` says `semantica.evals` supports. Palantir AIP Evals models exactly this: each metric has an **objective** (Boolean expected value, or numeric `maximize`/`minimize` direction with an optional threshold), and a test case passes when **all** its metrics meet their objectives.
|
||||
|
||||
## 2. Scope
|
||||
|
||||
In scope:
|
||||
|
||||
- A per-metric objective configuration consumed by the `evaluate()` runner.
|
||||
- Runner-level pass/fail re-decision for numeric scores and Boolean metrics.
|
||||
- Backward-compatible behavior when no objective is configured.
|
||||
- Tests and docs.
|
||||
|
||||
Out of scope:
|
||||
|
||||
- Changing the evaluator signature or the `EvalMetric` shape.
|
||||
- Multi-iteration test cases (AIP Evals has them; Semantica's runner is single-iteration per case).
|
||||
- Objective-aware aggregation beyond per-case `pass`/`fail` (existing `pass_rate` semantics are kept).
|
||||
|
||||
## 3. Design
|
||||
|
||||
### 3.1 Configuration surface
|
||||
|
||||
Objective is configured per evaluator inside the runner's `config`, under the evaluator name:
|
||||
|
||||
```python
|
||||
config = {
|
||||
"<evaluator_name>": {
|
||||
"objective": {
|
||||
"direction": "maximize" | "minimize",
|
||||
"threshold": <float>, # optional
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Boolean-form objective (shorthand): for metrics whose score is Boolean-like (0.0/1.0) or for semantic clarity, `{"objective": {"expect": true}}` / `{"objective": {"expect": false}}` is also supported.
|
||||
|
||||
### 3.2 Evaluation semantics
|
||||
|
||||
For each metric produced by an evaluator during a case run, if an objective exists for that evaluator name, the runner recomputes the metric's pass verdict:
|
||||
|
||||
- **maximize**: pass iff `score >= threshold`. If no `threshold` is given, the objective is treated as absent (evaluator's own verdict stands) — see 3.4 rule 2.
|
||||
- **minimize**: pass iff `score <= threshold` (threshold required, see 3.4 rule 1).
|
||||
- **expect**: pass iff `bool(score)` equals `expect` (for Boolean-style metrics).
|
||||
|
||||
When an objective is present, the runner **overrides** `metric.passed` with the objective verdict. When absent, `metric.passed` is used unchanged (existing behavior).
|
||||
|
||||
The `objective` key is a **reserved runner-level key**: it is consumed by the runner and is passed through to the evaluator function inside `eval_config` (evaluators already ignore unknown config keys via `cfg.get(...)`, so this is harmless); evaluators must not rely on it. The runner re-decision happens on the metric the evaluator returns, so no evaluator change is required.
|
||||
|
||||
### 3.3 Interaction with errors
|
||||
|
||||
An `EvalMetric` whose `meta` contains `"error"` remains classified as an error regardless of objective (error wins over fail, per the existing contract). Objectives only affect non-error metrics.
|
||||
|
||||
### 3.4 Ambiguity rules (explicit decisions)
|
||||
|
||||
1. **`minimize` without `threshold`** is rejected at config-validation time with a clear error (`ValueError`), because "lowest is best" has no absolute pass bar without a threshold. (AIP Evals allows direction-only; we require threshold to keep pass/fail well-defined.) — *Chosen for determinism; revisit if a use case demands direction-only minimize.*
|
||||
2. **`maximize` without `threshold`** behaves like no objective (pass iff evaluator's own `passed`), because the evaluator's default is already "higher is better".
|
||||
3. **`expect` with a numeric `direction`/`threshold`** is a config error (`ValueError`): pick one form.
|
||||
4. **Objective on a metric that errors** → the error wins (3.3), objective ignored.
|
||||
|
||||
### 3.5 Aggregation
|
||||
|
||||
Unchanged:
|
||||
|
||||
- Case `status`: `"error"` if any metric errored, else `"fail"` if any failed, else `"pass"`.
|
||||
- `pass_rate` = passed / total (1.0 on empty).
|
||||
- `metrics` dict holds the (possibly re-verdict'd) `EvalMetric`; the re-verdict is observable via `metric.passed`.
|
||||
- `details[name]` is populated when a metric ends up failed **after** objective re-decision (i.e. objective-failed metrics appear in `details`; metrics that pass under objective are not recorded there). This mirrors the existing "record failures in details" behavior applied to the final verdict.
|
||||
|
||||
### 3.6 Files
|
||||
|
||||
- `semantica/evals/runner.py` — add objective parsing/validation and re-decision inside the evaluator loop.
|
||||
- `tests/evals/test_runner.py` — new test class(es) for objective semantics.
|
||||
- `semantica/evals/usage.md` — document the objective config and examples.
|
||||
- `CHANGELOG.md` — `[Unreleased]` entry.
|
||||
|
||||
No new dependencies; Python ≥ 3.8 (stdlib `typing`).
|
||||
|
||||
## 4. Error handling
|
||||
|
||||
- Invalid objective config (`direction` not in {maximize, minimize}, both `expect` and `direction`, `minimize` without threshold, non-numeric threshold) → `ValueError` raised at runner config parse, before any evaluator runs. Deterministic, fail-fast.
|
||||
- These are programmer errors, not per-case data errors — no per-case `error` status involved.
|
||||
|
||||
## 5. Testing
|
||||
|
||||
New tests in `tests/evals/test_runner.py`:
|
||||
|
||||
1. maximize + threshold: score ≥ threshold → pass; below → fail.
|
||||
2. minimize + threshold: score ≤ threshold → pass; above → fail (e.g. levenshtein on a close pair).
|
||||
3. minimize without threshold → `ValueError`.
|
||||
4. expect=true / expect=false on a Boolean metric (exact_match) — pass/fail per expectation.
|
||||
5. no objective → existing behavior unchanged (evaluator's own verdict).
|
||||
6. objective + error metric → error wins (status=error, not fail).
|
||||
7. config error (bad direction) → `ValueError` raised by `evaluate()`.
|
||||
8. objective turns a passing metric into failing → `details` records it; case status becomes fail.
|
||||
9. backward-compat: all existing 62 tests keep passing.
|
||||
|
||||
## 6. Compatibility
|
||||
|
||||
- Public API (`evaluate`, `list_evaluators`, `get_evaluator`, types) unchanged in signature.
|
||||
- `EvalMetric` shape unchanged (score, passed, meta) — only `passed` may be recomputed by the runner.
|
||||
- Existing configs (no `objective` key) behave identically.
|
||||
+33
-7
@@ -8,8 +8,9 @@ Connects Claude Code, Cursor, Windsurf, Cline, Continue, VS Code (GitHub Copilot
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# From the repo root
|
||||
pip install -e ".[mcp]"
|
||||
# From the repo root — no extra install flag needed; the root mcp/ package is
|
||||
# part of the repository and does not require an external MCP SDK.
|
||||
pip install -e .
|
||||
|
||||
# Test the server (type a JSON-RPC request, press Enter)
|
||||
python -m mcp
|
||||
@@ -89,7 +90,14 @@ python -m mcp [--debug]
|
||||
|
||||
## Per-tool configuration
|
||||
|
||||
### Claude Code (`~/.claude/settings.json`)
|
||||
### Claude Code (`~/.claude.json` or `.mcp.json`)
|
||||
|
||||
Claude Code supports two MCP configuration scopes:
|
||||
|
||||
- **User scope** — `~/.claude.json` applies across all projects for your user account.
|
||||
- **Project scope** — `.mcp.json` in your project root applies only to that project.
|
||||
|
||||
Both files use the same `mcpServers` structure:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -97,15 +105,33 @@ python -m mcp [--debug]
|
||||
"semantica": {
|
||||
"command": "python",
|
||||
"args": ["-m", "mcp"],
|
||||
"cwd": "/path/to/semantica"
|
||||
"env": {
|
||||
"PYTHONPATH": "/path/to/semantica"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Or use the plugin bundle:
|
||||
> **Why `PYTHONPATH`?** The root `mcp/` package is intentionally not included in
|
||||
> the installed wheel, so `python -m mcp` only works when the repository is on
|
||||
> Python's import path. Setting `PYTHONPATH` here ensures this works regardless
|
||||
> of the working directory Claude uses when it launches the server.
|
||||
|
||||
Or add it via the CLI (user scope):
|
||||
|
||||
```bash
|
||||
claude mcp add semantica python -m mcp --cwd /path/to/semantica
|
||||
claude mcp add --scope user semantica \
|
||||
-e PYTHONPATH=/path/to/semantica \
|
||||
-- python -m mcp
|
||||
```
|
||||
|
||||
Or for project scope (omit `--scope user`):
|
||||
|
||||
```bash
|
||||
claude mcp add semantica \
|
||||
-e PYTHONPATH=/path/to/semantica \
|
||||
-- python -m mcp
|
||||
```
|
||||
|
||||
---
|
||||
@@ -216,7 +242,7 @@ Add to your Q Developer MCP config:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `SEMANTICA_KG_PATH` | *(in-memory)* | Path to persist/load the graph (JSON file) |
|
||||
| `SEMANTICA_KG_PATH` | *(in-memory only)* | Path to a JSON file used to **load** the graph on startup and **persist** mutations (record decisions, add entities/relationships) back to disk after each change. When unset the graph lives in memory only and is lost when the server exits. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
+35
-7
@@ -16,6 +16,13 @@ log = logging.getLogger("semantica.mcp.session")
|
||||
|
||||
_graph: Optional[Any] = None
|
||||
|
||||
# Tracks whether the last graph initialisation successfully loaded the
|
||||
# configured SEMANTICA_KG_PATH file. When True (or no path was configured)
|
||||
# mutation handlers are allowed to save. When False an existing file failed
|
||||
# to load; saving would overwrite the original data with an empty graph, so
|
||||
# persistence is blocked until the process is restarted with a readable file.
|
||||
_load_ok: bool = True
|
||||
|
||||
|
||||
def get_graph() -> Any:
|
||||
"""
|
||||
@@ -24,24 +31,45 @@ def get_graph() -> Any:
|
||||
The graph is created with advanced_analytics=True so all centrality,
|
||||
community-detection, and embedding features are available.
|
||||
"""
|
||||
global _graph
|
||||
global _graph, _load_ok
|
||||
if _graph is None:
|
||||
from semantica.context import ContextGraph
|
||||
|
||||
_graph = ContextGraph(advanced_analytics=True)
|
||||
_load_ok = True # default: safe to persist
|
||||
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH", "").strip()
|
||||
if kg_path and os.path.exists(kg_path):
|
||||
try:
|
||||
_graph.load(kg_path)
|
||||
log.info("Graph loaded from %s", kg_path)
|
||||
except Exception as exc:
|
||||
log.warning("Could not load graph from %s: %s", kg_path, exc)
|
||||
# Only attempt to load if the file has content. An empty file
|
||||
# means the path was just created (e.g. a fresh tempfile) and
|
||||
# should be treated as "start with empty graph" rather than a
|
||||
# corrupt-file failure.
|
||||
if os.path.getsize(kg_path) > 0:
|
||||
try:
|
||||
_graph.load_from_file(kg_path)
|
||||
log.info("Graph loaded from %s", kg_path)
|
||||
except Exception as exc:
|
||||
log.warning(
|
||||
"Could not load graph from %s: %s — persistence disabled "
|
||||
"to protect existing data; restart the server to retry.",
|
||||
kg_path, exc,
|
||||
)
|
||||
_load_ok = False # do not overwrite the original file
|
||||
|
||||
return _graph
|
||||
|
||||
|
||||
def is_persistence_safe() -> bool:
|
||||
"""Return True when it is safe to write mutations back to SEMANTICA_KG_PATH.
|
||||
|
||||
Returns False after a failed load so that mutation handlers do not
|
||||
overwrite the original (possibly intact) file with a fresh empty graph.
|
||||
"""
|
||||
return _load_ok
|
||||
|
||||
|
||||
def reset_graph() -> None:
|
||||
"""Reset the singleton (mainly useful in tests)."""
|
||||
global _graph
|
||||
global _graph, _load_ok
|
||||
_graph = None
|
||||
_load_ok = True
|
||||
|
||||
+35
-1
@@ -5,6 +5,7 @@ Decision intelligence tools — record, query, precedents, causal chain, impact.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from mcp.schemas import (
|
||||
ANALYZE_DECISION_IMPACT,
|
||||
@@ -13,7 +14,7 @@ from mcp.schemas import (
|
||||
QUERY_DECISIONS,
|
||||
RECORD_DECISION,
|
||||
)
|
||||
from mcp.session import get_graph
|
||||
from mcp.session import get_graph, is_persistence_safe
|
||||
|
||||
log = logging.getLogger("semantica.mcp.tools.decisions")
|
||||
|
||||
@@ -37,6 +38,39 @@ def handle_record_decision(args: dict) -> dict:
|
||||
valid_from=args.get("valid_from"),
|
||||
valid_until=args.get("valid_until"),
|
||||
)
|
||||
# Persist back to disk so the decision survives server restarts.
|
||||
# Skip when the initial load failed to avoid overwriting original data.
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH", "").strip()
|
||||
if kg_path:
|
||||
if not is_persistence_safe():
|
||||
# Roll back the in-memory mutation so the client-visible state
|
||||
# matches the persisted state (neither is saved).
|
||||
if hasattr(graph, "_decisions") and decision_id in graph._decisions:
|
||||
del graph._decisions[decision_id]
|
||||
if hasattr(graph, "_decision_index"):
|
||||
cat = args.get("category", "")
|
||||
if cat in graph._decision_index:
|
||||
graph._decision_index[cat].discard(decision_id)
|
||||
return {
|
||||
"error": (
|
||||
"Persistence blocked: the configured SEMANTICA_KG_PATH "
|
||||
"could not be loaded at startup. Restart the server with "
|
||||
"a readable graph file to re-enable persistence."
|
||||
)
|
||||
}
|
||||
try:
|
||||
graph.save_to_file(kg_path)
|
||||
except Exception as save_exc:
|
||||
# Atomic write failed. Roll back the in-memory mutation so the
|
||||
# client-visible and persisted states remain consistent.
|
||||
if hasattr(graph, "_decisions") and decision_id in graph._decisions:
|
||||
del graph._decisions[decision_id]
|
||||
if hasattr(graph, "_decision_index"):
|
||||
cat = args.get("category", "")
|
||||
if cat in graph._decision_index:
|
||||
graph._decision_index[cat].discard(decision_id)
|
||||
log.exception("save_to_file failed after record_decision; mutation rolled back")
|
||||
return {"error": f"Mutation rolled back: could not persist graph: {save_exc}"}
|
||||
return {
|
||||
"decision_id": decision_id,
|
||||
"status": "recorded",
|
||||
|
||||
+70
-1
@@ -5,9 +5,10 @@ Graph tools — add entities/relationships, search, analytics, summary.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from mcp.schemas import ADD_ENTITY, ADD_RELATIONSHIP, EMPTY, GET_ANALYTICS, SEARCH_GRAPH
|
||||
from mcp.session import get_graph
|
||||
from mcp.session import get_graph, is_persistence_safe
|
||||
|
||||
log = logging.getLogger("semantica.mcp.tools.graph")
|
||||
|
||||
@@ -25,6 +26,35 @@ def handle_add_entity(args: dict) -> dict:
|
||||
node_type=args.get("type", "Entity"),
|
||||
metadata=args.get("metadata", {}),
|
||||
)
|
||||
# Persist back to disk so the entity survives server restarts.
|
||||
# Skip when the initial load failed to avoid overwriting original data.
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH", "").strip()
|
||||
if kg_path:
|
||||
if not is_persistence_safe():
|
||||
# Roll back: remove the node we just added.
|
||||
try:
|
||||
with graph._lock:
|
||||
graph._drop_node_from_indexes(node_id)
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"error": (
|
||||
"Persistence blocked: the configured SEMANTICA_KG_PATH "
|
||||
"could not be loaded at startup. Restart the server with "
|
||||
"a readable graph file to re-enable persistence."
|
||||
)
|
||||
}
|
||||
try:
|
||||
graph.save_to_file(kg_path)
|
||||
except Exception as save_exc:
|
||||
# Roll back: remove the node so in-memory and persisted state agree.
|
||||
try:
|
||||
with graph._lock:
|
||||
graph._drop_node_from_indexes(node_id)
|
||||
except Exception:
|
||||
pass
|
||||
log.exception("save_to_file failed after add_entity; mutation rolled back")
|
||||
return {"error": f"Mutation rolled back: could not persist graph: {save_exc}"}
|
||||
return {"status": "added", "id": node_id, "type": args.get("type", "Entity")}
|
||||
except Exception as exc:
|
||||
log.exception("add_entity failed")
|
||||
@@ -46,6 +76,45 @@ def handle_add_relationship(args: dict) -> dict:
|
||||
edge_type=rel_type,
|
||||
metadata=args.get("metadata", {}),
|
||||
)
|
||||
# Persist back to disk so the relationship survives server restarts.
|
||||
# Skip when the initial load failed to avoid overwriting original data.
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH", "").strip()
|
||||
if kg_path:
|
||||
if not is_persistence_safe():
|
||||
# Roll back: remove the edge we just added (last matching edge).
|
||||
try:
|
||||
with graph._lock:
|
||||
for edge in reversed(list(graph.edges)):
|
||||
if (edge.source_id == source
|
||||
and edge.target_id == target
|
||||
and edge.edge_type == rel_type):
|
||||
graph._drop_edge_from_indexes(edge)
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"error": (
|
||||
"Persistence blocked: the configured SEMANTICA_KG_PATH "
|
||||
"could not be loaded at startup. Restart the server with "
|
||||
"a readable graph file to re-enable persistence."
|
||||
)
|
||||
}
|
||||
try:
|
||||
graph.save_to_file(kg_path)
|
||||
except Exception as save_exc:
|
||||
# Roll back: remove the edge so in-memory and persisted state agree.
|
||||
try:
|
||||
with graph._lock:
|
||||
for edge in reversed(list(graph.edges)):
|
||||
if (edge.source_id == source
|
||||
and edge.target_id == target
|
||||
and edge.edge_type == rel_type):
|
||||
graph._drop_edge_from_indexes(edge)
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
log.exception("save_to_file failed after add_relationship; mutation rolled back")
|
||||
return {"error": f"Mutation rolled back: could not persist graph: {save_exc}"}
|
||||
return {"status": "added", "source": source, "target": target, "type": rel_type}
|
||||
except Exception as exc:
|
||||
log.exception("add_relationship failed")
|
||||
|
||||
@@ -1203,8 +1203,30 @@ class ContextGraph:
|
||||
"links": links_data,
|
||||
}
|
||||
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
# Write atomically: serialize to a sibling temp file then replace the
|
||||
# destination in one OS-level rename. This guarantees the destination
|
||||
# is either the old contents or the new contents — never a partial write
|
||||
# — so a crash or disk-full error during json.dump cannot corrupt the
|
||||
# sole persisted copy of the graph.
|
||||
dest = Path(path)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp_path = tempfile.mkstemp(
|
||||
dir=dest.parent, prefix=".kg_tmp_", suffix=".json"
|
||||
)
|
||||
try:
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp_path, dest)
|
||||
except Exception:
|
||||
# Clean up the temp file on any failure so we don't litter the
|
||||
# directory with partial writes.
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
self.logger.info(f"Saved context graph to {path}")
|
||||
|
||||
|
||||
@@ -1,10 +1,21 @@
|
||||
"""
|
||||
Semantica Evals Module
|
||||
"""Semantica Evals — evaluation layer for decision intelligence outputs.
|
||||
|
||||
Coming Soon
|
||||
Provides a small library of deterministic and model-backed evaluators plus a
|
||||
runner for measuring decision records, audit trails, and reasoning output.
|
||||
"""
|
||||
|
||||
__version__ = "0.1.1"
|
||||
__status__ = "coming_soon"
|
||||
__all__ = []
|
||||
from . import decision_evaluators # noqa: F401 (registers decision_scores)
|
||||
from . import evaluators # noqa: F401 (registers the generic evaluators)
|
||||
from .registry import get_evaluator, list_evaluators
|
||||
from .runner import evaluate
|
||||
from .types import CaseResult, EvalMetric, EvalSummary
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__all__ = [
|
||||
"evaluate",
|
||||
"get_evaluator",
|
||||
"list_evaluators",
|
||||
"CaseResult",
|
||||
"EvalMetric",
|
||||
"EvalSummary",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Decision-specialized evaluator.
|
||||
|
||||
``decision_scores`` validates a ``Decision`` (or dict) against field-level and
|
||||
governance-level checks: expected outcome, confidence bounds, non-empty
|
||||
required fields, provenance presence, and (when configured) policy compliance
|
||||
via ``PolicyEngine.check_compliance``.
|
||||
"""
|
||||
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from .registry import register
|
||||
from .types import EvalMetric
|
||||
|
||||
|
||||
def _coerce_decision(actual: Any):
|
||||
"""Return a Decision or None; never raise for dict inputs."""
|
||||
from semantica.context.decision_models import Decision
|
||||
|
||||
if isinstance(actual, Decision):
|
||||
return actual
|
||||
if isinstance(actual, dict):
|
||||
try:
|
||||
return Decision(**actual)
|
||||
except (TypeError, ValueError, KeyError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
@register("decision_scores")
|
||||
def decision_scores(actual, expected=None, config=None, **kwargs):
|
||||
"""Composite evaluator over a Decision; see module docstring for sub-checks."""
|
||||
cfg = config or {}
|
||||
decision = _coerce_decision(actual)
|
||||
if decision is None:
|
||||
return EvalMetric(0.0, False, {"error": "input is not a valid Decision or dict"})
|
||||
|
||||
checks: Dict[str, bool] = {}
|
||||
reasons: Dict[str, str] = {}
|
||||
|
||||
expected_outcome = cfg.get("expected_outcome", expected)
|
||||
if expected_outcome is not None:
|
||||
checks["decision_outcome"] = decision.outcome == expected_outcome
|
||||
if not checks["decision_outcome"]:
|
||||
reasons["decision_outcome"] = f"expected {expected_outcome!r}, got {decision.outcome!r}"
|
||||
|
||||
lo = cfg.get("min_confidence", 0.0)
|
||||
hi = cfg.get("max_confidence", 1.0)
|
||||
checks["decision_confidence"] = lo <= decision.confidence <= hi
|
||||
if not checks["decision_confidence"]:
|
||||
reasons["decision_confidence"] = f"{decision.confidence} not in [{lo}, {hi}]"
|
||||
|
||||
for field in ("decision_maker", "reasoning", "scenario"):
|
||||
value = getattr(decision, field, None)
|
||||
checks[field] = isinstance(value, str) and bool(value.strip())
|
||||
if not checks[field]:
|
||||
reasons[field] = f"field {field!r} is empty"
|
||||
|
||||
metadata = decision.metadata if isinstance(decision.metadata, dict) else {}
|
||||
prov = metadata.get(cfg.get("provenance_key", "provenance"))
|
||||
checks["provenance"] = bool(prov)
|
||||
if not checks["provenance"]:
|
||||
reasons["provenance"] = "no provenance record found in metadata"
|
||||
|
||||
policy_engine = cfg.get("policy_engine")
|
||||
policy_id = cfg.get("policy_id")
|
||||
if policy_engine is not None and policy_id is not None:
|
||||
try:
|
||||
compliant = bool(policy_engine.check_compliance(decision, policy_id))
|
||||
checks["policy"] = compliant == cfg.get("expected_policy_compliant", True)
|
||||
if not checks["policy"]:
|
||||
reasons["policy"] = f"compliance={compliant}"
|
||||
except Exception as exc: # noqa: BLE001
|
||||
checks["policy"] = False
|
||||
reasons["policy"] = str(exc)
|
||||
|
||||
if cfg.get("causal_chain_exists"):
|
||||
raise NotImplementedError(
|
||||
"decision_scores causal_chain_exists is an interface slot reserved for V2"
|
||||
)
|
||||
|
||||
passed_count = sum(checks.values())
|
||||
total = len(checks)
|
||||
passed = total > 0 and passed_count == total
|
||||
meta = dict(checks)
|
||||
meta["reasons"] = reasons
|
||||
return EvalMetric(
|
||||
score=passed_count / total if total else 0.0,
|
||||
passed=passed,
|
||||
meta=meta,
|
||||
)
|
||||
@@ -0,0 +1,181 @@
|
||||
"""Generic (non-decision) evaluators for the evals module.
|
||||
|
||||
Each evaluator takes ``(actual, expected, config=None, **kwargs)`` and returns
|
||||
an ``EvalMetric``. Config uses ``min``/``max`` bounds where relevant.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .registry import register
|
||||
from .types import EvalMetric
|
||||
|
||||
|
||||
def _default_config(config):
|
||||
return config or {}
|
||||
|
||||
|
||||
@register("exact_match")
|
||||
def exact_match(actual, expected, config=None, **kwargs):
|
||||
"""Score 1.0 if ``actual`` equals ``expected`` (scalar or list)."""
|
||||
matched = actual == expected
|
||||
return EvalMetric(
|
||||
score=1.0 if matched else 0.0,
|
||||
passed=matched,
|
||||
meta={} if matched else {"reason": f"expected {expected!r}, got {actual!r}"},
|
||||
)
|
||||
|
||||
|
||||
@register("regex_match")
|
||||
def regex_match(actual, expected, config=None, **kwargs):
|
||||
"""Score 1.0 if string ``actual`` matches regex ``expected``."""
|
||||
import re
|
||||
try:
|
||||
matched = re.search(expected, actual) is not None
|
||||
return EvalMetric(
|
||||
score=1.0 if matched else 0.0,
|
||||
passed=matched,
|
||||
meta={} if matched else {"reason": f"'{actual}' does not match {expected}"},
|
||||
)
|
||||
except re.error as exc:
|
||||
return EvalMetric(0.0, False, {"error": str(exc)})
|
||||
|
||||
|
||||
@register("numeric_range")
|
||||
def numeric_range(actual, expected=None, config=None, **kwargs):
|
||||
"""Score 1.0 if number ``actual`` is within inclusive ``[min, max]``."""
|
||||
cfg = _default_config(config)
|
||||
lo, hi = cfg.get("min"), cfg.get("max")
|
||||
passed = lo is not None and hi is not None and lo <= actual <= hi
|
||||
return EvalMetric(
|
||||
score=1.0 if passed else 0.0,
|
||||
passed=passed,
|
||||
meta={} if passed else {"reason": f"{actual} not in [{lo}, {hi}]"},
|
||||
)
|
||||
|
||||
|
||||
@register("temporal_range")
|
||||
def temporal_range(actual, expected=None, config=None, **kwargs):
|
||||
"""Score 1.0 if datetime ``actual`` is within inclusive ISO-datetime window."""
|
||||
cfg = _default_config(config)
|
||||
try:
|
||||
stamp = datetime.fromisoformat(actual)
|
||||
lo = datetime.fromisoformat(cfg["min"])
|
||||
hi = datetime.fromisoformat(cfg["max"])
|
||||
passed = lo <= stamp <= hi
|
||||
return EvalMetric(
|
||||
score=1.0 if passed else 0.0,
|
||||
passed=passed,
|
||||
meta={} if passed else {"reason": f"{actual} not in [{cfg['min']}, {cfg['max']}]"},
|
||||
)
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
return EvalMetric(0.0, False, {"error": str(exc)})
|
||||
|
||||
|
||||
@register("length_range")
|
||||
def length_range(actual, expected=None, config=None, **kwargs):
|
||||
"""Score 1.0 if length of ``actual`` is within inclusive ``[min, max]``."""
|
||||
cfg = _default_config(config)
|
||||
size = len(actual)
|
||||
lo = cfg.get("min", 0)
|
||||
hi = cfg.get("max")
|
||||
passed = hi is not None and lo <= size <= hi
|
||||
return EvalMetric(
|
||||
score=1.0 if passed else 0.0,
|
||||
passed=passed,
|
||||
meta={} if passed else {"reason": f"length {size} not in [{lo}, {hi}]"},
|
||||
)
|
||||
|
||||
|
||||
@register("keyword_check")
|
||||
def keyword_check(actual, expected=None, config=None, **kwargs):
|
||||
"""Score 1.0 if all required terms appear in ``actual`` (word-boundary matching)."""
|
||||
cfg = _default_config(config)
|
||||
required = cfg.get("required") or (expected or [])
|
||||
import re
|
||||
tokens = set(re.findall(r"\w+", str(actual).lower()))
|
||||
missing = [term for term in required if str(term).lower() not in tokens]
|
||||
passed = not missing
|
||||
return EvalMetric(
|
||||
score=1.0 if passed else 0.0,
|
||||
passed=passed,
|
||||
meta={} if passed else {"missing": missing},
|
||||
)
|
||||
|
||||
|
||||
def _levenshtein(a: str, b: str) -> int:
|
||||
"""Classic Levenshtein edit distance."""
|
||||
if a == b:
|
||||
return 0
|
||||
if not a:
|
||||
return len(b)
|
||||
if not b:
|
||||
return len(a)
|
||||
prev = list(range(len(b) + 1))
|
||||
for i, ca in enumerate(a, 1):
|
||||
cur = [i]
|
||||
for j, cb in enumerate(b, 1):
|
||||
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
|
||||
prev = cur
|
||||
return prev[-1]
|
||||
|
||||
|
||||
@register("levenshtein")
|
||||
def levenshtein(actual, expected, config=None, **kwargs):
|
||||
"""Score normalized similarity (1 - distance/max_len) vs ``threshold`` (default 0.8)."""
|
||||
cfg = _default_config(config)
|
||||
threshold = cfg.get("threshold", 0.8)
|
||||
a, b = str(actual), str(expected)
|
||||
max_len = max(len(a), len(b))
|
||||
similarity = 1.0 if max_len == 0 else 1.0 - _levenshtein(a, b) / max_len
|
||||
passed = similarity >= threshold
|
||||
return EvalMetric(
|
||||
score=similarity,
|
||||
passed=passed,
|
||||
meta={"similarity": similarity},
|
||||
)
|
||||
|
||||
|
||||
def _tokenize(text: str) -> List[str]:
|
||||
import re
|
||||
return re.findall(r"\w+", str(text).lower())
|
||||
|
||||
|
||||
@register("rouge")
|
||||
def rouge(actual, expected, config=None, **kwargs):
|
||||
"""ROUGE-1 precision/recall/F1 over tokens; pass on F1 >= ``threshold`` (default 0.0)."""
|
||||
cfg = _default_config(config)
|
||||
threshold = cfg.get("threshold", 0.0)
|
||||
hyp, ref = _tokenize(actual), _tokenize(expected)
|
||||
from collections import Counter
|
||||
hyp_c, ref_c = Counter(hyp), Counter(ref)
|
||||
overlap = sum((hyp_c & ref_c).values())
|
||||
precision = overlap / len(hyp) if hyp else 0.0
|
||||
recall = overlap / len(ref) if ref else 0.0
|
||||
f1 = 0.0 if (precision + recall) == 0 else 2 * precision * recall / (precision + recall)
|
||||
passed = f1 > 0 and f1 >= threshold
|
||||
return EvalMetric(
|
||||
score=f1,
|
||||
passed=passed,
|
||||
meta={"precision": precision, "recall": recall, "f1": f1},
|
||||
)
|
||||
|
||||
|
||||
@register("llm_as_judge")
|
||||
def llm_as_judge(actual, expected, config=None, **kwargs):
|
||||
"""Score 1.0 when a caller-supplied ``judge_fn(actual, expected) -> bool`` passes.
|
||||
|
||||
The judge resolver stays lazy: no LLM backend is imported unless the caller
|
||||
provides one in config.
|
||||
"""
|
||||
cfg = _default_config(config)
|
||||
judge_fn = cfg.get("judge_fn")
|
||||
if judge_fn is None:
|
||||
return EvalMetric(
|
||||
0.0, False, {"error": "config['judge_fn'] required (callable(actual, expected) -> bool)"}
|
||||
)
|
||||
try:
|
||||
verdict = bool(judge_fn(actual, expected))
|
||||
return EvalMetric(score=1.0 if verdict else 0.0, passed=verdict)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return EvalMetric(0.0, False, {"error": str(exc)})
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Evaluator registry for the evals module.
|
||||
|
||||
Evaluators are plain functions ``fn(actual, expected, config=None, **kwargs)
|
||||
-> EvalMetric`` registered under a stable string name so the runner and users
|
||||
can select them by name without importing individual modules.
|
||||
"""
|
||||
|
||||
from typing import Callable, Dict, List
|
||||
|
||||
from .types import EvalMetric
|
||||
|
||||
EVALUATORS: Dict[str, Callable] = {}
|
||||
|
||||
|
||||
def register(name: str) -> Callable:
|
||||
"""Decorator registering an evaluator function under ``name``."""
|
||||
def _register(fn: Callable) -> Callable:
|
||||
if name in EVALUATORS:
|
||||
raise ValueError(f"evaluator already registered: {name}")
|
||||
EVALUATORS[name] = fn
|
||||
return fn
|
||||
return _register
|
||||
|
||||
|
||||
def list_evaluators() -> List[str]:
|
||||
"""Return sorted names of all registered evaluators."""
|
||||
return sorted(EVALUATORS)
|
||||
|
||||
|
||||
def get_evaluator(name: str) -> Callable:
|
||||
"""Look up an evaluator by name, raising ValueError with a hint otherwise."""
|
||||
if name not in EVALUATORS:
|
||||
raise ValueError(f"unknown evaluator '{name}'. Available: {list_evaluators()}")
|
||||
return EVALUATORS[name]
|
||||
@@ -0,0 +1,211 @@
|
||||
"""Evaluation runner: orchestrates evaluators over a list of cases."""
|
||||
|
||||
import math
|
||||
from typing import Any, Callable, Dict, List, Optional, Tuple, Union
|
||||
|
||||
from .registry import get_evaluator
|
||||
from .types import CaseResult, EvalMetric, EvalSummary
|
||||
|
||||
Case = Union[Dict[str, Any], Tuple[Any, Any]]
|
||||
|
||||
|
||||
def _coerce_threshold(name, threshold):
|
||||
"""Convert ``threshold`` to a finite float, raising ``ValueError`` otherwise.
|
||||
|
||||
Accepts any value that ``float()`` accepts (int, float, bool, numeric
|
||||
strings) as long as the result is finite. Raises ``ValueError`` — never
|
||||
``TypeError`` — for non-convertible types, NaN, and infinity so that
|
||||
all invalid objective config produces the same exception type.
|
||||
"""
|
||||
try:
|
||||
value = float(threshold)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'threshold' must be a finite number "
|
||||
f"(got {threshold!r})"
|
||||
) from exc
|
||||
if not math.isfinite(value):
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'threshold' must be a finite number "
|
||||
f"(got {threshold!r})"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _parse_objective(name, eval_config):
|
||||
"""Return the validated objective dict, or None when not configured.
|
||||
|
||||
Raises ValueError for invalid configurations (programmer error).
|
||||
"""
|
||||
objective = (eval_config or {}).get("objective")
|
||||
if objective is None:
|
||||
return None
|
||||
if not isinstance(objective, dict):
|
||||
raise ValueError(
|
||||
f"objective for '{name}': expected a dict, got {type(objective).__name__}"
|
||||
)
|
||||
direction = objective.get("direction")
|
||||
threshold = objective.get("threshold")
|
||||
expect = objective.get("expect")
|
||||
|
||||
if expect is not None:
|
||||
if not isinstance(expect, bool):
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'expect' must be a bool (got {expect!r})"
|
||||
)
|
||||
if direction is not None or threshold is not None:
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'expect' cannot be combined with "
|
||||
"'direction' or 'threshold'"
|
||||
)
|
||||
return {"expect": expect}
|
||||
if direction == "minimize":
|
||||
if threshold is None:
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'minimize' requires a 'threshold'"
|
||||
)
|
||||
return {"direction": "minimize", "threshold": _coerce_threshold(name, threshold)}
|
||||
if direction == "maximize":
|
||||
if threshold is None:
|
||||
# no bar to re-decide against; treat as absent (evaluator default stands)
|
||||
return None
|
||||
return {"direction": "maximize", "threshold": _coerce_threshold(name, threshold)}
|
||||
raise ValueError(
|
||||
f"objective for '{name}': 'direction' must be 'maximize' or 'minimize' "
|
||||
f"(got {direction!r})"
|
||||
)
|
||||
|
||||
|
||||
def _apply_objective(metric, objective):
|
||||
"""Return the objective-adjusted pass verdict for a non-error metric."""
|
||||
if "expect" in objective:
|
||||
return bool(metric.score) == objective["expect"]
|
||||
if objective["direction"] == "minimize":
|
||||
return metric.score <= objective["threshold"]
|
||||
return metric.score >= objective["threshold"]
|
||||
|
||||
|
||||
def _extract(case: Case, target_fn: Optional[Callable]):
|
||||
"""Return (case_id, expected, actual, config, per_case_target_fn)."""
|
||||
if isinstance(case, tuple):
|
||||
expected, actual = case[0], (case[1] if len(case) > 1 else None)
|
||||
return str(id(case)), expected, actual, {}, None
|
||||
case_id = case.get("id") or f"case-{id(case)}"
|
||||
expected = case.get("expected")
|
||||
actual = case.get("actual")
|
||||
config = case.get("config") or {}
|
||||
per_fn = case.get("target_fn")
|
||||
return case_id, expected, actual, config, per_fn
|
||||
|
||||
|
||||
def _merge_config(default_config: Dict[str, Any], case_config: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Deep-merge per-case config over the global config (two levels deep).
|
||||
|
||||
Level 1 (top-level keys, e.g. evaluator names): merged key-by-key so a
|
||||
per-case override of one evaluator's settings does not erase the whole
|
||||
global evaluator entry.
|
||||
|
||||
Level 2 (evaluator config keys, e.g. ``"objective"``): also merged
|
||||
key-by-key so a per-case override that specifies only some objective fields
|
||||
(e.g. just ``"threshold"``) inherits the rest from the global objective
|
||||
(e.g. ``"direction"``). Per-case values always take precedence.
|
||||
|
||||
Depth-3+ values are replaced wholesale, consistent with the previous
|
||||
single-level behaviour (no evaluator config currently nests beyond two
|
||||
levels). Neither the caller's global config nor the case config is
|
||||
mutated.
|
||||
"""
|
||||
merged = dict(default_config)
|
||||
for key, value in (case_config or {}).items():
|
||||
if isinstance(value, dict) and isinstance(merged.get(key), dict):
|
||||
# Merge level-1 dict (evaluator config) key-by-key.
|
||||
current = dict(merged[key])
|
||||
for k, v in value.items():
|
||||
if isinstance(v, dict) and isinstance(current.get(k), dict):
|
||||
# Merge level-2 dict (e.g. objective sub-dict) key-by-key.
|
||||
inner = dict(current[k])
|
||||
inner.update(v)
|
||||
current[k] = inner
|
||||
else:
|
||||
current[k] = v
|
||||
merged[key] = current
|
||||
else:
|
||||
merged[key] = value
|
||||
return merged
|
||||
|
||||
|
||||
def evaluate(
|
||||
cases: List[Case],
|
||||
evaluators: List[str],
|
||||
config: Optional[Dict[str, Any]] = None,
|
||||
target_fn: Optional[Callable] = None,
|
||||
) -> EvalSummary:
|
||||
"""Run named evaluators over each case and aggregate metrics.
|
||||
|
||||
A per-case or top-level ``target_fn`` produces ``actual`` when the case
|
||||
does not already carry one. Evaluator failures become ``error`` results.
|
||||
"""
|
||||
default_config = config or {}
|
||||
case_results: List[CaseResult] = []
|
||||
|
||||
# Validate objective config for every case up front so an invalid objective
|
||||
# rejects the run before any target_fn or evaluator executes (fail-fast),
|
||||
# regardless of which case carries it.
|
||||
pre_resolved = []
|
||||
for case in cases:
|
||||
_, _, _, case_config, _ = _extract(case, target_fn)
|
||||
merged = _merge_config(default_config, case_config)
|
||||
pre_resolved.append(
|
||||
{
|
||||
name: _parse_objective(name, merged.get(name) or {})
|
||||
for name in evaluators
|
||||
}
|
||||
)
|
||||
|
||||
for case, objective_by_name in zip(cases, pre_resolved):
|
||||
case_id, expected, actual, case_config, per_fn = _extract(case, target_fn)
|
||||
merged = _merge_config(default_config, case_config)
|
||||
if expected is None:
|
||||
expected = merged.get("expected")
|
||||
resolver = per_fn or target_fn
|
||||
if actual is None and resolver is not None:
|
||||
try:
|
||||
actual = resolver(case)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
case_results.append(
|
||||
CaseResult(case_id, "error", {}, {"target_fn": str(exc)})
|
||||
)
|
||||
continue
|
||||
metrics: Dict[str, EvalMetric] = {}
|
||||
details: Dict[str, Any] = {}
|
||||
failed, errored = False, False
|
||||
for name in evaluators:
|
||||
eval_config = merged.get(name) or {}
|
||||
try:
|
||||
metric = get_evaluator(name)(actual, expected, config=eval_config)
|
||||
objective = objective_by_name.get(name)
|
||||
if objective is not None and "error" not in metric.meta:
|
||||
metric = EvalMetric(metric.score, _apply_objective(metric, objective), metric.meta)
|
||||
metrics[name] = metric
|
||||
if "error" in metric.meta:
|
||||
errored = True
|
||||
details[name] = metric.meta
|
||||
elif not metric.passed:
|
||||
failed = True
|
||||
details[name] = metric.meta
|
||||
except Exception as exc: # noqa: BLE001
|
||||
errored = True
|
||||
metrics[name] = EvalMetric(0.0, False, {"error": str(exc)})
|
||||
details[name] = {"error": str(exc)}
|
||||
status = "error" if errored else ("fail" if failed else "pass")
|
||||
case_results.append(CaseResult(case_id, status, metrics, details))
|
||||
|
||||
total = len(case_results)
|
||||
passed = sum(1 for c in case_results if c.status == "pass")
|
||||
failed = sum(1 for c in case_results if c.status == "fail")
|
||||
errors = sum(1 for c in case_results if c.status == "error")
|
||||
pass_rate = (passed / total) if total else 1.0
|
||||
return EvalSummary(
|
||||
total, passed, failed, errors, pass_rate,
|
||||
cases=case_results,
|
||||
)
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Evals result data models.
|
||||
|
||||
Defines the metric and result shapes produced by the evals module.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, NamedTuple
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvalMetric:
|
||||
"""One evaluator's numeric score plus pass/fail verdict."""
|
||||
|
||||
score: float
|
||||
passed: bool
|
||||
meta: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class CaseResult(NamedTuple):
|
||||
"""Evaluation output for a single case."""
|
||||
|
||||
case_id: str
|
||||
status: str
|
||||
metrics: Dict[str, EvalMetric]
|
||||
details: Dict[str, Any]
|
||||
|
||||
|
||||
@dataclass
|
||||
class EvalSummary:
|
||||
"""Aggregate evaluation output across cases."""
|
||||
|
||||
total: int
|
||||
passed: int
|
||||
failed: int
|
||||
errors: int
|
||||
pass_rate: float
|
||||
cases: List[CaseResult] = field(default_factory=list)
|
||||
@@ -0,0 +1,176 @@
|
||||
# Semantica Evals — Usage
|
||||
|
||||
The evals module measures decision intelligence outputs: decision records,
|
||||
audit trails, and reasoning output — with deterministic and model-backed
|
||||
evaluators plus a small runner.
|
||||
|
||||
## Import
|
||||
|
||||
```python
|
||||
import semantica.evals as evals # through the root lazy proxy
|
||||
from semantica.evals import evaluate, list_evaluators
|
||||
```
|
||||
|
||||
## Discover evaluators
|
||||
|
||||
```python
|
||||
>>> evals.list_evaluators()
|
||||
['decision_scores', 'exact_match', 'keyword_check', 'length_range',
|
||||
'levenshtein', 'llm_as_judge', 'numeric_range', 'regex_match', 'rouge',
|
||||
'temporal_range']
|
||||
```
|
||||
|
||||
`list_evaluators` returns every name registered by importing the package —
|
||||
the import wiring runs each evaluator module's `register()` side effects.
|
||||
|
||||
## Run the runner over decision records
|
||||
|
||||
`evaluate(cases, evaluators, config=None)` accepts a list of cases; each case is
|
||||
a dict with `expected`, `actual`, optional `config`, and optional `id`. The
|
||||
`actual` can be a finished `Decision` object or its dict form.
|
||||
|
||||
```python
|
||||
from datetime import datetime
|
||||
from semantica.context.decision_models import Decision
|
||||
from semantica.evals import evaluate
|
||||
|
||||
decision = Decision(
|
||||
decision_id="d-1",
|
||||
category="loan",
|
||||
scenario="loan-request",
|
||||
reasoning="vetted by policy",
|
||||
outcome="approve",
|
||||
confidence=0.87,
|
||||
timestamp=datetime.now(),
|
||||
decision_maker="approver-a",
|
||||
metadata={"provenance": "workflow:loan/v3"},
|
||||
)
|
||||
|
||||
cases = [
|
||||
{
|
||||
"id": "loan-001",
|
||||
"actual": decision,
|
||||
"config": {
|
||||
"decision_scores": {
|
||||
"expected_outcome": "approve",
|
||||
"min_confidence": 0.7,
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
"id": "loan-002",
|
||||
"actual": {
|
||||
"decision_id": "d-2",
|
||||
"category": "loan",
|
||||
"scenario": "loan-request",
|
||||
"reasoning": "auto",
|
||||
"outcome": "reject",
|
||||
"confidence": 0.9,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"decision_maker": "system",
|
||||
"metadata": {},
|
||||
},
|
||||
"config": {
|
||||
"decision_scores": {
|
||||
"expected_outcome": "approve",
|
||||
"min_confidence": 0.7,
|
||||
}
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
summary = evaluate(cases, ["decision_scores"])
|
||||
```
|
||||
|
||||
`evaluate` also runs high-level names like `exact_match`, `keyword_check`, or
|
||||
`llm_as_judge`; per-case or top-level `config` may carry per-evaluator settings
|
||||
(e.g. `config={"exact_match": {...}}`).
|
||||
|
||||
## Set per-evaluator objectives
|
||||
|
||||
By default each evaluator decides its own pass/fail. To override that
|
||||
verdict at the run level, configure an **objective** per evaluator name:
|
||||
|
||||
```python
|
||||
from semantica.evals import evaluate
|
||||
|
||||
# Require a minimum similarity (levenshtein's default bar is >= 0.8; here we set 0.7):
|
||||
evaluate(
|
||||
[("apple", "aple")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.7}}},
|
||||
)
|
||||
|
||||
# Lower is better — override the direction:
|
||||
evaluate(
|
||||
[("night", "nacht")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.7}}},
|
||||
)
|
||||
|
||||
# Boolean expectation — the metric matches (score 1), but we expect it not to:
|
||||
evaluate(
|
||||
[("ok", "ok")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"expect": False}}},
|
||||
)
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `maximize` + `threshold`: pass iff `score >= threshold`. `maximize` without
|
||||
a threshold is a no-op (the evaluator's own verdict stands).
|
||||
- `minimize` + `threshold`: pass iff `score <= threshold`. `minimize`
|
||||
**requires** a threshold — omitting it or setting it to `None` raises
|
||||
`ValueError`.
|
||||
- `expect` (`true`/`false`): pass iff `bool(score)` matches; cannot be
|
||||
combined with `direction`/`threshold`. `expect` must be a real boolean
|
||||
(a string like `"false"` is rejected).
|
||||
- A metric whose `meta` contains `"error"` is always an error, never affected
|
||||
by an objective.
|
||||
- Invalid objective config (non-dict objective, bad `direction`, non-bool
|
||||
`expect`, missing `minimize` threshold) raises `ValueError` before any
|
||||
evaluator runs.
|
||||
|
||||
## Interpret the summary
|
||||
|
||||
```python
|
||||
>>> summary.total, summary.passed, summary.failed, summary.errors
|
||||
(2, 1, 1, 0)
|
||||
>>> summary.pass_rate
|
||||
0.5
|
||||
|
||||
>>> for case in summary.cases:
|
||||
... print(case.case_id, case.status)
|
||||
... for name, metric in case.metrics.items():
|
||||
... print(" ", name, metric.score, metric.passed)
|
||||
... print(" ", metric.meta.get("reasons"))
|
||||
loan-001 pass
|
||||
decision_scores 1.0 True
|
||||
{}
|
||||
loan-002 fail
|
||||
decision_scores 0.667 False
|
||||
{'decision_outcome': "expected 'approve', got 'reject'",
|
||||
'provenance': 'no provenance record found in metadata'}
|
||||
```
|
||||
|
||||
`EvalSummary` fields:
|
||||
|
||||
- `total` / `passed` / `failed` / `errors` — case counts by status.
|
||||
- `pass_rate` — `passed / total` (1.0 on an empty case list).
|
||||
- `cases` — one `CaseResult` per input case: `case_id`, `status`
|
||||
(`pass` | `fail` | `error`), `metrics` (name → `EvalMetric` with `score`,
|
||||
`passed`, `meta`), and `details`.
|
||||
|
||||
Evaluator failures do not crash the run; they surface as `status="error"` on
|
||||
the affected case with the exception text captured in the metric meta.
|
||||
|
||||
## Notes
|
||||
|
||||
- **`llm_as_judge` needs `config["judge_fn"]`**: a callable
|
||||
`judge_fn(actual, expected) -> bool` supplied by the caller. Without it the
|
||||
evaluator fails with `config['judge_fn'] required`.
|
||||
- **`decision_scores` governance checks are opt-in**: policy compliance is only
|
||||
evaluated when both `config["policy_engine"]` and `config["policy_id"]` are
|
||||
provided; otherwise those checks are skipped. The reserved
|
||||
`causal_chain_exists` slot is not yet implemented.
|
||||
@@ -72,19 +72,34 @@ os.environ["SEMANTICA_DISABLE_PROGRESS"] = "1"
|
||||
# ── lazy graph session ──────────────────────────────────────────────────────
|
||||
_graph: Any = None
|
||||
|
||||
# Tracks whether the last _get_graph() call successfully loaded the configured
|
||||
# SEMANTICA_KG_PATH file. When False (load failed) mutation handlers skip
|
||||
# save_to_file to avoid overwriting the original file with an empty graph.
|
||||
_kg_load_ok: bool = True
|
||||
|
||||
|
||||
def _get_graph():
|
||||
global _graph
|
||||
global _graph, _kg_load_ok
|
||||
if _graph is None:
|
||||
from semantica.context import ContextGraph
|
||||
_graph = ContextGraph(advanced_analytics=True)
|
||||
_kg_load_ok = True # default: safe to persist
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH")
|
||||
if kg_path and os.path.exists(kg_path):
|
||||
try:
|
||||
_graph.load_from_file(kg_path)
|
||||
log.info("Loaded graph from %s", kg_path)
|
||||
except Exception as exc:
|
||||
log.warning("Could not load graph from %s: %s", kg_path, exc)
|
||||
# Only attempt to load if the file has content. An empty file
|
||||
# means the path was just created (fresh destination) and should
|
||||
# be treated as "start with empty graph" not a corrupt-file failure.
|
||||
if os.path.getsize(kg_path) > 0:
|
||||
try:
|
||||
_graph.load_from_file(kg_path)
|
||||
log.info("Loaded graph from %s", kg_path)
|
||||
except Exception as exc:
|
||||
log.warning(
|
||||
"Could not load graph from %s: %s — persistence disabled "
|
||||
"to protect existing data; restart the server to retry.",
|
||||
kg_path, exc,
|
||||
)
|
||||
_kg_load_ok = False # do not overwrite the original file
|
||||
return _graph
|
||||
|
||||
|
||||
@@ -179,6 +194,35 @@ def _tool_record_decision(args: dict) -> dict:
|
||||
valid_from=args.get("valid_from"),
|
||||
valid_until=args.get("valid_until"),
|
||||
)
|
||||
# Persist back to disk so the decision survives server restarts.
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH")
|
||||
if kg_path:
|
||||
if not _kg_load_ok:
|
||||
# Roll back to keep in-memory state consistent with persisted state.
|
||||
if hasattr(graph, "_decisions") and decision_id in graph._decisions:
|
||||
del graph._decisions[decision_id]
|
||||
if hasattr(graph, "_decision_index"):
|
||||
cat = args.get("category", "")
|
||||
if cat in graph._decision_index:
|
||||
graph._decision_index[cat].discard(decision_id)
|
||||
return {
|
||||
"error": (
|
||||
"Persistence blocked: the configured SEMANTICA_KG_PATH "
|
||||
"could not be loaded at startup. Restart the server to retry."
|
||||
)
|
||||
}
|
||||
try:
|
||||
graph.save_to_file(kg_path)
|
||||
except Exception as save_exc:
|
||||
# Atomic write failed. Roll back to keep states consistent.
|
||||
if hasattr(graph, "_decisions") and decision_id in graph._decisions:
|
||||
del graph._decisions[decision_id]
|
||||
if hasattr(graph, "_decision_index"):
|
||||
cat = args.get("category", "")
|
||||
if cat in graph._decision_index:
|
||||
graph._decision_index[cat].discard(decision_id)
|
||||
log.exception("save_to_file failed after record_decision; mutation rolled back")
|
||||
return {"error": f"Mutation rolled back: could not persist graph: {save_exc}"}
|
||||
return {"decision_id": decision_id, "status": "recorded"}
|
||||
|
||||
|
||||
@@ -246,6 +290,31 @@ def _tool_add_entity(args: dict) -> dict:
|
||||
graph = _get_graph()
|
||||
graph.add_node(node_id=node_id, label=label, node_type=node_type,
|
||||
metadata=args.get("metadata", {}))
|
||||
# Persist back to disk so the entity survives server restarts.
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH")
|
||||
if kg_path:
|
||||
if not _kg_load_ok:
|
||||
try:
|
||||
with graph._lock:
|
||||
graph._drop_node_from_indexes(node_id)
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"error": (
|
||||
"Persistence blocked: the configured SEMANTICA_KG_PATH "
|
||||
"could not be loaded at startup. Restart the server to retry."
|
||||
)
|
||||
}
|
||||
try:
|
||||
graph.save_to_file(kg_path)
|
||||
except Exception as save_exc:
|
||||
try:
|
||||
with graph._lock:
|
||||
graph._drop_node_from_indexes(node_id)
|
||||
except Exception:
|
||||
pass
|
||||
log.exception("save_to_file failed after add_entity; mutation rolled back")
|
||||
return {"error": f"Mutation rolled back: could not persist graph: {save_exc}"}
|
||||
return {"status": "added", "id": node_id}
|
||||
|
||||
|
||||
@@ -259,6 +328,41 @@ def _tool_add_relationship(args: dict) -> dict:
|
||||
graph = _get_graph()
|
||||
graph.add_edge(source_id=source, target_id=target, edge_type=rel_type,
|
||||
metadata=args.get("metadata", {}))
|
||||
# Persist back to disk so the relationship survives server restarts.
|
||||
kg_path = os.environ.get("SEMANTICA_KG_PATH")
|
||||
if kg_path:
|
||||
if not _kg_load_ok:
|
||||
try:
|
||||
with graph._lock:
|
||||
for edge in reversed(list(graph.edges)):
|
||||
if (edge.source_id == source
|
||||
and edge.target_id == target
|
||||
and edge.edge_type == rel_type):
|
||||
graph._drop_edge_from_indexes(edge)
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"error": (
|
||||
"Persistence blocked: the configured SEMANTICA_KG_PATH "
|
||||
"could not be loaded at startup. Restart the server to retry."
|
||||
)
|
||||
}
|
||||
try:
|
||||
graph.save_to_file(kg_path)
|
||||
except Exception as save_exc:
|
||||
try:
|
||||
with graph._lock:
|
||||
for edge in reversed(list(graph.edges)):
|
||||
if (edge.source_id == source
|
||||
and edge.target_id == target
|
||||
and edge.edge_type == rel_type):
|
||||
graph._drop_edge_from_indexes(edge)
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
log.exception("save_to_file failed after add_relationship; mutation rolled back")
|
||||
return {"error": f"Mutation rolled back: could not persist graph: {save_exc}"}
|
||||
return {"status": "added", "source": source, "target": target, "type": rel_type}
|
||||
|
||||
|
||||
|
||||
@@ -661,6 +661,15 @@ class MilvusStore:
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _record_to_result(item: Dict[str, Any]) -> Dict[str, Any]:
|
||||
vec = item.get("vector")
|
||||
return {
|
||||
"id": str(item.get("id")),
|
||||
"metadata": item.get("metadata") or {},
|
||||
"vector": np.array(vec) if vec is not None else None,
|
||||
}
|
||||
|
||||
def filter_by_metadata(
|
||||
self, filters: Dict[str, Any], limit: int = 10
|
||||
) -> List[Dict[str, Any]]:
|
||||
@@ -705,21 +714,83 @@ class MilvusStore:
|
||||
limit=limit,
|
||||
output_fields=["id", "vector", "metadata"],
|
||||
)
|
||||
results = []
|
||||
for item in query_results:
|
||||
vec = item.get("vector")
|
||||
results.append(
|
||||
{
|
||||
"id": str(item.get("id")),
|
||||
"metadata": item.get("metadata") or {},
|
||||
"vector": np.array(vec) if vec is not None else None,
|
||||
}
|
||||
)
|
||||
return results
|
||||
return [self._record_to_result(item) for item in query_results]
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to query Milvus vectors by metadata expression: {e}")
|
||||
return []
|
||||
|
||||
def iter_all(self, batch_size: int = 500):
|
||||
"""
|
||||
Iterate over every stored entity using Milvus's query iterator.
|
||||
|
||||
Paginates by primary-key cursor rather than row offset, which is why
|
||||
this exists instead of scan_vectors(offset, limit). query(offset=...)
|
||||
is capped by the 16384 result window and would truncate anything
|
||||
larger.
|
||||
|
||||
Assumes the schema create_collection() builds: a VARCHAR `id` primary
|
||||
key plus vector and metadata fields, as get_vector() and
|
||||
filter_by_metadata() already do. get_collection() does not validate
|
||||
schema, so a collection with an integer key or no metadata field fails
|
||||
here.
|
||||
|
||||
Args:
|
||||
batch_size: Entities to request per iterator batch
|
||||
|
||||
Yields:
|
||||
Result dicts with 'id', 'metadata', and 'vector', in cursor order
|
||||
|
||||
Raises:
|
||||
ProcessingError: If the collection is not initialized, or the
|
||||
installed pymilvus does not expose query_iterator().
|
||||
"""
|
||||
if self.collection is None:
|
||||
raise ProcessingError(
|
||||
"Collection not initialized. Call create_collection() or get_collection() first."
|
||||
)
|
||||
|
||||
if not MILVUS_AVAILABLE:
|
||||
raise ProcessingError("Milvus not available")
|
||||
|
||||
query_iterator = getattr(self.collection.collection, "query_iterator", None)
|
||||
if not callable(query_iterator):
|
||||
raise ProcessingError(
|
||||
"This pymilvus version does not expose Collection.query_iterator(), "
|
||||
"which full enumeration requires. Falling back to query(offset=...) "
|
||||
"is not safe here: it is capped by the 16384 result window and would "
|
||||
"silently truncate a larger collection."
|
||||
)
|
||||
|
||||
# Query operations need a loaded collection. Idempotent, and once per
|
||||
# scan rather than per batch.
|
||||
self.collection.load()
|
||||
|
||||
# Milvus rejects an empty expression; this match-all form is what
|
||||
# filter_by_metadata() already uses.
|
||||
iterator = query_iterator(
|
||||
batch_size=batch_size,
|
||||
expr="id != ''",
|
||||
output_fields=["id", "vector", "metadata"],
|
||||
)
|
||||
|
||||
try:
|
||||
while True:
|
||||
batch = iterator.next()
|
||||
if not batch:
|
||||
return
|
||||
for item in batch:
|
||||
yield self._record_to_result(item)
|
||||
finally:
|
||||
# Release the server-side iterator even if the consumer stops early.
|
||||
# Swallowed so a broken connection at cleanup time doesn't replace
|
||||
# whatever real exception was already propagating out of the try.
|
||||
close = getattr(iterator, "close", None)
|
||||
if callable(close):
|
||||
try:
|
||||
close()
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to close Milvus query iterator: {e}")
|
||||
|
||||
def get_stats(self, collection_name: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""Get collection statistics."""
|
||||
if self.collection is None and collection_name:
|
||||
|
||||
@@ -299,6 +299,44 @@ class PineconeSearch:
|
||||
)
|
||||
|
||||
|
||||
def _pinecone_listed_ids(response: Any) -> List[str]:
|
||||
"""Extract vector IDs from a list_paginated() response.
|
||||
|
||||
Accepts record objects, bare id strings and dicts, since what listing
|
||||
returns has changed across pinecone SDK major versions.
|
||||
"""
|
||||
records = getattr(response, "vectors", None)
|
||||
if records is None and isinstance(response, dict):
|
||||
records = response.get("vectors")
|
||||
|
||||
ids: List[str] = []
|
||||
for record in records or []:
|
||||
if isinstance(record, str):
|
||||
ids.append(record)
|
||||
elif isinstance(record, dict):
|
||||
if record.get("id") is not None:
|
||||
ids.append(record["id"])
|
||||
else:
|
||||
record_id = getattr(record, "id", None)
|
||||
if record_id is not None:
|
||||
ids.append(record_id)
|
||||
return ids
|
||||
|
||||
|
||||
def _pinecone_next_token(response: Any) -> Optional[str]:
|
||||
"""Return the continuation token, or None when the listing is exhausted."""
|
||||
pagination = getattr(response, "pagination", None)
|
||||
if pagination is None and isinstance(response, dict):
|
||||
pagination = response.get("pagination")
|
||||
if pagination is None:
|
||||
return None
|
||||
|
||||
token = getattr(pagination, "next", None)
|
||||
if token is None and isinstance(pagination, dict):
|
||||
token = pagination.get("next")
|
||||
return token or None
|
||||
|
||||
|
||||
class PineconeStore:
|
||||
"""
|
||||
Pinecone store for vector storage and similarity search.
|
||||
@@ -735,6 +773,91 @@ class PineconeStore:
|
||||
self.logger.warning(f"Failed to filter Pinecone vectors by metadata: {e}")
|
||||
return []
|
||||
|
||||
def iter_all(self, batch_size: int = 500, namespace: str = ""):
|
||||
"""
|
||||
Iterate over every stored vector by listing IDs then fetching them.
|
||||
|
||||
Paginates with an opaque continuation token, which is why this exists
|
||||
instead of scan_vectors(offset, limit): the token for page N cannot be
|
||||
constructed without walking there.
|
||||
|
||||
Needs two calls per page, unlike the other backends, because listing
|
||||
returns IDs only. Both calls are namespace scoped and must agree, and
|
||||
listing covers one namespace rather than the whole index.
|
||||
|
||||
Args:
|
||||
batch_size: IDs to request per list_paginated() call
|
||||
namespace: Namespace to enumerate (default: the default namespace)
|
||||
|
||||
Yields:
|
||||
Result dicts with 'id', 'metadata', and 'vector', in listing order
|
||||
|
||||
Raises:
|
||||
ProcessingError: If the index is not initialized, if the installed
|
||||
SDK does not expose list_paginated(), or if the listing stops
|
||||
advancing.
|
||||
"""
|
||||
if self.index is None or not PINECONE_AVAILABLE:
|
||||
raise ProcessingError(
|
||||
"Index not initialized. Call create_index() or get_index() first."
|
||||
)
|
||||
|
||||
# list_paginated() rather than list(): list() is an auto-paging
|
||||
# iterator in current SDKs but reads as plain id lists in older
|
||||
# examples. Threading the token explicitly is version-agnostic.
|
||||
list_paginated = getattr(self.index.index, "list_paginated", None)
|
||||
if not callable(list_paginated):
|
||||
raise ProcessingError(
|
||||
"This pinecone SDK version does not expose Index.list_paginated(), "
|
||||
"which full enumeration requires."
|
||||
)
|
||||
|
||||
token = None
|
||||
while True:
|
||||
kwargs: Dict[str, Any] = {"limit": batch_size, "namespace": namespace}
|
||||
if token is not None:
|
||||
kwargs["pagination_token"] = token
|
||||
|
||||
response = list_paginated(**kwargs)
|
||||
vector_ids = _pinecone_listed_ids(response)
|
||||
|
||||
# A page listing zero ids is not necessarily exhaustion: Pinecone's
|
||||
# contract is that a scan ends only when there's no pagination
|
||||
# token, and a page can legitimately come back empty while
|
||||
# pagination.next is still set (sparse/filtered namespaces,
|
||||
# eventual-consistency windows on serverless indexes). Skip the
|
||||
# fetch (nothing to hydrate) but still fall through to the token
|
||||
# check below instead of returning early, or a gap like that
|
||||
# silently truncates the scan with no error.
|
||||
if vector_ids:
|
||||
fetched = self.index.fetch_vectors(vector_ids, namespace=namespace)
|
||||
vectors = fetched.get("vectors") or {}
|
||||
|
||||
for vector_id in vector_ids:
|
||||
entry = vectors.get(vector_id)
|
||||
if entry is None:
|
||||
# fetch() omits ids it cannot find: deleted since listing.
|
||||
continue
|
||||
values = entry.get("values")
|
||||
yield {
|
||||
"id": vector_id,
|
||||
"metadata": entry.get("metadata") or {},
|
||||
"vector": np.array(values) if values is not None else None,
|
||||
}
|
||||
|
||||
next_token = _pinecone_next_token(response)
|
||||
if not next_token:
|
||||
return
|
||||
if next_token == token:
|
||||
# Distinct from exhaustion above: a partial scan here would be
|
||||
# indistinguishable from a complete one.
|
||||
raise ProcessingError(
|
||||
"Pinecone returned the same pagination token twice, so the "
|
||||
"listing is not advancing. Refusing to return a truncated "
|
||||
"scan."
|
||||
)
|
||||
token = next_token
|
||||
|
||||
def fetch_vectors(
|
||||
self, vector_ids: List[str], namespace: str = "", **options
|
||||
) -> Dict[str, Any]:
|
||||
|
||||
@@ -488,6 +488,28 @@ class WeaviateStore:
|
||||
self.logger.debug(f"Could not build native Weaviate filter: {e}")
|
||||
return None
|
||||
|
||||
def _fetch_objects_offset_or_plain(self, kwargs: Dict[str, Any], scanned_count: int):
|
||||
"""Retry a failed `after`-cursor fetch_objects() call with `offset`, then
|
||||
with no pagination argument at all. Returns (objs, mode)."""
|
||||
kwargs = dict(kwargs)
|
||||
kwargs.pop("after", None)
|
||||
kwargs["offset"] = scanned_count
|
||||
try:
|
||||
return self.collection.query.fetch_objects(**kwargs), "offset"
|
||||
except TypeError:
|
||||
kwargs.pop("offset", None)
|
||||
return self.collection.query.fetch_objects(**kwargs), "single_page"
|
||||
|
||||
@staticmethod
|
||||
def _extract_vector(raw_vector: Any) -> Optional[np.ndarray]:
|
||||
"""weaviate-client v4 returns vector as {'default': [...]} rather than a
|
||||
bare list; older clients and mocks may still hand back a bare list."""
|
||||
if isinstance(raw_vector, dict):
|
||||
raw_vector = raw_vector.get("default")
|
||||
if raw_vector is None or len(raw_vector) == 0:
|
||||
return None
|
||||
return np.array(raw_vector)
|
||||
|
||||
def filter_by_metadata(
|
||||
self, filters: Dict[str, Any], limit: int = 10
|
||||
) -> List[Dict[str, Any]]:
|
||||
@@ -533,22 +555,11 @@ class WeaviateStore:
|
||||
try:
|
||||
objs = self.collection.query.fetch_objects(**kwargs)
|
||||
except TypeError:
|
||||
if "after" in kwargs:
|
||||
kwargs.pop("after", None)
|
||||
kwargs["offset"] = scanned_count
|
||||
try:
|
||||
objs = self.collection.query.fetch_objects(**kwargs)
|
||||
except TypeError:
|
||||
kwargs.pop("offset", None)
|
||||
objs = self.collection.query.fetch_objects(**kwargs)
|
||||
if "after" not in kwargs:
|
||||
raise
|
||||
objs, _ = self._fetch_objects_offset_or_plain(kwargs, scanned_count)
|
||||
elif "after" in kwargs:
|
||||
kwargs.pop("after", None)
|
||||
kwargs["offset"] = scanned_count
|
||||
try:
|
||||
objs = self.collection.query.fetch_objects(**kwargs)
|
||||
except TypeError:
|
||||
kwargs.pop("offset", None)
|
||||
objs = self.collection.query.fetch_objects(**kwargs)
|
||||
objs, _ = self._fetch_objects_offset_or_plain(kwargs, scanned_count)
|
||||
else:
|
||||
raise te
|
||||
except Exception as fe:
|
||||
@@ -611,6 +622,122 @@ class WeaviateStore:
|
||||
self.logger.warning(f"Failed to fetch Weaviate objects by metadata filter: {e}")
|
||||
return results if results else []
|
||||
|
||||
def iter_all(self, batch_size: int = 500):
|
||||
"""
|
||||
Iterate over every stored object using Weaviate's UUID cursor.
|
||||
|
||||
Paginates by the last object's UUID rather than a row offset, which is
|
||||
why this exists instead of scan_vectors(offset, limit). An empty page
|
||||
under that cursor falls back to offset pagination once before ending
|
||||
the scan, since an empty page isn't on its own proof there's nothing
|
||||
left past it (see the inline comment below).
|
||||
|
||||
Assumes a single unnamed vector per object, as get_vector() and
|
||||
filter_by_metadata() already do. Named-vector collections return a
|
||||
mapping and are not handled.
|
||||
|
||||
Args:
|
||||
batch_size: Objects to request per fetch_objects() call
|
||||
|
||||
Yields:
|
||||
Result dicts with 'id', 'metadata', and 'vector', in cursor order
|
||||
|
||||
Raises:
|
||||
ProcessingError: If the collection is not initialized, or if the
|
||||
scan cannot advance past a full page.
|
||||
"""
|
||||
if self.collection is None or not WEAVIATE_AVAILABLE:
|
||||
raise ProcessingError(
|
||||
"Collection not initialized. Call get_collection() first."
|
||||
)
|
||||
|
||||
after_cursor = None
|
||||
scanned_count = 0
|
||||
# Degrades cursor -> offset -> single_page as the client rejects each
|
||||
# form. Tracked across iterations, not just inside the except branch,
|
||||
# or later pages go out with no pagination argument at all.
|
||||
mode = "cursor"
|
||||
|
||||
while True:
|
||||
kwargs = {"limit": batch_size, "include_vector": True}
|
||||
if mode == "cursor" and after_cursor is not None:
|
||||
kwargs["after"] = after_cursor
|
||||
elif mode == "offset":
|
||||
kwargs["offset"] = scanned_count
|
||||
|
||||
try:
|
||||
objs = self.collection.query.fetch_objects(**kwargs)
|
||||
except TypeError:
|
||||
if mode == "cursor" and "after" in kwargs:
|
||||
objs, mode = self._fetch_objects_offset_or_plain(kwargs, scanned_count)
|
||||
elif mode == "offset":
|
||||
mode = "single_page"
|
||||
kwargs.pop("offset", None)
|
||||
objs = self.collection.query.fetch_objects(**kwargs)
|
||||
else:
|
||||
raise
|
||||
|
||||
batch_objects = getattr(objs, "objects", None) if objs else None
|
||||
if not batch_objects:
|
||||
# An empty page in "cursor" mode isn't necessarily the end.
|
||||
# Unlike an offset, `after` has no server-issued continuation
|
||||
# value of its own - it's derived client-side from the last
|
||||
# object's uuid - so an empty page gives nothing to advance
|
||||
# it with. If Weaviate's cursor walks internal storage
|
||||
# position rather than strict uuid order, a batch can in
|
||||
# principle land entirely on a gap (e.g. tombstoned objects)
|
||||
# with live data past it, the same risk already confirmed for
|
||||
# Qdrant's scroll cursor (#1316). Offset pagination doesn't
|
||||
# have that ambiguity - it addresses live rows by position -
|
||||
# so fall back to it once to confirm before ending the scan.
|
||||
if mode == "cursor":
|
||||
mode = "offset"
|
||||
continue
|
||||
return
|
||||
|
||||
page_full = len(batch_objects) >= batch_size
|
||||
next_cursor = after_cursor
|
||||
|
||||
# Checked before yielding: a page that can't advance is truncation,
|
||||
# not completion, and the caller shouldn't see any of it go out
|
||||
# before the error does.
|
||||
if page_full:
|
||||
if mode == "single_page":
|
||||
raise ProcessingError(
|
||||
"This Weaviate client accepts neither an `after` cursor nor a "
|
||||
"numeric offset, so the scan cannot advance past the first "
|
||||
"page. Refusing to return a truncated scan."
|
||||
)
|
||||
if mode == "cursor":
|
||||
last_uuid = getattr(batch_objects[-1], "uuid", None)
|
||||
if last_uuid is None:
|
||||
raise ProcessingError(
|
||||
"The last object of a full Weaviate page has no uuid, so the "
|
||||
"cursor cannot advance. Refusing to return a truncated scan."
|
||||
)
|
||||
next_cursor = str(last_uuid)
|
||||
if next_cursor == after_cursor:
|
||||
raise ProcessingError(
|
||||
"The Weaviate cursor stopped advancing, so the listing is "
|
||||
"repeating a page. Refusing to return a truncated scan."
|
||||
)
|
||||
|
||||
for obj in batch_objects:
|
||||
obj_uuid = getattr(obj, "uuid", None)
|
||||
yield {
|
||||
"id": str(obj_uuid) if obj_uuid is not None else None,
|
||||
"metadata": getattr(obj, "properties", None) or {},
|
||||
"vector": self._extract_vector(getattr(obj, "vector", None)),
|
||||
}
|
||||
|
||||
scanned_count += len(batch_objects)
|
||||
|
||||
if not page_full:
|
||||
return
|
||||
|
||||
if mode == "cursor":
|
||||
after_cursor = next_cursor
|
||||
|
||||
|
||||
def query_vectors(
|
||||
self,
|
||||
|
||||
@@ -1037,5 +1037,195 @@ class TestClearResetsDecisionIndexes(unittest.TestCase):
|
||||
self.assertEqual(g._decisions[did]["category"], "new")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Part 15: semantica.mcp_server mutation persistence (#1134)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestMCPServerMutationPersistence(unittest.TestCase):
|
||||
"""_tool_record_decision, _tool_add_entity, and _tool_add_relationship must
|
||||
each call save_to_file when SEMANTICA_KG_PATH is configured so mutations
|
||||
survive server restarts.
|
||||
|
||||
Mirrors update_node / delete_node which already had this behaviour from
|
||||
PR #967. These tests extend coverage to the three previously missing tools.
|
||||
"""
|
||||
|
||||
# ---- helpers --------------------------------------------------------
|
||||
|
||||
def _isolated_mcp_graph(self):
|
||||
"""Return a fresh ContextGraph injected as the mcp_server singleton."""
|
||||
import semantica.mcp_server as mcp_mod
|
||||
g = ContextGraph(advanced_analytics=False)
|
||||
self._original_graph = mcp_mod._graph
|
||||
mcp_mod._graph = g
|
||||
return g
|
||||
|
||||
def _restore_mcp_graph(self):
|
||||
import semantica.mcp_server as mcp_mod
|
||||
mcp_mod._graph = self._original_graph
|
||||
|
||||
# ---- record_decision ------------------------------------------------
|
||||
|
||||
def test_record_decision_persists_to_kg_path(self):
|
||||
"""_tool_record_decision must write the graph to SEMANTICA_KG_PATH."""
|
||||
from semantica.mcp_server import _tool_record_decision
|
||||
|
||||
g = self._isolated_mcp_graph()
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
result = _tool_record_decision({
|
||||
"category": "mcp_server_persist",
|
||||
"scenario": "Testing packaged server persistence",
|
||||
"reasoning": "save_to_file must be called on mutation",
|
||||
"outcome": "verified",
|
||||
"confidence": 0.99,
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertIn("decision_id", result)
|
||||
|
||||
# File must have been written.
|
||||
self.assertGreater(os.path.getsize(path), 0,
|
||||
"save_to_file must have written to the KG_PATH file")
|
||||
|
||||
# Simulate restart: reload into a fresh graph.
|
||||
g2 = ContextGraph(advanced_analytics=False)
|
||||
g2.load_from_file(path)
|
||||
decisions = list(g2.find_nodes(node_type="decision"))
|
||||
self.assertGreater(len(decisions), 0,
|
||||
"Decision must be present after save → load")
|
||||
cats = [d.get("category") or (d.get("metadata") or {}).get("category")
|
||||
for d in decisions]
|
||||
self.assertIn("mcp_server_persist", cats)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
finally:
|
||||
self._restore_mcp_graph()
|
||||
|
||||
def test_record_decision_works_without_kg_path(self):
|
||||
"""_tool_record_decision must succeed when SEMANTICA_KG_PATH is unset."""
|
||||
from semantica.mcp_server import _tool_record_decision
|
||||
|
||||
g = self._isolated_mcp_graph()
|
||||
try:
|
||||
env = {k: v for k, v in os.environ.items() if k != "SEMANTICA_KG_PATH"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
result = _tool_record_decision({
|
||||
"category": "no_path",
|
||||
"scenario": "no kg path",
|
||||
"reasoning": "in-memory only",
|
||||
"outcome": "ok",
|
||||
"confidence": 0.5,
|
||||
})
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertIn("decision_id", result)
|
||||
finally:
|
||||
self._restore_mcp_graph()
|
||||
|
||||
# ---- add_entity -----------------------------------------------------
|
||||
|
||||
def test_add_entity_persists_to_kg_path(self):
|
||||
"""_tool_add_entity must write the graph to SEMANTICA_KG_PATH."""
|
||||
from semantica.mcp_server import _tool_add_entity
|
||||
|
||||
g = self._isolated_mcp_graph()
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
result = _tool_add_entity({
|
||||
"id": "mcp_server_entity_test",
|
||||
"label": "Persistence Entity",
|
||||
"type": "TestEntity",
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
|
||||
self.assertGreater(os.path.getsize(path), 0)
|
||||
|
||||
g2 = ContextGraph(advanced_analytics=False)
|
||||
g2.load_from_file(path)
|
||||
self.assertTrue(g2.has_node("mcp_server_entity_test"),
|
||||
"Entity must be present after save → load")
|
||||
finally:
|
||||
os.unlink(path)
|
||||
finally:
|
||||
self._restore_mcp_graph()
|
||||
|
||||
def test_add_entity_works_without_kg_path(self):
|
||||
"""_tool_add_entity must succeed when SEMANTICA_KG_PATH is unset."""
|
||||
from semantica.mcp_server import _tool_add_entity
|
||||
|
||||
g = self._isolated_mcp_graph()
|
||||
try:
|
||||
env = {k: v for k, v in os.environ.items() if k != "SEMANTICA_KG_PATH"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
result = _tool_add_entity({"id": "ephemeral_ent", "label": "E"})
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
finally:
|
||||
self._restore_mcp_graph()
|
||||
|
||||
# ---- add_relationship -----------------------------------------------
|
||||
|
||||
def test_add_relationship_persists_to_kg_path(self):
|
||||
"""_tool_add_relationship must write the graph to SEMANTICA_KG_PATH."""
|
||||
from semantica.mcp_server import _tool_add_entity, _tool_add_relationship
|
||||
|
||||
g = self._isolated_mcp_graph()
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
_tool_add_entity({"id": "rel_src_mcp", "label": "Source"})
|
||||
_tool_add_entity({"id": "rel_tgt_mcp", "label": "Target"})
|
||||
result = _tool_add_relationship({
|
||||
"source": "rel_src_mcp",
|
||||
"target": "rel_tgt_mcp",
|
||||
"type": "PROVEN_BY",
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
|
||||
self.assertGreater(os.path.getsize(path), 0)
|
||||
|
||||
g2 = ContextGraph(advanced_analytics=False)
|
||||
g2.load_from_file(path)
|
||||
edges = list(g2.find_edges())
|
||||
self.assertTrue(any(e.get("type") == "PROVEN_BY" for e in edges),
|
||||
"PROVEN_BY edge must be present after save → load")
|
||||
finally:
|
||||
os.unlink(path)
|
||||
finally:
|
||||
self._restore_mcp_graph()
|
||||
|
||||
def test_add_relationship_works_without_kg_path(self):
|
||||
"""_tool_add_relationship must succeed when SEMANTICA_KG_PATH is unset."""
|
||||
from semantica.mcp_server import _tool_add_entity, _tool_add_relationship
|
||||
|
||||
g = self._isolated_mcp_graph()
|
||||
try:
|
||||
env = {k: v for k, v in os.environ.items() if k != "SEMANTICA_KG_PATH"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
_tool_add_entity({"id": "src_no_p", "label": "S"})
|
||||
_tool_add_entity({"id": "tgt_no_p", "label": "T"})
|
||||
result = _tool_add_relationship({
|
||||
"source": "src_no_p",
|
||||
"target": "tgt_no_p",
|
||||
"type": "RELATED_TO",
|
||||
})
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
finally:
|
||||
self._restore_mcp_graph()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
"""Tests for the decision_scores composite evaluator."""
|
||||
import pytest
|
||||
from datetime import datetime
|
||||
|
||||
from semantica.context.decision_models import Decision
|
||||
from semantica.evals import registry as reg
|
||||
|
||||
|
||||
def _decision(**overrides):
|
||||
base = dict(
|
||||
decision_id="d1",
|
||||
category="loan",
|
||||
scenario="mortgage application",
|
||||
reasoning="strong credit history",
|
||||
outcome="approved",
|
||||
confidence=0.95,
|
||||
timestamp=datetime(2026, 1, 1),
|
||||
decision_maker="loan_officer",
|
||||
)
|
||||
base.update(overrides)
|
||||
return Decision(**base)
|
||||
|
||||
|
||||
class TestDecisionScores:
|
||||
def test_full_pass(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
r = reg.get_evaluator("decision_scores")(
|
||||
d, config={"expected_outcome": "approved"}
|
||||
)
|
||||
assert r.passed
|
||||
assert r.meta["decision_outcome"] is True
|
||||
assert r.meta["provenance"] is True
|
||||
|
||||
def test_outcome_mismatch(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
r = reg.get_evaluator("decision_scores")(
|
||||
d, config={"expected_outcome": "denied"}
|
||||
)
|
||||
assert not r.passed
|
||||
assert r.meta["decision_outcome"] is False
|
||||
|
||||
def test_outcome_from_expected_argument(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
r = reg.get_evaluator("decision_scores")(d, expected="approved")
|
||||
assert r.passed
|
||||
assert r.meta["decision_outcome"] is True
|
||||
|
||||
def test_outcome_mismatch_via_expected_argument(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
r = reg.get_evaluator("decision_scores")(d, expected="denied")
|
||||
assert not r.passed
|
||||
assert r.meta["decision_outcome"] is False
|
||||
assert "decision_outcome" in r.meta["reasons"]
|
||||
|
||||
def test_outcome_check_skipped_when_no_expected(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
r = reg.get_evaluator("decision_scores")(d)
|
||||
assert "decision_outcome" not in r.meta
|
||||
|
||||
def test_confidence_out_of_range(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}}, confidence=0.4)
|
||||
r = reg.get_evaluator("decision_scores")(
|
||||
d, config={"expected_outcome": "approved", "min_confidence": 0.8}
|
||||
)
|
||||
assert not r.passed
|
||||
assert r.meta["decision_confidence"] is False
|
||||
|
||||
def test_missing_provenance_fails(self):
|
||||
d = _decision(metadata={})
|
||||
r = reg.get_evaluator("decision_scores")(d, config={"expected_outcome": "approved"})
|
||||
assert not r.passed
|
||||
assert r.meta["provenance"] is False
|
||||
|
||||
def test_missing_required_fields(self):
|
||||
d = _decision(reasoning="")
|
||||
r = reg.get_evaluator("decision_scores")(d, config={"expected_outcome": "approved"})
|
||||
assert not r.passed
|
||||
assert r.meta["reasoning"] is False
|
||||
|
||||
def test_dict_input_coerced(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
as_dict = d.to_dict()
|
||||
r = reg.get_evaluator("decision_scores")(
|
||||
as_dict, config={"expected_outcome": "approved"}
|
||||
)
|
||||
assert r.passed
|
||||
|
||||
def test_malformed_dict_is_error_not_crash(self):
|
||||
r = reg.get_evaluator("decision_scores")({"foo": "bar"}, config={})
|
||||
assert not r.passed
|
||||
assert r.meta.get("error")
|
||||
|
||||
def test_non_dict_metadata_is_error_not_crash(self):
|
||||
bad = _decision(metadata="not-a-dict")
|
||||
r = reg.get_evaluator("decision_scores")(bad, config={})
|
||||
assert not r.passed
|
||||
assert r.meta["provenance"] is False
|
||||
|
||||
def test_policy_compliance_check(self):
|
||||
class FakePolicyEngine:
|
||||
def check_compliance(self, decision, policy_id):
|
||||
return True
|
||||
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
r = reg.get_evaluator("decision_scores")(
|
||||
d, config={
|
||||
"expected_outcome": "approved",
|
||||
"policy_engine": FakePolicyEngine(),
|
||||
"policy_id": "p1",
|
||||
"expected_policy_compliant": True,
|
||||
}
|
||||
)
|
||||
assert r.meta["policy"] is True
|
||||
|
||||
def test_policy_mismatch_fails(self):
|
||||
class FakePolicyEngine:
|
||||
def check_compliance(self, decision, policy_id):
|
||||
return False
|
||||
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}})
|
||||
r = reg.get_evaluator("decision_scores")(
|
||||
d, config={
|
||||
"policy_engine": FakePolicyEngine(),
|
||||
"policy_id": "p1",
|
||||
"expected_policy_compliant": True,
|
||||
}
|
||||
)
|
||||
assert not r.passed
|
||||
assert r.meta["policy"] is False
|
||||
|
||||
def test_causal_chain_gate(self):
|
||||
d = _decision(metadata={"provenance": {"prov_record": "rid-1"}}, decision_id="only-decision")
|
||||
with pytest.raises(NotImplementedError):
|
||||
reg.get_evaluator("decision_scores")(
|
||||
d, config={"causal_chain_exists": True, "graph_store": object()}
|
||||
)
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Tests for generic evaluators: exact, regex, ranges, length."""
|
||||
import pytest
|
||||
|
||||
from semantica.evals import registry as reg
|
||||
|
||||
|
||||
class TestExactMatch:
|
||||
def test_exact_str(self):
|
||||
r = reg.get_evaluator("exact_match")("approved", "approved")
|
||||
assert r.passed and r.score == 1.0
|
||||
|
||||
def test_exact_str_negative(self):
|
||||
r = reg.get_evaluator("exact_match")("approved", "denied")
|
||||
assert not r.passed and r.score == 0.0
|
||||
|
||||
def test_exact_number(self):
|
||||
r = reg.get_evaluator("exact_match")(5, 5)
|
||||
assert r.passed
|
||||
|
||||
def test_exact_array(self):
|
||||
r = reg.get_evaluator("exact_match")([1, 2], [1, 2])
|
||||
assert r.passed
|
||||
|
||||
|
||||
class TestRegexMatch:
|
||||
def test_matching(self):
|
||||
r = reg.get_evaluator("regex_match")("abc123", r"^[a-z]+\d+$")
|
||||
assert r.passed
|
||||
|
||||
def test_non_matching(self):
|
||||
r = reg.get_evaluator("regex_match")("ABC", r"^[a-z]+$")
|
||||
assert not r.passed
|
||||
assert "ABC" in r.meta.get("reason", "")
|
||||
|
||||
def test_invalid_regex_is_error_metric(self):
|
||||
r = reg.get_evaluator("regex_match")("x", "[invalid")
|
||||
assert not r.passed
|
||||
assert r.meta.get("error")
|
||||
|
||||
|
||||
class TestNumericRange:
|
||||
def test_inside(self):
|
||||
r = reg.get_evaluator("numeric_range")(0.9, config={"min": 0.8, "max": 1.0})
|
||||
assert r.passed and r.score == 1.0
|
||||
|
||||
def test_outside(self):
|
||||
r = reg.get_evaluator("numeric_range")(0.5, config={"min": 0.8, "max": 1.0})
|
||||
assert not r.passed and r.score == 0.0
|
||||
|
||||
def test_bounds_inclusive(self):
|
||||
assert reg.get_evaluator("numeric_range")(0.8, config={"min": 0.8, "max": 0.8}).passed
|
||||
|
||||
|
||||
class TestTemporalRange:
|
||||
def test_inside_window(self):
|
||||
r = reg.get_evaluator("temporal_range")(
|
||||
"2026-01-15T10:00:00",
|
||||
config={"min": "2026-01-01T00:00:00", "max": "2026-02-01T00:00:00"},
|
||||
)
|
||||
assert r.passed
|
||||
|
||||
def test_outside_window(self):
|
||||
r = reg.get_evaluator("temporal_range")(
|
||||
"2026-03-01T00:00:00",
|
||||
config={"min": "2026-01-01T00:00:00", "max": "2026-02-01T00:00:00"},
|
||||
)
|
||||
assert not r.passed
|
||||
|
||||
|
||||
class TestLengthRange:
|
||||
def test_ok(self):
|
||||
r = reg.get_evaluator("length_range")("hello", config={"min": 3, "max": 5})
|
||||
assert r.passed
|
||||
|
||||
def test_too_long(self):
|
||||
r = reg.get_evaluator("length_range")([1, 2, 3], config={"min": 1, "max": 2})
|
||||
assert not r.passed
|
||||
|
||||
def test_min_not_given_defaults_zero(self):
|
||||
r = reg.get_evaluator("length_range")("abc", config={"max": 5})
|
||||
assert r.passed
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Tests for generic evaluators: keyword, levenshtein, rouge, llm-as-judge."""
|
||||
import pytest
|
||||
|
||||
from semantica.evals import registry as reg
|
||||
|
||||
|
||||
class TestKeywordCheck:
|
||||
def test_all_required_present(self):
|
||||
r = reg.get_evaluator("keyword_check")(
|
||||
"the loan was approved", expected=["loan", "approved"]
|
||||
)
|
||||
assert r.passed
|
||||
|
||||
def test_missing_keyword(self):
|
||||
r = reg.get_evaluator("keyword_check")(
|
||||
"the loan was approved", expected=["loan", "denied"]
|
||||
)
|
||||
assert not r.passed
|
||||
assert "denied" in r.meta.get("missing", [])
|
||||
|
||||
def test_short_words_ignored(self):
|
||||
r = reg.get_evaluator("keyword_check")("x and y", expected=["and"])
|
||||
assert r.passed
|
||||
|
||||
|
||||
class TestLevenshtein:
|
||||
def test_identical(self):
|
||||
r = reg.get_evaluator("levenshtein")("credit approved", "credit approved")
|
||||
assert r.passed
|
||||
|
||||
def test_close_above_threshold(self):
|
||||
r = reg.get_evaluator("levenshtein")(
|
||||
"credit approved", "credit denied", config={"threshold": 0.8}
|
||||
)
|
||||
assert not r.passed
|
||||
|
||||
def test_default_threshold(self):
|
||||
assert reg.get_evaluator("levenshtein")("a", "a").passed
|
||||
|
||||
|
||||
class TestRouge:
|
||||
def test_identical(self):
|
||||
r = reg.get_evaluator("rouge")("loan approved by committee", "loan approved by committee")
|
||||
assert r.passed
|
||||
assert r.meta["f1"] == pytest.approx(1.0)
|
||||
|
||||
def test_no_overlap(self):
|
||||
r = reg.get_evaluator("rouge")("one two three", "four five six")
|
||||
assert not r.passed
|
||||
|
||||
def test_partial_sets_meta(self):
|
||||
r = reg.get_evaluator("rouge")("a b c", "a b d", config={"threshold": 0.5})
|
||||
assert "precision" in r.meta and "recall" in r.meta
|
||||
|
||||
|
||||
class TestLlmAsJudge:
|
||||
def test_uses_supplied_judge(self):
|
||||
judge = lambda actual, expected: actual == expected # noqa: E731
|
||||
r = reg.get_evaluator("llm_as_judge")(
|
||||
"x", "x", config={"judge_fn": judge}
|
||||
)
|
||||
assert r.passed
|
||||
|
||||
def test_missing_judge_is_error(self):
|
||||
r = reg.get_evaluator("llm_as_judge")("x", "y", config={})
|
||||
assert not r.passed
|
||||
assert r.meta.get("error")
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Tests for the evals public package API."""
|
||||
from semantica import evals
|
||||
from semantica.evals import evaluate, get_evaluator, list_evaluators
|
||||
|
||||
|
||||
class TestPublicAPI:
|
||||
def test_imports(self):
|
||||
assert callable(evaluate)
|
||||
assert callable(list_evaluators)
|
||||
assert callable(get_evaluator)
|
||||
|
||||
def test_version_present(self):
|
||||
assert hasattr(evals, "__version__")
|
||||
|
||||
def test_module_proxy_via_root(self):
|
||||
# semantica.evals must resolve through the lazy proxy
|
||||
assert hasattr(evals, "evaluate")
|
||||
|
||||
def test_all_populated(self):
|
||||
assert len(evals.__all__) >= 2
|
||||
assert "evaluate" in evals.__all__
|
||||
assert "list_evaluators" in evals.__all__
|
||||
assert "get_evaluator" in evals.__all__
|
||||
|
||||
def test_register_discovery(self):
|
||||
names = evals.list_evaluators()
|
||||
for expected in (
|
||||
"exact_match", "regex_match", "numeric_range", "temporal_range",
|
||||
"length_range", "keyword_check", "levenshtein", "rouge",
|
||||
"llm_as_judge", "decision_scores",
|
||||
):
|
||||
assert expected in names
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Tests for the evaluator registry."""
|
||||
import pytest
|
||||
|
||||
from semantica.evals import registry as reg
|
||||
from semantica.evals.types import EvalMetric
|
||||
|
||||
# A unique name that will not collide with any production evaluator.
|
||||
_TEST_EVAL_NAME = "test_registry_demo_eval"
|
||||
|
||||
|
||||
class TestRegistry:
|
||||
def teardown_method(self, method):
|
||||
# Remove the test evaluator after each test that may have registered it,
|
||||
# so re-runs and randomised collection cannot see stale state.
|
||||
reg.EVALUATORS.pop(_TEST_EVAL_NAME, None)
|
||||
|
||||
def test_register_and_get(self):
|
||||
@reg.register(_TEST_EVAL_NAME)
|
||||
def demo(actual, expected, config=None, **kwargs):
|
||||
return EvalMetric(1.0, True)
|
||||
|
||||
assert reg.get_evaluator(_TEST_EVAL_NAME) is demo
|
||||
assert _TEST_EVAL_NAME in reg.list_evaluators()
|
||||
|
||||
def test_registration_is_immutable_after_commit(self):
|
||||
with pytest.raises(ValueError):
|
||||
reg.get_evaluator("does_not_exist")
|
||||
|
||||
def test_unknown_evaluator_failure_message(self):
|
||||
with pytest.raises(ValueError) as exc:
|
||||
reg.get_evaluator("nope")
|
||||
msg = str(exc.value)
|
||||
assert "nope" in msg
|
||||
# The error message lists available evaluators; verify using a name
|
||||
# that is always registered at import time (independent of test order).
|
||||
assert "exact_match" in msg
|
||||
@@ -0,0 +1,459 @@
|
||||
"""Tests for the evals runner."""
|
||||
import pytest
|
||||
|
||||
from semantica.evals.runner import evaluate
|
||||
|
||||
|
||||
class TestEvaluate:
|
||||
def test_raw_tuple_cases(self):
|
||||
result = evaluate(
|
||||
[("approved", "approved"), ("approved", "denied")],
|
||||
evaluators=["exact_match"],
|
||||
)
|
||||
assert result.total == 2
|
||||
assert result.passed == 1
|
||||
assert result.failed == 1
|
||||
assert result.errors == 0
|
||||
assert result.pass_rate == 0.5
|
||||
|
||||
def test_dict_cases_with_target_fn(self):
|
||||
def fn(case):
|
||||
return "ok" if case["id"] == "good" else "no"
|
||||
|
||||
result = evaluate(
|
||||
[{"id": "good"}, {"id": "bad"}],
|
||||
evaluators=["exact_match"],
|
||||
target_fn=fn,
|
||||
config={"expected": "ok"},
|
||||
)
|
||||
assert result.passed == 1
|
||||
assert result.failed == 1
|
||||
|
||||
def test_error_capture(self):
|
||||
result = evaluate([("x", "y")], evaluators=["does_not_exist"])
|
||||
assert result.errors == 1
|
||||
assert result.failed == 0
|
||||
assert result.pass_rate == 0.0
|
||||
|
||||
def test_error_metric_classified_as_error(self):
|
||||
result = evaluate(
|
||||
[("[invalid", "x")],
|
||||
evaluators=["regex_match"],
|
||||
)
|
||||
assert result.errors == 1
|
||||
assert result.failed == 0
|
||||
assert result.cases[0].status == "error"
|
||||
|
||||
def test_error_metric_and_fail_combine_as_error(self):
|
||||
result = evaluate(
|
||||
[("[invalid", "apple pie")],
|
||||
evaluators=["regex_match", "exact_match"],
|
||||
)
|
||||
assert result.errors == 1
|
||||
assert result.failed == 0
|
||||
assert result.cases[0].status == "error"
|
||||
|
||||
def test_per_case_details(self):
|
||||
result = evaluate([("a", "b")], evaluators=["exact_match"])
|
||||
case = result.cases[0]
|
||||
assert case.status == "fail"
|
||||
assert "exact_match" in case.details
|
||||
|
||||
def test_empty_cases(self):
|
||||
result = evaluate([], evaluators=["exact_match"])
|
||||
assert result.total == 0 and result.pass_rate == 1.0
|
||||
|
||||
def test_multiple_evaluators(self):
|
||||
result = evaluate(
|
||||
[("apple pie", "apple pie")],
|
||||
evaluators=["exact_match", "keyword_check"],
|
||||
config={"keyword_check": {"required": ["apple"]}},
|
||||
)
|
||||
assert result.passed == 1
|
||||
assert "exact_match" in result.cases[0].metrics
|
||||
assert "keyword_check" in result.cases[0].metrics
|
||||
|
||||
|
||||
class TestObjective:
|
||||
def test_maximize_with_threshold_pass(self):
|
||||
# levenshtein similarity 1.0 for identical, objective demands >= 0.5
|
||||
result = evaluate(
|
||||
[("apple", "apple")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.5}}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is True
|
||||
|
||||
def test_maximize_with_threshold_fail(self):
|
||||
result = evaluate(
|
||||
[("apple", "aple")], # similarity < 1.0
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.99}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is False
|
||||
assert "levenshtein" in result.cases[0].details
|
||||
|
||||
def test_minimize_with_threshold_pass(self):
|
||||
# levenshtein similarity 0.6 for ("night", "nacht"); objective: similarity <= 0.7
|
||||
result = evaluate(
|
||||
[("night", "nacht")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.7}}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is True
|
||||
|
||||
def test_minimize_with_threshold_fail(self):
|
||||
result = evaluate(
|
||||
[("night", "nacht")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.1}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
def test_expect_true_on_boolean_metric(self):
|
||||
result = evaluate(
|
||||
[("ok", "ok")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"expect": True}}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
|
||||
def test_expect_false_overrides_passing_metric(self):
|
||||
# exact_match passes (score 1.0) but expectation is false -> fail
|
||||
result = evaluate(
|
||||
[("ok", "ok")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"expect": False}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
assert result.cases[0].metrics["exact_match"].passed is False
|
||||
assert "exact_match" in result.cases[0].details
|
||||
|
||||
def test_maximize_without_threshold_is_noop(self):
|
||||
# identical behavior to no objective: evaluator's own verdict stands
|
||||
result = evaluate(
|
||||
[("ok", "no")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"direction": "maximize"}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
def test_minimize_without_threshold_raises(self):
|
||||
# direction-only minimize has no well-defined pass bar; must be rejected
|
||||
with pytest.raises(ValueError, match="'minimize' requires a 'threshold'"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize"}}},
|
||||
)
|
||||
|
||||
def test_minimize_with_explicit_none_threshold_raises(self):
|
||||
# explicit threshold=None is the same as omitting it; must also be rejected
|
||||
with pytest.raises(ValueError, match="'minimize' requires a 'threshold'"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": None}}},
|
||||
)
|
||||
|
||||
def test_bad_direction_raises(self):
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "sideways", "threshold": 0.5}}},
|
||||
)
|
||||
|
||||
def test_expect_with_direction_raises(self):
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"expect": True, "direction": "maximize"}}},
|
||||
)
|
||||
|
||||
def test_error_metric_wins_over_objective(self):
|
||||
result = evaluate(
|
||||
[("[invalid", "x")],
|
||||
evaluators=["regex_match"],
|
||||
config={"regex_match": {"objective": {"direction": "maximize", "threshold": 0.0}}},
|
||||
)
|
||||
assert result.cases[0].status == "error"
|
||||
assert result.errors == 1
|
||||
assert result.failed == 0
|
||||
|
||||
def test_no_objective_unchanged(self):
|
||||
result = evaluate([("ok", "no")], evaluators=["exact_match"])
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
def test_non_dict_objective_raises(self):
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": "maximize"}},
|
||||
)
|
||||
|
||||
def test_non_bool_expect_raises(self):
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"expect": "false"}}},
|
||||
)
|
||||
|
||||
def test_invalid_per_case_objective_fails_fast_before_target_fn(self):
|
||||
calls = []
|
||||
|
||||
def side_effectful_target_fn(case):
|
||||
calls.append(case)
|
||||
return "line"
|
||||
|
||||
with pytest.raises(ValueError):
|
||||
evaluate(
|
||||
[{"id": "c1"}, {"id": "c2", "config": {"levenshtein": {"objective": {"direction": "diagonal"}}}}],
|
||||
evaluators=["levenshtein"],
|
||||
target_fn=side_effectful_target_fn,
|
||||
)
|
||||
# validation must reject the run before any case is processed
|
||||
assert calls == []
|
||||
|
||||
def test_case_config_keeps_global_objective(self):
|
||||
# global objective on the evaluator must survive a per-case override
|
||||
# that touches other settings for the same evaluator (deep merge)
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd",
|
||||
"config": {"levenshtein": {"ignore_case": False}}}],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
|
||||
)
|
||||
# levenshtein("abc","abd") == 1 > 0 -> objective fails the case
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
|
||||
class TestMergeConfig:
|
||||
"""Focused tests for _merge_config two-level deep-merge semantics."""
|
||||
|
||||
def test_partial_per_case_objective_inherits_global_direction(self):
|
||||
# Per-case overrides only threshold; direction must come from global.
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd",
|
||||
"config": {"levenshtein": {"objective": {"threshold": 0.99}}}}],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
|
||||
)
|
||||
# Effective objective: minimize, threshold=0.99.
|
||||
# levenshtein("abc","abd") similarity ~0.667; 0.667 <= 0.99 -> pass.
|
||||
assert result.cases[0].status == "pass"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is True
|
||||
|
||||
def test_partial_per_case_objective_inherits_global_threshold(self):
|
||||
# Per-case overrides only direction; threshold must come from global.
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd",
|
||||
"config": {"levenshtein": {"objective": {"direction": "maximize"}}}}],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.99}}},
|
||||
)
|
||||
# Effective objective: maximize, threshold=0.99.
|
||||
# levenshtein("abc","abd") similarity ~0.667; 0.667 >= 0.99 -> fail.
|
||||
assert result.cases[0].status == "fail"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is False
|
||||
|
||||
def test_per_case_threshold_overrides_global_threshold(self):
|
||||
# Global: minimize, threshold=0.0 (would fail for any positive score).
|
||||
# Per-case: threshold=0.99 (almost everything passes minimize).
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd",
|
||||
"config": {"levenshtein": {"objective": {"threshold": 0.99}}}}],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
|
||||
)
|
||||
# Effective: minimize, threshold=0.99 -> ~0.667 <= 0.99 -> pass.
|
||||
assert result.cases[0].status == "pass"
|
||||
|
||||
def test_per_case_direction_overrides_global_direction(self):
|
||||
# Global: maximize, threshold=0.99 (would fail for ~0.667).
|
||||
# Per-case: direction=minimize (with inherited threshold=0.99).
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd",
|
||||
"config": {"levenshtein": {"objective": {"direction": "minimize"}}}}],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.99}}},
|
||||
)
|
||||
# Effective: minimize, threshold=0.99 -> ~0.667 <= 0.99 -> pass.
|
||||
assert result.cases[0].status == "pass"
|
||||
|
||||
def test_fully_specified_per_case_objective_replaces_global(self):
|
||||
# Both direction and threshold specified per-case; nothing from global.
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd",
|
||||
"config": {"levenshtein": {"objective": {"direction": "maximize", "threshold": 0.5}}}}],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
|
||||
)
|
||||
# Effective: maximize, threshold=0.5 -> ~0.667 >= 0.5 -> pass.
|
||||
assert result.cases[0].status == "pass"
|
||||
|
||||
def test_per_case_non_objective_keys_do_not_erase_global_objective(self):
|
||||
# Per-case touches only non-objective evaluator keys; global objective intact.
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd",
|
||||
"config": {"levenshtein": {"threshold": 0.5}}}],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.0}}},
|
||||
)
|
||||
# Effective: minimize, threshold=0.0 -> ~0.667 > 0.0 -> fail.
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
def test_no_objective_anywhere_unchanged(self):
|
||||
# No objectives anywhere; evaluator's own verdict stands throughout.
|
||||
result = evaluate(
|
||||
[{"id": "c1", "expected": "ok", "actual": "ok",
|
||||
"config": {"exact_match": {"some_key": "v"}}}],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"other_key": "w"}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
|
||||
def test_global_config_not_mutated(self):
|
||||
import copy
|
||||
global_config = {"levenshtein": {"objective": {"direction": "minimize", "threshold": 0.5}}}
|
||||
case_config = {"levenshtein": {"objective": {"threshold": 0.2}}}
|
||||
original_global = copy.deepcopy(global_config)
|
||||
original_case = copy.deepcopy(case_config)
|
||||
evaluate(
|
||||
[{"id": "c1", "expected": "abc", "actual": "abd", "config": case_config}],
|
||||
evaluators=["levenshtein"],
|
||||
config=global_config,
|
||||
)
|
||||
assert global_config == original_global
|
||||
assert case_config == original_case
|
||||
|
||||
|
||||
class TestThresholdValidation:
|
||||
"""Threshold coercion and validation: types, NaN, infinity."""
|
||||
|
||||
# --- valid numeric thresholds ---
|
||||
|
||||
def test_maximize_integer_threshold(self):
|
||||
# int is a valid threshold; coerced to float
|
||||
result = evaluate(
|
||||
[("apple", "apple")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": 1}}},
|
||||
)
|
||||
assert result.cases[0].status == "pass"
|
||||
assert result.cases[0].metrics["levenshtein"].passed is True
|
||||
|
||||
def test_minimize_integer_threshold(self):
|
||||
result = evaluate(
|
||||
[("night", "nacht")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": 1}}},
|
||||
)
|
||||
# similarity 0.6 <= 1 -> pass
|
||||
assert result.cases[0].status == "pass"
|
||||
|
||||
# --- invalid threshold types ---
|
||||
|
||||
def test_non_numeric_string_threshold_raises(self):
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": "high"}}},
|
||||
)
|
||||
|
||||
def test_list_threshold_raises_value_error(self):
|
||||
# Must be ValueError, not TypeError
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": [0.5]}}},
|
||||
)
|
||||
|
||||
def test_dict_threshold_raises_value_error(self):
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": {"v": 1}}}},
|
||||
)
|
||||
|
||||
# --- NaN and infinity ---
|
||||
|
||||
def test_nan_threshold_raises(self):
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": float("nan")}}},
|
||||
)
|
||||
|
||||
def test_positive_infinity_threshold_raises(self):
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": float("inf")}}},
|
||||
)
|
||||
|
||||
def test_negative_infinity_threshold_raises(self):
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "maximize", "threshold": float("-inf")}}},
|
||||
)
|
||||
|
||||
def test_nan_minimize_threshold_raises(self):
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": float("nan")}}},
|
||||
)
|
||||
|
||||
# --- preserved behaviors ---
|
||||
|
||||
def test_maximize_without_threshold_still_noop(self):
|
||||
# maximize without threshold remains a no-op regardless of threshold validation
|
||||
result = evaluate(
|
||||
[("ok", "no")],
|
||||
evaluators=["exact_match"],
|
||||
config={"exact_match": {"objective": {"direction": "maximize"}}},
|
||||
)
|
||||
assert result.cases[0].status == "fail"
|
||||
|
||||
def test_minimize_explicit_none_threshold_still_raises(self):
|
||||
# threshold=None for minimize hits the None check before coercion
|
||||
with pytest.raises(ValueError, match="'minimize' requires a 'threshold'"):
|
||||
evaluate(
|
||||
[("a", "b")],
|
||||
evaluators=["levenshtein"],
|
||||
config={"levenshtein": {"objective": {"direction": "minimize", "threshold": None}}},
|
||||
)
|
||||
|
||||
def test_threshold_errors_are_fail_fast(self):
|
||||
# Invalid threshold on case 2 must reject the whole run before case 1 executes
|
||||
calls = []
|
||||
|
||||
def recording_fn(case):
|
||||
calls.append(case)
|
||||
return "x"
|
||||
|
||||
with pytest.raises(ValueError, match="'threshold' must be a finite number"):
|
||||
evaluate(
|
||||
[
|
||||
{"id": "c1"},
|
||||
{"id": "c2", "config": {"levenshtein": {"objective": {"direction": "maximize", "threshold": [0.5]}}}},
|
||||
],
|
||||
evaluators=["levenshtein"],
|
||||
target_fn=recording_fn,
|
||||
)
|
||||
assert calls == []
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Tests for evals result models."""
|
||||
import pytest
|
||||
|
||||
from semantica.evals.types import CaseResult, EvalMetric, EvalSummary
|
||||
|
||||
|
||||
class TestEvalMetric:
|
||||
def test_construction(self):
|
||||
m = EvalMetric(score=1.0, passed=True, meta={"threshold": 1.0})
|
||||
assert m.score == 1.0 and m.passed and m.meta["threshold"] == 1.0
|
||||
|
||||
def test_default_meta(self):
|
||||
m = EvalMetric(0.0, False)
|
||||
assert m.meta == {}
|
||||
|
||||
def test_default_meta_is_not_shared(self):
|
||||
m1 = EvalMetric(0.0, False)
|
||||
m2 = EvalMetric(0.0, False)
|
||||
m1.meta["mutated"] = True
|
||||
assert "mutated" not in m2.meta
|
||||
|
||||
|
||||
class TestCaseResult:
|
||||
def test_status_fail_on_any_failed_metric(self):
|
||||
r = CaseResult(
|
||||
case_id="c1",
|
||||
status="fail",
|
||||
metrics={"exact_match": EvalMetric(0.0, False)},
|
||||
details={},
|
||||
)
|
||||
assert r.status == "fail"
|
||||
assert r.metrics["exact_match"].passed is False
|
||||
|
||||
|
||||
class TestEvalSummary:
|
||||
def test_pass_rate(self):
|
||||
s = EvalSummary(total=10, passed=8, failed=1, errors=1, pass_rate=0.8)
|
||||
assert s.pass_rate == 0.8
|
||||
|
||||
def test_cases_are_mutable(self):
|
||||
s = EvalSummary(0, 0, 0, 0, 1.0)
|
||||
s.cases.append(CaseResult("c", "pass", {}, {}))
|
||||
assert len(s.cases) == 1
|
||||
@@ -0,0 +1,294 @@
|
||||
"""Regression tests for root mcp/ graph persistence (issue #1134).
|
||||
|
||||
Covers:
|
||||
1. get_graph() loads an existing JSON file via load_from_file(), not the
|
||||
nonexistent .load() method (the original bug).
|
||||
2. get_graph() with a nonexistent / unset SEMANTICA_KG_PATH starts cleanly.
|
||||
3. handle_record_decision persists to SEMANTICA_KG_PATH and the mutation
|
||||
survives a fresh load_from_file() call.
|
||||
4. handle_add_entity persists to SEMANTICA_KG_PATH and survives reload.
|
||||
5. handle_add_relationship persists to SEMANTICA_KG_PATH and survives reload.
|
||||
6. All three mutation tools work correctly when SEMANTICA_KG_PATH is unset
|
||||
(no errors, no persistence attempt).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from semantica.context.context_graph import ContextGraph
|
||||
|
||||
import mcp.session as _session
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _fresh_graph() -> ContextGraph:
|
||||
"""Return a minimal ContextGraph ready for use in tests."""
|
||||
g = ContextGraph(advanced_analytics=False)
|
||||
g.add_node("seed_node", node_type="entity", label="Seed")
|
||||
return g
|
||||
|
||||
|
||||
class _IsolatedSession:
|
||||
"""Context manager that resets the mcp.session singleton before and after
|
||||
each test so tests are independent of process-level state."""
|
||||
|
||||
def __enter__(self):
|
||||
_session.reset_graph()
|
||||
return self
|
||||
|
||||
def __exit__(self, *_):
|
||||
_session.reset_graph()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. get_graph() loading — regression against _graph.load()
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestMCPSessionLoad(unittest.TestCase):
|
||||
"""get_graph() must load an existing file using load_from_file(), not .load()."""
|
||||
|
||||
def test_get_graph_loads_existing_kg_path(self):
|
||||
"""When SEMANTICA_KG_PATH points to a valid JSON file the graph must
|
||||
contain the persisted nodes after get_graph() returns."""
|
||||
g = _fresh_graph()
|
||||
g.add_node("persistent_node", node_type="entity", label="Should survive")
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
g.save_to_file(path)
|
||||
|
||||
with _IsolatedSession():
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
loaded = _session.get_graph()
|
||||
|
||||
self.assertTrue(
|
||||
loaded.has_node("persistent_node"),
|
||||
"Node saved before server start must be present after load",
|
||||
)
|
||||
self.assertTrue(
|
||||
loaded.has_node("seed_node"),
|
||||
"seed_node from the persisted graph must also be present",
|
||||
)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_get_graph_with_nonexistent_kg_path_starts_empty(self):
|
||||
"""When SEMANTICA_KG_PATH does not exist the graph initialises empty
|
||||
(no error) — matching pre-existing behaviour."""
|
||||
with _IsolatedSession():
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": "/nonexistent/path.json"}):
|
||||
loaded = _session.get_graph()
|
||||
|
||||
# An empty graph has no nodes; at minimum it must be a ContextGraph.
|
||||
self.assertIsNotNone(loaded)
|
||||
nodes = list(loaded.find_nodes())
|
||||
self.assertEqual(nodes, [], "Graph must be empty when KG_PATH does not exist")
|
||||
|
||||
def test_get_graph_without_kg_path_starts_empty(self):
|
||||
"""When SEMANTICA_KG_PATH is absent the graph initialises empty."""
|
||||
with _IsolatedSession():
|
||||
env = {k: v for k, v in os.environ.items() if k != "SEMANTICA_KG_PATH"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
loaded = _session.get_graph()
|
||||
|
||||
self.assertIsNotNone(loaded)
|
||||
|
||||
def test_get_graph_uses_load_from_file_not_load(self):
|
||||
"""Regression: ContextGraph has no .load() method; get_graph() must
|
||||
call load_from_file() or the AttributeError is silently swallowed and
|
||||
the graph silently stays empty. This test verifies the fix directly."""
|
||||
g = _fresh_graph()
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
g.save_to_file(path)
|
||||
|
||||
with _IsolatedSession():
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
# If the old _graph.load(path) bug were present the graph
|
||||
# would be empty (exception swallowed). With the fix the
|
||||
# node must be present.
|
||||
loaded = _session.get_graph()
|
||||
|
||||
self.assertTrue(
|
||||
loaded.has_node("seed_node"),
|
||||
"load_from_file must have been called; if .load() was used "
|
||||
"the AttributeError is swallowed and the graph stays empty",
|
||||
)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2–5. Mutation persistence
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestMCPPackageMutationPersistence(unittest.TestCase):
|
||||
"""Mutations via the root mcp/ tool handlers must persist to SEMANTICA_KG_PATH
|
||||
so the data survives a server restart (simulated by a fresh load_from_file)."""
|
||||
|
||||
# ---- record_decision ------------------------------------------------
|
||||
|
||||
def test_record_decision_persists_when_kg_path_set(self):
|
||||
"""handle_record_decision must write to disk when SEMANTICA_KG_PATH is set."""
|
||||
from mcp.tools.decisions import handle_record_decision
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
with _IsolatedSession():
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
result = handle_record_decision({
|
||||
"category": "test_persistence",
|
||||
"scenario": "Verifying mcp/ decision persistence",
|
||||
"reasoning": "KG_PATH must be written on mutation",
|
||||
"outcome": "verified",
|
||||
"confidence": 0.99,
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertIn("decision_id", result)
|
||||
|
||||
# The file must have been written (or overwritten from empty).
|
||||
self.assertTrue(os.path.exists(path), "save_to_file must create the file")
|
||||
self.assertGreater(os.path.getsize(path), 0, "Persisted file must not be empty")
|
||||
|
||||
# Simulate server restart: load into a fresh graph.
|
||||
g2 = ContextGraph(advanced_analytics=False)
|
||||
g2.load_from_file(path)
|
||||
decisions = list(g2.find_nodes(node_type="decision"))
|
||||
self.assertGreater(len(decisions), 0, "Decision must survive reload")
|
||||
cats = [d.get("category") or (d.get("metadata") or {}).get("category")
|
||||
for d in decisions]
|
||||
self.assertIn("test_persistence", cats,
|
||||
"Decision category must be present after reload")
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_record_decision_works_without_kg_path(self):
|
||||
"""handle_record_decision must succeed even when SEMANTICA_KG_PATH is unset."""
|
||||
from mcp.tools.decisions import handle_record_decision
|
||||
|
||||
with _IsolatedSession():
|
||||
env = {k: v for k, v in os.environ.items() if k != "SEMANTICA_KG_PATH"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
result = handle_record_decision({
|
||||
"category": "no_path",
|
||||
"scenario": "No persistence path configured",
|
||||
"reasoning": "Should still work in-memory",
|
||||
"outcome": "ok",
|
||||
"confidence": 0.5,
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertIn("decision_id", result)
|
||||
|
||||
# ---- add_entity -----------------------------------------------------
|
||||
|
||||
def test_add_entity_persists_when_kg_path_set(self):
|
||||
"""handle_add_entity must write to disk when SEMANTICA_KG_PATH is set."""
|
||||
from mcp.tools.graph import handle_add_entity
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
with _IsolatedSession():
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
result = handle_add_entity({
|
||||
"id": "entity_persist_test",
|
||||
"label": "Persistence Test Entity",
|
||||
"type": "TestType",
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
|
||||
self.assertTrue(os.path.exists(path))
|
||||
self.assertGreater(os.path.getsize(path), 0)
|
||||
|
||||
g2 = ContextGraph(advanced_analytics=False)
|
||||
g2.load_from_file(path)
|
||||
self.assertTrue(
|
||||
g2.has_node("entity_persist_test"),
|
||||
"Entity must be present in the graph after reload",
|
||||
)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_add_entity_works_without_kg_path(self):
|
||||
"""handle_add_entity must succeed when SEMANTICA_KG_PATH is unset."""
|
||||
from mcp.tools.graph import handle_add_entity
|
||||
|
||||
with _IsolatedSession():
|
||||
env = {k: v for k, v in os.environ.items() if k != "SEMANTICA_KG_PATH"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
result = handle_add_entity({"id": "no_path_entity", "label": "ephemeral"})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
|
||||
# ---- add_relationship -----------------------------------------------
|
||||
|
||||
def test_add_relationship_persists_when_kg_path_set(self):
|
||||
"""handle_add_relationship must write to disk when SEMANTICA_KG_PATH is set."""
|
||||
from mcp.tools.graph import handle_add_relationship
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".json", delete=False) as f:
|
||||
path = f.name
|
||||
try:
|
||||
with _IsolatedSession():
|
||||
with patch.dict(os.environ, {"SEMANTICA_KG_PATH": path}):
|
||||
# Nodes must exist before an edge can be added.
|
||||
from mcp.tools.graph import handle_add_entity
|
||||
handle_add_entity({"id": "rel_src", "label": "Source"})
|
||||
handle_add_entity({"id": "rel_tgt", "label": "Target"})
|
||||
result = handle_add_relationship({
|
||||
"source": "rel_src",
|
||||
"target": "rel_tgt",
|
||||
"type": "TESTED_BY",
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
|
||||
self.assertTrue(os.path.exists(path))
|
||||
self.assertGreater(os.path.getsize(path), 0)
|
||||
|
||||
g2 = ContextGraph(advanced_analytics=False)
|
||||
g2.load_from_file(path)
|
||||
edges = list(g2.find_edges())
|
||||
edge_types = [e.get("type") for e in edges]
|
||||
self.assertIn("TESTED_BY", edge_types,
|
||||
"Relationship must be present after reload")
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
def test_add_relationship_works_without_kg_path(self):
|
||||
"""handle_add_relationship must succeed when SEMANTICA_KG_PATH is unset."""
|
||||
from mcp.tools.graph import handle_add_entity, handle_add_relationship
|
||||
|
||||
with _IsolatedSession():
|
||||
env = {k: v for k, v in os.environ.items() if k != "SEMANTICA_KG_PATH"}
|
||||
with patch.dict(os.environ, env, clear=True):
|
||||
handle_add_entity({"id": "src_no_path", "label": "S"})
|
||||
handle_add_entity({"id": "tgt_no_path", "label": "T"})
|
||||
result = handle_add_relationship({
|
||||
"source": "src_no_path",
|
||||
"target": "tgt_no_path",
|
||||
"type": "RELATED_TO",
|
||||
})
|
||||
|
||||
self.assertNotIn("error", result, result)
|
||||
self.assertEqual(result.get("status"), "added")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,180 @@
|
||||
"""MCP stdio JSON-RPC framing regression test (#1134, point 1).
|
||||
|
||||
The original bug: progress output from the Semantica progress tracker was
|
||||
written to sys.stdout, which is also the MCP JSON-RPC transport channel.
|
||||
Interleaving progress text with JSON-RPC responses made every response
|
||||
unparseable and hung the client.
|
||||
|
||||
These tests exercise the *actual* root mcp/ server stdio framing loop
|
||||
(SemanticaMCPServer.run()) over a real subprocess pipe, not just the handler
|
||||
layer. They prove that:
|
||||
|
||||
1. Every non-empty stdout line produced by the running server is valid JSON.
|
||||
2. A valid JSON-RPC response is received for each request sent.
|
||||
3. No progress / non-JSON bytes appear on stdout even when a tool triggers
|
||||
the progress-producing code path (constructing a ContextGraph, which
|
||||
calls get_progress_tracker() and attempts to enable the tracker).
|
||||
|
||||
Tests that are already covered elsewhere are not duplicated here:
|
||||
- ConsoleProgressDisplay writing to stderr (test_progress_stream.py)
|
||||
- SEMANTICA_DISABLE_PROGRESS blocking re-enable (test_progress_tracker_regressions.py)
|
||||
- mcp import sets SEMANTICA_DISABLE_PROGRESS (test_mcp_package_export_graph.py)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _repo_root() -> str:
|
||||
return os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
|
||||
|
||||
|
||||
def _subprocess_env() -> dict[str, str]:
|
||||
"""Clean env with the repo on PYTHONPATH and no pre-set progress flag."""
|
||||
env = os.environ.copy()
|
||||
env["PYTHONPATH"] = _repo_root()
|
||||
env.pop("SEMANTICA_DISABLE_PROGRESS", None)
|
||||
return env
|
||||
|
||||
|
||||
def _jsonrpc(method: str, req_id: int | None, params: dict | None = None) -> bytes:
|
||||
msg: dict = {"jsonrpc": "2.0", "method": method}
|
||||
if req_id is not None:
|
||||
msg["id"] = req_id
|
||||
if params is not None:
|
||||
msg["params"] = params
|
||||
return (json.dumps(msg) + "\n").encode()
|
||||
|
||||
|
||||
def _assert_stdout_is_clean_json(test: unittest.TestCase,
|
||||
stdout: str,
|
||||
stderr: str = "") -> list[dict]:
|
||||
"""Assert every non-empty stdout line is valid JSON; return parsed objects.
|
||||
|
||||
Fails immediately with a useful diagnostic if any line is not JSON.
|
||||
"""
|
||||
lines = [ln for ln in stdout.splitlines() if ln.strip()]
|
||||
test.assertGreater(
|
||||
len(lines), 0,
|
||||
f"Expected at least one stdout line but got none.\nstderr={stderr!r}",
|
||||
)
|
||||
parsed = []
|
||||
for i, line in enumerate(lines):
|
||||
try:
|
||||
parsed.append(json.loads(line))
|
||||
except json.JSONDecodeError as exc:
|
||||
test.fail(
|
||||
f"stdout line {i} is not valid JSON (regression: progress leaked "
|
||||
f"to stdout?)\n line: {line!r}\n error: {exc}\n stderr={stderr!r}"
|
||||
)
|
||||
return parsed
|
||||
|
||||
|
||||
_INIT_REQUEST = _jsonrpc("initialize", 1, {
|
||||
"protocolVersion": "2024-11-05",
|
||||
"clientInfo": {"name": "test", "version": "0"},
|
||||
"capabilities": {},
|
||||
})
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main regression suite
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestMCPStdioFramingContract(unittest.TestCase):
|
||||
"""Run 'python -m mcp' exactly as an MCP client would, over a real pipe.
|
||||
|
||||
Each test sends a complete JSON-RPC session through stdin and asserts that
|
||||
every byte on stdout is valid JSON — catching the exact failure mode from
|
||||
#1134 where progress output corrupted the transport stream.
|
||||
"""
|
||||
|
||||
TIMEOUT = 30
|
||||
|
||||
def _run(self, *requests: bytes) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "mcp"],
|
||||
input=b"".join(requests),
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
timeout=self.TIMEOUT,
|
||||
cwd=_repo_root(),
|
||||
env=_subprocess_env(),
|
||||
check=False,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_initialize_stdout_is_valid_json_rpc(self):
|
||||
"""An initialize request must produce a single valid JSON-RPC response."""
|
||||
proc = self._run(_INIT_REQUEST)
|
||||
|
||||
self.assertEqual(proc.returncode, 0,
|
||||
f"server crashed:\n{proc.stderr.decode()}")
|
||||
responses = _assert_stdout_is_clean_json(
|
||||
self, proc.stdout.decode(), proc.stderr.decode()
|
||||
)
|
||||
init_resp = next((r for r in responses if r.get("id") == 1), None)
|
||||
self.assertIsNotNone(init_resp, f"No id=1 response in: {responses}")
|
||||
self.assertIn("serverInfo", init_resp.get("result", {}))
|
||||
|
||||
def test_tools_call_stdout_is_clean_json_rpc(self):
|
||||
"""A tools/call round-trip through the full stdio framing loop must keep
|
||||
stdout free of any non-JSON bytes.
|
||||
|
||||
run_reasoning is used because Reasoner.infer_with_results() explicitly
|
||||
calls self.progress_tracker.start_tracking(), making it the minimal
|
||||
deterministic tool path that exercises the progress-rendering code.
|
||||
Before the #1134 fix, that start_tracking call wrote a progress bar
|
||||
directly to stdout, corrupting the JSON-RPC framing. Every byte on
|
||||
stdout must still be valid JSON-RPC after the fix.
|
||||
"""
|
||||
proc = self._run(
|
||||
_INIT_REQUEST,
|
||||
_jsonrpc("notifications/initialized", None),
|
||||
_jsonrpc("tools/call", 2, {
|
||||
"name": "run_reasoning",
|
||||
"arguments": {
|
||||
"facts": ["Person(Alice)", "Employee(Alice)"],
|
||||
"rules": ["IF Employee(?x) THEN Worker(?x)"],
|
||||
},
|
||||
}),
|
||||
)
|
||||
self.assertEqual(proc.returncode, 0,
|
||||
f"server crashed:\n{proc.stderr.decode()}")
|
||||
|
||||
stdout = proc.stdout.decode()
|
||||
stderr = proc.stderr.decode()
|
||||
responses = _assert_stdout_is_clean_json(self, stdout, stderr)
|
||||
|
||||
tool_resp = next((r for r in responses if r.get("id") == 2), None)
|
||||
self.assertIsNotNone(
|
||||
tool_resp,
|
||||
f"No id=2 response in stdout.\nstdout={stdout!r}\nstderr={stderr!r}",
|
||||
)
|
||||
# The framing must be a valid JSON-RPC result object regardless of
|
||||
# whether the reasoner dependency is available in this environment.
|
||||
self.assertIn("jsonrpc", tool_resp)
|
||||
self.assertEqual(tool_resp["jsonrpc"], "2.0")
|
||||
self.assertIn("id", tool_resp)
|
||||
# If the tool succeeded the response must carry MCP content.
|
||||
if "result" in tool_resp:
|
||||
content = tool_resp["result"].get("content", [])
|
||||
self.assertGreater(len(content), 0,
|
||||
"Expected non-empty content list in result")
|
||||
# The embedded tool payload must itself be valid JSON.
|
||||
inner = json.loads(content[0]["text"])
|
||||
self.assertIn("derived_facts", inner)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,176 @@
|
||||
"""Tests for MilvusStore.iter_all() query-iterator enumeration.
|
||||
|
||||
pymilvus is not installed in this environment, so these drive the real
|
||||
MilvusStore against MagicMocks, following the pattern already used for milvus
|
||||
in test_backend_metadata_filtering.py.
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from semantica.utils.exceptions import ProcessingError
|
||||
from semantica.vector_store.milvus_store import MilvusStore
|
||||
|
||||
|
||||
def _store_with_batches(*batches):
|
||||
"""MilvusStore whose query_iterator yields the given batches then stops.
|
||||
|
||||
The attribute path is doubled here: the pymilvus Collection sits at
|
||||
wrapper.collection.
|
||||
"""
|
||||
store = MilvusStore()
|
||||
wrapper = MagicMock()
|
||||
inner = MagicMock()
|
||||
iterator = MagicMock()
|
||||
iterator.next.side_effect = list(batches)
|
||||
inner.query_iterator.return_value = iterator
|
||||
wrapper.collection = inner
|
||||
store.collection = wrapper
|
||||
return store, wrapper, inner, iterator
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_yields_batches_until_exhausted():
|
||||
"""Exhaustion is an empty list, not StopIteration."""
|
||||
store, _, _, iterator = _store_with_batches(
|
||||
[{"id": 1, "vector": [0.1], "metadata": {}}],
|
||||
[{"id": 2, "vector": [0.2], "metadata": {}}],
|
||||
[],
|
||||
)
|
||||
|
||||
result = list(store.iter_all(batch_size=1))
|
||||
|
||||
assert [item["id"] for item in result] == ["1", "2"]
|
||||
assert iterator.next.call_count == 3
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_requests_the_fields_needed_for_the_result_shape():
|
||||
store, _, inner, _ = _store_with_batches([])
|
||||
|
||||
list(store.iter_all(batch_size=64))
|
||||
|
||||
kwargs = inner.query_iterator.call_args[1]
|
||||
assert kwargs["batch_size"] == 64
|
||||
assert kwargs["output_fields"] == ["id", "vector", "metadata"]
|
||||
# Milvus rejects an empty expression, so a match-all form is required.
|
||||
assert kwargs["expr"] == "id != ''"
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_loads_the_collection_before_querying():
|
||||
"""Milvus requires a loaded collection for query operations."""
|
||||
store, wrapper, _, _ = _store_with_batches([])
|
||||
|
||||
list(store.iter_all())
|
||||
|
||||
assert wrapper.load.called
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_closes_the_iterator_on_exhaustion():
|
||||
store, _, _, iterator = _store_with_batches([])
|
||||
|
||||
list(store.iter_all())
|
||||
|
||||
assert iterator.close.called
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_closes_the_iterator_when_consumer_stops_early():
|
||||
"""Abandoning the generator early must still release the iterator."""
|
||||
store, _, _, iterator = _store_with_batches(
|
||||
[{"id": 1, "vector": [0.1], "metadata": {}}],
|
||||
[{"id": 2, "vector": [0.2], "metadata": {}}],
|
||||
[],
|
||||
)
|
||||
|
||||
generator = store.iter_all(batch_size=1)
|
||||
next(generator)
|
||||
assert not iterator.close.called
|
||||
generator.close()
|
||||
|
||||
assert iterator.close.called
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_converts_entities_to_the_shared_result_shape():
|
||||
store, _, _, _ = _store_with_batches(
|
||||
[{"id": 7, "vector": [0.1, 0.2, 0.3], "metadata": {"tag": "x"}}], []
|
||||
)
|
||||
|
||||
item = list(store.iter_all())[0]
|
||||
|
||||
assert item["id"] == "7"
|
||||
assert item["metadata"] == {"tag": "x"}
|
||||
np.testing.assert_allclose(item["vector"], np.array([0.1, 0.2, 0.3]))
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_handles_missing_vector_and_metadata():
|
||||
store, _, _, _ = _store_with_batches([{"id": 1, "vector": None, "metadata": None}], [])
|
||||
|
||||
item = list(store.iter_all())[0]
|
||||
|
||||
assert item["metadata"] == {}
|
||||
assert item["vector"] is None
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_empty_collection_yields_nothing():
|
||||
store, _, _, _ = _store_with_batches([])
|
||||
|
||||
assert list(store.iter_all()) == []
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_raises_when_query_iterator_is_unavailable():
|
||||
"""Older pymilvus lacks query_iterator; falling back to query(offset=...)
|
||||
would truncate at the 16384 window."""
|
||||
store = MilvusStore()
|
||||
wrapper = MagicMock()
|
||||
wrapper.collection = MagicMock(spec=["query"])
|
||||
store.collection = wrapper
|
||||
|
||||
with pytest.raises(ProcessingError, match="query_iterator"):
|
||||
list(store.iter_all())
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_raises_when_collection_not_initialized():
|
||||
"""Must fail loudly: an empty scan reads the same as an empty source."""
|
||||
store = MilvusStore()
|
||||
|
||||
with pytest.raises(ProcessingError, match="Collection not initialized"):
|
||||
list(store.iter_all())
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", False)
|
||||
def test_iter_all_raises_when_milvus_unavailable():
|
||||
store = MilvusStore()
|
||||
store.collection = MagicMock()
|
||||
|
||||
with pytest.raises(ProcessingError):
|
||||
list(store.iter_all())
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_propagates_iterator_errors():
|
||||
store, _, _, iterator = _store_with_batches()
|
||||
iterator.next.side_effect = RuntimeError("connection reset")
|
||||
|
||||
with pytest.raises(RuntimeError, match="connection reset"):
|
||||
list(store.iter_all())
|
||||
|
||||
|
||||
@patch("semantica.vector_store.milvus_store.MILVUS_AVAILABLE", True)
|
||||
def test_iter_all_closes_the_iterator_when_a_batch_fails():
|
||||
store, _, _, iterator = _store_with_batches()
|
||||
iterator.next.side_effect = RuntimeError("connection reset")
|
||||
|
||||
with pytest.raises(RuntimeError):
|
||||
list(store.iter_all())
|
||||
|
||||
assert iterator.close.called
|
||||
@@ -249,6 +249,191 @@ class TestPineconeIndex(unittest.TestCase):
|
||||
mock_index.query.assert_called_once()
|
||||
|
||||
|
||||
class TestPineconeIterAll(unittest.TestCase):
|
||||
"""PineconeStore.iter_all() list-then-fetch enumeration."""
|
||||
|
||||
def _page(self, ids, next_token):
|
||||
"""Stand-in for a list_paginated() response."""
|
||||
response = MagicMock()
|
||||
response.vectors = [MagicMock(id=vector_id) for vector_id in ids]
|
||||
response.pagination = MagicMock(next=next_token)
|
||||
return response
|
||||
|
||||
def _store(self, pages, fetch_results):
|
||||
store = PineconeStore()
|
||||
wrapper = MagicMock()
|
||||
raw_index = MagicMock()
|
||||
raw_index.list_paginated.side_effect = list(pages)
|
||||
wrapper.index = raw_index
|
||||
wrapper.fetch_vectors.side_effect = list(fetch_results)
|
||||
store.index = wrapper
|
||||
return store, wrapper, raw_index
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_threads_pagination_token_across_pages(self):
|
||||
store, _, raw_index = self._store(
|
||||
[self._page(["a", "b"], "token-1"), self._page(["c"], None)],
|
||||
[
|
||||
{"vectors": {"a": {"values": [0.1], "metadata": {}},
|
||||
"b": {"values": [0.2], "metadata": {}}}},
|
||||
{"vectors": {"c": {"values": [0.3], "metadata": {}}}},
|
||||
],
|
||||
)
|
||||
|
||||
result = list(store.iter_all(batch_size=2))
|
||||
|
||||
self.assertEqual([item["id"] for item in result], ["a", "b", "c"])
|
||||
calls = raw_index.list_paginated.call_args_list
|
||||
self.assertNotIn("pagination_token", calls[0][1])
|
||||
self.assertEqual(calls[1][1]["pagination_token"], "token-1")
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_hydrates_listed_ids_with_a_fetch(self):
|
||||
"""Listing returns ids only, so each page needs a fetch()."""
|
||||
store, wrapper, _ = self._store(
|
||||
[self._page(["a"], None)],
|
||||
[{"vectors": {"a": {"values": [0.1, 0.2], "metadata": {"tag": "x"}}}}],
|
||||
)
|
||||
|
||||
item = list(store.iter_all())[0]
|
||||
|
||||
self.assertEqual(item["id"], "a")
|
||||
self.assertEqual(item["metadata"], {"tag": "x"})
|
||||
np.testing.assert_allclose(item["vector"], np.array([0.1, 0.2]))
|
||||
wrapper.fetch_vectors.assert_called_once_with(["a"], namespace="")
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_list_and_fetch_use_the_same_namespace(self):
|
||||
store, wrapper, raw_index = self._store(
|
||||
[self._page(["a"], None)],
|
||||
[{"vectors": {"a": {"values": [0.1], "metadata": {}}}}],
|
||||
)
|
||||
|
||||
list(store.iter_all(namespace="prod"))
|
||||
|
||||
self.assertEqual(raw_index.list_paginated.call_args[1]["namespace"], "prod")
|
||||
wrapper.fetch_vectors.assert_called_once_with(["a"], namespace="prod")
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_skips_ids_deleted_between_list_and_fetch(self):
|
||||
"""fetch() omits ids it cannot find rather than returning blanks."""
|
||||
store, _, _ = self._store(
|
||||
[self._page(["a", "gone"], None)],
|
||||
[{"vectors": {"a": {"values": [0.1], "metadata": {}}}}],
|
||||
)
|
||||
|
||||
result = list(store.iter_all())
|
||||
|
||||
self.assertEqual([item["id"] for item in result], ["a"])
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_raises_when_pagination_token_repeats(self):
|
||||
"""A stalled token must not loop forever, nor quietly return a partial
|
||||
scan that reads as a complete one."""
|
||||
store, _, raw_index = self._store(
|
||||
[self._page(["a"], "same"), self._page(["b"], "same")],
|
||||
[
|
||||
{"vectors": {"a": {"values": [0.1], "metadata": {}}}},
|
||||
{"vectors": {"b": {"values": [0.2], "metadata": {}}}},
|
||||
],
|
||||
)
|
||||
|
||||
with self.assertRaises(ProcessingError):
|
||||
list(store.iter_all())
|
||||
|
||||
self.assertEqual(raw_index.list_paginated.call_count, 2)
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_empty_listing_yields_nothing_without_fetching(self):
|
||||
store, wrapper, _ = self._store([self._page([], None)], [])
|
||||
|
||||
self.assertEqual(list(store.iter_all()), [])
|
||||
wrapper.fetch_vectors.assert_not_called()
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_continues_past_an_empty_page_with_a_live_token(self):
|
||||
"""An empty page is not necessarily the end: Pinecone can legitimately
|
||||
list zero ids for a page while pagination.next is still set (sparse
|
||||
or filtered namespaces, eventual-consistency windows on serverless
|
||||
indexes). Only the absence of a next token means exhaustion."""
|
||||
store, wrapper, raw_index = self._store(
|
||||
[
|
||||
self._page(["a"], "token-1"),
|
||||
self._page([], "token-2"), # empty page, but the token still advances
|
||||
self._page(["b"], None),
|
||||
],
|
||||
[
|
||||
{"vectors": {"a": {"values": [0.1], "metadata": {}}}},
|
||||
{"vectors": {"b": {"values": [0.2], "metadata": {}}}},
|
||||
],
|
||||
)
|
||||
|
||||
result = list(store.iter_all(batch_size=1))
|
||||
|
||||
self.assertEqual([item["id"] for item in result], ["a", "b"])
|
||||
self.assertEqual(raw_index.list_paginated.call_count, 3)
|
||||
# Nothing to hydrate on the empty page, so only two fetches happen.
|
||||
self.assertEqual(wrapper.fetch_vectors.call_count, 2)
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_accepts_plain_string_ids_from_listing(self):
|
||||
"""SDK generations differ on what listing yields."""
|
||||
store, _, _ = self._store(
|
||||
[self._page([], None)],
|
||||
[{"vectors": {"a": {"values": [0.1], "metadata": {}}}}],
|
||||
)
|
||||
response = MagicMock()
|
||||
response.vectors = ["a"]
|
||||
response.pagination = MagicMock(next=None)
|
||||
store.index.index.list_paginated.side_effect = [response]
|
||||
|
||||
self.assertEqual([item["id"] for item in store.iter_all()], ["a"])
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_handles_missing_values_and_metadata(self):
|
||||
store, _, _ = self._store(
|
||||
[self._page(["a"], None)],
|
||||
[{"vectors": {"a": {"values": None, "metadata": None}}}],
|
||||
)
|
||||
|
||||
item = list(store.iter_all())[0]
|
||||
|
||||
self.assertIsNone(item["vector"])
|
||||
self.assertEqual(item["metadata"], {})
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_raises_when_list_paginated_unavailable(self):
|
||||
store = PineconeStore()
|
||||
wrapper = MagicMock()
|
||||
wrapper.index = MagicMock(spec=["query", "fetch"])
|
||||
store.index = wrapper
|
||||
|
||||
with self.assertRaises(ProcessingError):
|
||||
list(store.iter_all())
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_raises_when_index_not_initialized(self):
|
||||
"""Must fail loudly: an empty scan reads the same as an empty source."""
|
||||
with self.assertRaises(ProcessingError):
|
||||
list(PineconeStore().iter_all())
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', False)
|
||||
def test_raises_when_pinecone_unavailable(self):
|
||||
store = PineconeStore()
|
||||
store.index = MagicMock()
|
||||
|
||||
with self.assertRaises(ProcessingError):
|
||||
list(store.iter_all())
|
||||
|
||||
@patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
|
||||
def test_propagates_listing_errors(self):
|
||||
store, _, raw_index = self._store([], [])
|
||||
raw_index.list_paginated.side_effect = RuntimeError("connection reset")
|
||||
|
||||
with self.assertRaises(RuntimeError):
|
||||
list(store.iter_all())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
print("DEBUG: Starting unittest.main()")
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
"""Tests for WeaviateStore.iter_all() cursor enumeration.
|
||||
|
||||
weaviate-client is not installed in this environment, so these drive the real
|
||||
WeaviateStore against MagicMocks, following the pattern already used for
|
||||
weaviate in test_backend_metadata_filtering.py.
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from semantica.utils.exceptions import ProcessingError
|
||||
from semantica.vector_store.weaviate_store import WeaviateStore
|
||||
|
||||
|
||||
def _obj(uuid, properties=None, vector=None):
|
||||
"""Stand-in for a weaviate v4 returned object."""
|
||||
obj = MagicMock()
|
||||
obj.uuid = uuid
|
||||
obj.properties = properties
|
||||
obj.vector = vector
|
||||
return obj
|
||||
|
||||
|
||||
def _page(objects):
|
||||
"""Stand-in for a fetch_objects() response."""
|
||||
response = MagicMock()
|
||||
response.objects = objects
|
||||
return response
|
||||
|
||||
|
||||
def _store_with_pages(*pages):
|
||||
store = WeaviateStore()
|
||||
store.collection = MagicMock()
|
||||
store.collection.query.fetch_objects.side_effect = list(pages)
|
||||
return store
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_threads_uuid_cursor_across_pages():
|
||||
"""The next page must continue after the last object's UUID."""
|
||||
store = _store_with_pages(
|
||||
_page([_obj("uuid-1"), _obj("uuid-2")]),
|
||||
_page([_obj("uuid-3")]),
|
||||
)
|
||||
|
||||
result = list(store.iter_all(batch_size=2))
|
||||
|
||||
assert [item["id"] for item in result] == ["uuid-1", "uuid-2", "uuid-3"]
|
||||
calls = store.collection.query.fetch_objects.call_args_list
|
||||
assert "after" not in calls[0][1]
|
||||
assert calls[1][1]["after"] == "uuid-2"
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_stops_on_short_page():
|
||||
"""A page smaller than batch_size means the collection is exhausted."""
|
||||
store = _store_with_pages(_page([_obj("uuid-1")]))
|
||||
|
||||
result = list(store.iter_all(batch_size=5))
|
||||
|
||||
assert [item["id"] for item in result] == ["uuid-1"]
|
||||
assert store.collection.query.fetch_objects.call_count == 1
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_raises_when_cursor_stops_advancing():
|
||||
"""A stalled cursor must terminate, but not quietly: a partial scan reads
|
||||
as a complete one."""
|
||||
store = WeaviateStore()
|
||||
store.collection = MagicMock()
|
||||
store.collection.query.fetch_objects.return_value = _page(
|
||||
[_obj("same-uuid"), _obj("same-uuid")]
|
||||
)
|
||||
|
||||
with pytest.raises(ProcessingError, match="stopped advancing"):
|
||||
list(store.iter_all(batch_size=2))
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_continues_past_empty_page_in_cursor_mode():
|
||||
"""A full page followed by an empty page must not be read as the end of
|
||||
the collection: the empty page could be a gap (e.g. a window landing on
|
||||
tombstoned objects) with real data past it, the same failure mode
|
||||
already confirmed for Qdrant's scroll cursor (#1316). The `after` cursor
|
||||
has no server-issued value to advance past an empty page with, so this
|
||||
must fall back to offset pagination rather than silently stopping."""
|
||||
store = _store_with_pages(
|
||||
_page([_obj("uuid-1"), _obj("uuid-2")]), # full page, cursor -> uuid-2
|
||||
_page([]), # empty page: not the end
|
||||
_page([_obj("uuid-3")]), # real data past the gap
|
||||
)
|
||||
|
||||
result = [item["id"] for item in store.iter_all(batch_size=2)]
|
||||
|
||||
assert result == ["uuid-1", "uuid-2", "uuid-3"]
|
||||
calls = store.collection.query.fetch_objects.call_args_list
|
||||
assert len(calls) == 3
|
||||
assert calls[1][1]["after"] == "uuid-2" # the empty page still queried by cursor
|
||||
assert calls[2][1].get("offset") == 2 # then the fallback used position, not the cursor
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_offset_fallback_advances_across_pages():
|
||||
"""Regression: the offset was only set inside the except branch, so pages
|
||||
after the fallback went out with no pagination at all and the scan
|
||||
restarted from page one."""
|
||||
store = WeaviateStore()
|
||||
store.collection = MagicMock()
|
||||
calls = []
|
||||
|
||||
def _fetch(**kwargs):
|
||||
calls.append(dict(kwargs))
|
||||
if "after" in kwargs:
|
||||
raise TypeError("unexpected keyword argument 'after'")
|
||||
page_number = len(calls)
|
||||
if page_number < 4:
|
||||
return _page([_obj(f"u{page_number}a"), _obj(f"u{page_number}b")])
|
||||
return _page([_obj("last")])
|
||||
|
||||
store.collection.query.fetch_objects.side_effect = _fetch
|
||||
|
||||
ids = [item["id"] for item in store.iter_all(batch_size=2)]
|
||||
|
||||
assert len(set(ids)) == len(ids), f"duplicate ids means the scan restarted: {ids}"
|
||||
assert [c.get("offset") for c in calls] == [None, None, 2, 4]
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_raises_when_no_pagination_is_supported():
|
||||
"""A client rejecting both `after` and `offset` cannot page past the first
|
||||
result."""
|
||||
store = WeaviateStore()
|
||||
store.collection = MagicMock()
|
||||
|
||||
def _fetch(**kwargs):
|
||||
if "after" in kwargs or "offset" in kwargs:
|
||||
raise TypeError("unsupported")
|
||||
return _page([_obj("a"), _obj("b")])
|
||||
|
||||
store.collection.query.fetch_objects.side_effect = _fetch
|
||||
|
||||
with pytest.raises(ProcessingError, match="neither an .after. cursor nor a"):
|
||||
list(store.iter_all(batch_size=2))
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_empty_collection_yields_nothing():
|
||||
"""A genuinely empty collection needs two empty pages to confirm: the
|
||||
first (in cursor mode) triggers the offset fallback, and the second
|
||||
(in offset mode, which has no gap ambiguity) is what actually ends the
|
||||
scan. See test_iter_all_continues_past_empty_page_in_cursor_mode for the
|
||||
case where the first empty page is *not* the end."""
|
||||
store = _store_with_pages(_page([]), _page([]))
|
||||
|
||||
assert list(store.iter_all()) == []
|
||||
assert store.collection.query.fetch_objects.call_count == 2
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_converts_objects_to_the_shared_result_shape():
|
||||
store = _store_with_pages(
|
||||
_page([_obj("uuid-7", properties={"tag": "x"}, vector=[0.1, 0.2, 0.3])]),
|
||||
)
|
||||
|
||||
item = list(store.iter_all())[0]
|
||||
|
||||
assert item["id"] == "uuid-7"
|
||||
assert item["metadata"] == {"tag": "x"}
|
||||
np.testing.assert_allclose(item["vector"], np.array([0.1, 0.2, 0.3]))
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_handles_missing_properties_and_vector():
|
||||
store = _store_with_pages(_page([_obj("uuid-1", properties=None, vector=None)]))
|
||||
|
||||
item = list(store.iter_all())[0]
|
||||
|
||||
assert item["metadata"] == {}
|
||||
assert item["vector"] is None
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_treats_empty_vector_as_none():
|
||||
store = _store_with_pages(_page([_obj("uuid-1", vector=[])]))
|
||||
|
||||
assert list(store.iter_all())[0]["vector"] is None
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_requests_vectors():
|
||||
"""Weaviate omits vectors unless include_vector is set."""
|
||||
store = _store_with_pages(_page([]), _page([]))
|
||||
|
||||
list(store.iter_all(batch_size=64))
|
||||
|
||||
kwargs = store.collection.query.fetch_objects.call_args[1]
|
||||
assert kwargs["include_vector"] is True
|
||||
assert kwargs["limit"] == 64
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_falls_back_to_offset_when_after_unsupported():
|
||||
"""Older clients reject `after`; the scan degrades to numeric offset."""
|
||||
store = WeaviateStore()
|
||||
store.collection = MagicMock()
|
||||
seen = {"calls": 0}
|
||||
|
||||
def _fetch(**kwargs):
|
||||
if "after" in kwargs:
|
||||
raise TypeError("unexpected keyword argument 'after'")
|
||||
seen["calls"] += 1
|
||||
if seen["calls"] == 1:
|
||||
return _page([_obj("uuid-1"), _obj("uuid-2")])
|
||||
return _page([_obj("uuid-3")])
|
||||
|
||||
store.collection.query.fetch_objects.side_effect = _fetch
|
||||
|
||||
result = list(store.iter_all(batch_size=2))
|
||||
|
||||
assert [item["id"] for item in result] == ["uuid-1", "uuid-2", "uuid-3"]
|
||||
offsets = [
|
||||
c[1]["offset"]
|
||||
for c in store.collection.query.fetch_objects.call_args_list
|
||||
if "offset" in c[1]
|
||||
]
|
||||
assert offsets == [2]
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_raises_when_collection_not_initialized():
|
||||
"""Must fail loudly, not yield nothing.
|
||||
|
||||
An empty scan is indistinguishable from an empty source, which would let
|
||||
`store migrate` report success having copied nothing (issue #1083).
|
||||
"""
|
||||
store = WeaviateStore()
|
||||
|
||||
with pytest.raises(ProcessingError, match="Collection not initialized"):
|
||||
list(store.iter_all())
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", False)
|
||||
def test_iter_all_raises_when_weaviate_unavailable():
|
||||
store = WeaviateStore()
|
||||
store.collection = MagicMock()
|
||||
|
||||
with pytest.raises(ProcessingError):
|
||||
list(store.iter_all())
|
||||
|
||||
|
||||
@patch("semantica.vector_store.weaviate_store.WEAVIATE_AVAILABLE", True)
|
||||
def test_iter_all_propagates_fetch_errors():
|
||||
store = WeaviateStore()
|
||||
store.collection = MagicMock()
|
||||
store.collection.query.fetch_objects.side_effect = RuntimeError("connection reset")
|
||||
|
||||
with pytest.raises(RuntimeError, match="connection reset"):
|
||||
list(store.iter_all())
|
||||
Reference in New Issue
Block a user