mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into fix/1009-edge-labels

Browse Source
This commit is contained in:
Sameer Kadam
2026-08-18 13:11:35 +05:30
committed by GitHub
parent c58686b4ec 5c2901ae27
commit 55bde673c9
65 changed files with 7996 additions and 509 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/pull_request_template.md
+1 -1
View File
@@ -1,4 +1,4 @@
> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — comment on the issue and wait for assignment before opening a PR, to avoid duplicate work.
> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — wait for the issue to be assigned to you before opening a PR, to avoid duplicate work.
## Description
CHANGELOG.md
+45
View File
@@ -11,6 +11,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **First-class CrewAI integration** (#962)
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
- `integrations/crewai/SemanticaDecisionTool` — a CrewAI `BaseTool` wrapping `AgentContext` with 5 decision-intelligence actions (`record_decision`, `find_precedents`, `trace_causal_chain`, `analyze_impact`, `check_policy`)
- `integrations/crewai/SemanticaKnowledgeSource` — a CrewAI `BaseKnowledgeSource` that serializes a `ContextGraph` into crew knowledge storage; implements both the legacy `load_content()` and current `validate_content()`/`aadd()` contracts so it works across `crewai>=0.80.0`
- All three classes degrade gracefully when `crewai` is not installed (still importable, full Semantica API available)
- New `tests/integrations/crewai/`: 70 tests covering stub-based present-case behavior (Pydantic/BaseTool subclassing, every action, knowledge-source chunking/storage) plus a subprocess isolation test for the crewai-absent degradation path
- Docs: `docs/integrations/crewai.md` page, `docs.json` Integrations nav entry, and README integration-matrix/install updates
- **Hardened during code review**: live `graph`/`context`/extractor state is excluded from CrewAI JSON serialization (`model_dump(mode="json")`) with `model_post_init` self-healing defaults, so checkpoint/resume no longer raises `PydanticSerializationError`; `query_graph` now searches node content (not just ids/types); `trace_causal_chain` returns an explicit error instead of substituting similarity precedents when causal tracing is unavailable, and calls `trace_decision_causality(..., max_depth=...)` with the correct argument name; `find_precedents` propagates `max_precedents` as the backend `limit`; `add_to_graph` writes are serialized under a module lock so concurrent agents can't double-count duplicate adds; nameless entities are skipped instead of creating `repr()`-junk nodes
- **Hardened during second code review**: `check_policy` rules are now coerced type-aware — `bool("false")` was truthy, so `enabled == false` reported a violation for `enabled: false`, and string datums like `"0.90"` were compared lexicographically instead of numerically; `trace_causal_chain` no longer raises `AttributeError` (which escaped the tool) when the decision context has no `knowledge_graph`, returning honest error JSON instead; knowledge-source storage failures log an actionable ERROR (a missing crew embedder otherwise silently left agents with empty retrieval); `add_to_graph` uses a per-graph re-entrant lock instead of a process-global one (independent graphs no longer serialize each other, and re-entrant extractors can't deadlock); entity/relation `confidence=None` normalizes to `1.0` instead of failing the whole extraction; added a subprocess integration test against the real `crewai` package covering `Crew`-level serialization round-trip and restore
- **`ContextGraph` gains retraction and purge — the graph previously had no way to remove a node or edge without discarding everything via `clear()`** (#957, closes #955) by @pravit-amp, reviewed by @KaifAhmad1
- `retract_node()`/`retract_edge()` close an entity's validity window rather than deleting it, reusing the existing `valid_from`/`valid_until`/`state_at()` machinery: the entity drops out of `find_active_nodes()` and future `state_at()` queries going forward, but `state_at()` calls before the retraction time still return it, so decisions recorded against it stay explainable. A `("kind", id)`-keyed retraction record captures who/why/when, retrievable via `get_retraction()`/`list_retractions()`
- `purge_node()`/`purge_edge()` are the destructive counterpart: the entity is removed outright, from history as well as the active view, for erasure obligations retraction alone cannot satisfy (e.g. GDPR Article 17). Only a tombstone remains — that a purge happened, when, and why — deliberately never the purged content, via `get_tombstone()`/`list_tombstones()`. Purge is graph-scope only: `AgentMemory` and any bound vector store are not reached, so it is one step of an erasure workflow rather than the whole of it
@@ -62,6 +73,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305
- `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache
- Both now route through `load_spacy_model()`, sharing one cached `Language` instance per model name across `split_by_sentences()`, `SemanticChunker`, and `NERExtractor`; a missing model still falls back to regex/paragraph chunking without poisoning the cache for a later successful load
- **Fixed during review** (@Sameer6305): `NERExtractor.__init__()` still had a direct `spacy.load()` call site with the same cache-bypass issue, outside the two files named in #998 but sharing the same root cause; routed through the cache alongside stale test patch targets and a strengthened cache-configuration assertion
- **Fixed during review** (@KaifAhmad1): `SemanticChunker.__init__` only caught `OSError` around `load_spacy_model()`, while the sibling fix to `NERExtractor` in this same PR added a broader `except Exception` for a model that is installed but fails at runtime (e.g. a config incompatible with the installed spaCy version). A broken-but-present model crashed `SemanticChunker()` outright instead of degrading to fallback chunking like every other path in this PR. Added the matching `except Exception` branch, leaving `self.nlp` as `None`; new `test_semantic_chunker_falls_back_when_spacy_runtime_is_broken` mirrors the existing `NERExtractor` regression test for the same scenario
- New `tests/split/test_spacy_model_cache.py`: cache reuse across repeated calls/instances, shared cache between `split_by_sentences()`/`SemanticChunker`/`NERExtractor`, distinct model names loading separately, missing-model fallback without poisoning the cache, and the broken-runtime fallback added above
- `pytest tests/split/test_spacy_model_cache.py tests/split/test_splitter.py tests/split/test_chunkers.py`: all passing (3 pre-existing, unrelated `tests/test_ner_configurations.py` failures confirmed present on `main` before this PR)
- **`export_yaml` raised a raw `AttributeError` on list input, silently wrote empty exports for unrecognized dict keys, and graph payloads were reconciled differently by every exporter** (#958, closes #956, #952, #953) by @pravit-amp, reviewed by @Sameer6305
- Graph payloads circulate under two vocabularies, `entities`/`relationships` and `nodes`/`edges`, and each exporter reconciled them locally with a different idiom — `LPGExporter` in particular dropped every entity whenever `nodes` was present but empty, the exact shape `JSONExporter` emits. A new `normalize_graph_payload()` in `utils/helpers.py` centralizes that decision once, adopted by `LPGExporter`, `ArangoAQLExporter`, `Neo4jCSVExporter`, and both YAML exporters; `ContextGraph.to_dict()` now round-trips through YAML correctly as a result
- `export_yaml(records, path)` on a bare list previously failed with `AttributeError` from inside the exporter; it and the other YAML methods now reject non-mapping input with an actionable `ProcessingError` naming the expected keys, since these formats distinguish entities/relationships/triplets and guessing which one a list represents would mislabel the records
- `export_yaml({"data": [...]}, path)` previously wrote a structurally valid file with every collection empty, no exception, no warning, and the progress log reporting a completed export. `export_semantic_network`, `export_for_pipeline`, and `export_ontology_schema` now raise `ValidationError` when the payload shares no recognized key with what the method reads, or resolves to nothing while an unread key still holds records — an empty mapping is still accepted, since a genuinely empty graph has no records to lose
- **Breaking**: the two cases above, plus a bare list, now raise instead of returning cleanly with data silently dropped or a raw `AttributeError` from exporter internals. Migration: pass records under a recognized key (`{"entities": [...]}` / `{"nodes": [...]}` for `semantic_network`, `{"classes": [...]}` for `schema`)
- **Fixed during review** (Qodo): progress tracking could report a completed export before the output directory existed or the file was written; `export()` now creates the directory and serializes before starting tracking, so a rejected export leaves nothing behind
- **Fixed during review** (@Sameer6305, round 1): `normalize_graph_payload()`'s collection resolver treated any truthy value as a collection — `{"entities": "abc"}` silently became three single-character records, `{"entities": 42}` leaked a raw `TypeError` from inside `list()`. Collection values are now validated before conversion, rejecting strings/bytes/mappings/non-iterable scalars by name. Separately, `Neo4jCSVExporter._normalize_graph` called the shared resolver with `require_recognized=False`, so it alone kept accepting an unrecognized mapping as a silent empty export; the opt-out (introduced earlier in this same PR, with no other caller) was removed
- **Fixed during review** (@Sameer6305, round 2): `YAMLSchemaExporter`'s usable-schema check could treat scalar schema metadata (`version`, `uri`, `title`, `description`) as evidence records had been exported, letting records under an unread key drop silently; and `_is_record()` accepted modules and class/type objects through the generic `__dict__` path, which would have reached exporter internals instead of failing at the boundary. Both closed, with regression coverage
- **Fixed during final maintainer review** (before merge): four more gaps in the shared boundary that the earlier rounds didn't reach
- `LPGExporter`/`ArangoAQLExporter` called `normalize_graph_payload()` with no type guard, so non-mapping input raised `ValidationError` from inside the resolver — while YAML and `Neo4jCSVExporter` raised `ProcessingError` for the identical mistake, per this PR's own stated contract. The `_require_mapping()` guard that already existed in `yaml_exporter.py` is now shared from `utils/helpers.py` and used by all three
- `Neo4jCSVExporter._normalize_graph` checked `isinstance(graph, dict)`, so a non-dict `Mapping` (`MappingProxyType`, `ChainMap`) fell through to the object-attribute branch and was rejected, even though the identical payload exported fine via `LPGExporter`/`ArangoAQLExporter`/YAML. Now checks `isinstance(graph, Mapping)`
- `normalize_graph_payload()` accepts dataclass and attribute-bearing object records (`Neo4jCSVExporter._record_to_dict` reads them), but `LPGExporter`/`ArangoAQLExporter` call `.get(...)` directly on resolved entities — an object-shaped record passed validation only to crash with a raw `AttributeError` once used, the exact failure this PR's boundary exists to prevent. Records are now converted to plain dicts at the boundary (`_coerce_records` → new `_record_to_dict`), so every consumer gets a uniform shape regardless of which reading the caller used
- Two non-empty spellings of the same collection (e.g. `entities` and `nodes`) holding identical records in a different order were rejected as conflicting, since the check used plain list equality; a caller round-tripping through a dict-keyed cache or a set has no reason to preserve order. Comparison is now an order-independent multiset of each record's canonical JSON form
- New regression coverage in `tests/utils/test_normalize_graph_payload.py`: exception-type parity for non-mapping input across `export_lpg`/`export_arango`/`export_neo4j_csv`, dataclass-record conversion verified end-to-end through the same three exporters, `Neo4jCSVExporter` accepting a `MappingProxyType` payload, and reordered-alias equality (plus a duplicate-count case confirming the multiset check still catches real conflicts); 4 existing tests updated to assert the corrected dict-conversion behavior instead of the previous object passthrough
- `pytest tests/export tests/utils tests/context tests/test_export_module.py tests/test_export_methods_wrapper.py tests/test_notebooks_simulation.py`: 718 passed, 4 skipped (up from 641 passed, 62 subtests at PR submission); `black`/`isort`/`flake8 --max-line-length=88` clean on every line this PR touches; `python -m build`: succeeds
- **`ContextGraph.add_edge` had no dedupe — identical edges were stored repeatedly under one shared edge ID, and re-ingest doubled the edge set** (#926, closes #922) by @pravit-amp
- `_add_internal_edge` appended to `self.edges`, `edge_type_index`, and `_adjacency` unconditionally, with no check for an edge already present. Edge identity is content-derived (`_resolve_edge_identity` builds `edge_id` from `source_id`/`target_id`/`edge_type`/`weight`/`metadata`/`valid_from`/`valid_until`), so two identical `add_edge` calls produced two edge objects sharing one `edge_id` — the graph already considered them the same edge, it just kept both copies. `self.nodes` already deduped by ID; edges did not, so `stats()["edge_count"]` inflated, `density()` could exceed its mathematical maximum of `1.0`, and a refresh/restore job calling `build_from_entities_and_relationships()` (or reloading a saved graph) doubled the edge set on every cycle
- Added an `edge_id -> ContextEdge` index (`_edge_index`), mirroring how `self.nodes` dedupes by node ID. `_add_internal_edge` now returns `False` when the `edge_id` already exists, checked before touching `edges`/`edge_type_index`/`_adjacency` and before firing the mutation callback, so a repeat `add_edge` is a silent no-op with no phantom `ADD_EDGE` audit event
@@ -141,6 +176,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Security
- **`Authorization`/`Proxy-Authorization` credentials could leak to a different origin across HTTP redirects, and several ingest paths bypassed the shared SSRF/redirect guard entirely** (#1067, closes #947) by @Sameer6305, reviewed by @KaifAhmad1
- `request_with_ssrf_guard()` previously only stripped sensitive headers from per-request `kwargs["headers"]` on a cross-origin redirect; session-level `Authorization`/`Proxy-Authorization` headers, `session.auth`, and `session.trust_env` (`.netrc` lookup) could all still resurrect credentials on the hop to a foreign origin. All five credential sources are now stripped case-insensitively, kept stripped for the remainder of a multi-hop redirect chain (no resurrection even if a later hop returns to the original host), and unconditionally restored via `finally` — including on exceptions and redirect-limit errors
- `MCPClient._send_request_http()` and `PublicAPIIngestor.detect_public_api()`/`ingest_public_api()` called `httpx.post()`/`requests.post()`/`session.request()` directly, bypassing `request_with_ssrf_guard()` entirely. Both now route through the shared guard, including when `validate_no_auth=False`
- `SeedDataManager.load_from_api()` mutated the caller-supplied `headers` dict in place when adding an API-key `Authorization` header, silently leaking the key back into a dict the caller might reuse elsewhere. Now copies before modifying
- **Fixed during review** (@KaifAhmad1): `allow_private_ips=True` (used to let MCP servers run on localhost/internal networks) was applied to every redirect hop, not just the operator-configured host — a compromised or malicious MCP server could 302-redirect to an internal address (e.g. `169.254.169.254` cloud metadata) and the guard would follow it unchecked, defeating the SSRF protection this PR otherwise adds. Added `allow_private_ips_on_redirect` to `request_with_ssrf_guard()`: a redirect target inherits the original host's private-IP trust only when it matches that host; any other host falls back to strict validation. `MCPClient` now pins `allow_private_ips_on_redirect=False`, so only same-host redirects on a trusted MCP server keep working — a cross-host hop into private address space is blocked
- **Fixed during review** (@KaifAhmad1): `detect_public_api()` only caught `requests.exceptions.RequestException`, but `request_with_ssrf_guard()` raises `ValidationError` (a disjoint hierarchy) for SSRF-blocked hosts, blocked redirect targets, missing `Location`, or exceeded redirect limits — unlike its sibling `ingest_public_api()`, which already caught it. Callers (including `is_public_api()`) got an undocumented raw `ValidationError` instead of `ProcessingError`, and the error-logging call was skipped. Now catches `(ValidationError, ProcessingError)` and re-raises, matching the sibling method
- **Fixed during review** (@KaifAhmad1): `detect_public_api()`/`ingest_public_api()` forwarded `**options` into `request_with_ssrf_guard(..., session=self.session, allow_private_ips=self.allow_private_ips, **request_options)` without stripping `session`/`allow_private_ips` from `request_options` first — a caller passing either through the per-call `**options` (a plausible mistake, since `allow_private_ips` is also a documented constructor-level knob) got a raw `TypeError: got multiple values for keyword argument`. Both are now popped from `request_options` before the call
- New regression coverage added during review: `TestAllowPrivateIpsOnRedirect` (cross-host redirect into private space blocked, same-host redirect trust preserved, default behavior unchanged for existing callers that don't pass the new kwarg) and `TestMCPClientAuthRedirect::test_redirect_to_private_ip_is_blocked`/`test_same_host_redirect_on_private_mcp_server_is_not_blocked` in `tests/ingest/test_auth_header_redirect_security.py`; `test_detect_public_api_propagates_ssrf_validation_error` and duplicate-kwarg regression tests for both methods in `tests/ingest/test_public_api_ingestor.py`
- `pytest tests/ingest/test_auth_header_redirect_security.py tests/ingest/test_public_api_ingestor.py tests/test_seed_manager.py tests/ingest/test_submodules.py tests/ingest/test_cookbook_integration.py`: 111 passed
- **`FeedIngestor`/`FeedMonitor` (RSS/Atom feed ingestion) had no SSRF protection, allowing requests to internal/private network targets** (#928, closes #927) by @ZohaibHassan16
- `FeedIngestor.ingest_feed()`, `discover_feeds()` (link-tag fetch, common-path HEAD probe, and feed-validation GET), and `FeedMonitor.check_updates()` all called `requests.get()`/`requests.head()` directly with default redirect-following and no scheme allowlist or private/loopback/link-local IP validation — despite `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()` already existing and being used by `web_ingestor.py`/`api_ingestor.py`. `ingest_feed()`'s own URL check only verified `urlparse(url).scheme`/`.netloc` were non-empty, never that the scheme was http/https or that the resolved target IP was safe. Reachable via the public `ingest_feed()`/`ingest()` entry points with any caller-supplied feed URL
- All 5 call sites now route through `request_with_ssrf_guard()`, which validates scheme (http/https only) and resolved IP before the request, and re-validates every redirect `Location` before following it — closing both the direct-IP and redirect-chain SSRF paths. Added an `allow_private_ips` config option to both `FeedIngestor` and `FeedMonitor`, consistent with the other ingestors
CONTRIBUTING.md
+17 -3
View File
@@ -25,9 +25,9 @@ If you want to work on an open GitHub issue, please follow these steps to keep t
1. **Check the issue.** Look at the issue's assignees and recent comments. If someone is already actively working on it, consider a different issue or ask in the comments whether help is welcome.
2. **Comment before you start.** Leave a comment on the issue saying you'd like to work on it — something like *"I'd like to take this on"* is enough. This gives maintainers the context they need to assign the issue appropriately.
2. **Comment if you'd like the issue reserved.** Leaving a comment like *"I'd like to take this on"* is the fastest way to get assigned, but it isn't required — maintainers can also assign an issue directly to a contributor (e.g., based on recent activity in the repo) without waiting for a comment first.
3. **Wait for assignment.** A maintainer will review the request and assign the issue when appropriate. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
3. **Wait for assignment.** A maintainer will assign the issue when appropriate, whether or not a comment was left. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
4. **Create a branch and implement.** Once assigned, fork the repository (if you haven't already), create a dedicated branch, and begin your work.
@@ -37,12 +37,26 @@ If you want to work on an open GitHub issue, please follow these steps to keep t
5. **Open a focused PR and link the issue.** When you're ready, open a pull request and reference the issue in the description (e.g., `Closes #123`). Keep the PR scoped to the work described in the issue.
> **Why this matters:** Commenting before opening a PR helps maintainers track who is working on what, assign issues correctly, and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
> **Why this matters:** Assignment (with or without a comment) helps maintainers track who is working on what and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
Not sure where to start? Try a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue) or ask in [Discord](https://discord.gg/sV34vps5hH).
---
## 🔀 Duplicate PRs & Issue Priority
When more than one pull request targets the same issue, maintainers triage using this order of priority. These rules decide between PRs that are otherwise following the [assignment workflow above](#-working-on-an-existing-issue) — opening a PR before being assigned doesn't grant priority on its own, and an unassigned PR can still be closed as a duplicate once someone else is assigned to the issue.
1. **Contributor-raised issue with an existing PR.** If the person who opened the issue has also opened a PR for it, that PR is prioritized (they still need to be assigned before it's merged).
2. **Maintainer-raised issue with a claim comment.** If we opened the issue and someone has commented asking to work on it, we assign it to them and check their PR before picking up any other PR for the same issue.
3. **No prior assignment or comment.** If multiple PRs exist and no one was assigned or claimed the issue first, priority goes to whichever contributor has the most consistent activity in the repo over the last 60 days (e.g., merged PRs, substantive reviews, or issue triage participation) — not just PR volume.
4. **Late duplicate PRs.** If a PR is opened after another contributor has already been assigned to the issue, we close the duplicate early rather than let it sit open, and point the author to another open issue (or ask them to check `main` for newly opened ones). This avoids contributors spending time updating a PR that won't be merged.
5. **Overlapping scope.** If a PR covers multiple issues, or there's genuine overlap between competing PRs, maintainers discuss it on [Discord](https://discord.gg/sV34vps5hH) before deciding rather than resolving it unilaterally.
**Why this matters:** it keeps triage predictable, avoids wasted contributor effort on PRs that won't merge, and helps retain active contributors.
---
## 🎯 Ways to Contribute
### 💻 Code
README.md
+12 -12
View File
@@ -52,6 +52,8 @@ Most AI agents act without a trail. They store embeddings, not meaning: context
Semantica sits underneath your LLM, vector store, and agent framework as a deterministic infrastructure layer: no LLM required for graph construction, reasoning, or provenance.
> ⚠️ **System-level explainability, not foundation-model explainability.** Semantica does not expose or reconstruct what happens *inside* the LLM — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. Semantica explains what's *outside* the model: the context and data fed in, the decision produced, its provenance, relevant relationships, applied policies, and the full execution trail.
**Who it's for:**
- **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index
@@ -77,7 +79,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
- **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
- **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
- **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
- **Drop-in Integrations:** Native Agno support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
- **Drop-in Integrations:** Native Agno and CrewAI support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
---
@@ -1189,7 +1191,7 @@ Start with `semantica`, verify with `doctor`, build a graph, and explore the com
## Integrations
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno support for multi-agent shared context. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno and CrewAI support for agentic frameworks. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
@@ -1303,6 +1305,11 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<strong>Agno</strong><br/>
<sub>First-class · <code>pip install semantica[agno]</code></sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/crewAIInc/crewAI"><img src="https://github.com/crewAIInc.png?size=120" alt="CrewAI" width="48" height="48" /></a><br/>
<strong>CrewAI</strong><br/>
<sub>First-class · <code>pip install semantica[crewai]</code></sub>
</td>
</tr>
<tr>
<th colspan="8" align="left">Already Supported via REST API &amp; MCP</th>
@@ -1319,11 +1326,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<sub>REST API · MCP</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/crewAIInc/crewAI"><img src="https://github.com/crewAIInc.png?size=120" alt="CrewAI" width="48" height="48" /></a><br/>
<strong>CrewAI</strong><br/>
<sub>REST API · MCP</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/run-llama/llama_index"><img src="https://github.com/run-llama.png?size=120" alt="LlamaIndex" width="48" height="48" /></a><br/>
<strong>LlamaIndex</strong><br/>
<sub>REST API · MCP</sub>
@@ -1354,11 +1356,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<sub>Dedicated toolkit</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/crewAIInc/crewAI"><img src="https://github.com/crewAIInc.png?size=120" alt="CrewAI" width="48" height="48" /></a><br/>
<strong>CrewAI</strong><br/>
<sub>Dedicated toolkit</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/run-llama/llama_index"><img src="https://github.com/run-llama.png?size=120" alt="LlamaIndex" width="48" height="48" /></a><br/>
<strong>LlamaIndex</strong><br/>
<sub>Dedicated toolkit</sub>
@@ -1503,6 +1500,8 @@ Semantica is designed for environments where AI outputs must be explainable, aud
- **Cybersecurity:** Threat attribution, incident response timelines, and IOC provenance tracking
- **Autonomous Systems:** Decision logs, safety validation, and explainable AI for certification
> ⚠️ **This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits what the AI system did, not the LLM's private internal reasoning.
---
## Installation
@@ -1514,6 +1513,7 @@ pip install semantica[all] # everything
```bash
pip install semantica[agno] # Agno multi-agent integration
pip install semantica[crewai] # CrewAI integration
pip install semantica[llm-litellm] # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more
pip install semantica[graph-neo4j] # Neo4j graph store (LPG)
pip install semantica[graph-falkordb] # FalkorDB graph store (LPG)
docs/concepts.md
+3
View File
@@ -16,6 +16,9 @@ At its core, Semantica adds a **context and accountability layer** on top of you
- **Accountability Layer** — Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable.
- **Extension Layer** — `PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code.
<Warning>
**This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits *what the AI system did*, not the foundation model's private internal reasoning.
</Warning>
## Knowledge Graphs
docs/docs.json
+1
View File
@@ -102,6 +102,7 @@
"group": "Integrations",
"pages": [
"integrations/agno",
"integrations/crewai",
"integrations/docling",
"integrations/snowflake",
"integrations/databricks"
docs/faq.md
+10
View File
@@ -52,6 +52,16 @@ Semantica works alongside these frameworks, not against them.
</Accordion>
<Accordion title="Does Semantica explain an LLM's internal reasoning or chain-of-thought?" icon="triangle-exclamation">
No. This is **system-level explainability, not foundation-model explainability**. Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system.
What Semantica explains is *outside* the model: what context and data were used, what decision was produced, the provenance behind it, the relevant relationships, the policies applied, and the resulting decision trail.
In short: Semantica explains and audits *what the AI system did* — not the foundation model's private internal reasoning.
</Accordion>
<Accordion title="Is Semantica free?" icon="tag">
Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source.
docs/index.md
+5 -1
View File
@@ -192,7 +192,11 @@ decision_id = context.record_decision(
## Built for Where Mistakes Have Consequences
Semantica was designed for domains where every decision must be explainable and every fact must be traceable:
Semantica was designed for domains where every decision must be explainable and every fact must be traceable.
<Warning>
**This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. See [Core Concepts](concepts) for the full scope note.
</Warning>
**Healthcare & Life Sciences**
- Clinical decision support with full audit trails
docs/integrations/crewai.md
+147
View File
@@ -0,0 +1,147 @@
---
title: "CrewAI Integration"
description: "Give CrewAI crews a shared semantic knowledge graph, decision intelligence, and graph-based retrieval via three drop-in components."
icon: "users"
---
> Three drop-in components that bring Semantica's knowledge graph and decision intelligence into any CrewAI crew.
## Installation
```bash
pip install "semantica[crewai]"
```
Requires `crewai >= 0.80.0`. If `crewai` is not installed, the integration still imports — every class carries the full Semantica API and degrades gracefully, but cannot be passed to a `Crew`.
## Components at a Glance
- **SemanticaKGTool** — `Agent(tools=[…])`: 5 KG construction/query actions: extract entities, extract relations, add to graph, query graph, find related.
- **SemanticaDecisionTool** — `Agent(tools=[…])`: 5 decision intelligence actions: record decisions, find precedents, trace causal chains, analyze impact, check policies.
- **SemanticaKnowledgeSource** — `Crew(knowledge_sources=[…])`: Serializes a `ContextGraph` into CrewAI knowledge storage so every agent gets retrieval access to the graph.
## Component Details
<Tabs>
<Tab title="SemanticaKGTool">
Lets agents actively **build and query** a shared `ContextGraph` mid-reasoning.
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKGTool
graph = ContextGraph()
analyst = Agent(
role="Knowledge Analyst",
goal="Build and explore a knowledge graph from documents",
backstory="You map entities and relationships into a shared graph.",
tools=[SemanticaKGTool(graph=graph)],
)
crew = Crew(
agents=[analyst],
tasks=[Task(
description="Extract and link key entities from the brief",
expected_output="JSON",
agent=analyst,
)],
)
crew.kickoff()
```
| Tool | Description |
| :------ | :------------- |
| `extract_entities` | Extract named entities from `text` |
| `extract_relations` | Extract relationships between entities in `text` |
| `add_to_graph` | Extract entities/relations from `text` and add them to the shared graph |
| `query_graph` | Keyword-search the graph by node id, type, and content using `query` |
| `find_related` | Find concepts related to `entity` within `hops` hops |
All actions return JSON so agents get parseable results.
**Sharing a graph:** the tool reads/writes whatever `graph` you pass in. When no `graph` is given, a fresh in-memory `ContextGraph()` is created (and a warning is logged) — two tool instances that each auto-create their own graph do **not** share knowledge. Pass the same `ContextGraph` to every agent that must share state.
</Tab>
<Tab title="SemanticaDecisionTool">
Exposes Semantica's decision intelligence as a native CrewAI tool, backed by `AgentContext`.
```python
from crewai import Agent, Crew, Task
from integrations.crewai import SemanticaDecisionTool
planner = Agent(
role="Decision Planner",
goal="Make grounded, precedented decisions",
backstory="You record decisions and validate them against policy.",
tools=[SemanticaDecisionTool()],
)
crew = Crew(agents=[planner], tasks=[...])
```
When no `AgentContext` is passed, one is created in-memory with `decision_tracking=True` and its own `ContextGraph`, so decision actions work out of the box (a warning is logged — pass the same `AgentContext` to every agent that must share decision state). Missing optional fields in `record_decision` fall back to `category="general"`, `reasoning="agent decision"`, and `outcome="recorded"`. `find_precedents` returns up to `max_precedents` results. If a knowledge graph cannot trace causality, `trace_causal_chain` returns an explicit error rather than substituting similarity-based results.
| Tool | Description |
| :------ | :------------- |
| `record_decision` | Record a decision with reasoning, outcome, and confidence |
| `find_precedents` | Search for similar past decisions |
| `trace_causal_chain` | Trace the causal chain from a decision |
| `analyze_impact` | Assess downstream influence of a decision |
| `check_policy` | Validate a proposed decision against policy rules |
</Tab>
<Tab title="SemanticaKnowledgeSource">
Gives **every agent in the crew** retrieval access to a `ContextGraph`.
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKnowledgeSource
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="...")
researcher = Agent(
role="Policy Researcher",
goal="Answer questions from the knowledge base",
backstory="You retrieve from graph knowledge to answer accurately.",
)
crew = Crew(
agents=[researcher],
tasks=[...],
knowledge_sources=[SemanticaKnowledgeSource(graph=graph)],
)
```
On kickoff the graph's nodes and edges are serialized, chunked, and stored through CrewAI's knowledge pipeline.
> **Embedder required:** storing chunks goes through CrewAI's knowledge pipeline, which needs an embedder to be configured. Set `Crew(embedder=...)` (or provide the default credentials CrewAI falls back to, e.g. `OPENAI_API_KEY`). If no working embedder is configured, storage fails, an ERROR is logged, and agents will retrieve **nothing** — the crew still runs, but its knowledge queries return empty.
**Compatibility:** CrewAI's `BaseKnowledgeSource` contract changed between `0.80.x` and current releases (`load_content()` → `validate_content()`/`aadd()`). `SemanticaKnowledgeSource` implements both legacy and current methods, so it works across `crewai>=0.80.0`.
</Tab>
</Tabs>
## Checkpoints & Serialization
CrewAI serializes tools and knowledge sources to JSON for checkpointing/resume. Live Semantica state (`ContextGraph`, `AgentContext`, extractors) is **excluded from that serialization** — a restored tool/source comes back with a fresh in-memory `ContextGraph` and logs a warning. Until you re-attach the live graph/context, the restored objects answer queries against an **empty** graph, so re-wire them after resuming (e.g. `restored_tool.graph = live_graph`) before agents continue.
## API Reference
```python
from integrations.crewai import (
SemanticaKGTool, # BaseTool: KG construction/query actions
SemanticaDecisionTool, # BaseTool: decision intelligence actions
SemanticaKnowledgeSource, # BaseKnowledgeSource: graph → crew knowledge
CREWAI_AVAILABLE, # bool: True if crewai is installed
)
```
All three classes are usable without `crewai` installed: they carry the full Semantica API and degrade gracefully.
## See Also
- [Context Module](../reference/context) — AgentContext and ContextGraph backing the integration.
- [Semantic Extraction](../reference/semantic_extract) — NERExtractor / RelationExtractor used by SemanticaKGTool.
- [LLMs](../reference/llms) — Configure LLM providers for your crew's agents.
- [Vector Store](../reference/vector_store) — Vector backend used by SemanticaDecisionTool.
docs/reference/export.md
+13
View File
@@ -203,6 +203,13 @@ export_lpg(graph, "import.cypher", method="cypher")
exporter = SemanticNetworkYAMLExporter()
exporter.export(graph, "graph.yaml")
```
The YAML exporters read `entities`/`relationships`/`triplets` (with
`nodes`/`edges` accepted as aliases, so `ContextGraph.to_dict()` exports
directly). A non-empty mapping supplying none of them raises
`ValidationError` rather than writing a file with every collection empty,
as does one whose collection value is not a list of records
(`{"entities": "abc"}`).
</Tab>
<Tab title="Graph DB Import">
**LPGExporter** writes Cypher `CREATE` statements for Neo4j and Memgraph:
@@ -236,6 +243,12 @@ export_lpg(graph, "import.cypher", method="cypher")
Both exporters write to a file and return `None`.
`LPGExporter`, `ArangoAQLExporter`, and `Neo4jCSVExporter` resolve mapping
payloads on the same terms as the YAML exporters above, so an unrecognized
or malformed mapping is rejected instead of exported as an empty graph.
`Neo4jCSVExporter` still reads graph *objects* off their
`nodes`/`entities` and `edges`/`relationships` attributes.
<Warning>
**`ArangoAQLExporter.export()` and `LPGExporter.export()` write to a file and return `None`.** They do not return the AQL/Cypher string. Write to a file and read it back if you need the string.
</Warning>
integrations/crewai/README.md
+108
View File
@@ -0,0 +1,108 @@
# Semantica × CrewAI
First-class integration between Semantica and [CrewAI](https://github.com/crewAIInc/crewAI) — give your crews a shared semantic knowledge graph, decision intelligence, and graph-based retrieval.
## Installation
```bash
pip install semantica[crewai]
```
Requires `crewai >= 0.80.0`. If `crewai` is not installed, the integration still imports (classes degrade gracefully), but you can't pass the objects to a `Crew`.
> **⚠️ Security note:** crewai hard-requires `chromadb~=1.1.0`, which is currently affected by the unpatched pre-authentication code-injection advisory **CVE-2026-45829** (no fixed release — even the latest chromadb 1.5.9 is affected). Installing `semantica[crewai]` pulls that dependency into your environment. The `crewai` extra is intentionally **not** part of `semantica[all]` for this reason — only install it where you actually use CrewAI, and follow chromadb for a patched release.
## 1. SemanticaKGTool
A `BaseTool` that lets agents **build and query** a shared `ContextGraph` mid-reasoning:
- `extract_entities` — extract named entities from `text`
- `extract_relations` — extract relationships from `text`
- `add_to_graph` — extract entities/relations from `text` and add them to the shared graph
- `query_graph` — keyword-search the graph using `query`
- `find_related` — find concepts related to `entity` within `hops`
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKGTool
graph = ContextGraph()
analyst = Agent(
role="Knowledge Analyst",
goal="Build and explore a knowledge graph from documents",
backstory="You map entities and relationships into a shared graph.",
tools=[SemanticaKGTool(graph=graph)],
)
crew = Crew(
agents=[analyst],
tasks=[Task(description="Extract and link key entities from the brief", expected_output="JSON", agent=analyst)],
)
result = crew.kickoff()
```
All actions return JSON, so agents get parseable results.
## 2. SemanticaDecisionTool
A `BaseTool` that wraps `AgentContext` and exposes decision intelligence:
- `record_decision` — record a decision with reasoning and outcome
- `find_precedents` — retrieve past decisions similar to a scenario
- `trace_causal_chain` — trace the causal chain from a decision
- `analyze_impact` — assess downstream influence using graph centrality
- `check_policy` — validate a proposed decision against rule-based policies
```python
from crewai import Agent, Crew, Task
from integrations.crewai import SemanticaDecisionTool
planner = Agent(
role="Decision Planner",
goal="Make grounded, precedented decisions",
backstory="You record decisions and validate them against policy.",
tools=[SemanticaDecisionTool()],
)
crew = Crew(agents=[planner], tasks=[...])
```
When no `AgentContext` is passed, one is created in-memory with `decision_tracking=True`.
## 3. SemanticaKnowledgeSource
A `BaseKnowledgeSource` that serializes the current state of a `ContextGraph` (nodes, edges, metadata) into CrewAI's knowledge storage, giving **every agent in the crew** retrieval access to the graph:
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKnowledgeSource
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="...")
researcher = Agent(
role="Policy Researcher",
goal="Answer questions from the knowledge base",
backstory="You retrieve from graph knowledge to answer accurately.",
)
crew = Crew(
agents=[researcher],
tasks=[...],
knowledge_sources=[SemanticaKnowledgeSource(graph=graph)],
)
```
> **Embedder required:** storing chunks goes through CrewAI's knowledge pipeline, which needs an embedder. Set `Crew(embedder=...)` (or provide CrewAI's default credentials, e.g. `OPENAI_API_KEY`). Without a working embedder, storage fails, an ERROR is logged, and agents retrieve nothing — the crew still runs with empty knowledge queries.
### Compatibility note
CrewAI's `BaseKnowledgeSource` contract changed between `0.80.x` and current releases (`load_content()` → `validate_content()`/`aadd()`). `SemanticaKnowledgeSource` implements both the legacy and current methods, so it works across `crewai>=0.80.0`.
### Sharing state & checkpoints
- Each tool/source holds whatever `graph`/`context` you pass it. When omitted, a fresh in-memory object is created and a warning is logged — instances that auto-create their own state do **not** share knowledge, so pass the same object to every agent that must share.
- Live state (`ContextGraph`, `AgentContext`, extractors) is excluded from CrewAI's JSON serialization. After restoring from a checkpoint, re-attach the live graph/context to the restored objects.
integrations/crewai/__init__.py
+44
View File
@@ -0,0 +1,44 @@
"""
Semantica × CrewAI Integration
==============================
First-class integration between the Semantica semantic intelligence stack and
the `CrewAI <https://github.com/crewAIInc/crewAI>`_ agentic framework.
Public surface
--------------
SemanticaKGTool — CrewAI ``BaseTool`` exposing KG construction/query actions
SemanticaDecisionTool — CrewAI ``BaseTool`` exposing decision-intelligence actions
SemanticaKnowledgeSource— CrewAI ``BaseKnowledgeSource`` giving crews graph knowledge
Quick start
-----------
pip install semantica[crewai]
>>> from integrations.crewai import (
... SemanticaKGTool,
... SemanticaDecisionTool,
... SemanticaKnowledgeSource,
... )
Compatibility
-------------
Requires ``crewai >= 0.80.0``. All three classes degrade gracefully when
``crewai`` is not installed — they are still importable and carry the full
Semantica API, but cannot be passed to ``Crew`` / ``Agent`` constructors.
"""
from ._availability import CREWAI_AVAILABLE, CREWAI_IMPORT_ERROR
from .decision_tool import SemanticaDecisionTool
from .kg_tool import SemanticaKGTool
from .knowledge_source import SemanticaKnowledgeSource
__all__ = [
"SemanticaKGTool",
"SemanticaDecisionTool",
"SemanticaKnowledgeSource",
"CREWAI_AVAILABLE",
"CREWAI_IMPORT_ERROR",
]
__version__ = "0.1.0"
integrations/crewai/_availability.py
+24
View File
@@ -0,0 +1,24 @@
"""
Shared CrewAI availability probe.
Every integration module needs to know whether the real ``crewai`` package is
installed. Probing once here (instead of once per module) guarantees the
exported ``CREWAI_AVAILABLE`` flag means the *whole* integration is ready — a
caller gating on it will never see tools using CrewAI while a knowledge source
silently degrades (or vice versa).
"""
from typing import Optional
CREWAI_AVAILABLE = False
CREWAI_IMPORT_ERROR: Optional[str] = None
try:
from crewai.knowledge.source.base_knowledge_source import ( # noqa: F401
BaseKnowledgeSource,
)
from crewai.tools import BaseTool # noqa: F401
CREWAI_AVAILABLE = True
except ImportError as exc:
CREWAI_IMPORT_ERROR = str(exc)
integrations/crewai/decision_tool.py
+555
View File
@@ -0,0 +1,555 @@
"""
SemanticaDecisionTool — a CrewAI ``BaseTool`` exposing Semantica's decision
intelligence (``AgentContext``) to agents.
Lets agents record decisions with reasoning, retrieve past precedents, trace
causal chains, analyse downstream impact, and validate proposed decisions
against policy rules.
Install
-------
pip install semantica[crewai]
Example
-------
>>> from integrations.crewai import SemanticaDecisionTool
>>> from crewai import Agent, Crew, Task
>>> tool = SemanticaDecisionTool()
>>> crew = Crew(
... agents=[Agent(role="...", goal="...", backstory="...", tools=[tool])],
... tasks=[...],
... )
Tools exposed
-------------
record_decision — Record a decision with reasoning and outcome
find_precedents — Search past decisions similar to a scenario
trace_causal_chain— Trace the causal chain from a decision node
analyze_impact — Assess downstream influence of a decision
check_policy — Validate a proposed decision against policy rules
"""
from __future__ import annotations
import json
import re
from typing import Any, Dict, List, Literal, Optional, Type
from pydantic import BaseModel, Field
from semantica.utils.logging import get_logger
from ._availability import CREWAI_AVAILABLE
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: CrewAI BaseTool base class
# ---------------------------------------------------------------------------
_BaseTool: Any = object
if CREWAI_AVAILABLE:
from crewai.tools import BaseTool as _BaseTool # type: ignore
# ---------------------------------------------------------------------------
# Input schema
# ---------------------------------------------------------------------------
class SemanticaDecisionToolInput(BaseModel):
"""
Input schema for ``SemanticaDecisionTool``.
Exactly one action is dispatched per call; the remaining fields are only
used by the actions that need them.
"""
action: Literal[
"record_decision",
"find_precedents",
"trace_causal_chain",
"analyze_impact",
"check_policy",
] = Field(
...,
description=(
"Which decision-intelligence operation to run. One of: "
"'record_decision', 'find_precedents', 'trace_causal_chain', "
"'analyze_impact', 'check_policy'."
),
)
category: Optional[str] = Field(
None,
description="Domain category, e.g. 'loan_approval'. Used by 'record_decision'.",
)
scenario: Optional[str] = Field(
None,
description=(
"Short description of the situation. Used by 'record_decision' and "
"'find_precedents'."
),
)
reasoning: Optional[str] = Field(
None, description="Why this outcome was chosen. Used by 'record_decision'."
)
outcome: Optional[str] = Field(
None, description="The decision result. Used by 'record_decision'."
)
confidence: float = Field(
0.8,
ge=0.0,
le=1.0,
description="Confidence score in [0, 1]. Used by 'record_decision'.",
)
entities: Optional[str] = Field(
None,
description="Comma-separated entity names. Used by 'record_decision'.",
)
decision_id: Optional[str] = Field(
None,
description=(
"Identifier of a decision. Used by 'trace_causal_chain' and "
"'analyze_impact'."
),
)
depth: int = Field(
3,
ge=1,
le=20,
description="Maximum chain depth. Used by 'trace_causal_chain'.",
)
decision_data: Optional[str] = Field(
None,
description=(
"JSON object describing a proposed decision. Used by 'check_policy'."
),
)
policy_rules: Optional[str] = Field(
None,
description=(
"JSON list of rule strings like 'confidence >= 0.7'. Used by "
"'check_policy'."
),
)
# ---------------------------------------------------------------------------
# SemanticaDecisionTool
# ---------------------------------------------------------------------------
class SemanticaDecisionTool(_BaseTool): # type: ignore[misc]
"""
CrewAI tool that surfaces Semantica's decision intelligence as agent actions.
Parameters
----------
context:
A ``semantica.context.AgentContext`` (or compatible object exposing
``record_decision``, ``find_precedents_advanced``,
``analyze_decision_influence``). A fresh in-memory context is created
when ``None``.
max_precedents:
Default number of precedents returned by ``find_precedents``.
causal_depth:
Default chain depth used by ``trace_causal_chain``.
"""
name: str = "semantica_decision"
description: str = (
"Decision intelligence toolkit. Actions: 'record_decision' (record a "
"decision with category, scenario, reasoning, outcome, confidence), "
"'find_precedents' (search past decisions similar to 'scenario'), "
"'trace_causal_chain' (trace the causal chain from 'decision_id'), "
"'analyze_impact' (assess downstream influence of 'decision_id'), "
"'check_policy' (validate 'decision_data' JSON against 'policy_rules' "
"rules like 'confidence >= 0.7'). Returns JSON."
)
args_schema: Type[BaseModel] = SemanticaDecisionToolInput
context: Any = Field(default=None, exclude=True)
max_precedents: int = 5
causal_depth: int = 3
had_live_state: bool = False
reconstructed_state: bool = Field(default=False, exclude=True)
def __init__(
self,
context: Any = None,
max_precedents: int = 5,
causal_depth: int = 3,
**kwargs: Any,
) -> None:
if CREWAI_AVAILABLE:
super().__init__(
context=context,
max_precedents=max_precedents,
causal_depth=causal_depth,
**kwargs,
)
else:
super().__init__()
self.context = context
self.max_precedents = max_precedents
self.causal_depth = causal_depth
# Degraded mode is a plain class — no model_post_init lifecycle.
self._ensure_defaults()
logger.info("SemanticaDecisionTool initialised (crewai=%s)", CREWAI_AVAILABLE)
def model_post_init(self, __context: Any) -> None:
"""Re-create default state after validation/deserialisation.
``context`` is excluded from JSON serialisation (CrewAI checkpoints
serialise every tool via ``model_dump(mode="json")``), so a tool
restored from a checkpoint has ``None`` state until this runs.
"""
self._ensure_defaults()
super().model_post_init(__context)
def _ensure_defaults(self) -> None:
"""Lazy-import and build a real AgentContext when none is wired."""
if self.context is None:
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
self.context = AgentContext(
vector_store=VectorStore(backend="faiss"),
decision_tracking=True,
knowledge_graph=ContextGraph(),
)
if self.had_live_state:
self.reconstructed_state = True
logger.warning(
"SemanticaDecisionTool: the live decision context was lost "
"during serialization/checkpoint restore — an EMPTY "
"context was reconstructed; re-attach the original context "
"before continuing"
)
else:
logger.warning(
"SemanticaDecisionTool created a fresh in-memory "
"AgentContext — agents sharing decision state must be "
"wired to the same context"
)
self.had_live_state = True
# ------------------------------------------------------------------
# CrewAI entry points
# ------------------------------------------------------------------
def _run(
self,
action: str,
category: Optional[str] = None,
scenario: Optional[str] = None,
reasoning: Optional[str] = None,
outcome: Optional[str] = None,
confidence: float = 0.8,
entities: Optional[str] = None,
decision_id: Optional[str] = None,
depth: int = 3,
decision_data: Optional[str] = None,
policy_rules: Optional[str] = None,
**kwargs: Any,
) -> str:
valid = {
"record_decision",
"find_precedents",
"trace_causal_chain",
"analyze_impact",
"check_policy",
}
if action not in valid:
return json.dumps(
{
"error": f"Unknown action '{action}'. Valid actions: "
+ ", ".join(sorted(valid))
}
)
if action == "record_decision":
return self._record_decision(
category=category or "general",
scenario=scenario or "decision recorded",
reasoning=reasoning or "agent decision",
outcome=outcome or "recorded",
confidence=confidence,
entities=entities,
)
if action == "find_precedents":
return self._find_precedents(scenario=scenario or "", category=category)
if action == "trace_causal_chain":
return self._trace_causal_chain(decision_id or "", depth=depth)
if action == "analyze_impact":
return self._analyze_impact(decision_id or "")
return self._check_policy(decision_data or "", policy_rules)
async def _arun(self, action: str, **kwargs: Any) -> str:
"""Async variant of ``_run`` for CrewAI's async tool path."""
return self._run(action=action, **kwargs)
# ------------------------------------------------------------------
# Actions
# ------------------------------------------------------------------
def _record_decision(
self,
category: str,
scenario: str,
reasoning: str,
outcome: str,
confidence: float = 0.8,
entities: Optional[str] = None,
) -> str:
entity_list: Optional[List[str]] = None
if entities:
entity_list = [e.strip() for e in entities.split(",") if e.strip()]
try:
decision_id = self.context.record_decision(
category=category,
scenario=scenario,
reasoning=reasoning,
outcome=outcome,
confidence=float(confidence),
entities=entity_list,
)
result = {"decision_id": str(decision_id), "status": "recorded"}
logger.info("record_decision → %s", decision_id)
except Exception as exc:
result = {"error": str(exc), "status": "failed"}
logger.warning("record_decision failed: %s", exc)
return json.dumps(result)
def _find_precedents(
self,
scenario: str,
category: Optional[str] = None,
limit: Optional[int] = None,
) -> str:
k = limit if limit is not None else self.max_precedents
try:
precedents = self.context.find_precedents_advanced(
scenario=scenario,
category=category,
limit=k,
)
out: List[Dict[str, Any]] = []
for p in (precedents or [])[:k]:
if isinstance(p, dict):
out.append(p)
else:
out.append(
{
"scenario": getattr(p, "scenario", str(p)),
"outcome": getattr(p, "outcome", ""),
"confidence": getattr(p, "confidence", 0.0),
"category": getattr(p, "category", ""),
}
)
logger.info("find_precedents('%s') → %d results", scenario, len(out))
return json.dumps({"precedents": out, "count": len(out)})
except Exception as exc:
logger.warning("find_precedents failed: %s", exc)
return json.dumps({"precedents": [], "count": 0, "error": str(exc)})
def _trace_causal_chain(self, decision_id: str, depth: Optional[int] = None) -> str:
if not decision_id:
return json.dumps(
{
"error": "decision_id is required for trace_causal_chain",
"causal_chain": [],
"decision_id": "",
}
)
max_depth = depth or self.causal_depth
try:
graph = getattr(self.context, "knowledge_graph", None)
if graph is None:
return json.dumps(
{
"error": (
"causal tracing is not available on this knowledge "
"graph (the decision context has no knowledge_graph)"
),
"causal_chain": [],
"decision_id": decision_id,
}
)
trace = getattr(graph, "trace_decision_causality", None)
if trace is None:
return json.dumps(
{
"error": (
"causal tracing is not available on this knowledge graph "
"(graph.trace_decision_causality is not implemented)"
),
"causal_chain": [],
"decision_id": decision_id,
}
)
chain = trace(decision_id, max_depth=max_depth)
return json.dumps({"causal_chain": chain, "decision_id": decision_id})
except Exception as exc:
logger.warning("trace_causal_chain failed: %s", exc)
return json.dumps(
{"error": str(exc), "causal_chain": [], "decision_id": decision_id}
)
def _analyze_impact(self, decision_id: str) -> str:
try:
influence = self.context.analyze_decision_influence(decision_id)
if not isinstance(influence, dict):
influence = {"influence": str(influence)}
influence["decision_id"] = decision_id
return json.dumps(influence)
except Exception as exc:
logger.warning("analyze_impact failed: %s", exc)
return json.dumps({"error": str(exc), "decision_id": decision_id})
def _check_policy(
self,
decision_data: str,
policy_rules: Optional[str] = None,
) -> str:
try:
data = (
json.loads(decision_data)
if isinstance(decision_data, str)
else decision_data
)
except json.JSONDecodeError as exc:
return json.dumps(
{
"compliant": False,
"violations": [f"Invalid decision_data JSON: {exc}"],
"warnings": [],
}
)
if not isinstance(data, dict):
return json.dumps(
{
"compliant": False,
"violations": [
f"decision_data must decode to a JSON object, "
f"got {type(data).__name__}: {data!r}"
],
"warnings": [],
}
)
violations: List[str] = []
warnings: List[str] = []
rules: List[str] = []
if policy_rules:
try:
parsed_rules = json.loads(policy_rules)
except json.JSONDecodeError:
rules = [r.strip() for r in policy_rules.split(",") if r.strip()]
else:
if isinstance(parsed_rules, str):
rules = [parsed_rules]
elif isinstance(parsed_rules, list):
for item in parsed_rules:
if isinstance(item, str):
rules.append(item)
else:
warnings.append(
f"Ignoring non-string policy rule entry: {item!r}"
)
else:
warnings.append(
f"policy_rules must decode to a JSON list of rule strings, "
f"got {type(parsed_rules).__name__}: {parsed_rules!r}"
)
for rule in rules:
try:
if not self._eval_rule(rule, data):
violations.append(f"Rule violated: {rule}")
except Exception as exc:
warnings.append(f"Could not evaluate rule '{rule}': {exc}")
compliant = len(violations) == 0
logger.debug(
"check_policy: compliant=%s, violations=%d", compliant, len(violations)
)
return json.dumps(
{
"compliant": compliant,
"violations": violations,
"warnings": warnings,
}
)
def _eval_rule(self, rule: str, data: Dict[str, Any]) -> bool:
"""Evaluate a simple comparison rule (``field op value``) against data.
This is a small standalone evaluator for the tool's ``check_policy``
action — it is intentionally independent of Semantica's policy engine
so agents get a bounded, side-effect-free rule check. Rules are
``<field> <op> <value>`` comparisons only; there is no expression
evaluation (no ``eval``), so untrusted rule strings are safe to pass.
Values are coerced type-aware: ``true``/``false`` (and ``1``/``0``)
become booleans, numeric literals become numbers, and string values
that parse as numbers are compared numerically, so ``score == 0.9``
holds for ``score: "0.90"`` and ``enabled == false`` holds for
``enabled: false``. Field names may contain hyphens, dots and spaces
(e.g. ``risk-score >= 0.9``); they are matched against ``data`` keys
as-is.
"""
m = re.match(r"(.+?)\s*(>=|<=|!=|==|>|<)\s*(.+)$", rule.strip())
if not m:
raise ValueError(f"unrecognised rule format: {rule!r}")
field, op, val_str = m.group(1), m.group(2), m.group(3).strip().strip("\"'")
if field not in data:
raise ValueError(f"rule references undefined field {field!r}")
actual = data[field]
if actual is None:
raise ValueError(f"field {field!r} is null — cannot evaluate rule")
val = self._coerce_value(val_str)
if isinstance(actual, str):
actual = self._coerce_value(actual)
ops = {
">=": lambda a, b: a >= b,
"<=": lambda a, b: a <= b,
"!=": lambda a, b: a != b,
"==": lambda a, b: a == b,
">": lambda a, b: a > b,
"<": lambda a, b: a < b,
}
return ops[op](actual, val)
@staticmethod
def _coerce_value(value: str) -> Any:
"""Parse a rule literal into its most specific Python type."""
text = value.strip()
lowered = text.lower()
if lowered in ("true", "1"):
return True
if lowered in ("false", "0"):
return False
try:
return int(text)
except ValueError:
pass
try:
return float(text)
except ValueError:
pass
return text
# When crewai is absent there is no BaseTool to provide the public
# ``run``/``arun`` entry points, so expose them directly. With crewai
# installed these are left untouched so crewai's own implementations
# (usage tracking, ``result_as_answer``) win.
if not CREWAI_AVAILABLE:
def run(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool synchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
async def arun(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool asynchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
integrations/crewai/kg_tool.py
+573
View File
@@ -0,0 +1,573 @@
"""
SemanticaKGTool — a CrewAI ``BaseTool`` exposing Semantica's knowledge-graph
pipeline (``NERExtractor``, ``RelationExtractor``, ``ContextGraph``) to agents.
Lets agents build and query a shared ``ContextGraph`` as part of their
reasoning loop.
Install
-------
pip install semantica[crewai]
Example
-------
>>> from integrations.crewai import SemanticaKGTool
>>> from semantica.context import ContextGraph
>>> from crewai import Agent, Crew, Task
>>> graph = ContextGraph()
>>> tool = SemanticaKGTool(graph=graph)
>>> crew = Crew(
... agents=[Agent(role="...", goal="...", backstory="...", tools=[tool])],
... tasks=[...],
... )
Tools exposed
-------------
extract_entities — Extract named entities from text
extract_relations — Extract relationships between entities
add_to_graph — Extract entities/relations from text and add them to the graph
query_graph — Query the graph by keyword
find_related — Find concepts related to a given entity within ``hops``
"""
from __future__ import annotations
import json
import threading
import weakref
from typing import Any, Dict, List, Literal, Optional, Sequence, Type
from pydantic import BaseModel, Field
from semantica.utils.logging import get_logger
from ._availability import CREWAI_AVAILABLE, CREWAI_IMPORT_ERROR # noqa: F401
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: CrewAI BaseTool base class
# ---------------------------------------------------------------------------
_BaseTool: Any = object
if CREWAI_AVAILABLE:
from crewai.tools import BaseTool as _BaseTool # type: ignore
# One re-entrant lock per graph so concurrent tool invocations sharing a graph
# cannot double-count duplicate adds (check-then-act is not atomic), while
# independent graphs are never serialised against each other. An RLock also
# means an extractor callback that re-enters add_to_graph on the same graph
# cannot deadlock.
_graph_locks_guard = threading.Lock()
_graph_locks: "weakref.WeakKeyDictionary[Any, threading.RLock]" = (
weakref.WeakKeyDictionary()
)
# ---------------------------------------------------------------------------
# Input schema
# ---------------------------------------------------------------------------
class SemanticaKGToolInput(BaseModel):
"""
Input schema for ``SemanticaKGTool``.
Exactly one action is dispatched per call; the remaining fields are only
used by the actions that need them.
"""
action: Literal[
"extract_entities",
"extract_relations",
"add_to_graph",
"query_graph",
"find_related",
] = Field(
...,
description=(
"Which graph operation to run. One of: 'extract_entities', "
"'extract_relations', 'add_to_graph', 'query_graph', 'find_related'."
),
)
text: Optional[str] = Field(
None,
description=(
"Input text. Used by 'extract_entities', 'extract_relations' and "
"'add_to_graph'."
),
)
query: Optional[str] = Field(
None, description="Search query. Used by 'query_graph'."
)
entity: Optional[str] = Field(
None,
description="Root entity name. Used by 'find_related'.",
)
hops: int = Field(
1,
ge=1,
le=10,
description="Maximum relationship hops. Used by 'find_related'.",
)
# ---------------------------------------------------------------------------
# SemanticaKGTool
# ---------------------------------------------------------------------------
class SemanticaKGTool(_BaseTool): # type: ignore[misc]
"""
CrewAI tool that surfaces Semantica's KG pipeline as agent actions.
Parameters
----------
graph:
A ``semantica.context.ContextGraph`` to read/write. A fresh in-memory
graph is used when ``None``.
ner_extractor:
A ``semantica.semantic_extract.NERExtractor`` instance; auto-created
when ``None``.
relation_extractor:
A ``semantica.semantic_extract.RelationExtractor`` instance; auto-
created when ``None``.
"""
name: str = "semantica_knowledge_graph"
description: str = (
"Build and query a semantic knowledge graph. Actions: "
"'extract_entities' (extract named entities from 'text'), "
"'extract_relations' (extract relationships from 'text'), "
"'add_to_graph' (extract entities/relations from 'text' and add them "
"to the shared graph), 'query_graph' (keyword search using 'query'), "
"'find_related' (find concepts related to 'entity' within 'hops' "
"hops). Returns JSON."
)
args_schema: Type[BaseModel] = SemanticaKGToolInput
graph: Any = Field(default=None, exclude=True)
ner_extractor: Any = Field(default=None, exclude=True)
relation_extractor: Any = Field(default=None, exclude=True)
had_live_state: bool = False
reconstructed_state: bool = Field(default=False, exclude=True)
def __init__(
self,
graph: Any = None,
ner_extractor: Any = None,
relation_extractor: Any = None,
**kwargs: Any,
) -> None:
if CREWAI_AVAILABLE:
super().__init__(
graph=graph,
ner_extractor=ner_extractor,
relation_extractor=relation_extractor,
**kwargs,
)
else:
super().__init__()
self.graph = graph
self.ner_extractor = ner_extractor
self.relation_extractor = relation_extractor
# Degraded mode is a plain class — no model_post_init lifecycle.
self._ensure_defaults()
logger.info("SemanticaKGTool initialised (crewai=%s)", CREWAI_AVAILABLE)
def model_post_init(self, __context: Any) -> None:
"""Re-create default state after validation/deserialisation.
``graph``/extractors are excluded from JSON serialisation (CrewAI
checkpoints serialise every tool via ``model_dump(mode="json")``), so a
tool restored from a checkpoint has ``None`` state until this runs.
"""
self._ensure_defaults()
super().model_post_init(__context)
def _ensure_defaults(self) -> None:
"""Lazy-import and build defaults for any missing shared state."""
# Lazy imports keep the module importable without heavy deps
if self.graph is None:
from semantica.context import ContextGraph
self.graph = ContextGraph()
if self.had_live_state:
self.reconstructed_state = True
logger.warning(
"SemanticaKGTool: the live graph was lost during "
"serialization/checkpoint restore — an EMPTY graph was "
"reconstructed; re-attach the original graph before "
"continuing"
)
else:
logger.warning(
"SemanticaKGTool created a fresh in-memory ContextGraph — "
"agents sharing this tool's graph must be wired explicitly"
)
self.had_live_state = True
if self.ner_extractor is None:
from semantica.semantic_extract import NERExtractor
self.ner_extractor = NERExtractor()
if self.relation_extractor is None:
from semantica.semantic_extract import RelationExtractor
self.relation_extractor = RelationExtractor()
# ------------------------------------------------------------------
# CrewAI entry points
# ------------------------------------------------------------------
def _run(
self,
action: str,
text: Optional[str] = None,
query: Optional[str] = None,
entity: Optional[str] = None,
hops: int = 1,
**kwargs: Any,
) -> str:
"""
Dispatch a graph action. Always returns a JSON string so the agent
receives a structured, parseable result.
"""
valid = {
"extract_entities",
"extract_relations",
"add_to_graph",
"query_graph",
"find_related",
}
if action not in valid:
return json.dumps(
{
"error": f"Unknown action '{action}'. Valid actions: "
+ ", ".join(sorted(valid))
}
)
if action == "extract_entities":
return self._extract_entities(text or "")
if action == "extract_relations":
return self._extract_relations(text or "")
if action == "add_to_graph":
return self._add_from_text(text or "")
if action == "query_graph":
return self._query_graph(query or "")
return self._find_related(entity or "", hops=hops)
async def _arun(
self,
action: str,
text: Optional[str] = None,
query: Optional[str] = None,
entity: Optional[str] = None,
hops: int = 1,
**kwargs: Any,
) -> str:
"""
Async variant of ``_run`` for CrewAI's async tool path.
"""
return self._run(
action=action, text=text, query=query, entity=entity, hops=hops, **kwargs
)
# ------------------------------------------------------------------
# Entity/relation field access (handles both Semantica dataclasses and
# third-party shapes like MagicMock/plain dicts in stubs)
# ------------------------------------------------------------------
@staticmethod
def _first_str(obj: Any, attrs: Sequence[str]) -> str:
"""Return the first attribute value that is a non-empty string."""
for attr in attrs:
value = getattr(obj, attr, None)
if isinstance(value, str) and value:
return value
if isinstance(obj, dict):
for key in attrs:
value = obj.get(key)
if isinstance(value, str) and value:
return value
return ""
@classmethod
def _entity_name(cls, e: Any) -> str:
"""Best-effort name for an entity-like object."""
return cls._first_str(e, ("name", "text", "label", "node_id", "id"))
@classmethod
def _entity_type(cls, e: Any) -> str:
"""Best-effort type/label for an entity-like object."""
return cls._first_str(e, ("type", "label")) or "Entity"
@classmethod
def _relation_source(cls, r: Any) -> str:
"""Best-effort source of a relation-like object."""
src = cls._first_str(r, ("source",))
if not src:
src = cls._entity_name(getattr(r, "subject", None))
return src
@classmethod
def _relation_target(cls, r: Any) -> str:
"""Best-effort target of a relation-like object."""
tgt = cls._first_str(r, ("target",))
if not tgt:
tgt = cls._entity_name(getattr(r, "object", None))
return tgt
@classmethod
def _relation_type(cls, r: Any) -> str:
"""Best-effort relation type of a relation-like object."""
rtype = cls._first_str(r, ("type", "relation", "predicate"))
return rtype or "related_to"
@classmethod
def _confidence(cls, e: Any) -> float:
"""Normalise an entity/relation confidence value to a float."""
try:
val = getattr(e, "confidence", None)
if val is None:
return 1.0
return round(float(val), 4)
except (TypeError, ValueError):
return 1.0
@classmethod
def _graph_lock(cls, graph: Any) -> threading.RLock:
"""Return the re-entrant lock guarding a specific graph."""
with _graph_locks_guard:
lock = _graph_locks.get(graph)
if lock is None:
lock = threading.RLock()
_graph_locks[graph] = lock
return lock
# ------------------------------------------------------------------
# Actions
# ------------------------------------------------------------------
def _extract_entities(self, text: str) -> str:
"""Extract named entities from ``text``."""
try:
raw = self.ner_extractor.extract_entities(text) or []
entities = [
{
"name": self._entity_name(e),
"type": self._entity_type(e),
"confidence": self._confidence(e),
}
for e in raw
if self._entity_name(e)
]
logger.debug("extract_entities → %d entities", len(entities))
return json.dumps({"entities": entities, "count": len(entities)})
except Exception as exc:
logger.warning("extract_entities failed: %s", exc)
return json.dumps({"entities": [], "count": 0, "error": str(exc)})
def _extract_relations(self, text: str) -> str:
"""Extract relationships between entities in ``text``."""
try:
raw = self.relation_extractor.extract_relations(text) or []
relations = [
{
"source": self._relation_source(r),
"relation": self._relation_type(r),
"target": self._relation_target(r),
"confidence": self._confidence(r),
}
for r in raw
]
logger.debug("extract_relations → %d relations", len(relations))
return json.dumps({"relations": relations, "count": len(relations)})
except Exception as exc:
logger.warning("extract_relations failed: %s", exc)
return json.dumps({"relations": [], "count": 0, "error": str(exc)})
def _add_from_text(self, text: str) -> str:
"""
Extract entities and relations from ``text`` and add them to the graph.
Duplicate nodes/edges (same id, or same source/type/target) are
skipped so repeated calls are idempotent. Returns JSON with the
number of nodes/edges added.
"""
nodes_added = 0
edges_added = 0
try:
with self._graph_lock(self.graph):
existing_nodes = {
n.get("id") or n.get("node_id")
for n in (
self.graph.find_nodes() or [] # type: ignore[attr-defined]
)
if n.get("id") or n.get("node_id")
}
existing_edges = {
(e.get("source"), e.get("type") or "related_to", e.get("target"))
for e in (
self.graph.find_edges() or [] # type: ignore[attr-defined]
)
if e.get("source") and e.get("target")
}
raw_entities = self.ner_extractor.extract_entities(text) or []
entities: List[Any] = []
seen: set = set()
for e in raw_entities:
name = self._entity_name(e)
ntype = self._entity_type(e)
if not name or name in seen:
continue
seen.add(name)
entities.append(e)
if name in existing_nodes:
continue
try:
if self.graph.add_node(node_id=name, node_type=ntype):
nodes_added += 1
existing_nodes.add(name)
except Exception as exc:
logger.debug("add_node(%r) failed: %s", name, exc)
raw_relations = (
self.relation_extractor.extract_relations(text, entities=entities)
or []
)
for r in raw_relations:
src = self._relation_source(r)
tgt = self._relation_target(r)
rtype = self._relation_type(r)
if not src or not tgt:
continue
key = (src, rtype, tgt)
if key in existing_edges:
continue
try:
if self.graph.add_edge(
source_id=src, target_id=tgt, edge_type=rtype
):
edges_added += 1
existing_edges.add(key)
except Exception as exc:
logger.debug("add_edge(%r) failed: %s", key, exc)
logger.debug("add_to_graph: +%d nodes, +%d edges", nodes_added, edges_added)
return json.dumps({"nodes_added": nodes_added, "edges_added": edges_added})
except Exception as exc:
logger.warning("add_to_graph failed: %s", exc)
return json.dumps({"nodes_added": 0, "edges_added": 0, "error": str(exc)})
def _query_graph(self, query: str) -> str:
"""Keyword-search graph nodes by id, type and content."""
try:
q = (query or "").strip().lower()
out: List[dict] = []
seen: set = set()
query_method = getattr(self.graph, "query", None)
if query_method is not None:
for match in query_method(query) or []:
node = match.get("node") or {}
nid = node.get("id", "") or node.get("node_id", "")
if not nid or nid in seen:
continue
seen.add(nid)
content = match.get("content") or node.get("content", "")
out.append(
{
"id": nid,
"type": node.get("type", "") or node.get("node_type", ""),
"label": nid,
"content": str(content)[:500],
"score": round(float(match.get("score") or 0.0), 4),
}
)
if q:
for n in self.graph.find_nodes() or []: # type: ignore[attr-defined]
if isinstance(n, dict):
nid = n.get("id", "") or n.get("node_id", "")
ntype = n.get("type", "") or n.get("node_type", "")
content = str(
n.get("content")
or (n.get("properties") or {}).get("content", "")
or ""
)
else:
nid = getattr(n, "id", getattr(n, "label", ""))
ntype = getattr(n, "node_type", "")
content = str(getattr(n, "content", "") or "")
if not nid or nid in seen:
continue
if q in str(nid).lower() or q in str(ntype).lower():
seen.add(nid)
out.append(
{
"id": nid,
"type": ntype,
"label": nid,
"content": content[:500],
"score": 1.0,
}
)
return json.dumps({"results": out, "count": len(out)})
except Exception as exc:
logger.warning("query_graph failed: %s", exc)
return json.dumps({"results": [], "count": 0, "error": str(exc)})
def _find_related(self, entity: str, hops: int = 1) -> str:
"""Find concepts related to ``entity`` within ``hops`` graph hops.
Traversal is undirected — an edge counts as related regardless of
direction, so both outgoing and incoming edges are honored.
"""
try:
adjacency: Dict[str, List[str]] = {}
for edge in self.graph.find_edges() or []: # type: ignore[attr-defined]
if isinstance(edge, dict):
src = edge.get("source")
tgt = edge.get("target")
else:
src = getattr(edge, "source", None)
tgt = getattr(edge, "target", None)
if not src or not tgt:
continue
adjacency.setdefault(src, []).append(tgt)
adjacency.setdefault(tgt, []).append(src)
related: List[str] = []
frontier = [entity]
visited = {entity}
for _ in range(max(1, hops)):
next_frontier: List[str] = []
for e in frontier:
for n in adjacency.get(e, []):
if n in visited:
continue
visited.add(n)
next_frontier.append(n)
related.append(n)
frontier = next_frontier
logger.debug("find_related('%s', hops=%d) → %d", entity, hops, len(related))
return json.dumps(
{"entity": entity, "related": related, "count": len(related)}
)
except Exception as exc:
logger.warning("find_related failed: %s", exc)
return json.dumps(
{"entity": entity, "related": [], "count": 0, "error": str(exc)}
)
# When crewai is absent there is no BaseTool to provide the public
# ``run``/``arun`` entry points, so expose them directly. With crewai
# installed these are left untouched so crewai's own implementations
# (usage tracking, ``result_as_answer``) win.
if not CREWAI_AVAILABLE:
def run(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool synchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
async def arun(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool asynchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
integrations/crewai/knowledge_source.py
+331
View File
@@ -0,0 +1,331 @@
"""
SemanticaKnowledgeSource — expose a Semantica ``ContextGraph`` as a CrewAI
knowledge source.
Lets a ``Crew`` load the current state of a knowledge graph (nodes, edges,
metadata) into its knowledge storage, so every agent gets retrieval access to
graph knowledge during the kickoff.
Install
-------
pip install semantica[crewai]
Example
-------
>>> from integrations.crewai import SemanticaKnowledgeSource
>>> from semantica.context import ContextGraph
>>> from crewai import Agent, Crew, Task
>>> graph = ContextGraph()
>>> graph.add_node(node_id="privacy", node_type="policy")
>>> crew = Crew(
... agents=[...],
... tasks=[...],
... knowledge_sources=[SemanticaKnowledgeSource(graph=graph)],
... )
Compatibility
-------------
Works with ``crewai >= 0.80.0``. The ``BaseKnowledgeSource`` contract changed
between versions (``load_content`` → ``validate_content``/``aadd``), so this
source implements both legacy and current methods. It degrades gracefully
when ``crewai`` is not installed: the class is still importable and carries the
full Semantica API, but cannot be passed to a ``Crew``.
"""
from __future__ import annotations
import asyncio
from typing import Any, Dict, List, Optional
from pydantic import Field
from semantica.utils.logging import get_logger
from ._availability import CREWAI_AVAILABLE
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: CrewAI BaseKnowledgeSource base class
# ---------------------------------------------------------------------------
_BaseKnowledgeSource: Any = object
if CREWAI_AVAILABLE:
from crewai.knowledge.source.base_knowledge_source import (
BaseKnowledgeSource as _BaseKnowledgeSource, # type: ignore
)
def _chunk_text_manual(text: str, chunk_size: int, chunk_overlap: int) -> List[str]:
"""Fallback plain-text chunker for when CrewAI helpers are unavailable."""
if not text:
return []
if int(chunk_size) <= 0:
return [text]
size = max(1, int(chunk_size))
overlap = max(0, int(chunk_overlap))
if len(text) <= size:
return [text]
step = max(1, size - overlap)
return [text[i : i + size] for i in range(0, len(text), step)]
class SemanticaKnowledgeSource(_BaseKnowledgeSource): # type: ignore[misc]
"""
CrewAI knowledge source backed by a Semantica ``ContextGraph``.
On ``add()`` the graph's nodes and edges are serialised into readable text
and pushed through the standard CrewAI chunking / storage pipeline, making
graph knowledge retrievable by every agent in the crew.
Parameters
----------
graph:
A ``semantica.context.ContextGraph`` to expose. A fresh in-memory
graph is created when ``None``.
name:
Source name. Defaults to ``"semantica_knowledge_graph"``.
chunk_size:
Max characters per chunk (default 4000).
chunk_overlap:
Character overlap between adjacent chunks (default 200).
"""
name: str = "semantica_knowledge_graph"
graph: Any = Field(default=None, exclude=True)
chunk_size: int = 4000
chunk_overlap: int = 200
had_live_state: bool = False
reconstructed_state: bool = Field(default=False, exclude=True)
def __init__(
self,
graph: Any = None,
name: Optional[str] = None,
chunk_size: int = 4000,
chunk_overlap: int = 200,
**kwargs: Any,
) -> None:
if CREWAI_AVAILABLE:
# Do NOT eagerly build a graph here: pydantic calls this ``__init__``
# during ``model_validate`` (checkpoint restore), and the eager
# build would hide that a live graph was lost. ``model_post_init``
# rebuilds defaults and flags ``reconstructed_state`` instead.
super().__init__(
graph=graph,
name=name or "semantica_knowledge_graph",
chunk_size=int(chunk_size),
chunk_overlap=int(chunk_overlap),
**kwargs,
)
else:
if graph is None:
from semantica.context import ContextGraph
graph = ContextGraph()
super().__init__()
self.graph = graph
self.name = name or "semantica_knowledge_graph"
self.chunk_size = int(chunk_size)
self.chunk_overlap = int(chunk_overlap)
logger.info(
"SemanticaKnowledgeSource initialised (crewai=%s, chunk_size=%d)",
CREWAI_AVAILABLE,
self.chunk_size,
)
self.had_live_state = True
def model_post_init(self, __context: Any) -> None:
"""Re-create default state after validation/deserialisation.
``graph`` is excluded from JSON serialisation (CrewAI checkpoints
serialise their models via ``model_dump(mode="json")``), so a source
restored from a checkpoint has ``None`` state until this runs.
"""
if self.graph is None:
from semantica.context import ContextGraph
self.graph = ContextGraph()
if self.had_live_state:
self.reconstructed_state = True
logger.warning(
"SemanticaKnowledgeSource: the live graph was lost during "
"serialization/checkpoint restore — an EMPTY graph was "
"reconstructed; re-attach the original graph before "
"continuing"
)
else:
logger.warning(
"SemanticaKnowledgeSource created a fresh in-memory "
"ContextGraph — sources sharing knowledge must be wired to "
"the same graph explicitly"
)
self.had_live_state = True
super().model_post_init(__context)
# ------------------------------------------------------------------
# Content extraction
# ------------------------------------------------------------------
def load_content(self) -> Dict[str, str]:
"""
Serialise the graph into ``{id: readable_text}`` pairs.
Nodes are rendered with their type/content/metadata, edges with their
source, relation type and target. This satisfies the legacy CrewAI
``BaseKnowledgeSource.load_content`` contract.
"""
content: Dict[str, str] = {}
graph = self.graph
if graph is None:
return content
try:
for node in graph.find_nodes() or []: # type: ignore[attr-defined]
nid = node.get("id") or node.get("node_id") or ""
if not nid:
continue
parts = [
"Entity",
str(nid),
"type: " + str(node.get("type", "entity")),
]
if node.get("content"):
parts.append("content: " + str(node["content"]))
if node.get("metadata"):
try:
import json
parts.append("metadata: " + json.dumps(node["metadata"]))
except Exception:
parts.append("metadata: " + str(node["metadata"]))
content[str(nid)] = " | ".join(parts)
except Exception as exc:
logger.warning(
"SemanticaKnowledgeSource.load_content (nodes) failed: %s", exc
)
try:
for idx, edge in enumerate(
graph.find_edges() or [] # type: ignore[attr-defined]
):
src = edge.get("source")
tgt = edge.get("target")
if not src or not tgt:
continue
rel = edge.get("type") or edge.get("edge_type") or "related_to"
weight = edge.get("weight")
text = f"{src} -[{rel}]-> {tgt}"
if weight is not None:
text += f" (weight: {weight})"
content[f"edge-{idx}"] = text
except Exception as exc:
logger.warning(
"SemanticaKnowledgeSource.load_content (edges) failed: %s", exc
)
return content
def validate_content(self) -> Any:
"""
Validate that a readable graph is attached.
Satisfies the current CrewAI ``BaseKnowledgeSource.validate_content``
contract.
"""
if self.graph is None:
raise ValueError("SemanticaKnowledgeSource requires a ContextGraph.")
return True
# ------------------------------------------------------------------
# Chunking + storage (abstract in both CrewAI generations)
# ------------------------------------------------------------------
def _chunk(self, text: str) -> List[str]:
"""Chunk ``text`` using CrewAI's helper when available, else manual."""
helper = getattr(self, "_chunk_text", None)
if helper is not None:
try:
return list(helper(text) or [])
except Exception as exc:
logger.debug(
"SemanticaKnowledgeSource._chunk_text failed, falling back: %s", exc
)
return _chunk_text_manual(text, self.chunk_size, self.chunk_overlap)
def add(self) -> None:
"""
Process the graph into chunks and store them via CrewAI storage.
Sets both ``chunks`` (current CrewAI) and ``_chunks`` (legacy CrewAI)
so either ``_save_documents`` implementation picks them up. If no
storage has been wired (e.g. not yet attached to a ``Crew``), chunks
are kept in memory.
"""
content = self.load_content()
if not content:
logger.debug("SemanticaKnowledgeSource.add: empty graph — nothing to store")
return
chunks: List[str] = []
for _, text in content.items():
if text:
chunks.extend(self._chunk(text))
self.chunks = chunks
self._chunks = chunks
save = getattr(self, "_save_documents", None)
if save is not None:
if getattr(self, "storage", None) is None:
logger.debug(
"SemanticaKnowledgeSource.add: storage not wired — "
"keeping chunks in memory"
)
else:
try:
save()
logger.info(
"SemanticaKnowledgeSource.add: stored %d chunks", len(chunks)
)
return
except Exception as exc:
logger.error(
"SemanticaKnowledgeSource.add: storage save FAILED (%s) — "
"chunks are only kept in memory and agents will retrieve "
"nothing. Configure the Crew embedder (e.g. an OpenAI "
"embedder with OPENAI_API_KEY, or a local embedder) before "
"running the crew.",
exc,
)
logger.info(
"SemanticaKnowledgeSource.add: %d chunks ready in memory", len(chunks)
)
async def aadd(self) -> None:
"""
Asynchronous variant of ``add()`` (current CrewAI contract).
The graph serialisation is CPU-bound, so it runs in a thread pool to
avoid blocking the event loop.
"""
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, self.add)
# ------------------------------------------------------------------
# Inspection helpers
# ------------------------------------------------------------------
def get_content_summary(self) -> Dict[str, Any]:
"""
Summarise what the source exposes (helpful for debugging / testing).
"""
content = self.load_content()
return {
"name": self.name,
"source_count": len(content),
"chunks": len(getattr(self, "chunks", []) or []),
"crewai_available": CREWAI_AVAILABLE,
}
pyproject.toml
+8
View File
@@ -201,6 +201,10 @@ gpu = [
# ---- Agentic Framework Integrations ----
agno = ["agno>=1.0.0"]
# crewai core provides BaseTool and BaseKnowledgeSource; crewai-tools is not
# needed (it pulls vulnerable transitive deps like chromadb) and would only
# duplicate the prebuilt tooling users can install separately.
crewai = ["crewai>=0.80.0"]
# ---- File Watching ----
watch = ["watchdog>=6.0.0"]
@@ -242,6 +246,10 @@ explorer-lite = [
]
# Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
# NOTE: the ``crewai`` extra is intentionally NOT in ``all``: crewai hard-requires
# ``chromadb~=1.1.0``, which carries a pre-authentication code-injection advisory
# (CVE-2026-45829) with no fixed release — including it here would fail the CI
# dependency-audit/security gates. Install it explicitly via ``semantica[crewai]``.
all = [
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
requirements-ci.txt
+4 -4
View File
@@ -1,5 +1,5 @@
# This file was autogenerated by uv via the following command:
# uv pip compile -p 3.11 --extra all --generate-hashes -o requirements-ci.txt pyproject.toml
# uv pip compile pyproject.toml --python-version 3.11 --extra all --generate-hashes -o requirements-ci.txt
accelerate==1.14.0 \
--hash=sha256:41b9c4377a54e0b460a959b0defa1b736e4ca0a2373252d9a539964c2afe3c8d \
--hash=sha256:e94390c2863b873be18f623f9df48a0d8fe5eff13ea7f1a00092b0a7904888c6
@@ -4123,9 +4123,9 @@ pooch==1.9.0 \
--hash=sha256:de46729579b9857ffd3e741987a2f6d5e0e03219892c167c6578c0091fb511ed \
--hash=sha256:f265597baa9f760d25ceb29d0beb8186c243d6607b0f60b83ecf14078dbc703b
# via librosa
portalocker==3.2.0 \
--hash=sha256:1f3002956a54a8c3730586c5c77bf18fae4149e07eaf1c29fc3faf4d5a3f89ac \
--hash=sha256:3cdc5f565312224bc570c49337bd21428bba0ef363bbcf58b9ef4a9f11779968
portalocker==2.7.0 \
--hash=sha256:032e81d534a88ec1736d03f780ba073f047a06c478b06e2937486f334e955c51 \
--hash=sha256:a07c5b4f3985c3cf4798369631fb7011adb498e2a46d8440efc75a8f29a0f983
# via qdrant-client
pre-commit==4.6.2 \
--hash=sha256:8f5d7bfb021ecdbcd9d49d89847082dd24172ccde534390081a679ad046e2441 \
semantica/context/context_graph.py
+4 -4
View File
@@ -72,9 +72,9 @@ Example Usage:
... node_embeddings=True)
>>>
>>> # Basic graph operations
>>> graph.add_node("Python", type="language", properties={"popularity": "high"})
>>> graph.add_node("Programming", type="concept")
>>> graph.add_edge("Python", "Programming", type="related_to")
>>> graph.add_node("Python", "language", popularity="high")
>>> graph.add_node("Programming", "concept")
>>> graph.add_edge("Python", "Programming", "related_to")
>>> centrality = graph.get_node_centrality("Python")
>>> similar = graph.find_similar_nodes("Python", similarity_type="content")
>>> analysis = graph.analyze_graph_with_kg()
@@ -88,7 +88,7 @@ Example Usage:
... confidence=0.95,
... entities=["customer_123", "property_456"]
... )
>>> precedents = graph.find_precedents("loan_approval", limit=5)
>>> precedents = graph.find_precedents(decision_id, limit=5)
>>> influence = graph.analyze_decision_influence(decision_id)
>>> insights = graph.get_decision_insights()
>>> causality = graph.trace_decision_causality(decision_id)
semantica/export/arango_aql_exporter.py
+13 -11
View File
@@ -28,7 +28,7 @@ import json
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.helpers import ensure_directory
from ..utils.helpers import _require_mapping, ensure_directory, normalize_graph_payload
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -204,17 +204,19 @@ class ArangoAQLExporter:
self._generate_collection_creation(vertex_collection, edge_collection)
)
# Extract entities and relationships
entities = knowledge_graph.get("entities", [])
relationships = knowledge_graph.get("relationships", [])
nodes = knowledge_graph.get("nodes", entities)
edges = knowledge_graph.get("edges", relationships)
# A non-mapping payload cannot reach normalize_graph_payload(): it
# raises ValidationError for that case, which would leave this
# exporter alone in raising a different exception type than the YAML
# and Neo4j exporters raise for the identical mistake.
_require_mapping(
knowledge_graph, ("entities", "relationships", "nodes", "edges")
)
# Use nodes/edges if entities/relationships are empty
if not entities and nodes:
entities = nodes
if not relationships and edges:
relationships = edges
# Accept either vocabulary; resolution is centralized so every
# exporter agrees on what a given payload means.
normalized = normalize_graph_payload(knowledge_graph)
entities = normalized["entities"]
relationships = normalized["relationships"]
# Generate vertex INSERT statements
vertex_statements = self._generate_vertex_inserts(entities, vertex_collection)
semantica/export/export_usage.md
+68
View File
@@ -297,6 +297,57 @@ export_yaml(semantic_network, "network.yaml", method="semantic_network")
export_yaml(schema, "schema.yaml", method="schema")
```
### Accepted Input
Both YAML exporters read their payload by key, so the input must be a mapping;
anything else raises `ProcessingError`. A bare list is rejected rather than
wrapped, since these formats distinguish entities from relationships from
triplets and guessing which one a list holds would mislabel the records.
Each exporter then reads a fixed set of keys, and raises `ValidationError` on a
non-empty mapping that supplies none of them — such a payload would otherwise
serialize to a valid file with every collection empty. Naming a recognized key
is not enough on its own: `{"entities": [], "data": [...]}` also raises, since
nothing resolves while the records sit under a key the exporter never reads.
| Method | Recognized keys |
| :--- | :--- |
| `"semantic_network"` | `entities` (alias `nodes`), `relationships` (alias `edges`), `triplets` |
| `"schema"` | `classes`, `properties`, `namespaces`, `uri`, `title`, `description`, `version` |
`metadata` is carried through on both, but does not by itself make a payload
recognized — an `export_json` envelope (`{"data": [...], "count": N,
"metadata": {...}}`) carries one and is rejected.
```python
# ContextGraph.to_dict() exports directly via the nodes/edges aliases
export_yaml(context_graph.to_dict(), "graph.yaml")
# A bare list has no unambiguous meaning here
export_yaml(records, "out.yaml") # ProcessingError
# An export_json payload is refused rather than written out empty
export_yaml({"data": records}, "out.yaml") # ValidationError
# ...and so is one that names a recognized key but leaves it empty
export_yaml({"entities": [], "data": records}, "out.yaml") # ValidationError
```
The value under a recognized key must be a collection of records — a list or
tuple of mappings or objects. A string, a bare mapping, or a scalar raises
`ValidationError` naming the key, rather than being iterated into
character-sized "records" or surfacing as a `TypeError` from inside the
exporter. `None` is read as an absent collection, the same as `[]`.
```python
export_yaml({"entities": "abc"}, "out.yaml") # ValidationError
export_yaml({"entities": 42}, "out.yaml") # ValidationError
export_yaml({"nodes": {"id": "n1"}}, "out.yaml") # ValidationError — wrap it in a list
```
An empty mapping is still accepted: an empty graph is a legitimate export and
has no records to lose.
## OWL Export
### OWL/XML Format
@@ -479,6 +530,23 @@ Pass `validate=True` to run a post-export integrity check before returning:
export_neo4j_csv(kg, "neo4j_import/", validate=True)
```
#### Accepted Input
Mapping payloads are read on the same terms as the YAML exporters (see [Accepted
Input](#accepted-input) above): `entities`/`relationships`, with `nodes`/`edges`
accepted as aliases. A non-empty mapping that supplies neither — or that supplies
a malformed collection value — raises `ValidationError` rather than writing
header-only CSVs indistinguishable from a genuinely exported empty graph. The
payload is normalized before any file is opened, so a rejected export writes
nothing.
Graph *objects* are unaffected: they are still read off `nodes`/`entities` and
`edges`/`relationships` attributes.
```python
export_neo4j_csv({"data": [{"id": "e1"}]}, "neo4j_import/") # ValidationError
```
#### Importing into Neo4j
Once the CSV files are generated, they can be imported into a new Neo4j database using the `neo4j-admin database import` command:
semantica/export/lpg_exporter.py
+28 -12
View File
@@ -24,7 +24,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory
from ..utils.helpers import _require_mapping, ensure_directory, normalize_graph_payload
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -154,15 +154,26 @@ class LPGExporter:
"""
queries = []
# Generate indexes if requested
if self.include_indexes:
queries.extend(self._generate_indexes(knowledge_graph))
# A non-mapping payload cannot reach normalize_graph_payload(): it
# raises ValidationError for that case, which would leave this
# exporter alone in raising a different exception type than the YAML
# and Neo4j exporters raise for the identical mistake.
_require_mapping(
knowledge_graph, ("entities", "relationships", "nodes", "edges")
)
# Extract entities and relationships
entities = knowledge_graph.get("entities", [])
relationships = knowledge_graph.get("relationships", [])
nodes = knowledge_graph.get("nodes", entities)
edges = knowledge_graph.get("edges", relationships)
# Accept either vocabulary. Reading 'nodes' with 'entities' as the
# default dropped every entity when 'nodes' was present but empty --
# the shape JSONExporter emits -- so resolution is centralized.
normalized = normalize_graph_payload(knowledge_graph)
nodes = normalized["entities"]
edges = normalized["relationships"]
# Generate indexes if requested. Fed the normalized entities so index
# generation sees the same records as node generation; reading
# 'entities' directly here skipped indexes for nodes/edges payloads.
if self.include_indexes:
queries.extend(self._generate_indexes(nodes))
# Generate node creation queries
node_queries = self._generate_node_queries(nodes)
@@ -174,13 +185,18 @@ class LPGExporter:
return queries
def _generate_indexes(self, knowledge_graph: Dict[str, Any]) -> List[str]:
"""Generate Cypher index and constraint creation queries."""
def _generate_indexes(self, entities: List[Dict[str, Any]]) -> List[str]:
"""Generate Cypher index and constraint creation queries.
Args:
entities: Entity records, already resolved from whichever
vocabulary the caller supplied.
"""
indexes = []
# Get unique entity types for labels
entity_types = set()
for entity in knowledge_graph.get("entities", []):
for entity in entities:
entity_type = entity.get("type") or entity.get("entity_type")
if entity_type:
entity_types.add(entity_type)
semantica/export/methods.py
+20 -2
View File
@@ -494,7 +494,7 @@ def export_graph(
def export_yaml(
data: Union[Dict[str, Any], List[Dict[str, Any]]],
data: Dict[str, Any],
file_path: Union[str, Path],
method: str = "semantic_network",
**kwargs,
@@ -504,14 +504,32 @@ def export_yaml(
This is a user-friendly wrapper that exports data to YAML format.
Unlike :func:`export_json` and :func:`export_csv`, which treat a list as
opaque records, both YAML methods are keyed formats: they distinguish
entities from relationships from triplets (and classes from properties
for ``method="schema"``). A bare list is therefore rejected rather than
guessed at, since inferring which collection it represents would silently
mislabel the records.
Args:
data: Data to export (semantic network, entities, relationships)
data: Data to export, as a mapping. For ``method="semantic_network"``,
keyed by 'entities'/'relationships'/'triplets'; for
``method="schema"``, by 'classes'/'properties'.
file_path: Output YAML file path
method: Export method (default: "semantic_network")
- "semantic_network": Semantic network YAML export
- "schema": Schema YAML export
**kwargs: Additional options passed to YAML exporters
Raises:
ProcessingError: if ``data`` is not a mapping, or if ``method`` is not
a known YAML export method.
ValidationError: if ``data`` is a mapping whose keys the selected
exporter does not read -- an ``export_json`` envelope
(``{"data": [...], "count": N, "metadata": {...}}``) is the
common case. Such a payload used to be written out as a valid
YAML file with every collection empty.
Examples:
>>> from semantica.export.methods import export_yaml
>>> export_yaml(semantic_network, "network.yaml", method="semantic_network")
semantica/export/neo4j_csv_exporter.py
+28 -4
View File
@@ -32,12 +32,13 @@ from __future__ import annotations
import csv
import hashlib
import json
from collections.abc import Mapping
from dataclasses import asdict, is_dataclass
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Sequence, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory
from ..utils.helpers import ensure_directory, normalize_graph_payload
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -173,6 +174,19 @@ class Neo4jCSVExporter:
Returns:
Mapping with ``"nodes"`` and ``"relationships"`` output paths.
Raises:
ValidationError: if a mapping payload carries no recognized graph
key, resolves to nothing while an unread key still holds
records, or holds something other than records under one --
see
:func:`~semantica.utils.helpers.normalize_graph_payload`.
Each would otherwise be written out as header-only CSVs
indistinguishable from a genuinely empty graph. The payload is
normalized before any file is opened, so a rejected export
writes nothing.
ProcessingError: if a non-mapping payload exposes none of the
graph attributes.
"""
output_dir = Path(output_dir)
ensure_directory(output_dir)
@@ -494,9 +508,19 @@ class Neo4jCSVExporter:
return prepared
def _normalize_graph(self, graph: Any) -> Dict[str, List[Dict[str, Any]]]:
if isinstance(graph, dict):
nodes = graph.get("nodes") or graph.get("entities") or []
relationships = graph.get("edges") or graph.get("relationships") or []
if isinstance(graph, Mapping):
# Mapping payloads go through the shared resolver on its default
# terms, so this backend cannot drift from the others: an
# unrecognized mapping raises here rather than writing header-only
# CSVs that read as a successful export of an empty graph. Checked
# against Mapping rather than dict, so a non-dict Mapping (a
# MappingProxyType, a ChainMap) takes this path too, instead of
# falling through to the attribute branch below and being rejected
# as an unrecognized object -- the LPG, Arango, and YAML exporters
# already accept such payloads via the same resolver.
resolved = normalize_graph_payload(graph)
nodes = resolved["entities"]
relationships = resolved["relationships"]
else:
nodes = getattr(graph, "nodes", None)
if nodes is None:
semantica/export/yaml_exporter.py
+178 -17
View File
@@ -21,15 +21,83 @@ Author: Semantica Contributors
License: MIT
"""
from collections.abc import Mapping
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory
from ..utils.exceptions import ValidationError
from ..utils.helpers import (
_require_mapping,
_require_nothing_dropped,
_require_recognized_keys,
ensure_directory,
normalize_graph_payload,
)
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
# Keys YAMLSchemaExporter.export_ontology_schema reads. Graph payloads use the
# recognized set owned by normalize_graph_payload() instead; schemas are a
# separate vocabulary with no aliasing, so the set lives here.
_SCHEMA_KEYS = (
"classes",
"properties",
"namespaces",
"uri",
"title",
"description",
"version",
)
def _require_usable_schema(ontology: Mapping) -> None:
"""Reject a schema mapping this exporter cannot read.
Two ways an ontology mapping produces an empty file: it shares no key with
the recognized set at all, or it names a recognized key that is empty
while the real records sit under a key this exporter does not read
(``{"classes": [], "nodes": [...]}``). Both are refused, using the same
checks the graph payloads go through, so the two vocabularies cannot drift
apart in what they consider a silent-empty export.
An empty mapping is allowed through: it carries nothing that could be
lost, and an empty export is a legitimate result.
Note the deliberate split in exception types, which the codebase already
makes: a wrong *type* cannot be exported at all and raises
ProcessingError, matching ``Neo4jCSVExporter._normalize_graph``; a mapping
whose *contents* are unusable raises ValidationError, matching
``normalize_graph_payload``.
Args:
ontology: Mapping already checked by :func:`_require_mapping`.
Raises:
ValidationError: if the mapping shares no key with ``_SCHEMA_KEYS``,
or resolves to nothing while an unread key still holds records.
"""
_require_recognized_keys(ontology, _SCHEMA_KEYS, what="Ontology schema")
# Only non-empty list/tuple values from recognized schema keys count as
# evidence that records survived export. Scalar metadata fields such as
# 'uri', 'title', 'description', and 'version' are truthy strings, but
# their presence does not mean the caller's record collections were
# exported -- passing them as ``resolved`` would let any scalar value
# short-circuit the dropped-records check and silently discard a list
# under an unread key alongside e.g. {"version": "1.0", "nodes": [...]}.
resolved = [
v
for key in _SCHEMA_KEYS
for v in (ontology.get(key),)
if isinstance(v, (list, tuple)) and v
]
_require_nothing_dropped(
ontology,
_SCHEMA_KEYS,
resolved,
what="Ontology schema",
)
class SemanticNetworkYAMLExporter:
"""
@@ -90,15 +158,39 @@ class SemanticNetworkYAMLExporter:
Args:
semantic_network: Semantic network dictionary containing:
- entities: List of entity dictionaries
- entities: List of entity dictionaries (alias: 'nodes')
- relationships: List of relationship dictionaries
(alias: 'edges')
- triplets: List of triplet dictionaries (optional)
- metadata: Metadata dictionary (optional)
Key resolution is delegated to
:func:`~semantica.utils.helpers.normalize_graph_payload`, so
``ContextGraph.to_dict()`` output ('nodes'/'edges') exports
directly.
**options: Additional export options (unused)
Returns:
String containing YAML representation of semantic network
Raises:
ProcessingError: if ``semantic_network`` is not a mapping. A bare
list of records cannot be exported here because this format
distinguishes entities, relationships, and triplets, and
guessing which one a list represents would silently mislabel
it.
ValidationError: if the mapping carries both spellings of a
collection with different contents; if it is non-empty and
shares no key with the recognized set; or if it resolves to
nothing while an unread key still holds records
(``{"entities": [], "data": [...]}``). Each previously
serialized to a file with every collection empty while the log
reported success. An empty mapping is still accepted -- it has
no records to lose. Note that 'metadata' alone is not a
recognized key: an ``export_json`` envelope carries one, and
accepting it would readmit the silent-empty export it is the
most likely source of.
Example:
>>> network = {
... "entities": [...],
@@ -107,6 +199,8 @@ class SemanticNetworkYAMLExporter:
... }
>>> yaml_str = exporter.export_semantic_network(network)
"""
_require_mapping(semantic_network, ("entities", "relationships", "triplets"))
# Track YAML export
tracking_id = self.progress_tracker.start_tracking(
file=None,
@@ -119,15 +213,14 @@ class SemanticNetworkYAMLExporter:
self.progress_tracker.update_tracking(
tracking_id, message="Preparing YAML data..."
)
records = normalize_graph_payload(semantic_network)
yaml_data = {
"metadata": {
"exported_at": datetime.now().isoformat(),
"version": "1.0",
**semantic_network.get("metadata", {}),
},
"entities": semantic_network.get("entities", []),
"relationships": semantic_network.get("relationships", []),
"triplets": semantic_network.get("triplets", []),
**records,
}
self.progress_tracker.update_tracking(
@@ -140,7 +233,7 @@ class SemanticNetworkYAMLExporter:
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message="Exported semantic network to YAML",
message="Serialized semantic network to YAML",
)
return result
@@ -160,16 +253,46 @@ class SemanticNetworkYAMLExporter:
data: Data to export
file_path: Output file path
**options: Additional options
Raises:
ProcessingError: if ``data`` is not a mapping.
ValidationError: on the mappings :meth:`export_semantic_network`
rejects. Serialization runs before the output directory is
created, so a rejected export leaves nothing behind.
OSError: if the file cannot be written. The write is tracked
separately from serialization, so no progress entry reports a
completed export until the bytes are on disk.
"""
file_path = Path(file_path)
ensure_directory(file_path.parent)
yaml_content = self.export_semantic_network(data, **options)
with open(file_path, "w", encoding="utf-8") as f:
f.write(yaml_content)
# Serialization reports its own completion, but it says nothing about
# the file: without this second span, a failing write would leave the
# tracker showing a completed export and no output.
tracking_id = self.progress_tracker.start_tracking(
file=str(file_path),
module="export",
submodule="SemanticNetworkYAMLExporter",
message=f"Writing YAML to {file_path}",
)
self.logger.info(f"Exported YAML to: {file_path}")
try:
ensure_directory(file_path.parent)
with open(file_path, "w", encoding="utf-8") as f:
f.write(yaml_content)
self.logger.info(f"Exported YAML to: {file_path}")
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Exported YAML to: {file_path}",
)
except Exception as e:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e)
)
raise
def export_entities(
self, entities: List[Dict[str, Any]], include_metadata: bool = True, **options
@@ -263,18 +386,34 @@ class SemanticNetworkYAMLExporter:
• Structure for definition generation
• Include extraction metadata
• Return pipeline-ready YAML
Args:
extracted_data: Semantic network mapping, read through
:func:`~semantica.utils.helpers.normalize_graph_payload` on
the same terms as :meth:`export_semantic_network`.
pipeline_stage: Stage number recorded in the output.
**options: Additional export options (unused)
Returns:
Pipeline-ready YAML string.
Raises:
ProcessingError: if ``extracted_data`` is not a mapping.
ValidationError: on the same mappings as
:meth:`export_semantic_network` -- this method built its
nested semantic network from the same defaulted lookups and
so had the same silent-empty failure.
"""
_require_mapping(extracted_data, ("entities", "relationships", "triplets"))
semantic_network = normalize_graph_payload(extracted_data)
yaml_data = {
"pipeline_stage": pipeline_stage,
"metadata": {
"extracted_at": datetime.now().isoformat(),
**extracted_data.get("metadata", {}),
},
"semantic_network": {
"entities": extracted_data.get("entities", []),
"relationships": extracted_data.get("relationships", []),
"triplets": extracted_data.get("triplets", []),
},
"semantic_network": semantic_network,
}
return self.yaml.dump(yaml_data, default_flow_style=False, sort_keys=False)
@@ -308,7 +447,29 @@ class YAMLSchemaExporter:
• Include hierarchies and constraints
• Structure for easy editing
• Return YAML schema
Args:
ontology: Ontology mapping keyed by any of 'classes',
'properties', 'namespaces', 'uri', 'title', 'description',
'version'.
**options: Additional export options (unused)
Returns:
YAML schema string.
Raises:
ProcessingError: if ``ontology`` is not a mapping.
ValidationError: if ``ontology`` is a non-empty mapping sharing
no key with the recognized set, or resolves to nothing while
an unread key still holds records
(``{"classes": [], "nodes": [...]}``) -- each previously
produced a file with empty 'classes', 'properties' and
'namespaces' and no indication anything was dropped. An empty
mapping is still accepted.
"""
_require_mapping(ontology, ("classes", "properties"))
_require_usable_schema(ontology)
yaml_data = {
"ontology": {
"uri": ontology.get("uri", ""),
semantica/ingest/mcp_client.py
+26 -23
View File
@@ -41,6 +41,7 @@ from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from .ssrf import request_with_ssrf_guard
@dataclass
@@ -341,36 +342,38 @@ class MCPClient:
raise
def _send_request_http(self, request: Dict[str, Any]) -> Optional[Dict[str, Any]]:
"""Send request via HTTP."""
try:
import httpx
"""Send request via HTTP, with redirect-safe credential handling.
response = httpx.post(
Uses ``request_with_ssrf_guard`` so that:
* ``Authorization`` / ``Proxy-Authorization`` headers are **not**
forwarded to a different origin if the MCP server issues a redirect
(issue #947).
* The redirect chain is bounded (default 10 hops).
``allow_private_ips=True`` is set because MCP servers are explicitly
configured by the operator and frequently run on localhost or an
internal network — the same trust model as ``allow_private_ips`` opt-in
in the other ingestors. That trust covers only ``self.url`` itself:
``allow_private_ips_on_redirect=False`` keeps redirect targets held to
the normal public-address check, so a compromised or malicious MCP
server cannot use a redirect to route the client into private/
internal address space (e.g. cloud metadata) that the operator never
configured. Scheme validation (http/https only) and the
auth-stripping logic remain active regardless of these flags.
"""
try:
response = request_with_ssrf_guard(
"POST",
self.url,
json=request,
headers=self.headers,
json=request,
timeout=self.config.get("timeout", 30.0),
allow_private_ips=True,
allow_private_ips_on_redirect=False,
)
response.raise_for_status()
return response.json()
except (ImportError, OSError):
# Fallback to requests if httpx not available
try:
import requests
response = requests.post(
self.url,
json=request,
headers=self.headers,
timeout=self.config.get("timeout", 30.0),
)
response.raise_for_status()
return response.json()
except (ImportError, OSError):
raise ProcessingError(
"HTTP transport requires 'httpx' or 'requests' package. "
"Install with: pip install httpx or pip install requests"
)
except Exception as e:
self.logger.error(f"Failed to send HTTP request: {e}")
raise
semantica/ingest/public_api_ingestor.py
+32 -6
View File
@@ -45,6 +45,7 @@ except ModuleNotFoundError: # pragma: no cover - fallback for minimal installs
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from .api_ingestor import APIData, RESTIngestor
from .ssrf import request_with_ssrf_guard
AUTH_HEADER_NAMES = {
"authorization",
@@ -359,18 +360,31 @@ class PublicAPIIngestor(RESTIngestor):
request_options = options.copy()
timeout = request_options.pop("timeout", self.config.get("timeout", 30))
rate_limit_delay = request_options.pop("rate_limit_delay", None)
# session and allow_private_ips are always supplied explicitly below;
# drop any caller-provided copies so request_with_ssrf_guard() does
# not receive duplicate keyword arguments.
request_options.pop("session", None)
request_options.pop("allow_private_ips", None)
request_headers = self._merged_headers(headers)
try:
self._wait_if_needed(rate_limit_delay=rate_limit_delay)
response = self.session.request(
method=method,
url=endpoint,
# Route through the SSRF guard so that:
# * redirects to private/loopback IPs are blocked, and
# * Authorization / Proxy-Authorization are stripped on
# cross-origin redirects (issue #947).
response = request_with_ssrf_guard(
method,
endpoint,
session=self.session,
headers=request_headers,
params=params,
timeout=timeout,
allow_private_ips=self.allow_private_ips,
**request_options,
)
except (ValidationError, ProcessingError):
raise
except requests.exceptions.RequestException as exc:
self.logger.error(f"Failed to detect public API {endpoint}: {exc}")
raise ProcessingError(f"Failed to detect public API: {exc}") from exc
@@ -440,18 +454,30 @@ class PublicAPIIngestor(RESTIngestor):
request_options = options.copy()
timeout = request_options.pop("timeout", self.config.get("timeout", 30))
# session and allow_private_ips are always supplied explicitly below;
# drop any caller-provided copies so request_with_ssrf_guard() does
# not receive duplicate keyword arguments.
request_options.pop("session", None)
request_options.pop("allow_private_ips", None)
request_headers = self._merged_headers(headers)
try:
self._wait_if_needed(rate_limit_delay=rate_limit_delay)
response = self.session.request(
method=method,
url=endpoint,
# Route through the SSRF guard so that:
# * redirects to private/loopback IPs are blocked, and
# * Authorization / Proxy-Authorization are stripped on
# cross-origin redirects even when validate_no_auth=False
# (issue #947).
response = request_with_ssrf_guard(
method,
endpoint,
session=self.session,
headers=request_headers,
params=params,
data=data,
json=json_data,
timeout=timeout,
allow_private_ips=self.allow_private_ips,
**request_options,
)
semantica/ingest/ssrf.py
+184 -43
View File
@@ -268,6 +268,7 @@ def request_with_ssrf_guard(
*,
session: Optional[requests.Session] = None,
allow_private_ips: bool = False,
allow_private_ips_on_redirect: Optional[bool] = None,
max_redirects: int = _DEFAULT_MAX_REDIRECTS,
**kwargs: Any,
) -> requests.Response:
@@ -277,10 +278,65 @@ def request_with_ssrf_guard(
public URL to bounce into private/loopback/link-local space. This helper
disables automatic redirects and re-validates each ``Location`` target
before issuing the next hop.
``allow_private_ips`` trusts the caller's own *url* (e.g. an
operator-configured internal endpoint). That trust follows a redirect
only when the redirect target's host matches the original host (e.g. a
same-host path redirect on a private/localhost server); a redirect to a
*different* host is validated with ``allow_private_ips_on_redirect``
instead, which defaults to ``allow_private_ips`` for backward
compatibility but can be pinned to ``False`` by callers that want to
trust only the original host and never extend private-IP eligibility to
any other host a redirect chain might reach — otherwise a private-IP-
eligible endpoint could be tricked into redirecting into arbitrary
internal address space (e.g. cloud metadata) the caller never
configured.
Authorization / credential-header handling (issue #947)
--------------------------------------------------------
Credentials are stripped from **all** sources that ``requests`` can use to
attach an ``Authorization`` header whenever a redirect changes origin:
1. ``kwargs["headers"]`` — per-request header dict (already handled).
2. ``session.headers`` — session-level headers that ``requests`` merges
automatically; cleared for the hop and restored via ``finally``.
3. ``kwargs["auth"]`` — per-request auth tuple/callable; removed from the
local ``kwargs`` copy when stripping is required. This copy never
escapes to the caller, so there is nothing to restore.
4. ``session.auth`` — session-level auth handler that ``requests`` merges
via ``merge_setting(auth, self.auth)`` inside ``prepare_request``;
cleared for the hop and restored via ``finally``.
5. ``session.trust_env`` — when ``True``, ``requests`` reads ``~/.netrc``
for the *redirect target* host and calls ``prepare_auth()`` with those
credentials even after sources 3 and 4 are cleared; disabled for
cross-origin hops and restored via ``finally``.
Leaving any one of these intact allows ``requests`` to re-attach
credentials on the hop to the foreign origin, defeating the header-level
strip.
Session state that was removed is unconditionally restored in a ``finally``
block so the session is left in its original state after this call returns,
regardless of how it exits (normal return, exception, redirect cap). The
loop is sequential and single-threaded within one call, so the mutation is
safe as long as the caller does not share the session across concurrent
threads (the standard Semantica pattern: one session per ingestor instance).
Once credentials have been stripped for a cross-origin hop they are NOT
re-added for subsequent hops in the same chain, even if a later hop
happens to point back to the original host. This prevents credential
resurrection via crafted multi-hop redirect chains.
"""
kwargs = dict(kwargs)
kwargs.pop("allow_redirects", None)
redirect_allow_private_ips = (
allow_private_ips
if allow_private_ips_on_redirect is None
else allow_private_ips_on_redirect
)
_original_host = (urlparse(url).hostname or "").lower()
validate_url_for_request(url, allow_private_ips=allow_private_ips)
requester = session.request if session is not None else requests.request
@@ -288,56 +344,141 @@ def request_with_ssrf_guard(
current_method = method.upper()
redirects_followed = 0
while True:
response = requester(
current_method,
current_url,
allow_redirects=False,
**kwargs,
)
# -- issue #947: snapshot every session-level credential source so we can
# restore them unconditionally when this call exits.
_SENSITIVE = ("Authorization", "Proxy-Authorization")
_session_auth_backup: dict = {}
_session_auth_handler_backup: Any = None # session.auth backup
_session_trust_env_backup: bool = True # session.trust_env backup
if response.status_code not in _REDIRECT_STATUS_CODES:
return response
if session is not None:
for _h in _SENSITIVE:
# requests stores session headers in a case-insensitive dict;
# .get() matches regardless of the casing used at insertion time.
_val = session.headers.get(_h)
if _val is not None:
_session_auth_backup[_h] = _val
# Snapshot session.auth (HTTPBasicAuth, tuple, callable, or None).
_session_auth_handler_backup = session.auth
# Snapshot session.trust_env (controls .netrc / env proxy lookup).
_session_trust_env_backup = session.trust_env
if redirects_followed >= max_redirects:
response.close()
raise ValidationError(
f"Exceeded maximum redirects ({max_redirects}) while "
f"fetching '{url}'"
# Track whether credentials have been stripped for this redirect chain.
# Once stripped they must not reappear on any subsequent hop.
_auth_stripped = False
try:
while True:
response = requester(
current_method,
current_url,
allow_redirects=False,
**kwargs,
)
location = response.headers.get("Location")
if not location or not str(location).strip():
response.close()
raise ValidationError(
f"Redirect from '{current_url}' is missing a Location header"
if response.status_code not in _REDIRECT_STATUS_CODES:
return response
if redirects_followed >= max_redirects:
response.close()
raise ValidationError(
f"Exceeded maximum redirects ({max_redirects}) while "
f"fetching '{url}'"
)
location = response.headers.get("Location")
if not location or not str(location).strip():
response.close()
raise ValidationError(
f"Redirect from '{current_url}' is missing a Location header"
)
next_url = urljoin(current_url, str(location).strip())
next_host = (urlparse(next_url).hostname or "").lower()
# A redirect back to the original host inherits the caller's
# trust in that host (e.g. a same-host path redirect on a
# private/localhost MCP server). A redirect to a *different*
# host must not inherit that trust, even if the original host
# was private/internal — otherwise a compromised or malicious
# endpoint could redirect into arbitrary private address space
# (e.g. cloud metadata) the caller never configured.
hop_allow_private_ips = (
allow_private_ips
if next_host and next_host == _original_host
else redirect_allow_private_ips
)
validate_url_for_request(next_url, allow_private_ips=hop_allow_private_ips)
next_url = urljoin(current_url, str(location).strip())
validate_url_for_request(next_url, allow_private_ips=allow_private_ips)
# Do not leak sensitive headers or auth handlers to a different
# origin on redirects. All four credential sources are cleared:
# • kwargs["headers"] — per-request header dict
# • session.headers — session-level header dict
# • kwargs["auth"] — per-request auth tuple/callable
# • session.auth — session-level auth handler
#
# Once stripped (_auth_stripped=True), credentials stay absent for
# the remainder of the chain — even if a later hop targets the
# original host — to prevent credential resurrection.
if _auth_stripped or _should_strip_auth(current_url, next_url):
_auth_stripped = True
# Do not leak sensitive headers to a different origin on redirects:
# reuse the caller's headers only while host, port, and scheme keep
# the credential safe, mirroring requests' should_strip_auth.
if _should_strip_auth(current_url, next_url):
kwargs = dict(kwargs)
headers = dict(kwargs.get("headers") or {})
for sensitive in ("Authorization", "Proxy-Authorization"):
headers.pop(sensitive, None)
kwargs["headers"] = headers
# 1. Strip from per-request kwargs headers.
kwargs = dict(kwargs)
headers = dict(kwargs.get("headers") or {})
for sensitive in _SENSITIVE:
headers.pop(sensitive, None)
# Also remove any case variant the caller may have used
# (e.g. "authorization" or "AUTHORIZATION").
for key in list(headers):
if key.lower() == sensitive.lower():
del headers[key]
kwargs["headers"] = headers
# Match requests' historical method rewriting for 301/302/303.
if (
response.status_code in _STRIP_BODY_ON_REDIRECT
and current_method not in {"GET", "HEAD"}
):
current_method = "GET"
for key in ("data", "json", "files"):
kwargs.pop(key, None)
# 2. Strip per-request auth kwarg so requests cannot call
# prepare_auth() with the caller's credential on this hop.
kwargs.pop("auth", None)
# Params apply to the original request URL only; Location is authoritative.
kwargs.pop("params", None)
# 3. Strip session-level headers so requests cannot re-inject
# them when merging session + per-request headers for this hop.
if session is not None:
for sensitive in _SENSITIVE:
# CaseInsensitiveDict.pop(key, None) handles any casing.
session.headers.pop(sensitive, None)
response.close()
current_url = next_url
redirects_followed += 1
# 4. Clear session.auth so prepare_request's merge_setting()
# cannot fall back to the session-level auth handler and
# reattach credentials on the foreign-origin hop.
session.auth = None
# 5. Disable .netrc / environment-proxy credential lookup so
# requests cannot inject credentials from ~/.netrc for the
# redirect target host on this hop.
session.trust_env = False
# Match requests' historical method rewriting for 301/302/303.
if (
response.status_code in _STRIP_BODY_ON_REDIRECT
and current_method not in {"GET", "HEAD"}
):
current_method = "GET"
for key in ("data", "json", "files"):
kwargs.pop(key, None)
# Params apply to the original request URL only; Location is authoritative.
kwargs.pop("params", None)
response.close()
current_url = next_url
redirects_followed += 1
finally:
# Unconditionally restore every session credential source we touched,
# so the session is in its original state after this call returns or raises.
if session is not None:
if _session_auth_backup:
for _h, _v in _session_auth_backup.items():
session.headers[_h] = _v
# Restore session.auth to whatever it was before this call.
session.auth = _session_auth_handler_backup
# Restore session.trust_env (.netrc / env-proxy lookup flag).
session.trust_env = _session_trust_env_backup
semantica/kg/_graph_view.py
+206
View File
@@ -0,0 +1,206 @@
"""Internal graph view helpers shared by KG analytics modules."""
from dataclasses import dataclass
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
@dataclass
class GraphView:
"""Normalized node and edge view used by graph analytics."""
nodes: List[Any]
edges: List[Tuple[Any, Any]]
def build_graph_view(graph: Any) -> GraphView:
"""Build a graph view without dropping explicitly declared nodes.
Graph analytics accepts graph dictionaries, ContextGraph-like objects, and
NetworkX graphs. Nodes declared without an incident edge remain in the
returned view so callers can choose how to handle isolated nodes.
"""
nodes: List[Any] = []
edges: List[Tuple[Any, Any]] = []
seen_nodes: Set[Any] = set()
seen_edges: Set[Tuple[Any, Any]] = set()
def add_node(value: Any) -> Optional[Any]:
node_id = _node_id(value)
if node_id is None or node_id == "":
return None
if node_id not in seen_nodes:
seen_nodes.add(node_id)
nodes.append(node_id)
return node_id
for node in _extract_nodes(graph):
add_node(node)
for raw_edge in _extract_edges(graph):
edge = _edge_endpoints(raw_edge)
if edge is None:
continue
source, target = edge
source = add_node(source)
target = add_node(target)
if source is None or target is None:
continue
if (source, target) not in seen_edges:
seen_edges.add((source, target))
edges.append((source, target))
return GraphView(nodes=nodes, edges=edges)
def build_adjacency(graph: Any, directed: bool = False) -> Dict[Any, List[Any]]:
"""Build an adjacency list while preserving isolated graph nodes."""
view = build_graph_view(graph)
adjacency: Dict[Any, List[Any]] = {node: [] for node in view.nodes}
for source, target in view.edges:
if target not in adjacency[source]:
adjacency[source].append(target)
if not directed and source not in adjacency[target]:
adjacency[target].append(source)
return adjacency
def _extract_nodes(graph: Any) -> Iterable[Any]:
if isinstance(graph, dict):
raw_nodes: List[Any] = []
for key in ("entities", "nodes"):
values = graph.get(key, [])
if isinstance(values, dict):
raw_nodes.extend(values.keys())
elif values:
raw_nodes.extend(values)
return raw_nodes
raw_nodes = getattr(graph, "nodes", None)
if callable(raw_nodes):
return raw_nodes()
if isinstance(raw_nodes, dict):
return raw_nodes.keys()
if raw_nodes is not None:
return raw_nodes
get_nodes = getattr(graph, "get_nodes", None)
if callable(get_nodes):
return get_nodes()
return []
def _extract_edges(graph: Any) -> Iterable[Any]:
if isinstance(graph, dict):
raw_edges: List[Any] = []
for key in ("relationships", "edges"):
values = graph.get(key, [])
if values:
raw_edges.extend(values)
return raw_edges
raw_edges: List[Any] = []
relationships = getattr(graph, "relationships", None)
if relationships is not None:
raw_edges.extend(relationships)
edges = getattr(graph, "edges", None)
if callable(edges):
raw_edges.extend(edges())
elif edges is not None:
raw_edges.extend(edges)
if raw_edges:
return raw_edges
get_relationships = getattr(graph, "get_relationships", None)
if callable(get_relationships):
return get_relationships()
return []
def _edge_endpoints(edge: Any) -> Optional[Tuple[Any, Any]]:
if isinstance(edge, (tuple, list)) and len(edge) >= 2:
return edge[0], edge[1]
if isinstance(edge, dict):
source = _first_value(
edge,
"source",
"source_id",
"subject",
"start",
"start_id",
"from",
"src",
"START_ID",
":START_ID",
)
target = _first_value(
edge,
"target",
"target_id",
"object",
"end",
"end_id",
"to",
"dst",
"END_ID",
":END_ID",
)
else:
source = _first_attribute(
edge,
"source_id",
"source",
"subject",
"start",
"start_id",
"from_id",
)
target = _first_attribute(
edge,
"target_id",
"target",
"object",
"end",
"end_id",
"to_id",
)
if source is None or target is None:
return None
return source, target
def _node_id(value: Any) -> Any:
if isinstance(value, dict):
value = _first_value(
value, "id", "node_id", "entity_id", "key", "name", "text"
)
elif not isinstance(value, (str, int, float, bool, bytes, tuple)):
value = _first_attribute(
value, "node_id", "id", "entity_id", "key", "name", "text"
)
if value is None:
return None
try:
hash(value)
except TypeError:
return str(value)
return value
def _first_value(mapping: Dict[str, Any], *keys: str) -> Any:
for key in keys:
if key in mapping and mapping[key] not in (None, ""):
return mapping[key]
return None
def _first_attribute(value: Any, *names: str) -> Any:
for name in names:
attribute = getattr(value, name, None)
if attribute not in (None, ""):
return attribute
return None
semantica/kg/centrality_calculator.py
+6 -66
View File
@@ -43,7 +43,7 @@ Author: Semantica Contributors
License: MIT
"""
from collections import defaultdict, deque
from collections import deque
from typing import Any, Dict, List, Optional
import numpy as np
@@ -51,6 +51,7 @@ from scipy import sparse
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ._graph_view import build_adjacency, build_graph_view
class CentralityCalculator:
@@ -518,76 +519,15 @@ class CentralityCalculator:
def _build_adjacency(self, graph) -> Dict[str, List[str]]:
"""Build adjacency list from graph."""
adjacency = defaultdict(list)
# Extract relationships
relationships = []
if hasattr(graph, "relationships"):
relationships = graph.relationships
elif hasattr(graph, "get_relationships"):
relationships = graph.get_relationships()
elif isinstance(graph, dict):
relationships = graph.get("relationships", graph.get("edges", []))
elif hasattr(graph, "edges") and not callable(graph.edges):
# ContextGraph-style: edges is a list of dataclass objects with source_id/target_id
for edge in (graph.edges or []):
if isinstance(edge, dict):
src = edge.get("source") or edge.get("source_id")
tgt = edge.get("target") or edge.get("target_id")
else:
src = getattr(edge, "source_id", None) or getattr(edge, "source", None)
tgt = getattr(edge, "target_id", None) or getattr(edge, "target", None)
if src and tgt:
src, tgt = str(src), str(tgt)
if tgt not in adjacency[src]:
adjacency[src].append(tgt)
if src not in adjacency[tgt]:
adjacency[tgt].append(src)
return dict(adjacency)
# Build adjacency
for rel in relationships:
# Handle tuple/list edges (e.g., from NetworkX)
if isinstance(rel, (tuple, list)) and len(rel) >= 2:
source, target = str(rel[0]), str(rel[1])
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
continue
source = rel.get("source") or rel.get("subject")
target = rel.get("target") or rel.get("object")
# Extract IDs if objects are passed
if source and not isinstance(source, (str, int, float)):
if isinstance(source, dict):
source = source.get("id") or source.get("entity_id") or source.get("text") or str(source)
else:
source = getattr(source, "id", getattr(source, "text", str(source)))
if target and not isinstance(target, (str, int, float)):
if isinstance(target, dict):
target = target.get("id") or target.get("entity_id") or target.get("text") or str(target)
else:
target = getattr(target, "id", getattr(target, "text", str(target)))
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
return dict(adjacency)
return build_adjacency(graph)
def _to_networkx(self, graph):
"""Convert graph to NetworkX format."""
adjacency = self._build_adjacency(graph)
view = build_graph_view(graph)
nx_graph = self.nx.Graph()
for source, targets in adjacency.items():
for target in targets:
nx_graph.add_edge(source, target)
nx_graph.add_nodes_from(view.nodes)
nx_graph.add_edges_from(view.edges)
return nx_graph
semantica/kg/community_detector.py
+47 -80
View File
@@ -49,6 +49,16 @@ from typing import Any, Dict, List, Optional
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ._graph_view import build_adjacency, build_graph_view
def _is_hashable(value: Any) -> bool:
"""Return whether a community identifier can be used in a set."""
try:
hash(value)
except TypeError:
return False
return True
class CommunityDetector:
@@ -157,17 +167,18 @@ class CommunityDetector:
nx_graph = self._to_networkx(graph)
# Check if graph is empty or has no edges
# An empty graph has no communities. A graph with nodes but
# no edges still has singleton communities.
num_nodes = nx_graph.number_of_nodes()
num_edges = nx_graph.number_of_edges()
self.logger.debug(f"Graph stats: nodes={num_nodes}, edges={num_edges}")
if num_nodes == 0 or num_edges == 0:
self.logger.warning("Graph is empty or has no edges, returning 0 communities")
if num_nodes == 0:
self.logger.warning("Graph is empty, returning 0 communities")
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message="Detected 0 communities (empty graph/no edges)",
message="Detected 0 communities (empty graph)",
)
return {
"communities": [],
@@ -350,17 +361,7 @@ class CommunityDetector:
adjacency = self._build_adjacency(graph)
# Extract community structure
if isinstance(communities, dict):
node_communities = communities
elif isinstance(communities, dict) and "node_assignments" in communities:
node_communities = communities["node_assignments"]
else:
# Convert list of communities to node assignments
node_communities = {}
for i, community in enumerate(communities):
for node in community:
node_communities[node] = i
node_communities = self._to_node_assignments(communities)
# Calculate metrics
num_communities = len(set(node_communities.values()))
@@ -408,16 +409,7 @@ class CommunityDetector:
metrics = self.calculate_community_metrics(graph, communities)
# Extract node assignments
if isinstance(communities, dict) and "node_assignments" in communities:
node_communities = communities["node_assignments"]
elif isinstance(communities, dict):
node_communities = communities
else:
node_communities = {}
for i, community in enumerate(communities):
for node in community:
node_communities[node] = i
node_communities = self._to_node_assignments(communities)
# Analyze connectivity between communities
adjacency = self._build_adjacency(graph)
@@ -440,6 +432,32 @@ class CommunityDetector:
"edge_ratio": intra_community_edges / (inter_community_edges + 1),
}
@staticmethod
def _to_node_assignments(communities: Any) -> Dict[Any, Any]:
"""Normalize community results to a node-to-community mapping."""
if isinstance(communities, dict):
assignments = communities.get("node_assignments")
if isinstance(assignments, dict):
return assignments
detected_communities = communities.get("communities")
if isinstance(detected_communities, (list, tuple)):
communities = detected_communities
elif "communities" in communities:
raise ValueError("Community results must contain a list of communities")
elif not all(_is_hashable(value) for value in communities.values()):
raise ValueError(
"Community assignments must map nodes to hashable community IDs"
)
else:
return communities
node_assignments: Dict[Any, Any] = {}
for community_id, community in enumerate(communities or []):
for node in community:
node_assignments[node] = community_id
return node_assignments
def detect_communities(
self, graph: Any, algorithm: str = "louvain", method: str = None, **options
) -> Dict[str, Any]:
@@ -478,57 +496,7 @@ class CommunityDetector:
def _build_adjacency(self, graph) -> Dict[str, List[str]]:
"""Build adjacency list from graph."""
from collections import defaultdict
adjacency = defaultdict(list)
# Extract relationships
relationships = []
raw_edges = [] # flat (u, v) tuples
if hasattr(graph, "relationships"):
relationships = graph.relationships
elif hasattr(graph, "get_relationships"):
relationships = graph.get_relationships()
elif isinstance(graph, dict):
relationships = graph.get("relationships", [])
# Also handle 'edges' key (list of tuples or dicts)
for edge in graph.get("edges", []):
if isinstance(edge, (list, tuple)) and len(edge) >= 2:
raw_edges.append((str(edge[0]), str(edge[1])))
elif isinstance(edge, dict):
relationships.append(edge)
# Add raw (u, v) edges
for u, v in raw_edges:
if u and v:
adjacency[u].append(v)
adjacency[v].append(u)
# Build adjacency
for rel in relationships:
source = rel.get("source") or rel.get("subject")
target = rel.get("target") or rel.get("object")
# Extract IDs if objects are passed
if source and not isinstance(source, (str, int, float)):
if isinstance(source, dict):
source = source.get("id") or source.get("entity_id") or source.get("text") or str(source)
else:
source = getattr(source, "id", getattr(source, "text", str(source)))
if target and not isinstance(target, (str, int, float)):
if isinstance(target, dict):
target = target.get("id") or target.get("entity_id") or target.get("text") or str(target)
else:
target = getattr(target, "id", getattr(target, "text", str(target)))
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
return dict(adjacency)
return build_adjacency(graph)
def _to_networkx(self, graph):
"""Convert graph to NetworkX format."""
@@ -536,12 +504,11 @@ class CommunityDetector:
if hasattr(graph, 'nodes') and hasattr(graph, 'edges') and hasattr(graph, 'number_of_nodes'):
return graph
adjacency = self._build_adjacency(graph)
view = build_graph_view(graph)
nx_graph = self.nx.Graph()
for source, targets in adjacency.items():
for target in targets:
nx_graph.add_edge(source, target)
nx_graph.add_nodes_from(view.nodes)
nx_graph.add_edges_from(view.edges)
return nx_graph
semantica/kg/connectivity_analyzer.py
+3 -46
View File
@@ -48,11 +48,12 @@ Author: Semantica Contributors
License: MIT
"""
from collections import defaultdict, deque
from collections import deque
from typing import Any, Dict, List, Optional, Set, Tuple
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ._graph_view import build_adjacency
class ConnectivityAnalyzer:
@@ -385,51 +386,7 @@ class ConnectivityAnalyzer:
def _build_adjacency(self, graph) -> Dict[str, List[str]]:
"""Build adjacency list from graph."""
adjacency = defaultdict(list)
# Extract relationships
relationships = []
if hasattr(graph, "relationships"):
relationships = graph.relationships
elif hasattr(graph, "get_relationships"):
relationships = graph.get_relationships()
elif isinstance(graph, dict):
relationships = graph.get("relationships", graph.get("edges", []))
# Build adjacency
for rel in relationships:
# Handle tuple/list edges (e.g., from NetworkX)
if isinstance(rel, (tuple, list)) and len(rel) >= 2:
source, target = str(rel[0]), str(rel[1])
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
continue
source = rel.get("source") or rel.get("subject")
target = rel.get("target") or rel.get("object")
# Extract IDs if objects are passed
if source and not isinstance(source, (str, int, float)):
if isinstance(source, dict):
source = source.get("id") or source.get("entity_id") or source.get("text") or str(source)
else:
source = getattr(source, "id", getattr(source, "text", str(source)))
if target and not isinstance(target, (str, int, float)):
if isinstance(target, dict):
target = target.get("id") or target.get("entity_id") or target.get("text") or str(target)
else:
target = getattr(target, "id", getattr(target, "text", str(target)))
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
return dict(adjacency)
return build_adjacency(graph)
def _bfs_shortest_path(
self, adjacency: Dict[str, List[str]], source: str, target: str
semantica/normalize/number_normalizer.py
+12 -5
View File
@@ -562,6 +562,11 @@ class CurrencyNormalizer:
"SEK",
"NOK",
"DKK",
"RUB",
"KRW",
"ILS",
"NGN",
"PKR",
]
self.logger.debug("Currency normalizer initialized")
@@ -606,13 +611,15 @@ class CurrencyNormalizer:
# Check for currency code
if not currency_code:
for code in self.currency_codes:
if code in currency_input.upper():
match = re.search(
rf"(?<![A-Z]){re.escape(code)}(?![A-Z])",
currency_input.upper(),
)
if match:
currency_code = code
amount_str = (
currency_input.replace(code, "")
.replace(code.lower(), "")
.strip()
)
currency_input[: match.start()] + currency_input[match.end() :]
).strip()
amount_str = amount_str.replace(",", "").replace(" ", "")
try:
amount = float(amount_str)
semantica/parse/excel_parser.py
+1
View File
@@ -37,6 +37,7 @@ from openpyxl import load_workbook
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@dataclass
semantica/seed/seed_manager.py
+5 -2
View File
@@ -501,8 +501,11 @@ class SeedDataManager:
else:
full_url = api_url
# Prepare headers
request_headers = headers or {}
# Prepare headers — copy the caller's dict so we never mutate it in-place.
# Without the copy, adding "Authorization" here would silently modify the
# caller's original dict and potentially leak the key to subsequent calls
# that reuse the same dict without expecting it to contain credentials.
request_headers = dict(headers) if headers else {}
if api_key:
request_headers["Authorization"] = f"Bearer {api_key}"
semantica/semantic_extract/ner_extractor.py
+6 -1
View File
@@ -144,7 +144,12 @@ class NERExtractor:
self._ml_runtime_usable = True
if "ml" in self.method and SPACY_AVAILABLE:
try:
self.nlp = spacy.load(self.model_name)
# Deferred import: keeps semantic_extract.methods out of the
# module-level import graph and routes loading through the
# process-level cache so repeated NERExtractor constructions
# never pay the ~120 ms spacy.load() cost more than once.
from .methods import load_spacy_model
self.nlp = load_spacy_model(self.model_name)
except OSError:
self.logger.warning(
f"spaCy model {self.model_name} not found. ML method will fallback."
semantica/split/methods.py
+3 -2
View File
@@ -97,7 +97,7 @@ from .semantic_chunker import Chunk
logger = get_logger("split_methods")
# Try to import optional dependencies
spacy, SPACY_AVAILABLE = safe_import("spacy")
_, SPACY_AVAILABLE = safe_import("spacy")
nltk, NLTK_AVAILABLE = safe_import("nltk")
tiktoken, TIKTOKEN_AVAILABLE = safe_import("tiktoken")
@@ -336,7 +336,8 @@ def split_by_sentences(
# Try spaCy first
if SPACY_AVAILABLE and kwargs.get("use_spacy", True):
try:
nlp = spacy.load("en_core_web_sm")
from ..semantic_extract.methods import load_spacy_model
nlp = load_spacy_model("en_core_web_sm")
doc = nlp(text)
sentences = [sent.text for sent in doc.sents]
except Exception:
semantica/split/semantic_chunker.py
+11 -2
View File
@@ -36,7 +36,8 @@ from ..utils.helpers import safe_import
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
spacy, SPACY_AVAILABLE = safe_import("spacy")
_, SPACY_AVAILABLE = safe_import("spacy")
@dataclass
@@ -79,11 +80,19 @@ class SemanticChunker:
if SPACY_AVAILABLE:
model_name = config.get("model", "en_core_web_sm")
try:
self.nlp = spacy.load(model_name)
from ..semantic_extract.methods import load_spacy_model
self.nlp = load_spacy_model(model_name)
except OSError:
self.logger.warning(
f"spaCy model {model_name} not found. Using fallback chunking."
)
except Exception:
self.logger.warning(
"spaCy model %s failed to initialize and will be disabled "
"for this chunker instance. Using fallback chunking.",
model_name,
exc_info=True,
)
def chunk(self, text: str, **options) -> List[Chunk]:
"""
semantica/utils/__init__.py
+2
View File
@@ -80,6 +80,7 @@ from .helpers import (
hash_data,
merge_dicts,
normalize_entities,
normalize_graph_payload,
parse_timestamp,
read_json_file,
retry_on_error,
@@ -183,6 +184,7 @@ __all__ = [
"format_data",
"clean_text",
"normalize_entities",
"normalize_graph_payload",
"hash_data",
"safe_filename",
"ensure_directory",
semantica/utils/helpers.py
+392 -9
View File
@@ -63,9 +63,16 @@ import importlib
import json
import os
import re
import types
from collections import Counter
from collections.abc import Iterable as IterableABC
from collections.abc import Mapping
from dataclasses import asdict, is_dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Type, Union
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple, Type, Union
from .exceptions import ProcessingError, ValidationError
def format_data(data: Any, format_type: str = "json") -> str:
@@ -391,9 +398,7 @@ def chunk_list(items: List[Any], chunk_size: int) -> List[List[Any]]:
Returns:
List of chunks
"""
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)]
def flatten_dict(
d: Dict[str, Any], parent_key: str = "", sep: str = "."
) -> Dict[str, Any]:
@@ -407,18 +412,32 @@ def flatten_dict(
Returns:
Flattened dictionary
Raises:
ValueError: If two input paths produce the same flattened key.
"""
items = []
result = {}
for k, v in d.items():
new_key = f"{parent_key}{sep}{k}" if parent_key else k
if isinstance(v, dict):
items.extend(flatten_dict(v, new_key, sep=sep).items())
else:
items.append((new_key, v))
nested = flatten_dict(v, new_key, sep=sep)
return dict(items)
for key, value in nested.items():
if key in result:
raise ValueError(
f"Key collision while flattening dictionary: {key}"
)
result[key] = value
else:
if new_key in result:
raise ValueError(
f"Key collision while flattening dictionary: {new_key}"
)
result[new_key] = v
return result
def get_nested_value(
@@ -584,3 +603,367 @@ def classify_path_distance(hop_count: int) -> str:
if hop_count <= 6:
return "mid-range"
return "distant"
# Graph payloads circulate under two vocabularies: 'entities'/'relationships'
# (kg builders, most exporters) and 'nodes'/'edges' (ContextGraph.to_dict,
# Neo4jCSVExporter, the Explorer routes). Consumers each reconciled them
# locally, with at least three competing idioms, so the same payload could be
# exported, silently dropped, or rejected depending on which consumer read it.
# This is the single place that decision is made.
_ENTITY_KEYS = ("entities", "nodes")
_RELATIONSHIP_KEYS = ("relationships", "edges")
_TRIPLET_KEYS = ("triplets",)
# Keys that legitimately travel alongside the collections without being
# records themselves, so their presence is never evidence that records were
# dropped: ContextGraph.to_dict() carries 'statistics', JSON envelopes carry
# 'metadata' and 'count'.
_CONTEXT_KEYS = ("metadata", "statistics", "count")
def _require_recognized_keys(
payload: Mapping, recognized_keys: Sequence[str], *, what: str
) -> None:
"""Reject a mapping that shares no key with the recognized set.
A consumer that reads a fixed set of keys turns an unrecognized mapping
into an empty result that looks like a legitimate one. An empty mapping is
allowed through -- it carries nothing that could be lost.
Args:
payload: Mapping to check.
recognized_keys: Keys the consumer reads.
what: Noun for the error message, e.g. ``"Graph payload"``.
Raises:
ValidationError: if ``payload`` is non-empty and shares no key with
``recognized_keys``.
"""
if not payload or any(key in payload for key in recognized_keys):
return
supplied = ", ".join(f"'{key}'" for key in sorted(map(str, payload)))
expected = ", ".join(f"'{key}'" for key in recognized_keys)
raise ValidationError(
f"{what} has no recognized key. Supplied: {supplied}. "
f"Expected at least one of: {expected}."
)
def _require_nothing_dropped(
payload: Mapping,
recognized_keys: Sequence[str],
resolved: Iterable[Any],
*,
what: str,
) -> None:
"""Reject a mapping that resolved to nothing while still holding records.
Checking that a recognized key is *present* is not enough:
``{"entities": [], "data": [...]}`` clears that bar and still resolves to
empty, dropping every record under 'data'. Presence answers "did the
caller use our vocabulary"; this answers the question that actually
matters, "did anything the caller supplied survive".
Only non-empty lists count as evidence of dropped records. A payload can
carry scalars and dicts that are not collections -- ContextGraph.to_dict()
always includes 'statistics' -- and an empty graph must stay exportable.
Args:
payload: Mapping to check.
recognized_keys: Keys the consumer reads.
resolved: The collections the consumer resolved from ``payload``.
what: Noun for the error message, e.g. ``"Graph payload"``.
Raises:
ValidationError: if nothing resolved and an unread key holds a
non-empty list.
"""
if any(resolved):
return
dropped = sorted(
str(key)
for key, value in payload.items()
if key not in recognized_keys
and key not in _CONTEXT_KEYS
and isinstance(value, (list, tuple))
and value
)
if not dropped:
return
named = ", ".join(f"'{key}'" for key in dropped)
expected = ", ".join(f"'{key}'" for key in recognized_keys)
raise ValidationError(
f"{what} resolved to nothing, but {named} still holds records. "
f"Exporting it would drop them silently. Supply the records under "
f"one of: {expected}."
)
def _is_record(value: Any) -> bool:
"""Report whether a value can stand in for a graph record.
Consumers read records either as mappings (``entity.get("type")`` in the
LPG and Arango exporters) or as objects with attributes
(``Neo4jCSVExporter._record_to_dict`` accepts dataclasses and anything
carrying a ``__dict__``). Both are legitimate, so both are accepted here;
strings, numbers, and nested sequences are not records under either
reading.
Modules and class/type objects are excluded even though they carry
``__dict__``: they are not graph records under any supported reading, and
passing them through the boundary would produce ``AttributeError`` inside
exporters rather than a ``ValidationError`` at the boundary where the
problem is visible.
"""
return isinstance(value, Mapping) or is_dataclass(value) or (
hasattr(value, "__dict__")
and not isinstance(value, (types.ModuleType, type))
)
def _record_to_dict(record: Any) -> Dict[str, Any]:
"""Convert an accepted record to a plain dict.
:func:`_is_record` accepts mappings, dataclasses, and objects carrying
``__dict__`` as legitimate record shapes, but consumers of
:func:`normalize_graph_payload` -- YAML serialization, ``entity.get(...)``
in the LPG and Arango exporters -- read records as dicts. Converting here,
at the boundary, means every exporter gets the same shape regardless of
which reading the caller used; previously only ``Neo4jCSVExporter``
converted object-shaped records locally, so a dataclass record passed
validation for the other exporters only to crash with a raw
``AttributeError`` once used.
"""
if isinstance(record, Mapping):
return dict(record)
if is_dataclass(record):
return asdict(record)
return {
key: value for key, value in vars(record).items() if not key.startswith("_")
}
def _coerce_records(key: str, value: Any) -> List[Any]:
"""Validate one collection value and materialize it as a list of records.
This runs before any truthiness or ``list()`` call, because both mislead
on malformed input: ``list("abc")`` quietly turns a string into three
single-character "records", and ``list(42)`` raises a bare ``TypeError``
from deep inside the exporter that named the exporter rather than the
offending payload key. Neither reaches the caller as an actionable
message, so the shapes that produce them are rejected by name instead.
``None`` is deliberately not rejected: JSON round-trips an absent
collection to null, and treating that as "no records under this key" is
the same answer an explicit ``[]`` gets. It is not silent data loss --
a null collection alongside records under an unread key is still caught
by :func:`_require_nothing_dropped`.
Args:
key: Payload key the value came from, for the error message.
value: The raw value stored under ``key``.
Returns:
The records as a new list, so the result never aliases the input.
Raises:
ValidationError: if ``value`` is a string, bytes, a mapping, or any
non-iterable scalar; or if any element is not a record.
"""
if value is None:
return []
if isinstance(value, (str, bytes, bytearray)):
raise ValidationError(
f"Graph payload key '{key}' holds a {type(value).__name__}, not a "
f"collection of records. Iterating it would yield characters, not "
f"records. Supply a list of records."
)
if isinstance(value, Mapping):
raise ValidationError(
f"Graph payload key '{key}' holds a mapping, not a collection of "
f"records. If it is a single record, wrap it in a list; if it is "
f"keyed by ID, supply its values as a list."
)
if not isinstance(value, IterableABC):
raise ValidationError(
f"Graph payload key '{key}' holds a "
f"{type(value).__name__}, not a collection of records. Supply a "
f"list of records."
)
records = list(value)
for index, record in enumerate(records):
if not _is_record(record):
raise ValidationError(
f"Graph payload key '{key}' holds a "
f"{type(record).__name__} at index {index}, not a record. "
f"Records must be mappings or objects with attributes."
)
return [_record_to_dict(record) for record in records]
def _canonical_record_multiset(records: List[Dict[str, Any]]) -> "Counter[str]":
"""Represent records as an order-independent multiset for equality checks.
Two spellings of the same collection (``entities`` and ``nodes``) can
legitimately list identical records in a different order -- a caller
round-tripping through a dict-keyed cache or a set has no reason to
preserve list order. Comparing with plain list equality would treat that
as a conflict and reject a payload that carries no real data loss, so
records are compared as a multiset of their canonical JSON form instead.
"""
return Counter(
json.dumps(record, sort_keys=True, default=str) for record in records
)
def _resolve_collection(
payload: Dict[str, Any], keys: Tuple[str, ...]
) -> List[Dict[str, Any]]:
"""Pick one collection from a payload that may use either vocabulary.
Both spellings may legitimately be present: ``JSONExporter`` writes
'entities' and 'nodes' side by side, so a round-trip of its output carries
both, one of them empty. Where only one holds records, that one wins.
Two non-empty, unequal spellings are a different matter -- there is no
basis for preferring either, and picking one would silently discard the
other -- so that is refused rather than guessed at.
Every spelling present is validated, not just the one that wins: a
malformed 'nodes' alongside a well-formed 'entities' is a payload the
caller should hear about, and validating only the winner would let it
through on the strength of the other key.
Args:
payload: Mapping to read from.
keys: Accepted spellings, most canonical first.
Returns:
The resolved collection, or an empty list if no spelling is present.
Raises:
ValidationError: if a spelling holds something other than a collection
of records; or if two spellings are both present, both non-empty,
and hold different records, order ignored.
"""
present = {
key: _coerce_records(key, payload[key]) for key in keys if key in payload
}
populated = {key: value for key, value in present.items() if value}
if len(populated) > 1:
values = list(populated.values())
canonical = [_canonical_record_multiset(value) for value in values]
if any(entry != canonical[0] for entry in canonical[1:]):
named = " and ".join(f"'{key}'" for key in populated)
raise ValidationError(
f"Graph payload carries {named} with different contents; "
f"cannot determine which to export. Supply one, or make them "
f"identical."
)
for key in keys:
value = present.get(key)
if value:
# Already a fresh list from _coerce_records, so the result cannot
# alias the caller's collection.
return value
# Every spelling present is empty (or none is): an explicit empty
# collection is a legitimate answer, distinct from "unrecognized".
return []
def _require_mapping(data: Any, expected_keys: Sequence[str]) -> None:
"""Reject non-mapping export input with an actionable error.
Shared by every consumer of :func:`normalize_graph_payload` so that a
wrong *type* fails the same way everywhere. Handed a sequence (or any
other non-mapping), every downstream key lookup would fail with a bare
``AttributeError: 'list' object has no attribute 'get'``, which tells the
caller nothing about the shape expected -- and ``normalize_graph_payload``
itself raises ``ValidationError`` for this case, which would leave
exporters that skip this guard raising a different exception type than
the ones that call it, for the identical mistake.
A list is rejected rather than wrapped: these formats distinguish
entities from relationships from triplets (or nodes/edges), so inferring
which one a bare list represents would silently mislabel the records.
Args:
data: Candidate export payload.
expected_keys: Key names the caller reads, named in the error so the
caller learns the expected shape.
Raises:
ProcessingError: if ``data`` is not a mapping.
"""
if not isinstance(data, Mapping):
keys = "/".join(f"'{key}'" for key in expected_keys)
raise ProcessingError(
f"Cannot export object of type '{type(data).__name__}': "
f"expected a dict with {keys}."
)
def normalize_graph_payload(
payload: Dict[str, Any],
) -> Dict[str, List[Dict[str, Any]]]:
"""Reduce a graph payload to one canonical vocabulary.
Accepts either 'entities'/'relationships' or 'nodes'/'edges' (or a mix)
and returns the canonical spelling, so consumers read one shape instead of
reimplementing the reconciliation.
This is the validation boundary for graph payloads: it either returns
collections of records or raises. Nothing that reaches an exporter through
it needs re-checking, and nothing malformed passes through it as a
valid-looking empty graph.
Args:
payload: Graph payload mapping.
Returns:
``{"entities": [...], "relationships": [...], "triplets": [...]}``.
Raises:
ValidationError: if ``payload`` is not a mapping; if a recognized key
holds something other than a collection of records; if two
spellings of the same collection are both non-empty and differ; if
a non-empty mapping contains no recognized key; or if it resolves
to nothing while an unread key still holds records. The last two
would otherwise hand the caller a valid-looking result with their
records silently dropped.
Example:
>>> normalize_graph_payload({"nodes": [{"id": "n1"}], "edges": []})
{'entities': [{'id': 'n1'}], 'relationships': [], 'triplets': []}
"""
if not isinstance(payload, Mapping):
raise ValidationError(
f"Cannot normalize graph payload of type "
f"'{type(payload).__name__}': expected a mapping."
)
recognized = _ENTITY_KEYS + _RELATIONSHIP_KEYS + _TRIPLET_KEYS
_require_recognized_keys(payload, recognized, what="Graph payload")
resolved = {
"entities": _resolve_collection(payload, _ENTITY_KEYS),
"relationships": _resolve_collection(payload, _RELATIONSHIP_KEYS),
"triplets": _resolve_collection(payload, _TRIPLET_KEYS),
}
_require_nothing_dropped(
payload, recognized, resolved.values(), what="Graph payload"
)
return resolved
tests/context/test_context_graph_docstring_example.py
+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env python3
"""Regression tests for the ContextGraph module docstring example.
The "Example Usage" block in ``semantica/context/context_graph.py`` previously
called ``add_node``/``add_edge`` with keyword arguments those methods do not
accept (``type=`` and ``properties=``), so the documented example raised
``TypeError`` -- and the near-miss variants silently nested the properties dict
instead of failing.
These tests keep the documented example executable and pin the two behaviours
that made the original mistake easy to miss.
"""
import doctest
import re
from typing import Dict, List
import pytest
import semantica.context.context_graph as context_graph_module
from semantica.context.context_graph import ContextGraph
# The example block runs to the next top-level section header (a line starting
# in column 0, e.g. "Production Use Cases:") or the end of the docstring.
# Terminating on the next header rather than on a blank line keeps the capture
# intact when the example gains blank lines or extra paragraphs.
_EXAMPLE_BLOCK_RE = re.compile(r"^Example Usage:\n(.*?)(?=^\S|\Z)", re.DOTALL | re.MULTILINE)
# ``type=`` as its own keyword, but not the legitimate ``node_type=``/``edge_type=``.
_BARE_TYPE_KWARG_RE = re.compile(r"(?<![\w])type\s*=")
def _example_block() -> str:
"""Return the 'Example Usage' block from the module docstring."""
doc = context_graph_module.__doc__ or ""
match = _EXAMPLE_BLOCK_RE.search(doc)
assert match, "module docstring no longer contains an 'Example Usage:' block"
block = match.group(1).strip()
assert block, "the 'Example Usage:' block in the module docstring is empty"
return block
def _example_statements() -> List[str]:
"""Return the documented ``>>>`` statements, continuation lines included."""
statements = [example.source for example in doctest.DocTestParser().get_examples(_example_block())]
assert statements, "the 'Example Usage:' block no longer contains any '>>>' statements"
return statements
def _statements_calling(method: str) -> List[str]:
"""Return the documented statements that call ``graph.<method>(``."""
return [stmt for stmt in _example_statements() if "graph.{}(".format(method) in stmt]
def _run_example() -> Dict[str, object]:
"""Execute the documented example verbatim and return its namespace."""
source = "".join(_example_statements())
namespace: Dict[str, object] = {}
exec(compile(source, "<context_graph module docstring>", "exec"), namespace)
return namespace
class TestDocstringExampleIsRunnable:
"""The documented example must execute exactly as written."""
def test_documented_calls_execute(self):
# Run the docstring text itself so this test cannot drift from the docs.
ns = _run_example()
graph = ns["graph"]
assert "Python" in graph.nodes
assert "Programming" in graph.nodes
assert graph.nodes["Python"].node_type == "language"
assert graph.nodes["Programming"].node_type == "concept"
neighbors = graph.get_neighbors("Python", hops=1)
assert any(n["id"] == "Programming" for n in neighbors)
# record_decision must return a non-empty string ID.
assert isinstance(ns["decision_id"], str) and ns["decision_id"]
# find_precedents must be called with that ID and return a list.
assert isinstance(ns["precedents"], list)
def test_node_properties_are_stored_flat(self):
"""``popularity`` must land as a top-level property, not nested.
Passing the previously documented ``properties={...}`` does not raise --
it stores a dict *inside* the properties dict, which is why the original
docs bug could reach a user's graph unnoticed.
"""
graph = ContextGraph(advanced_analytics=False)
graph.add_node("Python", "language", popularity="high")
assert graph.nodes["Python"].properties == {"popularity": "high"}
assert graph.find_node("Python")["metadata"]["popularity"] == "high"
assert "properties" not in graph.nodes["Python"].properties
def test_edge_type_is_positional_not_a_property(self):
"""``related_to`` must be the edge type, not a stray metadata key."""
graph = ContextGraph(advanced_analytics=False)
graph.add_node("Python", "language")
graph.add_node("Programming", "concept")
graph.add_edge("Python", "Programming", "related_to")
edge = graph.edges[0]
assert edge.edge_type == "related_to"
assert "type" not in edge.metadata
class TestDocstringExampleDoesNotRegress:
"""Guard the docstring text itself, not just equivalent code."""
def test_add_node_example_supplies_node_type_positionally(self):
calls = _statements_calling("add_node")
assert calls, "the 'Example Usage:' block no longer calls graph.add_node()"
for call in calls:
assert not _BARE_TYPE_KWARG_RE.search(call), (
f"add_node example passes type= as a keyword: {call!r}. "
"node_type is positional-required; type= falls through to "
"**properties and the call raises TypeError."
)
assert "properties=" not in call, (
f"add_node example passes properties=: {call!r}. "
"add_node has no properties parameter; extra properties are "
"passed as **kwargs."
)
def test_add_edge_example_supplies_edge_type_positionally(self):
calls = _statements_calling("add_edge")
assert calls, "the 'Example Usage:' block no longer calls graph.add_edge()"
for call in calls:
assert not _BARE_TYPE_KWARG_RE.search(call), (
f"add_edge example passes type= as a keyword: {call!r}. "
"The parameter is edge_type; type= is silently absorbed into "
"**properties and pollutes edge metadata."
)
def test_broken_form_still_raises(self):
"""Pin the signature contract the example has to respect."""
graph = ContextGraph(advanced_analytics=False)
with pytest.raises(TypeError, match="node_type"):
graph.add_node("Python", type="language", properties={"popularity": "high"})
tests/export/test_neo4j_csv_exporter.py
+57
View File
@@ -301,3 +301,60 @@ def test_nested_properties_are_json_serialized(tmp_path):
by_id = {row[0]: row for row in rows[1:]}
assert by_id["node1"][2] == '{"k":"v"}'
assert by_id["node1"][3] == "[1,2,3]"
def test_unrecognized_mapping_is_refused_rather_than_exported_empty(tmp_path):
"""The Neo4j path reads mappings on the shared normalizer's default terms.
An ``export_json`` envelope names no graph key, so it resolves to nothing.
Written out, that is a pair of header-only CSVs indistinguishable from a
genuinely empty graph -- the silent-empty export the shared contract
exists to prevent.
"""
exporter = Neo4jCSVExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export({"data": [{"id": "e1"}]}, tmp_path)
message = str(excinfo.value)
assert "data" in message
assert "entities" in message
assert not (tmp_path / "nodes.csv").exists()
assert not (tmp_path / "relationships.csv").exists()
def test_records_under_an_unread_key_are_not_dropped_silently(tmp_path):
"""Naming a recognized key is not enough if nothing resolves from it."""
exporter = Neo4jCSVExporter()
with pytest.raises(ValidationError):
exporter.export({"nodes": [], "data": [{"id": "e1"}]}, tmp_path)
assert not (tmp_path / "nodes.csv").exists()
def test_malformed_collection_value_is_refused(tmp_path):
"""``list("abc")`` would otherwise export one node per character."""
exporter = Neo4jCSVExporter()
for value in ("abc", 42, {"id": "n1"}):
with pytest.raises(ValidationError) as excinfo:
exporter.export({"nodes": value}, tmp_path)
assert "nodes" in str(excinfo.value)
assert not (tmp_path / "nodes.csv").exists()
def test_graph_objects_still_use_the_attribute_path(tmp_path):
"""Only mappings changed; objects are not mappings and are unaffected."""
class Graph:
def __init__(self):
self.nodes = [{"id": "e1", "type": "Person", "name": "Acme"}]
self.edges = []
exporter = Neo4jCSVExporter()
exporter.export(Graph(), tmp_path)
assert "Acme" in (tmp_path / "nodes.csv").read_text(encoding="utf-8")
tests/export/test_yaml_exporter_input_validation.py
+181
View File
@@ -0,0 +1,181 @@
"""Regression tests for YAML export input validation (issue #952).
``export_yaml`` declared ``Union[Dict[str, Any], List[Dict[str, Any]]]`` but
both YAML exporters read their payload by key, so a list reached
``semantic_network.get(...)`` and surfaced as a bare
``AttributeError: 'list' object has no attribute 'get'`` from inside the
exporter — an error that names neither the offending argument nor the shape
expected.
A list is rejected rather than wrapped. These formats distinguish entities
from relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records; and wrapping it under an
unrecognised key would write a structurally valid file with every collection
empty, trading a loud failure for silent data loss.
Both directions are pinned: non-mappings raise ``ProcessingError`` with an
actionable message, and every mapping that worked before still exports.
"""
import os
import shutil
import tempfile
import unittest
from collections import OrderedDict, defaultdict
import yaml
from semantica.export.methods import export_yaml
from semantica.export.yaml_exporter import (
SemanticNetworkYAMLExporter,
YAMLSchemaExporter,
)
from semantica.utils.exceptions import ProcessingError
# Non-mapping payloads that must be rejected. A list of dicts is the shape
# from #952; the rest guard the same path against other sequence/scalar types.
NON_MAPPINGS = {
"list_of_dicts": [{"id": "1", "name": "Acme"}],
"empty_list": [],
"tuple_of_dicts": ({"id": "1"},),
"list_of_scalars": ["a", "b"],
"string": "entities",
"bytes": b"entities",
"int": 42,
"none": None,
"set": {"a"},
}
# Both YAML methods, with a minimal valid payload and the key names the
# corresponding error message must mention.
METHODS = {
"semantic_network": {
"valid": {
"entities": [{"id": "1", "name": "Acme"}],
"relationships": [],
"triplets": [],
},
"expected_key": "entities",
"top_level_key": "entities",
},
"schema": {
"valid": {"classes": [{"name": "Thing"}], "properties": []},
"expected_key": "classes",
"top_level_key": "classes",
},
}
class TestExportYamlRejectsNonMappings(unittest.TestCase):
"""Non-mapping input fails loudly, through the public wrapper."""
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True)
def _path(self, name="out.yaml"):
return os.path.join(self.tmpdir, name)
def test_fixture_tables_are_populated(self):
"""Guard against a vacuous suite.
Every test below iterates a table; emptying or renaming one would let
those loops pass without asserting anything.
"""
self.assertGreaterEqual(len(NON_MAPPINGS), 9)
self.assertEqual(set(METHODS), {"semantic_network", "schema"})
def test_non_mapping_raises_processing_error(self):
for method in METHODS:
for label, payload in NON_MAPPINGS.items():
with self.subTest(method=method, case=label):
with self.assertRaises(ProcessingError):
export_yaml(payload, self._path(), method=method)
def test_error_names_the_offending_type_and_expected_keys(self):
"""The message must be actionable, not just the right exception type."""
for method, spec in METHODS.items():
with self.subTest(method=method):
with self.assertRaises(ProcessingError) as ctx:
export_yaml([{"id": "1"}], self._path(), method=method)
message = str(ctx.exception)
self.assertIn("list", message)
self.assertIn(spec["expected_key"], message)
def test_no_file_is_written_when_input_is_rejected(self):
"""A rejected export must not leave a partial or empty artefact."""
for method in METHODS:
with self.subTest(method=method):
path = self._path(f"{method}_rejected.yaml")
with self.assertRaises(ProcessingError):
export_yaml([{"id": "1"}], path, method=method)
self.assertFalse(os.path.exists(path))
def test_exporter_classes_reject_non_mappings_directly(self):
"""Validation lives in the exporters, not only the convenience wrapper.
Callers using the classes directly get the same contract.
"""
for label, payload in NON_MAPPINGS.items():
with self.subTest(exporter="SemanticNetworkYAMLExporter", case=label):
with self.assertRaises(ProcessingError):
SemanticNetworkYAMLExporter().export_semantic_network(payload)
with self.subTest(exporter="YAMLSchemaExporter", case=label):
with self.assertRaises(ProcessingError):
YAMLSchemaExporter().export_ontology_schema(payload)
class TestExportYamlStillAcceptsMappings(unittest.TestCase):
"""Everything that exported before must still export."""
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True)
def _path(self, name="out.yaml"):
return os.path.join(self.tmpdir, name)
def _load(self, path):
with open(path, encoding="utf-8") as handle:
return yaml.safe_load(handle)
def test_valid_mapping_exports_for_each_method(self):
for method, spec in METHODS.items():
with self.subTest(method=method):
path = self._path(f"{method}.yaml")
export_yaml(spec["valid"], path, method=method)
self.assertTrue(os.path.exists(path))
loaded = self._load(path)
self.assertIn(spec["top_level_key"], loaded)
def test_semantic_network_records_survive_the_round_trip(self):
path = self._path("network.yaml")
export_yaml(METHODS["semantic_network"]["valid"], path)
loaded = self._load(path)
self.assertEqual(loaded["entities"], [{"id": "1", "name": "Acme"}])
def test_empty_mapping_is_still_accepted(self):
"""An empty dict is a mapping; rejecting it would be a behaviour change."""
for method in METHODS:
with self.subTest(method=method):
path = self._path(f"{method}_empty.yaml")
export_yaml({}, path, method=method)
self.assertTrue(os.path.exists(path))
def test_mapping_subclasses_are_accepted(self):
"""Validation is by Mapping, not dict, so these must keep working."""
valid = METHODS["semantic_network"]["valid"]
subclasses = {
"OrderedDict": OrderedDict(valid),
"defaultdict": defaultdict(list, valid),
}
for label, payload in subclasses.items():
with self.subTest(case=label):
path = self._path(f"{label}.yaml")
export_yaml(payload, path)
loaded = self._load(path)
self.assertEqual(loaded["entities"], valid["entities"])
if __name__ == "__main__":
unittest.main()
tests/export/test_yaml_exporter_key_recognition.py
+434
View File
@@ -0,0 +1,434 @@
"""Tests for YAML export key recognition (issue #953).
``SemanticNetworkYAMLExporter`` built its output from ``.get(key, [])``
lookups, so a mapping keyed by anything it did not read -- an ``export_json``
envelope, a typo'd 'entitys', ``ContextGraph.to_dict()``'s 'nodes'/'edges' --
serialized to a structurally valid file with every collection empty. Nothing
signalled the loss: no exception, no warning, and the progress log reported a
completed export. ``YAMLSchemaExporter`` had the same defect over a different
key set.
The exporters are run for real rather than mocked, and the written files are
parsed back, since the behaviour under test is what actually lands on disk.
"""
from pathlib import Path
import pytest
import yaml
from semantica.context.context_graph import ContextGraph
from semantica.export.methods import export_json, export_yaml
from semantica.export.yaml_exporter import (
SemanticNetworkYAMLExporter,
YAMLSchemaExporter,
)
from semantica.utils.exceptions import ProcessingError, ValidationError
ENTITIES = [{"id": "e1", "name": "Acme"}, {"id": "e2", "name": "Beta"}]
RELATIONSHIPS = [{"id": "r1", "source": "e1", "target": "e2", "type": "PARTNER"}]
TRIPLETS = [{"subject": "e1", "predicate": "partner_of", "object": "e2"}]
def _load(path):
with open(path, "r", encoding="utf-8") as handle:
return yaml.safe_load(handle)
class TestSemanticNetworkKeyRecognition:
"""An unrecognized mapping is refused instead of silently emptied."""
def test_export_json_envelope_is_rejected(self, tmp_path):
"""The realistic trigger: re-exporting an export_json payload.
``export_json`` wraps records as ``{"data": [...], "count": N,
"metadata": {...}}``. Feeding that straight to ``export_yaml`` used to
write a file with every record gone. Note the envelope's 'metadata'
key is deliberately not enough to make the payload recognized --
treating it as sufficient would readmit exactly this case.
"""
json_path = tmp_path / "records.json"
export_json(ENTITIES, json_path)
envelope = yaml.safe_load(json_path.read_text(encoding="utf-8"))
assert "data" in envelope and "metadata" in envelope
yaml_path = tmp_path / "records.yaml"
with pytest.raises(ValidationError) as excinfo:
export_yaml(envelope, yaml_path)
message = str(excinfo.value)
assert "'data'" in message, "error should name the supplied keys"
assert "'entities'" in message, "error should name the expected keys"
assert not yaml_path.exists(), "a rejected export must write nothing"
@pytest.mark.parametrize(
"payload",
[
{"records": ENTITIES},
{"entitys": ENTITIES},
{"data": ENTITIES},
{"metadata": {"source": "test"}},
],
ids=["records", "typo", "data", "metadata-only"],
)
def test_unrecognized_mappings_are_rejected(self, payload):
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export_semantic_network(payload)
@pytest.mark.parametrize(
"payload",
[
{"entities": [], "data": ENTITIES},
{"nodes": [], "edges": [], "records": ENTITIES},
{"triplets": [], "data": ENTITIES, "metadata": {"source": "test"}},
],
ids=["entities-empty", "nodes-edges-empty", "triplets-empty"],
)
def test_recognized_but_empty_with_records_elsewhere_is_rejected(self, payload):
"""Presence of a recognized key is not proof the records survived.
``{"entities": [], "data": [...]}`` clears a presence-only check and
still resolves to empty, dropping everything under 'data' -- the same
silent-empty export by a narrower route.
"""
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export_semantic_network(payload)
message = str(excinfo.value)
assert "holds records" in message
assert "'entities'" in message, "error should name where records belong"
def test_empty_graph_with_non_record_keys_still_exports(self, tmp_path):
"""The rejection must key on dropped *records*, not on unread keys.
``ContextGraph.to_dict()`` always carries a populated 'statistics'
dict, so an empty graph would be refused if any unread key counted.
"""
graph = ContextGraph()
path = tmp_path / "empty_graph.yaml"
export_yaml(graph.to_dict(), path)
written = _load(path)
assert written["entities"] == []
assert written["relationships"] == []
def test_empty_mapping_still_exports(self, tmp_path):
"""An empty graph is legitimate and carries nothing that could be lost."""
path = tmp_path / "empty.yaml"
export_yaml({}, path)
written = _load(path)
assert written["entities"] == []
assert written["relationships"] == []
assert written["triplets"] == []
def test_recognized_keys_still_export(self, tmp_path):
path = tmp_path / "network.yaml"
export_yaml(
{
"entities": ENTITIES,
"relationships": RELATIONSHIPS,
"triplets": TRIPLETS,
"metadata": {"source": "test"},
},
path,
)
written = _load(path)
assert written["entities"] == ENTITIES
assert written["relationships"] == RELATIONSHIPS
assert written["triplets"] == TRIPLETS
assert written["metadata"]["source"] == "test"
def test_nodes_edges_alias_exports_records(self, tmp_path):
path = tmp_path / "aliased.yaml"
export_yaml({"nodes": ENTITIES, "edges": RELATIONSHIPS}, path)
written = _load(path)
assert written["entities"] == ENTITIES
assert written["relationships"] == RELATIONSHIPS
def test_context_graph_to_dict_round_trips(self, tmp_path):
"""The most direct path from this library's own graph type to YAML.
Built from a real ``ContextGraph`` rather than a hand-written
'nodes'/'edges' dict, so the test breaks if ``to_dict()`` changes
vocabulary.
"""
graph = ContextGraph()
graph.add_node("n1", node_type="Person", content="Alice")
graph.add_node("n2", node_type="Org", content="Acme")
graph.add_edge("n1", "n2", "WORKS_FOR")
path = tmp_path / "context.yaml"
export_yaml(graph.to_dict(), path)
written = _load(path)
assert len(written["entities"]) == 2
assert len(written["relationships"]) == 1
def test_conflicting_spellings_are_refused(self):
"""Two populated spellings of one collection: no basis to pick either."""
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export_semantic_network(
{"entities": ENTITIES, "nodes": [{"id": "other"}]}
)
def test_non_mapping_raises_processing_error(self):
"""A wrong type is a different failure from a wrong-keyed mapping.
ProcessingError says the object cannot be exported at all;
ValidationError says the mapping's contents are unusable. Pinned here
so the two do not quietly converge.
"""
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ProcessingError):
exporter.export_semantic_network(ENTITIES)
def test_rejected_export_creates_no_output_directory(self, tmp_path):
"""Validation runs before the output directory is created."""
target = tmp_path / "nested" / "out.yaml"
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export({"data": ENTITIES}, target)
assert not target.parent.exists()
class TestPipelineExportKeyRecognition:
"""export_for_pipeline read the same defaulted lookups, so it had the bug too."""
def test_unrecognized_mapping_is_rejected(self):
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export_for_pipeline({"data": ENTITIES})
def test_non_mapping_raises_processing_error(self):
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ProcessingError):
exporter.export_for_pipeline(ENTITIES)
def test_aliases_resolve_into_the_semantic_network(self):
exporter = SemanticNetworkYAMLExporter()
written = yaml.safe_load(
exporter.export_for_pipeline({"nodes": ENTITIES, "edges": RELATIONSHIPS})
)
assert written["semantic_network"]["entities"] == ENTITIES
assert written["semantic_network"]["relationships"] == RELATIONSHIPS
def test_metadata_is_preserved(self):
exporter = SemanticNetworkYAMLExporter()
written = yaml.safe_load(
exporter.export_for_pipeline(
{"entities": ENTITIES, "metadata": {"source": "test"}}
)
)
assert written["metadata"]["source"] == "test"
assert written["semantic_network"]["entities"] == ENTITIES
class TestSchemaKeyRecognition:
"""method="schema" emitted empty classes/properties/namespaces the same way."""
def test_unrecognized_mapping_is_rejected(self, tmp_path):
path = tmp_path / "schema.yaml"
with pytest.raises(ValidationError) as excinfo:
export_yaml({"nodes": [{"id": "1"}]}, path, method="schema")
message = str(excinfo.value)
assert "'nodes'" in message
assert "'classes'" in message
assert not path.exists()
def test_non_mapping_raises_processing_error(self):
exporter = YAMLSchemaExporter()
with pytest.raises(ProcessingError):
exporter.export_ontology_schema([{"id": "1"}])
def test_recognized_but_empty_with_records_elsewhere_is_rejected(self):
"""The schema path had the same presence-only hole."""
exporter = YAMLSchemaExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export_ontology_schema({"classes": [], "nodes": [{"id": "1"}]})
assert "holds records" in str(excinfo.value)
def test_ontology_metadata_without_records_still_exports(self):
"""A schema described only by its identity is not a dropped export."""
exporter = YAMLSchemaExporter()
written = yaml.safe_load(
exporter.export_ontology_schema(
{"uri": "http://example.org/o", "classes": []}
)
)
assert written["ontology"]["uri"] == "http://example.org/o"
assert written["classes"] == []
def test_empty_mapping_still_exports(self, tmp_path):
path = tmp_path / "schema.yaml"
export_yaml({}, path, method="schema")
written = _load(path)
assert written["classes"] == []
assert written["properties"] == []
assert written["namespaces"] == {}
@pytest.mark.parametrize(
"payload",
[
{"classes": ["Person"], "properties": ["WORKS_FOR"]},
{"namespaces": {"ex": "http://example.org/"}},
{"uri": "http://example.org/ontology"},
],
ids=["classes-properties", "namespaces-only", "uri-only"],
)
def test_recognized_keys_still_export(self, payload, tmp_path):
path = tmp_path / "schema.yaml"
export_yaml(payload, path, method="schema")
written = _load(path)
assert written["classes"] == payload.get("classes", [])
assert written["properties"] == payload.get("properties", [])
assert written["ontology"]["uri"] == payload.get("uri", "")
# ── Fix regression: scalar recognized keys must not short-circuit the ──
# ── dropped-records check (version, uri, title, description). ──────────
@pytest.mark.parametrize(
"scalar_key, scalar_value",
[
("version", "1.0"),
("uri", "http://example.org/ontology"),
("title", "My Ontology"),
("description", "A test ontology"),
],
ids=["version", "uri", "title", "description"],
)
def test_scalar_recognized_key_does_not_excuse_records_under_unread_key(
self, scalar_key, scalar_value
):
"""A truthy scalar such as version='1.0' must not silence the dropped-
records check. Before the fix, any truthy value from _SCHEMA_KEYS
would make _require_nothing_dropped believe something resolved and
return early, silently discarding a list under an unread key.
"""
exporter = YAMLSchemaExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export_ontology_schema(
{scalar_key: scalar_value, "nodes": [{"id": "c1"}]}
)
assert "holds records" in str(excinfo.value), str(excinfo.value)
def test_valid_classes_with_scalar_metadata_is_accepted(self):
"""classes/properties populated alongside version/uri must still work."""
exporter = YAMLSchemaExporter()
written = yaml.safe_load(
exporter.export_ontology_schema(
{
"classes": [{"id": "Person"}],
"properties": [{"id": "name"}],
"version": "2.0",
"uri": "http://example.org/o",
}
)
)
assert written["classes"] == [{"id": "Person"}]
assert written["properties"] == [{"id": "name"}]
assert written["ontology"]["version"] == "2.0"
assert written["ontology"]["uri"] == "http://example.org/o"
class TestFailureIsObservable:
"""The complaint in #953 was that the logs affirmatively reported success."""
def test_no_success_is_logged_for_a_rejected_export(self, tmp_path, caplog):
path = tmp_path / "out.yaml"
with caplog.at_level("DEBUG"):
with pytest.raises(ValidationError):
export_yaml({"data": ENTITIES}, path)
assert "Exported YAML to" not in caplog.text
assert any(
record.levelname in ("WARNING", "ERROR", "CRITICAL")
for record in caplog.records
), "a rejected export should leave something at warning or above"
class _RecordingTracker:
"""Records the exporter's own progress calls, which are what is under test."""
def __init__(self):
self.stopped = []
self._next_id = 0
def start_tracking(self, **kwargs):
self._next_id += 1
return str(self._next_id)
def update_tracking(self, tracking_id, **kwargs):
pass
def stop_tracking(self, tracking_id, status=None, message=None):
self.stopped.append((status, message))
class TestProgressReflectsTheWrite:
"""Serialization completing is not the same as the file landing on disk."""
def test_failed_write_is_not_reported_as_completed(self, tmp_path):
"""A write failure after serialization must not leave a clean tracker.
The path's parent is an existing *file*, so directory creation fails
after `export_semantic_network` has already reported its own
completion.
"""
blocker = tmp_path / "blocker"
blocker.write_text("not a directory", encoding="utf-8")
target = blocker / "nested" / "out.yaml"
exporter = SemanticNetworkYAMLExporter()
tracker = _RecordingTracker()
exporter.progress_tracker = tracker
with pytest.raises(OSError):
exporter.export({"entities": ENTITIES}, target)
assert not target.exists()
statuses = [status for status, _ in tracker.stopped]
assert "failed" in statuses, f"write failure went unreported: {tracker.stopped}"
assert not any(
status == "completed" and "Exported YAML" in (message or "")
for status, message in tracker.stopped
), "no span may claim a completed export when nothing was written"
def test_successful_write_is_reported_as_completed(self, tmp_path):
target = tmp_path / "out.yaml"
exporter = SemanticNetworkYAMLExporter()
tracker = _RecordingTracker()
exporter.progress_tracker = tracker
exporter.export({"entities": ENTITIES}, target)
assert target.exists()
assert all(status == "completed" for status, _ in tracker.stopped)
assert any(
"Exported YAML" in (message or "") for _, message in tracker.stopped
), "the write should report its own completion, not just serialization"
class TestUnaffectedExporters:
"""export_json's own behaviour is untouched -- only the YAML path changed."""
def test_export_json_still_accepts_a_bare_list(self, tmp_path):
path = tmp_path / "records.json"
export_json(ENTITIES, path)
assert Path(path).exists()
tests/ingest/conftest.py
+35
View File
@@ -0,0 +1,35 @@
"""
Shared pytest fixtures for the ingest test suite.
The ``mock_dns`` fixture is applied to *every* test in this directory
(``autouse=True``). It stubs out ``socket.getaddrinfo`` inside the SSRF
guard module so that unit tests that mock ``requests.Session.request`` do not
accidentally hit the network for DNS resolution — which would fail in offline
CI environments and cause intermittent timeouts.
Tests that explicitly need to exercise DNS-related behaviour (e.g. checking
that a hostname resolving to a private IP is blocked) override this fixture
by patching ``semantica.ingest.ssrf.socket.getaddrinfo`` with their own
``side_effect`` *inside* the test body; that inner patch wins because
``unittest.mock.patch`` applies patches in innermost-last order.
"""
from __future__ import annotations
import socket
from unittest.mock import patch
import pytest
_PUBLIC_IP = "93.184.216.34" # example.com — a safe, routable public address
@pytest.fixture(autouse=True)
def mock_dns():
"""Map every hostname to a safe public IP for the duration of each test."""
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[
(socket.AF_INET, socket.SOCK_STREAM, 6, "", (_PUBLIC_IP, 0))
],
):
yield
tests/ingest/test_auth_header_redirect_security.py
+1063
View File
File diff suppressed because it is too large Load Diff
tests/ingest/test_cookbook_integration.py
+20 -20
View File
@@ -10,19 +10,20 @@ class TestCookbookIntegration:
@pytest.fixture
def mock_mcp_server(self):
# We need to patch both httpx and requests because MCPClient tries httpx first
with patch("httpx.post") as mock_httpx_post, \
patch("requests.post") as mock_requests_post:
def side_effect(url, json=None, **kwargs):
# MCPClient._send_request_http now routes through request_with_ssrf_guard,
# which calls requests.request (not httpx.post / requests.post directly).
# Patch at the point where the guard issues the actual HTTP call.
with patch("semantica.ingest.ssrf.requests.request") as mock_request:
def side_effect(method, url, json=None, **kwargs):
if not json:
return MagicMock()
method = json.get("method")
rpc_method = json.get("method")
response_mock = MagicMock()
response_mock.status_code = 200
if method == "initialize":
if rpc_method == "initialize":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -32,7 +33,7 @@ class TestCookbookIntegration:
"serverInfo": {"name": "test_server", "version": "1.0"}
}
}
elif method == "resources/list":
elif rpc_method == "resources/list":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -44,7 +45,7 @@ class TestCookbookIntegration:
]
}
}
elif method == "tools/list":
elif rpc_method == "tools/list":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -56,7 +57,7 @@ class TestCookbookIntegration:
]
}
}
elif method == "resources/read":
elif rpc_method == "resources/read":
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -66,13 +67,13 @@ class TestCookbookIntegration:
]
}
}
elif method == "tools/call":
elif rpc_method == "tools/call":
tool_name = json.get("params", {}).get("name")
content = [{"type": "text", "text": "Tool Output"}]
if tool_name == "query_inventory":
content = [{"type": "text", "text": '{"warehouse_id": "WH001", "level": 100}'}]
response_mock.json.return_value = {
"jsonrpc": "2.0",
"id": json.get("id"),
@@ -86,12 +87,11 @@ class TestCookbookIntegration:
"id": json.get("id"),
"result": {}
}
return response_mock
mock_httpx_post.side_effect = side_effect
mock_requests_post.side_effect = side_effect
yield mock_httpx_post
mock_request.side_effect = side_effect
yield mock_request
def test_financial_data_integration(self, mock_mcp_server):
"""
tests/ingest/test_public_api_ingestor.py
+78 -7
View File
@@ -198,15 +198,82 @@ def test_public_api_detection_reports_auth_required() -> None:
headers={"WWW-Authenticate": "Bearer"},
)
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://api.example.com/private"
)
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://api.example.com/private"
)
assert detection.is_public is False
assert detection.requires_auth is True
assert detection.response_status == 401
def test_detect_public_api_propagates_ssrf_validation_error() -> None:
"""detect_public_api() must surface ValidationError, not swallow it.
request_with_ssrf_guard() raises ValidationError (not
requests.exceptions.RequestException) for SSRF-blocked hosts, so
detect_public_api()'s error handling must catch it explicitly like its
sibling ingest_public_api() already does.
"""
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
mock_session.headers = {}
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("127.0.0.1", 0))],
):
with pytest.raises(ValidationError):
PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://blocked.example.com/data"
)
def test_detect_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None:
"""Passing session/allow_private_ips through **options must not crash.
Both are always supplied explicitly to request_with_ssrf_guard(); caller
copies must be dropped from **options rather than causing a
'got multiple values for keyword argument' TypeError.
"""
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
mock_session.headers = {}
mock_session.request.return_value = _mock_response(
headers={"Content-Type": "application/json"}
)
detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api(
"https://jsonplaceholder.typicode.com/posts",
allow_private_ips=True,
session=object(),
)
assert detection.is_public is True
def test_ingest_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None:
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
mock_session.headers = {}
mock_session.request.return_value = _mock_response(
json_payload=[{"id": 1}],
headers={"Content-Type": "application/json"},
)
result = PublicAPIIngestor(rate_limit_delay=0).ingest_public_api(
"https://jsonplaceholder.typicode.com/posts",
allow_private_ips=True,
session=object(),
)
assert result.response_status == 200
def test_public_api_ingestor_rejects_authentication_inputs() -> None:
with patch("requests.Session") as mock_session_class:
mock_session = mock_session_class.return_value
@@ -238,10 +305,14 @@ def test_public_api_ingestor_parses_string_boolean_config() -> None:
config={"validate_no_auth": "false"},
rate_limit_delay=0,
)
result = ingestor.ingest_public_api(
"https://api.example.com/data",
headers={"Authorization": "Bearer token"},
)
with patch(
"semantica.ingest.ssrf.socket.getaddrinfo",
return_value=[(None, None, None, None, ("93.184.216.34", 0))],
):
result = ingestor.ingest_public_api(
"https://api.example.com/data",
headers={"Authorization": "Bearer token"},
)
assert ingestor.validate_no_auth is False
assert result.data == payload
tests/ingest/test_submodules.py
+54 -81
View File
@@ -195,89 +195,62 @@ class TestMCPIngestor:
class TestMCPClient:
def test_call_tool(self):
# Patch requests.post globally if requests is used, or httpx.post if httpx is used.
# The code tries importing httpx, then requests.
# We should patch both or ensure we catch the right one.
# Simpler to patch sys.modules to simulate httpx missing, then patch requests.
with patch.dict(sys.modules, {'httpx': None}):
with patch("requests.post") as mock_post:
mock_response = MagicMock()
mock_response.status_code = 200
# Sequence of calls:
# 1. connect() calls _connect_http() -> calls _initialize() -> calls _send_request()
# _send_request() calls requests.post with method="initialize"
# 2. call_tool() calls _send_request() with method="tools/call"
# Response for initialize
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1
}
# Response for tool call
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2
}
mock_response.json.side_effect = [init_response, tool_response]
mock_post.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
result = client.call_tool("my_tool", {"arg": "val"})
# result is the dict returned by tool call?
# call_tool returns dict?
# Check MCPClient.call_tool implementation
# It calls _send_request, which returns response.json().
# But wait, call_tool might process the result.
# Let's check call_tool implementation in mcp_client.py (not read yet, but assumed).
# Wait, I read mcp_client.py but didn't check call_tool specifically.
# Assuming call_tool returns result part or whole response.
# Actually, let's verify call_tool in mcp_client.py
pass
# MCPClient._send_request_http now routes through request_with_ssrf_guard,
# which calls requests.request (not requests.post) with allow_redirects=False.
# Patch the requests.request call inside ssrf.py.
with patch("semantica.ingest.ssrf.requests.request") as mock_request:
mock_response = MagicMock()
mock_response.status_code = 200
# Response for initialize
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1,
}
# Response for tool call
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2,
}
mock_response.json.side_effect = [init_response, tool_response]
mock_request.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
client.call_tool("my_tool", {"arg": "val"})
def test_call_tool_mock_check(self):
# Redoing the test with more specific mocking logic
with patch.dict(sys.modules, {'httpx': None}):
with patch("requests.post") as mock_post:
mock_response = MagicMock()
mock_response.status_code = 200
# initialize response
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1
}
# tool call response - Assuming call_tool returns the 'result' part of JSON-RPC response
# If call_tool implementation wraps it, we need to know.
# Let's assume standard behavior for now.
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2
}
mock_response.json.side_effect = [init_response, tool_response]
mock_post.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
result = client.call_tool("my_tool", {"arg": "val"})
# Verify result.
# If call_tool returns the 'result' dict from JSON-RPC:
assert result["content"] == [{"type": "text", "text": "Tool Result"}]
# Redo with the corrected patch target.
with patch("semantica.ingest.ssrf.requests.request") as mock_request:
mock_response = MagicMock()
mock_response.status_code = 200
init_response = {
"jsonrpc": "2.0",
"result": {"serverInfo": {"name": "test", "version": "1.0"}},
"id": 1,
}
tool_response = {
"jsonrpc": "2.0",
"result": {"content": [{"type": "text", "text": "Tool Result"}]},
"id": 2,
}
mock_response.json.side_effect = [init_response, tool_response]
mock_request.return_value = mock_response
client = MCPClient(url="http://localhost:8000")
client.connect()
result = client.call_tool("my_tool", {"arg": "val"})
assert result["content"] == [{"type": "text", "text": "Tool Result"}]
class TestGDriveIngestor:
def test_init_raises_if_no_google_libs(self):
tests/integrations/crewai/conftest.py
+151
View File
@@ -0,0 +1,151 @@
"""
Shared pytest configuration for CrewAI integration tests.
Installs comprehensive crewai stubs into sys.modules before any test in this
directory runs, so every test file can import the integration modules with
``CREWAI_AVAILABLE == True`` and exercise the real subclassing code paths
without a real crewai installation.
The stubs mirror the current CrewAI contracts:
- ``crewai.tools.BaseTool`` — Pydantic ``BaseModel`` (arbitrary types allowed)
- ``crewai.knowledge.source.base_knowledge_source.BaseKnowledgeSource`` —
Pydantic model with ``validate_content``/``add``/``aadd`` abstract methods
and ``_chunk_text``/``_save_documents`` helpers.
The graceful-degradation path (crewai genuinely absent) is covered separately
in ``test_degradation.py`` via a subprocess, so this stub never has to be torn
down mid-session.
"""
from __future__ import annotations
import sys
import types
from typing import Any, Optional
from pydantic import BaseModel, ConfigDict, Field, field_serializer, field_validator
def _install_crewai_stubs() -> None:
"""Install a full set of crewai stubs into sys.modules."""
# -----------------------------------------------------------------------
# crewai.tools — BaseTool
# -----------------------------------------------------------------------
class BaseTool(BaseModel): # noqa: D101
"""Stub mirroring crewai.tools.base_tool.BaseTool."""
model_config = ConfigDict(arbitrary_types_allowed=True)
name: str = "base_tool"
description: str = ""
args_schema: Any = None
result_as_answer: bool = False
@field_serializer("args_schema", when_used="json")
def _ser_args_schema(self, schema): # noqa: D102
if schema is None:
return None
return {"__schema__": f"{schema.__module__}.{schema.__qualname__}"}
@field_validator("args_schema", mode="before")
@classmethod
def _restore_args_schema(cls, v): # noqa: D102
if isinstance(v, dict) and "__schema__" in v:
import importlib
mod_name, cls_name = v["__schema__"].rsplit(".", 1)
return getattr(importlib.import_module(mod_name), cls_name)
return v
def run(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
return self._run(*args, **kwargs)
async def arun(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
return await self._arun(*args, **kwargs)
def _run(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
raise NotImplementedError
async def _arun(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
raise NotImplementedError
tools_mod = types.ModuleType("crewai.tools")
tools_mod.BaseTool = BaseTool # type: ignore[attr-defined]
tools_base_mod = types.ModuleType("crewai.tools.base_tool")
tools_base_mod.BaseTool = BaseTool # type: ignore[attr-defined]
# -----------------------------------------------------------------------
# crewai.knowledge.source.base_knowledge_source — BaseKnowledgeSource
# -----------------------------------------------------------------------
class BaseKnowledgeSource(BaseModel): # noqa: D101
"""Stub mirroring crewai.knowledge.source.base_knowledge_source."""
model_config = ConfigDict(arbitrary_types_allowed=True)
chunk_size: int = 4000
chunk_overlap: int = 200
chunks: list = Field(default_factory=list)
chunk_embeddings: list = Field(default_factory=list, exclude=True)
storage: Any = None
metadata: dict = Field(default_factory=dict)
collection_name: Optional[str] = None
def _chunk_text(self, text: str) -> list: # noqa: D102
return [
text[i : i + self.chunk_size]
for i in range(0, len(text), self.chunk_size - self.chunk_overlap)
]
def _save_documents(self) -> None: # noqa: D102
if self.storage is not None:
self.storage.save(self.chunks)
else:
raise ValueError("No storage found to save documents.")
async def _asave_documents(self) -> None: # noqa: D102
if self.storage is not None:
await self.storage.asave(self.chunks)
else:
raise ValueError("No storage found to save documents.")
def validate_content(self) -> Any: # noqa: D102
raise NotImplementedError
def add(self) -> None: # noqa: D102
raise NotImplementedError
async def aadd(self) -> None: # noqa: D102
raise NotImplementedError
knowledge_pkg = types.ModuleType("crewai.knowledge")
source_pkg = types.ModuleType("crewai.knowledge.source")
source_base_mod = types.ModuleType("crewai.knowledge.source.base_knowledge_source")
source_base_mod.BaseKnowledgeSource = ( # type: ignore[attr-defined]
BaseKnowledgeSource
)
source_pkg.BaseKnowledgeSource = BaseKnowledgeSource # type: ignore[attr-defined]
knowledge_pkg.source = source_pkg
# -----------------------------------------------------------------------
# Register everything
# -----------------------------------------------------------------------
crewai = types.ModuleType("crewai")
crewai.tools = tools_mod # type: ignore[attr-defined]
crewai.knowledge = knowledge_pkg # type: ignore[attr-defined]
_mods = {
"crewai": crewai,
"crewai.tools": tools_mod,
"crewai.tools.base_tool": tools_base_mod,
"crewai.knowledge": knowledge_pkg,
"crewai.knowledge.source": source_pkg,
"crewai.knowledge.source.base_knowledge_source": source_base_mod,
}
for name, mod in _mods.items():
sys.modules[name] = mod
# Install once at import time (conftest is imported before any test file)
_install_crewai_stubs()
tests/integrations/crewai/test_decision_tool.py
+562
View File
@@ -0,0 +1,562 @@
"""
Tests for SemanticaDecisionTool — decision intelligence CrewAI tool.
Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is
``True`` and the real Pydantic/BaseTool subclassing path is exercised. A
MagicMock ``AgentContext`` is used so no vector store / faiss is required.
"""
from __future__ import annotations
import json
import unittest
from unittest.mock import MagicMock
from integrations.crewai import SemanticaDecisionTool
from integrations.crewai.decision_tool import (
CREWAI_AVAILABLE,
SemanticaDecisionToolInput,
)
def _make_context() -> MagicMock:
ctx = MagicMock()
ctx.record_decision.return_value = "dec-test-001"
ctx.find_precedents_advanced.return_value = [
{
"scenario": "past loan",
"outcome": "approved",
"confidence": 0.9,
"category": "loan",
}
]
ctx.analyze_decision_influence.return_value = {"centrality": 0.75, "influenced": 3}
ctx.knowledge_graph = MagicMock()
ctx.knowledge_graph.trace_decision_causality = MagicMock(
return_value=["step1", "step2"]
)
return ctx
class TestSemanticaDecisionToolInit(unittest.TestCase):
def test_crewai_available_via_stub(self):
self.assertTrue(CREWAI_AVAILABLE)
def test_is_base_tool_subclass(self):
from crewai.tools import BaseTool
self.assertTrue(issubclass(SemanticaDecisionTool, BaseTool))
def test_creates_with_explicit_context(self):
ctx = _make_context()
tool = SemanticaDecisionTool(context=ctx)
self.assertIs(tool.context, ctx)
def test_creates_context_when_none(self):
tool = SemanticaDecisionTool()
self.assertIsNotNone(tool.context)
def test_default_metadata(self):
tool = SemanticaDecisionTool(context=_make_context())
self.assertEqual(tool.name, "semantica_decision")
self.assertTrue(tool.description)
self.assertEqual(tool.args_schema, SemanticaDecisionToolInput)
def test_input_schema_validates(self):
inp = SemanticaDecisionToolInput(action="record_decision", confidence=0.5)
self.assertEqual(inp.confidence, 0.5)
with self.assertRaises(Exception):
SemanticaDecisionToolInput(action="bogus")
def test_max_precedents_and_causal_depth_defaults(self):
tool = SemanticaDecisionTool(context=_make_context())
self.assertEqual(tool.max_precedents, 5)
self.assertEqual(tool.causal_depth, 3)
class TestSemanticaDecisionToolSerialization(unittest.TestCase):
"""CrewAI checkpoints serialise tools via ``model_dump(mode="json")`` — the
live context must not break that (regression for PydanticSerializationError
on arbitrary state objects)."""
def test_model_dump_json_excludes_context(self):
tool = SemanticaDecisionTool(context=_make_context())
dumped = tool.model_dump(mode="json")
self.assertNotIn("context", dumped)
self.assertEqual(dumped["max_precedents"], 5)
self.assertEqual(dumped["causal_depth"], 3)
def test_model_validate_restores_defaults(self):
tool = SemanticaDecisionTool(context=_make_context())
restored = SemanticaDecisionTool.model_validate(tool.model_dump(mode="json"))
self.assertIsNotNone(restored.context)
self.assertEqual(restored.max_precedents, 5)
self.assertEqual(restored.causal_depth, 3)
def test_restore_flags_lost_live_state(self):
"""A tool restored from a checkpoint must signal that its live context
was excluded and an empty one reconstructed (``reconstructed_state``)."""
tool = SemanticaDecisionTool(context=_make_context())
dumped = tool.model_dump(mode="json")
self.assertTrue(dumped["had_live_state"])
self.assertNotIn("reconstructed_state", dumped)
restored = SemanticaDecisionTool.model_validate(dumped)
self.assertTrue(restored.reconstructed_state)
self.assertFalse(SemanticaDecisionTool().reconstructed_state)
class TestRecordDecision(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_decision_id(self):
result = json.loads(
self.tool._run(
action="record_decision",
category="loan",
scenario="Customer A loan application",
reasoning="Good credit score 740",
outcome="approved",
confidence=0.95,
)
)
self.assertEqual(result["decision_id"], "dec-test-001")
self.assertEqual(result["status"], "recorded")
def test_delegates_to_context(self):
self.tool._run(
action="record_decision",
category="content",
scenario="Moderation check",
reasoning="No violations",
outcome="allowed",
confidence=0.88,
)
self.ctx.record_decision.assert_called_once()
def test_parses_entities_string(self):
self.tool._run(
action="record_decision",
category="hr",
scenario="Hire decision",
reasoning="Qualified",
outcome="hired",
confidence=0.9,
entities="Alice, ACME Corp, Senior Engineer",
)
call_kwargs = self.ctx.record_decision.call_args[1]
self.assertIsInstance(call_kwargs["entities"], list)
self.assertEqual(len(call_kwargs["entities"]), 3)
def test_returns_error_json_on_failure(self):
self.ctx.record_decision.side_effect = RuntimeError("DB unavailable")
result = json.loads(
self.tool._run(
action="record_decision",
category="x",
scenario="y",
reasoning="z",
outcome="failed",
)
)
self.assertEqual(result["status"], "failed")
self.assertIn("error", result)
def test_default_confidence_used(self):
self.tool._run(
action="record_decision",
category="test",
scenario="Default confidence test",
reasoning="N/A",
outcome="pass",
)
call_kwargs = self.ctx.record_decision.call_args[1]
self.assertEqual(call_kwargs["confidence"], 0.8)
def test_malformed_confidence_returns_error_json(self):
"""A non-numeric confidence must not crash the tool — it is coerced
inside ``_record_decision``'s error handling and reported as JSON."""
for bad in ("high", None, "0.9"):
result = json.loads(
self.tool._run(
action="record_decision",
category="x",
scenario="y",
reasoning="z",
outcome="failed",
confidence=bad,
)
)
if bad == "0.9":
self.assertEqual(result["status"], "recorded")
else:
self.assertEqual(result["status"], "failed")
self.assertIn("error", result)
def test_missing_fields_get_sane_defaults(self):
"""record_decision must not hard-fail when the agent omits optional
fields — category/reasoning/outcome get defaults."""
result = json.loads(self.tool._run(action="record_decision"))
self.assertEqual(result["status"], "recorded")
call_kwargs = self.ctx.record_decision.call_args[1]
self.assertEqual(call_kwargs["category"], "general")
self.assertEqual(call_kwargs["scenario"], "decision recorded")
self.assertEqual(call_kwargs["reasoning"], "agent decision")
self.assertEqual(call_kwargs["outcome"], "recorded")
class TestRealAutoCreatedContext(unittest.TestCase):
"""The no-context path builds a real AgentContext with a knowledge graph so
decision tracking is actually enabled (regression for the live
'Decision tracking is not enabled' failure)."""
def setUp(self):
self.tool = SemanticaDecisionTool()
def test_context_is_real_agent_context(self):
from semantica.context import AgentContext
self.assertIsInstance(self.tool.context, AgentContext)
self.assertIsNotNone(self.tool.context.knowledge_graph)
def test_record_decision_actually_records(self):
result = json.loads(
self.tool.run(
action="record_decision",
scenario="ship v2",
reasoning="user demand",
confidence=0.9,
)
)
self.assertEqual(result["status"], "recorded")
self.assertTrue(result["decision_id"])
def test_find_precedents_runs_against_real_context(self):
result = json.loads(self.tool.run(action="find_precedents", scenario="ship v2"))
self.assertIn("precedents", result)
def test_trace_causal_chain_runs_against_real_context(self):
"""Regression: trace_decision_causality takes ``max_depth``, not
``depth`` — must not raise against a real ContextGraph."""
rec = json.loads(
self.tool.run(
action="record_decision",
scenario="ship v2",
reasoning="user demand",
confidence=0.9,
)
)
trace = json.loads(
self.tool.run(action="trace_causal_chain", decision_id=rec["decision_id"])
)
self.assertIn("causal_chain", trace)
self.assertEqual(trace["decision_id"], rec["decision_id"])
class TestFindPrecedents(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_precedents(self):
result = json.loads(
self.tool._run(action="find_precedents", scenario="new loan application")
)
self.assertIn("precedents", result)
self.assertIsInstance(result["precedents"], list)
def test_count_in_result(self):
result = json.loads(
self.tool._run(action="find_precedents", scenario="test scenario")
)
self.assertEqual(result["count"], len(result["precedents"]))
def test_category_filter_passed(self):
self.tool._run(
action="find_precedents", scenario="scenario", category="finance"
)
call_kwargs = self.ctx.find_precedents_advanced.call_args[1]
self.assertEqual(call_kwargs.get("category"), "finance")
def test_limit_propagated_to_backend(self):
self.tool.max_precedents = 20
self.tool._run(action="find_precedents", scenario="scenario")
call_kwargs = self.ctx.find_precedents_advanced.call_args[1]
self.assertEqual(call_kwargs.get("limit"), 20)
def test_handles_exception_gracefully(self):
self.ctx.find_precedents_advanced.side_effect = RuntimeError("fail")
result = json.loads(self.tool._run(action="find_precedents", scenario="broken"))
self.assertEqual(result["precedents"], [])
self.assertIn("error", result)
class TestTraceCausalChain(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_causal_chain(self):
result = json.loads(
self.tool._run(action="trace_causal_chain", decision_id="dec-001")
)
self.assertIn("causal_chain", result)
self.assertEqual(result["decision_id"], "dec-001")
def test_honest_error_when_causal_trace_unavailable(self):
"""When the graph cannot trace causality, the tool must say so — it
must NOT substitute similarity-based precedents as a causal chain."""
del self.ctx.knowledge_graph.trace_decision_causality
result = json.loads(
self.tool._run(action="trace_causal_chain", decision_id="dec-002")
)
self.assertEqual(result["causal_chain"], [])
self.assertIn("error", result)
self.ctx.knowledge_graph.find_precedents.assert_not_called()
def test_missing_decision_id_reports_error(self):
result = json.loads(self.tool._run(action="trace_causal_chain"))
self.assertIn("error", result)
self.assertEqual(result["causal_chain"], [])
def test_depth_used(self):
self.tool._run(action="trace_causal_chain", decision_id="dec-001", depth=5)
self.ctx.knowledge_graph.trace_decision_causality.assert_called_once_with(
"dec-001", max_depth=5
)
def test_graceful_error_when_context_has_no_knowledge_graph(self):
"""Regression: an unguarded ``self.context.knowledge_graph`` read raised
AttributeError out of ``_run`` and could hard-fail a crew task. It must
return honest error JSON instead."""
del self.ctx.knowledge_graph
result = json.loads(
self.tool._run(action="trace_causal_chain", decision_id="dec-003")
)
self.assertEqual(result["causal_chain"], [])
self.assertIn("error", result)
class TestAnalyzeImpact(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_decision_id(self):
result = json.loads(
self.tool._run(action="analyze_impact", decision_id="dec-001")
)
self.assertEqual(result["decision_id"], "dec-001")
def test_includes_influence_metrics(self):
result = json.loads(
self.tool._run(action="analyze_impact", decision_id="dec-001")
)
self.assertIn("centrality", result)
class TestCheckPolicy(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_compliant_key(self):
decision = json.dumps(
{"category": "loan", "outcome": "approved", "confidence": 0.9}
)
result = json.loads(
self.tool._run(action="check_policy", decision_data=decision)
)
self.assertIn("compliant", result)
def test_invalid_json_returns_error(self):
result = json.loads(
self.tool._run(action="check_policy", decision_data="{not valid json}")
)
self.assertFalse(result["compliant"])
self.assertGreater(len(result["violations"]), 0)
def test_rule_violation_detected(self):
decision = json.dumps({"confidence": 0.5})
rules = json.dumps(["confidence >= 0.9"])
result = json.loads(
self.tool._run(
action="check_policy", decision_data=decision, policy_rules=rules
)
)
self.assertFalse(result["compliant"])
self.assertEqual(len(result["violations"]), 1)
def test_bool_false_rule_is_compliant(self):
"""Regression: ``enabled == false`` with ``enabled: false`` must be
compliant — bool("false") is truthy, so the old coercion inverted it."""
decision = json.dumps({"enabled": False, "confidence": 0.95})
rules = json.dumps(["enabled == false"])
result = json.loads(
self.tool._run(
action="check_policy", decision_data=decision, policy_rules=rules
)
)
self.assertTrue(result["compliant"])
self.assertEqual(result["violations"], [])
def test_bool_true_rule_is_compliant(self):
decision = json.dumps({"enabled": True})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["enabled == true"]),
)
)
self.assertTrue(result["compliant"])
def test_whitespace_padded_strings_are_trimmed(self):
"""Regression: ``_coerce_value`` must return the *stripped* string for
non-numeric literals, or padded decision_data fields never match."""
decision = json.dumps({"status": " approved "})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["status == approved"]),
)
)
self.assertTrue(result["compliant"])
self.assertEqual(result["violations"], [])
def test_bool_false_rule_violated_when_true(self):
decision = json.dumps({"enabled": True})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["enabled == false"]),
)
)
self.assertFalse(result["compliant"])
self.assertEqual(len(result["violations"]), 1)
def test_zero_one_flag_parsed_as_bool(self):
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"flag": 1}),
policy_rules=json.dumps(["flag != 0"]),
)
)
self.assertTrue(result["compliant"])
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"flag": 0}),
policy_rules=json.dumps(["flag != 0"]),
)
)
self.assertFalse(result["compliant"])
def test_numeric_string_value_compared_numerically(self):
"""Regression: a string datum like "0.90" must compare numerically to
rule literal 0.9, not lexicographically."""
decision = json.dumps({"score": "0.90"})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["score == 0.9"]),
)
)
self.assertTrue(result["compliant"])
def test_numeric_string_ordering(self):
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"pct": "0.95"}),
policy_rules=json.dumps(["pct >= 0.9"]),
)
)
self.assertTrue(result["compliant"])
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"pct": "0.85"}),
policy_rules=json.dumps(["pct >= 0.9"]),
)
)
self.assertFalse(result["compliant"])
def test_field_names_with_hyphens_dots_spaces(self):
"""Rule field names are not limited to ``\\w+`` — hyphenated/dotted
(and space-containing) JSON keys must be addressable."""
decision = json.dumps({"risk-score": 0.95, "max.risk": 0.2, "min score": 0.4})
compliant = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(
["risk-score >= 0.9", "max.risk <= 0.5", "min score >= 0.3"]
),
)
)
self.assertTrue(compliant["compliant"])
self.assertEqual(compliant["violations"], [])
violated = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["max.risk >= 0.5"]),
)
)
self.assertFalse(violated["compliant"])
self.assertEqual(len(violated["violations"]), 1)
def test_rule_missing_field_warns_not_silently_compliant(self):
decision = json.dumps({"confidence": 0.95})
rules = json.dumps(["minimum_score >= 0.9"])
result = json.loads(
self.tool._run(
action="check_policy", decision_data=decision, policy_rules=rules
)
)
self.assertTrue(result["compliant"])
self.assertEqual(result["violations"], [])
self.assertEqual(len(result["warnings"]), 1)
self.assertIn("minimum_score", result["warnings"][0])
def test_decision_data_non_object_rejected(self):
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps(["confidence", 0.95]),
policy_rules=json.dumps(["confidence >= 0.9"]),
)
)
self.assertFalse(result["compliant"])
self.assertEqual(len(result["violations"]), 1)
self.assertIn("JSON object", result["violations"][0])
def test_unknown_action_returns_error(self):
result = json.loads(self.tool._run(action="nope"))
self.assertIn("error", result)
def test_run_entrypoint(self):
result = json.loads(
self.tool.run(
action="check_policy",
decision_data=json.dumps({"confidence": 0.95}),
policy_rules=json.dumps(["confidence >= 0.9"]),
)
)
self.assertTrue(result["compliant"])
if __name__ == "__main__":
unittest.main()
tests/integrations/crewai/test_degradation.py
+103
View File
@@ -0,0 +1,103 @@
"""
Graceful-degradation tests for the CrewAI integration.
These run the integration modules in a fresh subprocess (no conftest crewai
stubs, no real crewai) to prove that every public class remains importable and
functional when ``crewai`` is absent. A subprocess is used because the other
test files in this directory install crewai stubs into ``sys.modules`` for the
whole pytest session; a subprocess keeps the two scenarios isolated.
"""
from __future__ import annotations
import os
import subprocess
import sys
import unittest
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
_SCRIPT = r"""
import json
import sys
try:
import crewai # noqa: F401
real_crewai = True
except ImportError:
real_crewai = False
from integrations.crewai import (
CREWAI_AVAILABLE,
SemanticaKGTool,
SemanticaDecisionTool,
SemanticaKnowledgeSource,
)
from semantica.context import ContextGraph
assert CREWAI_AVAILABLE == real_crewai, (
f"CREWAI_AVAILABLE={CREWAI_AVAILABLE} but real crewai={real_crewai}"
)
# --- SemanticaKGTool: importable + functional without crewai -----------------
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="privacy policy doc")
tool = SemanticaKGTool(graph=graph)
assert tool.name == "semantica_knowledge_graph"
assert tool.args_schema is not None
res = json.loads(tool._run(action="query_graph", query="privacy"))
assert res["count"] == 1, res
res = json.loads(tool._run(action="find_related", entity="ghost", hops=1))
assert res["count"] == 0, res
# The public run()/arun() entry points must exist without crewai too.
res = json.loads(tool.run(action="query_graph", query="privacy"))
assert res["count"] == 1, res
import asyncio
res = json.loads(asyncio.run(tool.arun(action="query_graph", query="privacy")))
assert res["count"] == 1, res
# --- SemanticaKnowledgeSource: importable + functional without crewai --------
src = SemanticaKnowledgeSource(graph=graph, chunk_size=40, chunk_overlap=5)
assert src.load_content() != {}
assert src.validate_content() is True
src.add() # must not raise; chunks kept in memory
assert len(src.chunks) > 0
# --- SemanticaDecisionTool: importable, builds its own context --------------
dt = SemanticaDecisionTool()
assert dt.name == "semantica_decision"
res = json.loads(dt.run(action="find_precedents", scenario="x"))
assert "precedents" in res, res
res = json.loads(asyncio.run(dt.arun(action="find_precedents", scenario="x")))
assert "precedents" in res, res
print("DEGRADATION_OK")
"""
class TestDegradation(unittest.TestCase):
def test_importable_and_functional_without_crewai(self):
result = subprocess.run(
[sys.executable, "-c", _SCRIPT],
cwd=REPO_ROOT,
capture_output=True,
text=True,
timeout=180,
)
self.assertEqual(
result.returncode,
0,
msg=(
f"subprocess failed:\nSTDOUT:\n{result.stdout}\n"
f"STDERR:\n{result.stderr}"
),
)
self.assertIn("DEGRADATION_OK", result.stdout)
if __name__ == "__main__":
unittest.main()
tests/integrations/crewai/test_kg_tool.py
+453
View File
@@ -0,0 +1,453 @@
"""
Tests for SemanticaKGTool — knowledge graph CrewAI tool.
Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is
``True`` and the real Pydantic/BaseTool subclassing path is exercised.
"""
from __future__ import annotations
import asyncio
import json
import unittest
from unittest.mock import MagicMock
from integrations.crewai import SemanticaKGTool as ImportedSemanticaKGTool
from integrations.crewai.kg_tool import (
CREWAI_AVAILABLE,
CREWAI_IMPORT_ERROR,
SemanticaKGTool,
SemanticaKGToolInput,
)
from semantica.context import ContextGraph
# ---------------------------------------------------------------------------
# Fakes
# ---------------------------------------------------------------------------
def _fake_entity(name="Tesla", etype="ORG", conf=0.9):
e = MagicMock()
e.name = name
e.type = etype
e.confidence = conf
return e
def _fake_relation(src="Tesla", rel="FOUNDED_BY", tgt="Elon Musk", conf=0.85):
r = MagicMock()
r.source = src
r.type = rel
r.target = tgt
r.confidence = conf
return r
class _FakeNER:
def extract_entities(self, text):
return [_fake_entity("Tesla"), _fake_entity("Elon Musk", "PERSON")]
class _FakeRelExtractor:
def extract_relations(self, text, entities=None):
return [_fake_relation()]
class _DataclassNER:
"""Returns Semantica's real ``Entity`` dataclass shape (text/label, no name)."""
def extract_entities(self, text):
from semantica.semantic_extract.types import Entity
return [
Entity(text="Tesla", label="ORG", start_char=0, end_char=5),
Entity(text="Elon Musk", label="PERSON", start_char=17, end_char=26),
]
class _DataclassRelExtractor:
"""Returns Semantica's real ``Relation`` dataclass shape (subject/object)."""
def __init__(self):
self.received_entities = None
def extract_relations(self, text, entities=None):
from semantica.semantic_extract.types import Entity, Relation
self.received_entities = entities
return [
Relation(
subject=Entity(text="Tesla", label="ORG", start_char=0, end_char=5),
predicate="FOUNDED_BY",
object=Entity(
text="Elon Musk", label="PERSON", start_char=17, end_char=26
),
)
]
class TestSemanticaKGToolInit(unittest.TestCase):
def test_crewai_available_via_stub(self):
self.assertTrue(CREWAI_AVAILABLE)
self.assertIsNone(CREWAI_IMPORT_ERROR)
def test_is_base_tool_subclass(self):
from crewai.tools import BaseTool
self.assertTrue(issubclass(SemanticaKGTool, BaseTool))
def test_exposed_from_package_init(self):
self.assertIs(ImportedSemanticaKGTool, SemanticaKGTool)
def test_creates_with_explicit_graph(self):
graph = ContextGraph()
tool = SemanticaKGTool(graph=graph)
self.assertIs(tool.graph, graph)
def test_creates_fresh_graph_when_none(self):
tool = SemanticaKGTool(
ner_extractor=_FakeNER(), relation_extractor=_FakeRelExtractor()
)
self.assertIsNotNone(tool.graph)
self.assertIsInstance(tool.graph, ContextGraph)
def test_default_metadata(self):
tool = SemanticaKGTool(
ner_extractor=_FakeNER(), relation_extractor=_FakeRelExtractor()
)
self.assertEqual(tool.name, "semantica_knowledge_graph")
self.assertTrue(tool.description)
self.assertEqual(tool.args_schema, SemanticaKGToolInput)
def test_input_schema_validates(self):
inp = SemanticaKGToolInput(action="query_graph", query="privacy", hops=2)
self.assertEqual(inp.hops, 2)
with self.assertRaises(Exception):
SemanticaKGToolInput(action="bogus")
def test_custom_kwargs_forwarded(self):
tool = SemanticaKGTool(
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
result_as_answer=True,
)
self.assertTrue(tool.result_as_answer)
class TestSemanticaKGToolSerialization(unittest.TestCase):
"""CrewAI checkpoints serialise tools via ``model_dump(mode="json")`` — the
live graph/extractors must not break that (regression for
PydanticSerializationError on arbitrary state objects)."""
def setUp(self):
self.tool = SemanticaKGTool(
graph=ContextGraph(),
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
)
def test_model_dump_json_excludes_shared_state(self):
dumped = self.tool.model_dump(mode="json")
self.assertNotIn("graph", dumped)
self.assertNotIn("ner_extractor", dumped)
self.assertNotIn("relation_extractor", dumped)
self.assertEqual(dumped["name"], "semantica_knowledge_graph")
def test_model_validate_restores_defaults(self):
restored = SemanticaKGTool.model_validate(self.tool.model_dump(mode="json"))
self.assertIsInstance(restored.graph, ContextGraph)
self.assertIs(restored.args_schema, SemanticaKGToolInput)
self.assertEqual(restored.name, "semantica_knowledge_graph")
def test_model_validate_restored_tool_still_runs(self):
restored = SemanticaKGTool.model_validate(self.tool.model_dump(mode="json"))
restored.graph.add_node(node_id="privacy", node_type="policy")
result = json.loads(restored._run(action="query_graph", query="privacy"))
self.assertEqual(result["count"], 1)
def test_restore_flags_lost_live_state(self):
"""A tool restored from a checkpoint must signal that its live graph
was excluded and an empty one reconstructed (``reconstructed_state``)."""
dumped = self.tool.model_dump(mode="json")
self.assertTrue(dumped["had_live_state"])
self.assertNotIn("reconstructed_state", dumped)
restored = SemanticaKGTool.model_validate(dumped)
self.assertTrue(restored.reconstructed_state)
self.assertFalse(SemanticaKGTool().reconstructed_state)
class TestSemanticaKGToolActions(unittest.TestCase):
def setUp(self):
self.graph = ContextGraph()
self.tool = SemanticaKGTool(
graph=self.graph,
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
)
def test_extract_entities(self):
result = json.loads(
self.tool._run(
action="extract_entities", text="Tesla was founded by Elon Musk"
)
)
self.assertEqual(result["count"], 2)
self.assertEqual(result["entities"][0]["name"], "Tesla")
self.assertEqual(result["entities"][0]["type"], "ORG")
def test_extract_relations(self):
result = json.loads(
self.tool._run(
action="extract_relations", text="Tesla was founded by Elon Musk"
)
)
self.assertEqual(result["count"], 1)
self.assertEqual(result["relations"][0]["source"], "Tesla")
self.assertEqual(result["relations"][0]["target"], "Elon Musk")
def test_add_to_graph_populates_graph(self):
result = json.loads(
self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk")
)
self.assertGreaterEqual(result["nodes_added"], 2)
self.assertGreaterEqual(result["edges_added"], 1)
nodes = self.graph.find_nodes()
node_ids = {n["id"] for n in nodes}
self.assertIn("Tesla", node_ids)
self.assertIn("Elon Musk", node_ids)
def test_add_to_graph_is_idempotent(self):
self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk")
second = json.loads(
self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk")
)
self.assertEqual(second["nodes_added"], 0)
self.assertEqual(second["edges_added"], 0)
def test_query_graph_finds_matching_node(self):
self.graph.add_node(
node_id="privacy", node_type="policy", content="privacy policy doc"
)
result = json.loads(self.tool._run(action="query_graph", query="privacy"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["results"][0]["id"], "privacy")
def test_query_graph_no_match(self):
result = json.loads(
self.tool._run(action="query_graph", query="nothing-matches")
)
self.assertEqual(result["count"], 0)
self.assertEqual(result["results"], [])
def test_query_graph_searches_node_content(self):
"""query_graph must match node content, not just ids/types."""
self.graph.add_node(
node_id="n1",
node_type="policy",
content="all refunds must be processed within 30 days",
)
result = json.loads(self.tool._run(action="query_graph", query="refunds"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["results"][0]["id"], "n1")
def test_query_graph_matches_type(self):
self.graph.add_node(node_id="n2", node_type="risk")
result = json.loads(self.tool._run(action="query_graph", query="risk"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["results"][0]["id"], "n2")
def test_query_graph_result_shape_is_consistent(self):
"""Every result — content match or id/type match — must carry the same
keys (id, type, label, content, score) so agents get one schema."""
self.graph.add_node(
node_id="n1",
node_type="policy",
content="all refunds within 30 days",
)
by_content = json.loads(self.tool._run(action="query_graph", query="refunds"))[
"results"
][0]
expected_keys = {"id", "type", "label", "content", "score"}
self.assertEqual(set(by_content.keys()), expected_keys)
by_id = json.loads(self.tool._run(action="query_graph", query="n1"))["results"][
0
]
self.assertEqual(set(by_id.keys()), expected_keys)
self.assertEqual(by_id["content"], "all refunds within 30 days")
self.assertEqual(by_id["score"], 1.0)
def test_extract_entities_skips_nameless_entities(self):
class _NamelessNER:
def extract_entities(self, text):
e = MagicMock()
e.name = None
e.type = "MISC"
e.confidence = 0.5
return [e]
tool = SemanticaKGTool(
graph=self.graph,
ner_extractor=_NamelessNER(),
relation_extractor=_FakeRelExtractor(),
)
result = json.loads(tool._run(action="extract_entities", text="text"))
self.assertEqual(result["count"], 0)
self.assertEqual(result["entities"], [])
def test_find_related_multi_hop(self):
self.graph.add_node(node_id="A", node_type="concept")
self.graph.add_node(node_id="B", node_type="concept")
self.graph.add_node(node_id="C", node_type="concept")
self.graph.add_edge(source_id="A", target_id="B", edge_type="related_to")
self.graph.add_edge(source_id="B", target_id="C", edge_type="related_to")
result = json.loads(self.tool._run(action="find_related", entity="A", hops=2))
self.assertEqual(result["count"], 2)
self.assertIn("B", result["related"])
self.assertIn("C", result["related"])
def test_find_related_unknown_entity(self):
result = json.loads(
self.tool._run(action="find_related", entity="Ghost", hops=1)
)
self.assertEqual(result["count"], 0)
self.assertEqual(result["related"], [])
def test_find_related_honors_incoming_edges(self):
"""find_related must be undirected: a node whose only edge is
incoming (A -> B) is still related to A."""
self.graph.add_node(node_id="OpenAI", node_type="ORG")
self.graph.add_node(node_id="Google", node_type="ORG")
self.graph.add_edge(
source_id="OpenAI", target_id="Google", edge_type="related_to"
)
result = json.loads(self.tool._run(action="find_related", entity="Google"))
self.assertEqual(result["related"], ["OpenAI"])
result_out = json.loads(self.tool._run(action="find_related", entity="OpenAI"))
self.assertEqual(result_out["related"], ["Google"])
def test_unknown_action_returns_error(self):
result = json.loads(self.tool._run(action="do_something_else"))
self.assertIn("error", result)
self.assertIn("do_something_else", result["error"])
def test_extract_entities_empty_text_is_graceful(self):
result = json.loads(self.tool._run(action="extract_entities", text=""))
self.assertIn("entities", result)
def test_extract_entities_confidence_none_defaults_to_one(self):
"""A single entity with ``confidence=None`` must not nuke the whole
extract result — it normalises to 1.0 instead of raising float(None)."""
class _NoneConfNER:
def extract_entities(self, text):
e = MagicMock()
e.name = "X"
e.type = "MISC"
e.confidence = None
return [e]
tool = SemanticaKGTool(
graph=self.graph,
ner_extractor=_NoneConfNER(),
relation_extractor=_FakeRelExtractor(),
)
result = json.loads(tool._run(action="extract_entities", text="text"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["entities"][0]["name"], "X")
self.assertEqual(result["entities"][0]["confidence"], 1.0)
self.assertNotIn("error", result)
def test_graph_lock_is_per_graph(self):
"""Independent graphs must not share a batch lock."""
g2 = ContextGraph()
lock_a = self.tool._graph_lock(self.graph)
lock_a_again = self.tool._graph_lock(self.graph)
lock_b = self.tool._graph_lock(g2)
self.assertIs(lock_a, lock_a_again)
self.assertIsNot(lock_a, lock_b)
class TestSemanticaKGToolDataclassShapes(unittest.TestCase):
"""Real Semantica ``Entity``/``Relation`` dataclasses (text/label,
subject/object) instead of MagicMock-shaped fakes."""
def setUp(self):
self.ner = _DataclassNER()
self.rel = _DataclassRelExtractor()
self.graph = ContextGraph()
self.tool = SemanticaKGTool(
graph=self.graph, ner_extractor=self.ner, relation_extractor=self.rel
)
def test_extract_entities_reads_text_label(self):
result = json.loads(
self.tool._run(action="extract_entities", text="Tesla founded by Elon Musk")
)
self.assertEqual(result["count"], 2)
self.assertEqual(result["entities"][0]["name"], "Tesla")
self.assertEqual(result["entities"][0]["type"], "ORG")
self.assertEqual(result["entities"][1]["name"], "Elon Musk")
self.assertEqual(result["entities"][1]["type"], "PERSON")
def test_extract_relations_reads_subject_object(self):
result = json.loads(
self.tool._run(
action="extract_relations", text="Tesla founded by Elon Musk"
)
)
self.assertEqual(result["count"], 1)
self.assertEqual(result["relations"][0]["source"], "Tesla")
self.assertEqual(result["relations"][0]["relation"], "FOUNDED_BY")
self.assertEqual(result["relations"][0]["target"], "Elon Musk")
def test_add_to_graph_passes_entity_objects_to_relation_extractor(self):
result = json.loads(
self.tool._run(action="add_to_graph", text="Tesla founded by Elon Musk")
)
self.assertEqual(result["nodes_added"], 2)
self.assertEqual(result["edges_added"], 1)
from semantica.semantic_extract.types import Entity
self.assertIsNotNone(self.rel.received_entities)
for e in self.rel.received_entities:
self.assertIsInstance(e, Entity)
node_ids = {n["id"] for n in self.graph.find_nodes()}
self.assertIn("Tesla", node_ids)
self.assertIn("Elon Musk", node_ids)
edge_keys = {
(e["source"], e["type"], e["target"]) for e in self.graph.find_edges()
}
self.assertIn(("Tesla", "FOUNDED_BY", "Elon Musk"), edge_keys)
class TestSemanticaKGToolCrewAIEntrypoints(unittest.TestCase):
def setUp(self):
self.tool = SemanticaKGTool(
graph=ContextGraph(),
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
)
def test_run_delegates_to_run(self):
result = json.loads(
self.tool.run(action="extract_entities", text="Tesla led by Elon Musk")
)
self.assertEqual(result["count"], 2)
def test_arun_async(self):
async def _call():
return await self.tool.arun(action="query_graph", query="x")
result = json.loads(asyncio.run(_call()))
self.assertIn("results", result)
def test_run_returns_string(self):
out = self.tool.run(action="extract_entities", text="hello world")
self.assertIsInstance(out, str)
if __name__ == "__main__":
unittest.main()
tests/integrations/crewai/test_knowledge_source.py
+228
View File
@@ -0,0 +1,228 @@
"""
Tests for SemanticaKnowledgeSource — CrewAI knowledge source backed by a
Semantica ContextGraph.
Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is
``True`` and the real Pydantic/BaseKnowledgeSource subclassing path (including
the current ``validate_content`` / ``add`` / ``aadd`` contract) is exercised.
"""
from __future__ import annotations
import asyncio
import unittest
from integrations.crewai import SemanticaKnowledgeSource
from integrations.crewai.knowledge_source import CREWAI_AVAILABLE, _chunk_text_manual
from semantica.context import ContextGraph
class _FakeStorage:
def __init__(self):
self.saved_chunks: list = []
def save(self, chunks: list) -> None:
self.saved_chunks.extend(chunks)
async def asave(self, chunks: list) -> None:
self.saved_chunks.extend(chunks)
class _RaisingStorage(_FakeStorage):
"""Mirrors real crewai: storage is wired but ``save`` raises ``ValueError``
(e.g. the embedder has no credentials configured)."""
def save(self, chunks: list) -> None:
raise ValueError("The OPENAI_API_KEY environment variable is not set.")
async def asave(self, chunks: list) -> None:
raise ValueError("The OPENAI_API_KEY environment variable is not set.")
def _build_graph() -> ContextGraph:
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="privacy policy doc")
graph.add_node(node_id="fraud", node_type="risk", content="fraud detection rules")
graph.add_edge(source_id="privacy", target_id="fraud", edge_type="constrains")
return graph
class TestSemanticaKnowledgeSourceInit(unittest.TestCase):
def test_crewai_available_via_stub(self):
self.assertTrue(CREWAI_AVAILABLE)
def test_is_base_knowledge_source_subclass(self):
from crewai.knowledge.source import BaseKnowledgeSource
self.assertTrue(issubclass(SemanticaKnowledgeSource, BaseKnowledgeSource))
def test_creates_with_explicit_graph(self):
graph = _build_graph()
src = SemanticaKnowledgeSource(graph=graph)
self.assertIs(src.graph, graph)
def test_creates_fresh_graph_when_none(self):
src = SemanticaKnowledgeSource()
self.assertIsNotNone(src.graph)
self.assertIsInstance(src.graph, ContextGraph)
def test_default_metadata(self):
src = SemanticaKnowledgeSource(graph=_build_graph())
self.assertEqual(src.name, "semantica_knowledge_graph")
self.assertEqual(src.chunk_size, 4000)
self.assertEqual(src.chunk_overlap, 200)
def test_custom_chunking_params(self):
src = SemanticaKnowledgeSource(
graph=_build_graph(), chunk_size=50, chunk_overlap=10
)
self.assertEqual(src.chunk_size, 50)
self.assertEqual(src.chunk_overlap, 10)
class TestLoadContent(unittest.TestCase):
def setUp(self):
self.graph = _build_graph()
self.src = SemanticaKnowledgeSource(graph=self.graph)
def test_nodes_serialized(self):
content = self.src.load_content()
text = "\n".join(content.values())
self.assertIn("privacy", text)
self.assertIn("fraud", text)
self.assertIn("policy", text)
def test_edges_serialized(self):
content = self.src.load_content()
text = "\n".join(content.values())
self.assertIn("-[" + "constrains" + "]->", text)
def test_empty_graph_returns_empty(self):
src = SemanticaKnowledgeSource(graph=ContextGraph())
self.assertEqual(src.load_content(), {})
def test_validate_content_passes(self):
self.assertTrue(self.src.validate_content())
def test_validate_content_raises_without_graph(self):
self.src.graph = None
with self.assertRaises(ValueError):
self.src.validate_content()
class TestAdd(unittest.TestCase):
def setUp(self):
self.graph = _build_graph()
self.src = SemanticaKnowledgeSource(
graph=self.graph, chunk_size=40, chunk_overlap=5
)
def test_add_saves_chunks_to_storage(self):
storage = _FakeStorage()
self.src.storage = storage
self.src.add()
self.assertGreater(len(storage.saved_chunks), 0)
self.assertTrue(all(isinstance(c, str) and c for c in storage.saved_chunks))
def test_add_without_storage_keeps_chunks_in_memory(self):
self.src.add()
self.assertGreater(len(self.src.chunks), 0)
self.assertGreater(len(self.src._chunks), 0)
def test_add_wired_storage_failure_logs_error_not_debug(self):
"""Regression: real crewai raises ``ValueError`` for a missing embedder
even though storage IS wired. That used to fall into the "storage not
wired" DEBUG branch, silently hiding the failure — it must log an
actionable ERROR instead."""
self.src.storage = _RaisingStorage()
with self.assertLogs(
f"semantica.{SemanticaKnowledgeSource.__module__}", level="ERROR"
) as caught:
self.src.add()
joined = "\n".join(caught.output)
self.assertIn("storage save FAILED", joined)
self.assertIn("OPENAI_API_KEY", joined)
self.assertGreater(len(self.src.chunks), 0)
def test_add_empty_graph_no_chunks(self):
src = SemanticaKnowledgeSource(
graph=ContextGraph(), chunk_size=40, chunk_overlap=5
)
src.add()
self.assertEqual(src.chunks, [])
def test_aadd_async(self):
storage = _FakeStorage()
self.src.storage = storage
asyncio.run(self.src.aadd())
self.assertGreater(len(storage.saved_chunks), 0)
def test_content_summary(self):
summary = self.src.get_content_summary()
self.assertEqual(summary["name"], "semantica_knowledge_graph")
self.assertGreater(summary["source_count"], 0)
self.assertTrue(summary["crewai_available"])
class TestSemanticaKnowledgeSourceSerialization(unittest.TestCase):
"""CrewAI checkpoints serialise their models via ``model_dump(mode="json")``
— the live graph must not break that (regression for
PydanticSerializationError on arbitrary state objects)."""
def test_model_dump_json_excludes_graph(self):
src = SemanticaKnowledgeSource(graph=_build_graph())
dumped = src.model_dump(mode="json")
self.assertNotIn("graph", dumped)
self.assertEqual(dumped["name"], "semantica_knowledge_graph")
def test_model_validate_restores_graph(self):
src = SemanticaKnowledgeSource(graph=_build_graph())
restored = SemanticaKnowledgeSource.model_validate(src.model_dump(mode="json"))
self.assertIsInstance(restored.graph, ContextGraph)
def test_restored_source_still_loads_content(self):
"""A checkpoint-restored source gets a fresh graph (the live graph is
excluded from serialisation); once a graph is attached it works."""
src = SemanticaKnowledgeSource(graph=_build_graph())
restored = SemanticaKnowledgeSource.model_validate(src.model_dump(mode="json"))
restored.graph = _build_graph()
self.assertNotEqual(restored.load_content(), {})
def test_restore_flags_lost_live_state(self):
"""A source restored from a checkpoint must signal that its live graph
was excluded and an empty one reconstructed (``reconstructed_state``).
Regression: an eager graph build in ``__init__`` used to hide this."""
src = SemanticaKnowledgeSource(graph=_build_graph())
dumped = src.model_dump(mode="json")
self.assertTrue(dumped["had_live_state"])
self.assertNotIn("reconstructed_state", dumped)
restored = SemanticaKnowledgeSource.model_validate(dumped)
self.assertTrue(restored.reconstructed_state)
self.assertFalse(SemanticaKnowledgeSource().reconstructed_state)
self.assertIsInstance(SemanticaKnowledgeSource().graph, ContextGraph)
class TestManualChunker(unittest.TestCase):
def test_short_text_single_chunk(self):
self.assertEqual(_chunk_text_manual("hello", 40, 5), ["hello"])
def test_empty_text(self):
self.assertEqual(_chunk_text_manual("", 40, 5), [])
def test_long_text_overlaps(self):
chunks = _chunk_text_manual("a" * 100, 40, 10)
self.assertGreater(len(chunks), 1)
self.assertTrue(all(len(c) <= 40 for c in chunks))
# Overlap means consecutive chunks share tail/head content
self.assertIn("a" * 10, chunks[0][-10:] + chunks[1][:10])
def test_zero_chunk_size_guarded(self):
self.assertEqual(_chunk_text_manual("hello world", 0, 5), ["hello world"])
if __name__ == "__main__":
unittest.main()
tests/integrations/crewai/test_real_crewai_integration.py
+123
View File
@@ -0,0 +1,123 @@
"""
End-to-end integration tests against the REAL crewai package.
These run in a subprocess because the stubs in ``conftest.py`` install a fake
``crewai`` module into ``sys.modules`` for the whole pytest session — the same
interpreter can never see both. Each test launches a fresh interpreter; if
crewai is genuinely not installed there, the test is skipped.
This covers the failure class the stubs cannot: ``Crew``-level serialization
(list[BaseTool] inside Agent.tools), checkpoint restore via ``model_validate``,
and knowledge-source behaviour with a real ``Crew``.
"""
from __future__ import annotations
import subprocess
import sys
import textwrap
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[3]
_SCRIPT = textwrap.dedent(
"""
import os
import json
import sys
sys.path.insert(0, os.getcwd())
try:
import crewai
except ImportError:
print("CREWAI_IMPORT_FAILED")
sys.exit(2)
import crewai as crewai_mod
from crewai import Agent, Task, Crew
from semantica.context import ContextGraph
from integrations.crewai import (
SemanticaKGTool,
SemanticaDecisionTool,
SemanticaKnowledgeSource,
)
os.environ["CREWAI_DESERIALIZE_CALLBACKS"] = "1"
# --- 1. Crew-level serialization round-trip ------------------------------
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy",
content="privacy policy: no data sharing")
tool = SemanticaKGTool(graph=graph)
decision_ctx = SemanticaDecisionTool()
decision_tool = SemanticaDecisionTool(context=decision_ctx.context)
agent = Agent(role="researcher", goal="answer questions",
backstory="retrieves from a knowledge graph",
tools=[tool, decision_tool])
task = Task(description="answer", expected_output="an answer", agent=agent)
crew = Crew(agents=[agent], tasks=[task])
dump = crew.model_dump(mode="json")
agents = dump["agents"]
assert len(agents) == 1, f"expected 1 agent, got {len(agents)}"
dumped_tools = agents[0]["tools"]
assert len(dumped_tools) == 2, f"expected 2 tools, got {len(dumped_tools)}"
for t in dumped_tools:
assert isinstance(t, dict), f"tool not serialized to dict: {type(t)}"
assert "graph" not in t, "live graph leaked into serialized tool"
assert "context" not in t, "live context leaked into serialized tool"
assert "ner_extractor" not in t, "extractor leaked into serialized tool"
# --- 2. Restore a tool from the crew dump --------------------------------
kg_dump = dumped_tools[0]
assert kg_dump["name"] == "semantica_knowledge_graph", kg_dump["name"]
restored = SemanticaKGTool.model_validate(kg_dump)
assert restored.graph is not None, "restored tool did not self-heal a graph"
q = json.loads(restored._run(action="query_graph", query="privacy"))
assert "results" in q, f"restored tool query_graph failed: {q}"
# --- 3. Knowledge source with no embedder must not crash a Crew ----------
ks = SemanticaKnowledgeSource(graph=graph)
agent2 = Agent(role="researcher2", goal="answer",
backstory="retrieves from knowledge")
task2 = Task(description="q", expected_output="a", agent=agent2)
crew2 = Crew(agents=[agent2], tasks=[task2],
knowledge_sources=[ks])
assert ks.chunks, "knowledge source retained no chunks in memory"
assert crew2.knowledge is not None, "crew.knowledge not created"
print("REAL_CREWAI_OK")
"""
)
class TestRealCrewAIIntegration(unittest.TestCase):
def _run(self) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, "-c", _SCRIPT],
cwd=str(REPO_ROOT),
capture_output=True,
text=True,
timeout=240,
)
def test_crew_level_round_trip_with_real_crewai(self):
proc = self._run()
if proc.returncode == 2:
self.skipTest("real crewai is not installed in this environment")
self.assertEqual(
proc.returncode,
0,
msg=f"subprocess failed:\n{proc.stdout}\n{proc.stderr}",
)
self.assertIn("REAL_CREWAI_OK", proc.stdout)
if __name__ == "__main__":
unittest.main()
tests/kg/test_analytics_node_scope.py
+112
View File
@@ -0,0 +1,112 @@
"""Regression tests for KG analytics node scope handling."""
import networkx as nx
from semantica.kg.centrality_calculator import CentralityCalculator
from semantica.kg.community_detector import CommunityDetector
from semantica.kg.connectivity_analyzer import ConnectivityAnalyzer
def _graph_with_isolated_node():
return {
"entities": [{"id": "A"}, {"id": "B"}, {"id": "C"}],
"relationships": [{"source": "A", "target": "B"}],
}
def test_centrality_keeps_declared_isolated_nodes():
result = CentralityCalculator().calculate_degree_centrality(
_graph_with_isolated_node()
)
assert result["total_nodes"] == 3
assert result["centrality"]["C"] == 0.0
def test_connectivity_reports_declared_isolated_nodes():
result = ConnectivityAnalyzer().analyze_connectivity(
_graph_with_isolated_node()
)
assert result["num_nodes"] == 3
assert result["num_components"] == 2
assert ["C"] in result["components"]
assert result["is_connected"] is False
def test_community_detection_keeps_declared_isolated_nodes():
detector = CommunityDetector()
result = detector.detect_communities(_graph_with_isolated_node())
assert set(result["node_assignments"]) == {"A", "B", "C"}
metrics = detector.calculate_community_metrics(
_graph_with_isolated_node(), result
)
assert metrics["num_communities"] == 2
structure = detector.analyze_community_structure(
_graph_with_isolated_node(), result
)
assert structure["num_communities"] == 2
def test_community_detection_returns_singletons_for_edgeless_graph():
graph = {"entities": [{"id": "A"}, {"id": "B"}], "relationships": []}
result = CommunityDetector().detect_communities(graph)
assert {frozenset(community) for community in result["communities"]} == {
frozenset({"A"}),
frozenset({"B"}),
}
def test_networkx_graph_keeps_isolated_nodes_for_analytics():
graph = nx.Graph()
graph.add_nodes_from(["A", "B", "C"])
graph.add_edge("A", "B")
centrality = CentralityCalculator().calculate_degree_centrality(graph)
connectivity = ConnectivityAnalyzer().analyze_connectivity(graph)
assert centrality["total_nodes"] == 3
assert centrality["centrality"]["C"] == 0.0
assert connectivity["num_nodes"] == 3
assert connectivity["num_components"] == 2
def test_nodes_edges_payload_keeps_declared_isolated_nodes():
graph = {
"nodes": [{"id": "A"}, {"id": "B"}, {"id": "C"}],
"edges": [("A", "B")],
}
result = CentralityCalculator().calculate_degree_centrality(graph)
assert result["total_nodes"] == 3
assert result["centrality"]["C"] == 0.0
def test_name_and_text_nodes_are_kept_when_ids_are_missing():
graph = {
"entities": [{"name": "Alice"}, {"text": "Bob"}],
"relationships": [],
}
result = CentralityCalculator().calculate_degree_centrality(graph)
assert result["total_nodes"] == 2
assert set(result["centrality"]) == {"Alice", "Bob"}
def test_community_metrics_accepts_communities_payload():
detector = CommunityDetector()
graph = {
"entities": [{"id": "A"}, {"id": "B"}, {"id": "C"}],
"relationships": [{"source": "A", "target": "B"}],
}
result = {"communities": [["A", "B"], ["C"]]}
metrics = detector.calculate_community_metrics(graph, result)
assert metrics["num_communities"] == 2
assert metrics["community_sizes"] == {0: 2, 1: 1}
tests/normalize/test_number_normalizer.py
+18
View File
@@ -49,6 +49,24 @@ class TestCurrencyNormalizer(unittest.TestCase):
self.assertEqual(result["amount"], 100.0)
self.assertEqual(result["currency"], "EUR")
def test_symbol_currencies_are_validated_as_supported_codes(self):
for symbol, expected_code in self.normalizer.currency_symbols.items():
result = self.normalizer.normalize_currency(f"{symbol}100")
self.assertEqual(result["currency"], expected_code)
self.assertTrue(self.normalizer.validate_currency_code(result["currency"]))
def test_currency_codes_match_boundaries_without_matching_words(self):
for value in ("RUB100", "100 RUB", "rub 100"):
result = self.normalizer.normalize_currency(value)
self.assertEqual(result["amount"], 100.0)
self.assertEqual(result["currency"], "RUB")
for value in ("ruby 100", "wilson 100"):
result = self.normalizer.normalize_currency(value)
self.assertEqual(result["amount"], 100.0)
self.assertEqual(result["currency"], "USD")
class TestScientificNotationHandler(unittest.TestCase):
def setUp(self):
self.handler = ScientificNotationHandler()
tests/parse/test_parser_construction.py
+73
View File
@@ -0,0 +1,73 @@
"""Construction coverage for the parse module's public parser classes.
Regression tests for #1014: ``ExcelParser.__init__`` called ``get_progress_tracker()``
without importing it, so every instantiation raised ``NameError``. The class was
covered by an import-only test, which passes regardless of whether ``__init__``
works, so nothing caught it. #530 was the same bug in ``SimilarityCalculator``.
These tests deliberately do **not** patch ``get_logger``/``get_progress_tracker``.
``tests/parse/test_parse_comprehensive.py`` patches both into every parse module
that exposes them, which would mock away the exact interaction under test here and
let the regression back in silently.
"""
import unittest
import semantica.parse as parse_module
from semantica.parse.excel_parser import ExcelParser
def _exported_parser_classes():
"""Public parser classes, taken from the package's own ``__all__``.
Driven off ``__all__`` rather than a hand-written list so a parser added later
is covered without anyone remembering to update this file.
"""
return [
(name, getattr(parse_module, name))
for name in parse_module.__all__
if name.endswith("Parser")
]
class TestExcelParserConstruction(unittest.TestCase):
"""ExcelParser must be constructible -- see #1014."""
def test_excel_parser_constructs(self):
parser = ExcelParser()
self.assertIsNotNone(parser)
def test_excel_parser_wires_progress_tracker(self):
"""The missing import was for the tracker, so assert it is actually set.
A bare construction check would pass against a version that dropped the
tracker call entirely; this pins the attribute the import exists to provide.
"""
parser = ExcelParser()
self.assertIsNotNone(parser.progress_tracker)
class TestExportedParsersConstruct(unittest.TestCase):
"""Every parser the package exports must survive ``__init__``."""
def test_all_exported_parsers_construct(self):
classes = _exported_parser_classes()
self.assertGreater(len(classes), 0, "no exported parser classes found")
for name, cls in classes:
with self.subTest(parser=name):
try:
self.assertIsNotNone(cls())
except ImportError as exc:
# Parsers backed by an optional dependency raise a deliberate,
# actionable ImportError when it is absent (e.g. DoclingParser
# without `docling`). That is correct behavior, not a defect.
self.assertIn(
"install",
str(exc).lower(),
f"{name} raised ImportError without install guidance: {exc}",
)
if __name__ == "__main__":
unittest.main()
tests/split/test_spacy_model_cache.py
+321
View File
@@ -0,0 +1,321 @@
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
from semantica.semantic_extract import methods as se_methods
from semantica.split import methods as split_methods
from semantica.split import semantic_chunker
from semantica.semantic_extract import ner_extractor as ner_extractor_module
from semantica.semantic_extract.ner_extractor import NERExtractor
@pytest.fixture(autouse=True)
def clear_cache():
se_methods.clear_spacy_model_cache()
yield
se_methods.clear_spacy_model_cache()
@pytest.fixture(autouse=True)
def force_spacy_available(monkeypatch):
# split.methods, split.semantic_chunker, and ner_extractor each compute
# their own SPACY_AVAILABLE flag from the real environment at import time;
# force all true so these tests exercise the spaCy branch regardless of
# whether spaCy is actually installed where they run.
monkeypatch.setattr(split_methods, "SPACY_AVAILABLE", True)
monkeypatch.setattr(semantic_chunker, "SPACY_AVAILABLE", True)
monkeypatch.setattr(ner_extractor_module, "SPACY_AVAILABLE", True)
def _fake_spacy(load):
return SimpleNamespace(
load=load,
util=SimpleNamespace(is_package=lambda _name: True),
)
def _nlp_mock(sentences=("Hello world.",)):
"""A stand-in spaCy Language object: callable, returns a doc with .sents."""
nlp = MagicMock()
nlp.return_value = SimpleNamespace(
sents=[SimpleNamespace(text=s) for s in sentences]
)
return nlp
class TestSpacyModelCache:
"""split.methods and split.semantic_chunker must share the cached model
defined in semantic_extract.methods instead of each calling spacy.load()
independently.
"""
def test_split_by_sentences_reuses_cached_model(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("Hello world. Bye world.")
split_methods.split_by_sentences("Another sentence here.")
split_methods.split_by_sentences("A third call.")
assert len(calls) == 1, "spacy.load should run once, not once per call"
assert calls[0][0] == "en_core_web_sm"
def test_semantic_chunker_reuses_cached_model_across_instances(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
chunker1 = semantic_chunker.SemanticChunker()
chunker2 = semantic_chunker.SemanticChunker()
assert len(calls) == 1, "each new SemanticChunker should not reload the model"
assert chunker1.nlp is chunker2.nlp
def test_split_methods_and_semantic_chunker_share_the_cache(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("Test sentence for split.methods.")
semantic_chunker.SemanticChunker()
assert len(calls) == 1, (
"split.methods and split.semantic_chunker must share one cached "
"model instead of each loading their own"
)
def test_distinct_model_names_load_separately(self, monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append((name, kwargs))
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
sm_chunker = semantic_chunker.SemanticChunker(model="en_core_web_sm")
lg_chunker = semantic_chunker.SemanticChunker(model="en_core_web_lg")
sm_chunker_again = semantic_chunker.SemanticChunker(model="en_core_web_sm")
assert [name for name, _ in calls] == ["en_core_web_sm", "en_core_web_lg"]
assert sm_chunker.nlp is sm_chunker_again.nlp
assert sm_chunker.nlp is not lg_chunker.nlp
def test_no_disable_kwarg_requested(self, monkeypatch):
"""split.methods and split.semantic_chunker both want the full
pipeline (they need .sents, which requires the parser/senter). If
either one later starts requesting a trimmed pipeline (e.g.
disable=["ner"]), the name-only cache key in load_spacy_model would
silently hand back a cached model built for a different config --
this test should catch that the moment it happens.
"""
calls = []
def fake_load(_name, **kwargs):
calls.append(kwargs)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("Hello world.")
se_methods.clear_spacy_model_cache()
semantic_chunker.SemanticChunker()
assert len(calls) == 2
assert all(kwargs == {} for kwargs in calls), (
"neither caller should pass any pipeline-configuration kwargs; "
"the name-only cache key in load_spacy_model cannot distinguish "
"models loaded with different component configs"
)
def test_missing_model_falls_back_without_poisoning_cache(self, monkeypatch):
attempts = []
def failing_load(name, **_kwargs):
attempts.append(name)
raise OSError(f"Can't find model '{name}'")
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
# split_by_sentences should fall back to regex splitting, not raise
chunks = split_methods.split_by_sentences("Hello world. Bye world.")
assert chunks, "fallback splitting should still produce chunks"
# SemanticChunker should leave .nlp as None rather than propagate
chunker = semantic_chunker.SemanticChunker()
assert chunker.nlp is None
assert len(attempts) == 2, "a failed load must not be cached"
# Once the model is available, both callers should now get it, and
# share a single successful load.
def working_load(name, **_kwargs):
attempts.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
chunker2 = semantic_chunker.SemanticChunker()
split_methods.split_by_sentences("One more sentence.")
assert len(attempts) == 3, (
"the model should load once after it becomes available"
)
assert chunker2.nlp is not None
def test_semantic_chunker_falls_back_when_spacy_runtime_is_broken(
self, monkeypatch
):
"""A spaCy model that is installed but unusable at runtime (e.g. a
config incompatible with the installed spaCy version) must degrade
SemanticChunker to fallback chunking, not crash __init__ -- mirrors
TestNERExtractorSpacyModelCache's equivalent broken-runtime test.
"""
def broken_load(name, **_kwargs):
raise RuntimeError("ConfigSchemaNlp is not fully defined")
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(broken_load))
chunker = semantic_chunker.SemanticChunker()
assert chunker.nlp is None
class TestNERExtractorSpacyModelCache:
"""NERExtractor(method="ml") must reuse the centralized cache in
semantic_extract.methods, not call spacy.load() on every construction.
These tests mirror TestSpacyModelCache but focus on the NERExtractor path,
confirming that all three callers (split_by_sentences, SemanticChunker, and
NERExtractor) draw from the same process-level cache.
"""
def test_ner_extractor_reuses_cached_model_across_instances(self, monkeypatch):
"""Two NERExtractor(method='ml') constructions with the same model name
must cause exactly one underlying spacy.load() call."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
e1 = NERExtractor(method="ml")
e2 = NERExtractor(method="ml")
e3 = NERExtractor(method="ml", model="en_core_web_sm")
assert len(calls) == 1, (
"repeated NERExtractor constructions should not reload the model"
)
assert e1.nlp is e2.nlp is e3.nlp
def test_ner_extractor_and_split_callers_share_one_cached_model(self, monkeypatch):
"""NERExtractor, SemanticChunker, and split_by_sentences must all use
the same cached Language object for the same model name."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
split_methods.split_by_sentences("First sentence.")
semantic_chunker.SemanticChunker()
NERExtractor(method="ml")
assert len(calls) == 1, (
"split_by_sentences, SemanticChunker, and NERExtractor must share "
"one cached model instead of each loading their own"
)
def test_ner_extractor_distinct_model_names_load_separately(self, monkeypatch):
"""Different model names must produce separate cache entries."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
sm = NERExtractor(method="ml", model="en_core_web_sm")
lg = NERExtractor(method="ml", model="en_core_web_lg")
sm_again = NERExtractor(method="ml", model="en_core_web_sm")
assert calls == ["en_core_web_sm", "en_core_web_lg"]
assert sm.nlp is sm_again.nlp
assert sm.nlp is not lg.nlp
def test_ner_extractor_failed_load_not_cached_and_retried(self, monkeypatch):
"""A missing model must not poison the cache. A subsequent construction
after the model becomes available must succeed and share the loaded model."""
attempts = []
def failing_load(name, **_kwargs):
attempts.append(name)
raise OSError(f"Can't find model '{name}'")
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load))
# Construction with missing model: nlp must remain None, no crash
extractor1 = NERExtractor(method="ml")
assert extractor1.nlp is None
assert len(attempts) == 1, "one load attempt expected for the missing model"
# Second construction: must retry (cache must not hold the failure)
extractor2 = NERExtractor(method="ml")
assert extractor2.nlp is None
assert len(attempts) == 2, "a failed load must not be cached"
# Now install a working model and verify recovery
def working_load(name, **_kwargs):
attempts.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load))
extractor3 = NERExtractor(method="ml")
extractor4 = NERExtractor(method="ml")
assert extractor3.nlp is not None
assert extractor3.nlp is extractor4.nlp
assert len(attempts) == 3, (
"exactly one successful load expected after the model becomes available"
)
def test_ner_extractor_non_ml_method_does_not_load_model(self, monkeypatch):
"""NERExtractor with a non-ml method must not touch the spaCy cache."""
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return _nlp_mock()
monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load))
NERExtractor(method="pattern")
NERExtractor(method="llm")
NERExtractor(method="regex")
assert calls == [], "non-ml methods must not trigger any spacy.load()"
if __name__ == "__main__":
pytest.main([__file__])
tests/split/test_splitter.py
+7 -9
View File
@@ -30,18 +30,16 @@ class TestSplitter(unittest.TestCase):
splitter = TextSplitter(method=["recursive", "token"])
self.assertEqual(splitter.methods, ["recursive", "token"])
@patch('semantica.split.semantic_chunker.spacy')
@patch('semantica.semantic_extract.methods.spacy')
def test_semantic_chunker_initialization(self, mock_spacy):
# Mock spacy.load to return a mock nlp object
# SemanticChunker now loads spaCy through the centralized
# load_spacy_model() in semantic_extract.methods, so we patch
# methods.spacy rather than the removed semantic_chunker.spacy binding.
mock_nlp = MagicMock()
mock_spacy.load.return_value = mock_nlp
# We need to ensure SPACY_AVAILABLE is True for this test context if possible,
# but it is imported at module level.
# If spacy is not installed, it sets SPACY_AVAILABLE = False.
# We might need to patch the module attribute or just test fallback if spacy missing.
chunker = SemanticChunker(chunk_size=100)
with patch('semantica.split.semantic_chunker.SPACY_AVAILABLE', True):
chunker = SemanticChunker(chunk_size=100)
self.assertEqual(chunker.chunk_size, 100)
def test_chunk_dataclass(self):
tests/test_030_context_graph_realworld_extended.py
+28 -28
View File
@@ -54,6 +54,11 @@ from semantica.context.decision_models import (
validate_decision,
)
# ── Export module ──────────────────────────────────────────────────────────────
# Set by the exporter's own `import pyarrow` attempt; False when pyarrow is
# missing or unimportable.
from semantica.export.parquet_exporter import PARQUET_AVAILABLE
# ── KG module ──────────────────────────────────────────────────────────────────
from semantica.kg import (
CentralityCalculator,
@@ -981,6 +986,17 @@ class TestParquetExportRealData:
Requires: pyarrow (optional dep — tests skip if not installed).
"""
# ParquetExporter imports fine without pyarrow and only raises ImportError
# when an export actually runs, so guarding on that import never skips
# anything. Guard on the exporter's own availability flag instead: it is set
# by the same `import pyarrow` / `import pyarrow.parquet` the exporter gates
# on, so the skip condition cannot drift from the runtime check — including
# when pyarrow is present on the path but fails to import.
pytestmark = pytest.mark.skipif(
not PARQUET_AVAILABLE,
reason="pyarrow not installed",
)
@pytest.fixture
def kg_data(self):
return {
@@ -1000,16 +1016,12 @@ class TestParquetExportRealData:
}
def test_parquet_exporter_importable(self):
try:
from semantica.export import ParquetExporter
except ImportError as e:
pytest.skip(f"ParquetExporter not available: {e}")
from semantica.export import ParquetExporter
assert ParquetExporter is not None
def test_parquet_export_entities_to_file(self, kg_data, tmp_path):
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
from semantica.export import ParquetExporter
exporter = ParquetExporter(compression="snappy")
out_path = tmp_path / "github_entities.parquet"
@@ -1018,10 +1030,7 @@ class TestParquetExportRealData:
assert out_path.stat().st_size > 0
def test_parquet_export_relationships_to_file(self, kg_data, tmp_path):
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
from semantica.export import ParquetExporter
exporter = ParquetExporter(compression="gzip")
out_path = tmp_path / "github_relationships.parquet"
@@ -1030,10 +1039,7 @@ class TestParquetExportRealData:
assert out_path.stat().st_size > 0
def test_parquet_export_knowledge_graph(self, kg_data, tmp_path):
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
from semantica.export import ParquetExporter
exporter = ParquetExporter(compression="snappy")
base_path = tmp_path / "github_kg"
@@ -1043,30 +1049,24 @@ class TestParquetExportRealData:
assert len(files) >= 1
def test_parquet_export_snappy_compression(self, kg_data, tmp_path):
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
from semantica.export import ParquetExporter
exporter = ParquetExporter(compression="snappy")
out_path = tmp_path / "snappy_test.parquet"
exporter.export_entities(kg_data["entities"], str(out_path))
assert out_path.exists()
def test_parquet_export_none_compression(self, kg_data, tmp_path):
try:
from semantica.export import ParquetExporter
except ImportError:
pytest.skip("pyarrow not installed")
from semantica.export import ParquetExporter
exporter = ParquetExporter(compression="none")
out_path = tmp_path / "uncompressed_test.parquet"
exporter.export_entities(kg_data["entities"], str(out_path))
assert out_path.exists()
def test_parquet_convenience_function(self, kg_data, tmp_path):
try:
from semantica.export.methods import export_parquet
except ImportError:
pytest.skip("pyarrow not installed")
from semantica.export.methods import export_parquet
out_path = tmp_path / "convenience_test.parquet"
export_parquet(kg_data["entities"], str(out_path))
assert out_path.exists()
tests/test_ner_configurations.py
+21 -8
View File
@@ -101,9 +101,15 @@ class TestNERConfigurations(unittest.TestCase):
self.assertEqual(entities[0].metadata["extraction_method"], "ml")
self.assertEqual(entities[0].metadata["model"], "en_core_web_trf")
@patch('semantica.semantic_extract.ner_extractor.spacy')
@patch('semantica.semantic_extract.methods.spacy')
def test_ner_ml_init_falls_back_when_spacy_runtime_is_broken(self, mock_spacy):
"""Test NER init does not crash when spaCy is installed but unusable at runtime."""
"""Test NER init does not crash when spaCy is installed but unusable at runtime.
The model load now goes through load_spacy_model() in semantic_extract.methods,
so we patch methods.spacy (not ner_extractor.spacy) to inject the failure.
"""
from semantica.semantic_extract.methods import clear_spacy_model_cache
clear_spacy_model_cache()
mock_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
with patch('semantica.semantic_extract.ner_extractor.SPACY_AVAILABLE', True):
@@ -112,17 +118,23 @@ class TestNERConfigurations(unittest.TestCase):
self.assertIsNone(extractor.nlp)
self.assertFalse(extractor._ml_runtime_usable)
@patch('semantica.semantic_extract.ner_extractor.spacy')
@patch('semantica.semantic_extract.methods.get_entity_method')
@patch('semantica.semantic_extract.methods.spacy')
def test_ner_ml_runtime_failure_disables_repeated_ml_load_attempts(
self,
mock_methods_spacy,
mock_get_method,
mock_init_spacy,
):
"""Test degraded ML mode skips repeated spaCy load attempts after init failure."""
mock_init_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
"""Test degraded ML mode skips repeated spaCy load attempts after init failure.
The model load at construction time now goes through load_spacy_model() in
semantic_extract.methods, so methods.spacy is the single mock target for the
init-time failure. After the RuntimeError is raised, _ml_runtime_usable is
False and no further spacy.load (or extract_entities_ml) calls are made.
"""
from semantica.semantic_extract.methods import clear_spacy_model_cache
clear_spacy_model_cache()
mock_methods_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined")
mock_ml_method = MagicMock(return_value=[])
mock_get_method.side_effect = lambda name: mock_ml_method if name == "ml" else (lambda *_args, **_kwargs: [])
@@ -132,8 +144,9 @@ class TestNERConfigurations(unittest.TestCase):
entities = extractor.extract_entities(self.text)
self.assertFalse(extractor._ml_runtime_usable)
self.assertEqual(mock_init_spacy.load.call_count, 1)
self.assertEqual(mock_methods_spacy.load.call_count, 0)
# methods.spacy.load called once during __init__ (the RuntimeError); not again
# during extract_entities because _filter_unusable_methods removes "ml".
self.assertEqual(mock_methods_spacy.load.call_count, 1)
self.assertEqual(mock_ml_method.call_count, 0)
self.assertIsInstance(entities, list)
tests/test_seed_manager.py
+54
View File
@@ -209,6 +209,60 @@ def test_load_from_api_allows_private_when_configured(mock_guard, seed_manager):
call_kwargs = mock_guard.call_args[1]
assert call_kwargs["allow_private_ips"] is True
@patch("semantica.seed.seed_manager.request_with_ssrf_guard")
def test_load_from_api_does_not_mutate_caller_headers_dict(mock_guard, seed_manager):
"""Regression test for issue #947 audit: load_from_api must not mutate the
caller's headers dict in-place when api_key is provided.
Before the fix, ``request_headers = headers or {}`` aliased the caller's dict.
Writing ``request_headers["Authorization"] = ...`` then silently modified the
caller's original dict, potentially leaking credentials to subsequent calls
that reused the same headers dict without expecting it to carry Authorization.
"""
mock_response = MagicMock()
mock_response.json.return_value = {"results": []}
mock_guard.return_value = mock_response
# Caller owns this dict and expects it to be unchanged after the call.
original_headers = {"X-Custom-Header": "value"}
headers_before = dict(original_headers) # snapshot
seed_manager.load_from_api(
api_url="http://api.example.com",
api_key="secret-key",
headers=original_headers,
)
# The caller's dict must be unchanged — Authorization must NOT have been added.
assert original_headers == headers_before, (
"load_from_api must not mutate the caller's headers dict; "
f"expected {headers_before!r}, got {original_headers!r}"
)
# The guard must still have received Authorization (in its own copy).
call_kwargs = mock_guard.call_args[1]
guard_headers = call_kwargs.get("headers", {})
assert guard_headers.get("Authorization") == "Bearer secret-key"
@patch("semantica.seed.seed_manager.request_with_ssrf_guard")
def test_load_from_api_does_not_mutate_empty_headers_dict(mock_guard, seed_manager):
"""When headers=None, a fresh dict is created — no aliasing to a shared mutable default."""
mock_response = MagicMock()
mock_response.json.return_value = {"results": []}
mock_guard.return_value = mock_response
seed_manager.load_from_api(
api_url="http://api.example.com",
api_key="key",
headers=None,
)
call_kwargs = mock_guard.call_args[1]
guard_headers = call_kwargs.get("headers", {})
assert guard_headers.get("Authorization") == "Bearer key"
def test_load_source(seed_manager, temp_data_dir):
json_file = temp_data_dir / "source.json"
with open(json_file, "w") as f:
tests/utils/test_normalize_graph_payload.py
+487
View File
@@ -0,0 +1,487 @@
"""Tests for the shared graph-payload normalizer (issue #956).
Graph payloads circulate under two vocabularies -- 'entities'/'relationships'
and 'nodes'/'edges' -- and consumers each reconciled them locally with at
least three competing idioms. The same payload could therefore be exported,
silently dropped, or rejected depending on which consumer read it:
``export_lpg`` dropped every entity when 'nodes' was present but empty, which
is precisely the shape ``JSONExporter`` emits.
The end-to-end assertions run the real exporters rather than mocking them,
since the behaviour under test is that the exporters now agree.
"""
import os
import shutil
import tempfile
import unittest
from dataclasses import dataclass
from semantica.export import methods as export_methods
from semantica.utils import normalize_graph_payload
from semantica.utils.exceptions import ValidationError
ENTITY = {"id": "e1", "name": "Acme"}
RELATIONSHIP = {"id": "r1", "source": "e1", "target": "e2"}
class TestVocabularyResolution(unittest.TestCase):
def test_canonical_keys_pass_through(self):
result = normalize_graph_payload(
{"entities": [ENTITY], "relationships": [RELATIONSHIP]}
)
self.assertEqual(result["entities"], [ENTITY])
self.assertEqual(result["relationships"], [RELATIONSHIP])
self.assertEqual(result["triplets"], [])
def test_aliases_are_mapped_to_canonical_keys(self):
result = normalize_graph_payload({"nodes": [ENTITY], "edges": [RELATIONSHIP]})
self.assertEqual(result["entities"], [ENTITY])
self.assertEqual(result["relationships"], [RELATIONSHIP])
def test_empty_alias_does_not_mask_a_populated_canonical_key(self):
"""The JSONExporter round-trip shape, and the #956 data-loss case."""
result = normalize_graph_payload(
{"entities": [ENTITY], "nodes": [], "relationships": [], "edges": []}
)
self.assertEqual(result["entities"], [ENTITY])
def test_empty_canonical_key_does_not_mask_a_populated_alias(self):
result = normalize_graph_payload({"entities": [], "nodes": [ENTITY]})
self.assertEqual(result["entities"], [ENTITY])
def test_identical_spellings_are_accepted(self):
result = normalize_graph_payload({"entities": [ENTITY], "nodes": [ENTITY]})
self.assertEqual(result["entities"], [ENTITY])
def test_conflicting_spellings_are_refused(self):
"""No basis to prefer either, and picking one would lose the other."""
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload(
{"entities": [ENTITY], "nodes": [{"id": "different"}]}
)
message = str(ctx.exception)
self.assertIn("entities", message)
self.assertIn("nodes", message)
def test_reordered_identical_spellings_are_accepted(self):
"""Same records, different order, is not a conflict.
A caller round-tripping through a dict-keyed cache or a set has no
reason to preserve list order; comparing spellings with plain list
equality rejected this as if the records differed.
"""
other = {"id": "e2", "name": "Beta"}
result = normalize_graph_payload(
{"entities": [ENTITY, other], "nodes": [other, ENTITY]}
)
self.assertCountEqual(result["entities"], [ENTITY, other])
def test_reordered_spellings_with_duplicate_records_still_conflict(self):
"""Multiset comparison must still catch a real count mismatch."""
with self.assertRaises(ValidationError):
normalize_graph_payload({"entities": [ENTITY, ENTITY], "nodes": [ENTITY]})
def test_triplets_are_carried_through(self):
result = normalize_graph_payload({"triplets": [{"s": "a", "p": "b", "o": "c"}]})
self.assertEqual(result["triplets"], [{"s": "a", "p": "b", "o": "c"}])
def test_missing_collections_default_to_empty_lists(self):
result = normalize_graph_payload({"entities": [ENTITY]})
self.assertEqual(result["relationships"], [])
self.assertEqual(result["triplets"], [])
def test_result_does_not_alias_the_input_collections(self):
payload = {"entities": [ENTITY]}
result = normalize_graph_payload(payload)
result["entities"].append({"id": "e2"})
self.assertEqual(len(payload["entities"]), 1)
class TestUnrecognizedInput(unittest.TestCase):
def test_unrecognized_keys_raise_by_default(self):
for payload in ({"data": [ENTITY]}, {"records": [ENTITY]}, {"foo": "bar"}):
with self.subTest(payload=payload):
with self.assertRaises(ValidationError):
normalize_graph_payload(payload)
def test_error_names_supplied_and_expected_keys(self):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"data": [ENTITY]})
message = str(ctx.exception)
self.assertIn("data", message)
self.assertIn("entities", message)
self.assertIn("nodes", message)
def test_empty_mapping_is_accepted(self):
"""An empty graph is legitimate and carries nothing that could be lost."""
result = normalize_graph_payload({})
self.assertEqual(result, {"entities": [], "relationships": [], "triplets": []})
def test_non_mapping_input_raises(self):
for payload in ([ENTITY], (ENTITY,), "entities", 42, None):
with self.subTest(payload=repr(payload)):
with self.assertRaises(ValidationError):
normalize_graph_payload(payload)
class TestExportersAgree(unittest.TestCase):
"""The divergence from #956, run against the real exporters."""
# export_csv is excluded: it writes entities/relationships/nodes/edges to
# four separate files by design, so it is not resolving two spellings of
# one collection and is out of scope for this change.
EXPORTERS = ("export_json", "export_arango", "export_neo4j_csv", "export_lpg")
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True)
def _export_and_read(self, name, payload):
outdir = os.path.join(self.tmpdir, name)
os.makedirs(outdir, exist_ok=True)
getattr(export_methods, name)(payload, os.path.join(outdir, "out"))
blob = ""
for root, _, files in os.walk(outdir):
for filename in files:
with open(os.path.join(root, filename), errors="ignore") as handle:
blob += handle.read()
return blob
def test_exporter_list_is_populated(self):
"""Guard against a vacuous suite if the list is emptied."""
self.assertGreaterEqual(len(self.EXPORTERS), 4)
def test_every_exporter_keeps_records_when_an_alias_is_empty(self):
payload = {
"entities": [ENTITY],
"nodes": [],
"relationships": [],
"edges": [],
}
for name in self.EXPORTERS:
with self.subTest(exporter=name):
self.assertIn(
"Acme",
self._export_and_read(name, payload),
f"{name} dropped the entity when 'nodes' was present but empty",
)
def test_every_exporter_accepts_the_alias_vocabulary(self):
payload = {"nodes": [ENTITY], "edges": []}
for name in self.EXPORTERS:
with self.subTest(exporter=name):
self.assertIn(
"Acme",
self._export_and_read(name, payload),
f"{name} dropped the entity supplied as 'nodes'",
)
def test_every_exporter_raises_processing_error_for_non_mapping_input(self):
"""A wrong-type payload is rejected the same way everywhere.
export_yaml and export_neo4j_csv raised ProcessingError for a bare
list; export_lpg and export_arango called normalize_graph_payload()
directly with no type guard, so they alone raised ValidationError
(from inside the resolver) for the identical mistake.
"""
from semantica.utils.exceptions import ProcessingError
for name in ("export_arango", "export_neo4j_csv", "export_lpg"):
with self.subTest(exporter=name):
outdir = os.path.join(self.tmpdir, name + "_bad_type")
os.makedirs(outdir, exist_ok=True)
with self.assertRaises(ProcessingError):
getattr(export_methods, name)([ENTITY], os.path.join(outdir, "out"))
def test_every_exporter_converts_object_shaped_records(self):
"""A dataclass record must not merely pass validation.
normalize_graph_payload() accepts dataclass/attribute-bearing
records (Neo4jCSVExporter reads them off attributes), but
export_lpg and export_arango read records with ``.get(...)``. A
record that passed validation unconverted crashed with a raw
AttributeError once used -- the exact failure the boundary exists
to prevent.
"""
@dataclass
class Node:
id: str
name: str
payload = {"entities": [Node(id="e1", name="Acme")], "relationships": []}
for name in ("export_arango", "export_neo4j_csv", "export_lpg"):
with self.subTest(exporter=name):
self.assertIn("Acme", self._export_and_read(name, payload))
def test_neo4j_accepts_non_dict_mappings(self):
"""Neo4jCSVExporter's mapping path must not be narrower than the rest.
_normalize_graph checked isinstance(graph, dict), so a non-dict
Mapping (a MappingProxyType, a ChainMap) fell into the
object-attribute branch and was rejected as an unrecognized object,
even though the identical payload exports fine via LPG/Arango/YAML.
"""
import types
payload = types.MappingProxyType({"entities": [ENTITY], "relationships": []})
self.assertIn("Acme", self._export_and_read("export_neo4j_csv", payload))
class TestRecordsCannotBeDroppedSilently(unittest.TestCase):
"""Presence of a recognized key is not proof the records survived.
``{"entities": [], "data": [...]}`` clears a presence-only check and still
resolves to empty, so the records under 'data' would be dropped with no
signal -- the same failure the recognition check exists to prevent.
"""
def test_empty_recognized_key_does_not_excuse_records_elsewhere(self):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [], "data": [ENTITY]})
message = str(ctx.exception)
self.assertIn("'data'", message)
self.assertIn("holds records", message)
def test_check_applies_to_every_recognized_spelling(self):
for key in ("entities", "nodes", "relationships", "edges", "triplets"):
with self.subTest(key=key):
with self.assertRaises(ValidationError):
normalize_graph_payload({key: [], "records": [ENTITY]})
def test_non_record_keys_are_not_mistaken_for_dropped_records(self):
"""ContextGraph.to_dict() always carries 'statistics'.
An empty graph must stay exportable, so only a non-empty list counts
as evidence that records were dropped.
"""
result = normalize_graph_payload(
{"nodes": [], "edges": [], "statistics": {"node_count": 0}}
)
self.assertEqual(result["entities"], [])
self.assertEqual(result["relationships"], [])
def test_records_alongside_a_populated_collection_are_not_refused(self):
"""Something resolved, so the export is not silently empty."""
result = normalize_graph_payload({"entities": [ENTITY], "statistics": {"n": 1}})
self.assertEqual(result["entities"], [ENTITY])
class TestCollectionValuesAreValidated(unittest.TestCase):
"""A recognized key is not proof its value is a collection of records.
Resolving on truthiness alone let ``{"entities": "abc"}`` through as three
single-character "records" and let ``{"entities": 42}`` surface as a raw
``TypeError`` from ``list()`` inside an exporter, naming the exporter
rather than the payload key at fault. Both are rejected here, at the
boundary that owns the question.
"""
COLLECTION_KEYS = ("entities", "nodes", "relationships", "edges", "triplets")
# Every public export path that reads its payload through the normalizer.
# export_json is excluded: it treats the payload as opaque records rather
# than resolving graph collections, so it never calls the normalizer.
NORMALIZING_EXPORTERS = (
"export_arango",
"export_neo4j_csv",
"export_lpg",
"export_yaml",
)
def test_string_value_is_not_treated_as_a_collection(self):
for key in self.COLLECTION_KEYS:
with self.subTest(key=key):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({key: "abc"})
message = str(ctx.exception)
self.assertIn(f"'{key}'", message)
self.assertIn("str", message)
def test_bytes_value_is_not_treated_as_a_collection(self):
for value in (b"abc", bytearray(b"abc")):
with self.subTest(value=repr(value)):
with self.assertRaises(ValidationError):
normalize_graph_payload({"entities": value})
def test_scalar_value_raises_validation_error_not_type_error(self):
for key in self.COLLECTION_KEYS:
for value in (42, 3.5, True, object()):
with self.subTest(key=key, value=repr(value)):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({key: value})
self.assertIn(f"'{key}'", str(ctx.exception))
def test_mapping_value_is_not_treated_as_a_collection(self):
"""``{"nodes": {"id": "n1"}}`` -- a single record, or an ID index."""
for payload in (
{"nodes": {"id": "n1"}},
{"entities": {"e1": ENTITY}},
{"edges": {"id": "r1"}},
):
with self.subTest(payload=payload):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload(payload)
self.assertIn("mapping", str(ctx.exception))
def test_non_record_elements_are_rejected(self):
for value in (["Acme"], [ENTITY, "Acme"], [42], [None], [[ENTITY]]):
with self.subTest(value=repr(value)):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": value})
self.assertIn("'entities'", str(ctx.exception))
def test_error_names_the_offending_index(self):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [ENTITY, ENTITY, "Acme"]})
self.assertIn("index 2", str(ctx.exception))
def test_object_records_are_accepted(self):
"""Attribute-bearing objects are accepted and converted to dicts.
LPGExporter and ArangoAQLExporter read records with ``.get(...)``, so
an object record that merely passed validation unconverted would
still crash with AttributeError once used; the boundary converts it.
"""
class Node:
def __init__(self):
self.id = "e1"
self.name = "Acme"
node = Node()
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "e1", "name": "Acme"}])
def test_dataclass_records_are_accepted(self):
@dataclass
class Node:
id: str
node = Node(id="e1")
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "e1"}])
def test_tuple_collections_are_accepted_and_materialized(self):
result = normalize_graph_payload({"entities": (ENTITY,)})
self.assertEqual(result["entities"], [ENTITY])
def test_none_is_read_as_an_absent_collection(self):
"""JSON round-trips an absent collection to null."""
result = normalize_graph_payload(
{"entities": None, "relationships": [RELATIONSHIP]}
)
self.assertEqual(result["entities"], [])
self.assertEqual(result["relationships"], [RELATIONSHIP])
def test_null_collection_still_cannot_hide_dropped_records(self):
with self.assertRaises(ValidationError):
normalize_graph_payload({"entities": None, "data": [ENTITY]})
def test_every_spelling_is_validated_not_just_the_winner(self):
"""A malformed alias is a defect even when the canonical key resolves."""
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [ENTITY], "nodes": "abc"})
self.assertIn("'nodes'", str(ctx.exception))
def test_malformed_value_reaches_no_exporter(self):
"""The end-to-end half: no exporter sees a TypeError from list()."""
tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, tmpdir, ignore_errors=True)
for name in self.NORMALIZING_EXPORTERS:
for value in ("abc", 42, {"id": "n1"}):
with self.subTest(exporter=name, value=repr(value)):
outdir = os.path.join(tmpdir, f"{name}_{type(value).__name__}")
os.makedirs(outdir, exist_ok=True)
with self.assertRaises(ValidationError):
getattr(export_methods, name)(
{"entities": value}, os.path.join(outdir, "out")
)
class TestIsRecordBoundary(unittest.TestCase):
"""_is_record gates the validation boundary introduced by this PR.
Modules and class/type objects carry ``__dict__`` but are not graph
records. Passing them through previously produced ``AttributeError``
inside exporters rather than a ``ValidationError`` at the boundary.
"""
def test_python_module_in_entities_raises_validation_error(self):
"""import math; {"entities": [math]} must be rejected at the boundary."""
import math
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [math]})
self.assertIn("'entities'", str(ctx.exception))
def test_class_object_in_entities_raises_validation_error(self):
"""A class (type object) is not a graph record."""
class MyNode:
pass
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [MyNode]})
self.assertIn("'entities'", str(ctx.exception))
def test_user_defined_instance_with_attributes_is_accepted(self):
"""Attribute-bearing instances are the legitimate use-case, converted
to a dict so every exporter -- not just Neo4jCSVExporter -- can read
it with ``.get(...)``."""
class Node:
def __init__(self):
self.id = "n1"
self.name = "Alice"
node = Node()
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "n1", "name": "Alice"}])
def test_dataclass_instance_is_accepted(self):
"""Dataclasses are a common record type used by Neo4jCSVExporter,
converted to a dict at the boundary so LPGExporter and
ArangoAQLExporter can read it too."""
node = dataclass_node()
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "dc1"}])
def test_mapping_record_is_accepted(self):
"""Plain dicts are the canonical record shape."""
result = normalize_graph_payload({"entities": [ENTITY]})
self.assertEqual(result["entities"], [ENTITY])
def test_module_rejected_through_normalizing_exporter(self):
"""End-to-end: a module element must not reach an exporter's internals."""
import math
tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, tmpdir, ignore_errors=True)
for name in ("export_arango", "export_neo4j_csv", "export_lpg"):
with self.subTest(exporter=name):
outdir = os.path.join(tmpdir, name)
os.makedirs(outdir, exist_ok=True)
with self.assertRaises(ValidationError):
getattr(export_methods, name)(
{"entities": [math]}, os.path.join(outdir, "out")
)
@dataclass
class _DataclassNode:
id: str
def dataclass_node():
return _DataclassNode(id="dc1")
if __name__ == "__main__":
unittest.main()
tests/utils/test_utils.py
+15
View File
@@ -31,6 +31,21 @@ class TestHelpers(unittest.TestCase):
dict2 = {"b": {"d": 3}, "e": 4}
merged = helpers.merge_dicts(dict1, dict2, deep=True)
self.assertEqual(merged, {"a": 1, "b": {"c": 2, "d": 3}, "e": 4})
def test_flatten_dict(self):
data = {"a": {"b": 1, "c": 2}}
result = helpers.flatten_dict(data)
self.assertEqual(result, {"a.b": 1, "a.c": 2})
def test_flatten_dict_key_collision(self):
data = {
"a.b": 1,
"a": {
"b": 2
}
}
with self.assertRaises(ValueError):
helpers.flatten_dict(data)
def test_safe_import_returns_module_and_flag(self):
module, available = helpers.safe_import("json")