From 5579851208ae5adbc813c787be8c8581d8bd2aed Mon Sep 17 00:00:00 2001 From: pravit-amp <43916793+pravit-amp@users.noreply.github.com> Date: Sat, 15 Aug 2026 09:40:04 -0700 Subject: [PATCH 01/19] fix(export): harden YAML export input handling (#958) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(export): centralize graph-payload key normalization Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it. Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade. Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings. Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed. CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files. * fix(export): reject non-mapping input to the YAML exporters export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both YAML exporters read their payload by key, so a list reached .get() and surfaced as a bare AttributeError from inside the exporter, naming neither the offending argument nor the shape expected. Reject rather than wrap. These formats distinguish entities from relationships from triplets, so inferring which collection a bare list represents would silently mislabel the records, and wrapping it under an unrecognised key would write a structurally valid file with every collection empty - trading a loud failure for silent data loss. Validate in the exporters, matching the existing precedent in Neo4jCSVExporter._normalize_graph, so direct users of the classes get the same contract as callers of the convenience wrapper. Narrow the wrapper type hint to Dict[str, Any] to match. * fix(export): address YAML exporter review findings - semantica/export/yaml_exporter.py — import Sequence from typing instead of collections.abc. `Sequence[str]` in _require_mapping's annotation is evaluated at function-definition time; collections.abc.Sequence only became subscriptable in Python 3.9, so on the 3.8 this project declares support for, importing this module raised TypeError. typing.Sequence has supported subscripting since 3.5.3. Mapping stays imported from collections.abc since it's only used for isinstance. - tests/export/test_yaml_exporter_input_validation.py — clean up each test's tempfile.mkdtemp() dir via addCleanup instead of leaking it, and read exported YAML through a context manager instead of an unclosed yaml.safe_load(open(...)). * fix(export): reject YAML export payloads with no recognized key Both YAML exporters built their output from a fixed set of `.get(key, [])` lookups, so a mapping keyed by anything else serialized to a structurally valid file with every collection empty. Nothing signalled the loss: no exception, no warning, and the progress log reported a completed export. The only way to notice was to open the file. The realistic trigger is re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}` envelope drops every record. - SemanticNetworkYAMLExporter.export_semantic_network now resolves its collections through normalize_graph_payload(), which raises rather than returning empty collections for an unrecognized mapping. Adopting the shared resolver rather than repeating the check locally also brings the 'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path from this library's own graph type to YAML, used in examples/capability_gap_context_graphs_example.py — exports its records instead of an empty file. - export_for_pipeline built its nested semantic network from the same defaulted lookups and had the same defect; it goes through the resolver too. - YAMLSchemaExporter.export_ontology_schema gets the equivalent check over its own key set. Schemas are a separate vocabulary with no aliasing, so _require_recognized_keys lives in this module rather than in the shared graph resolver. - 'metadata' is deliberately not sufficient to make a payload recognized. An export_json envelope carries one, so accepting it would readmit the case this fix is most likely to be needed for. - An empty mapping is still exported: an empty graph is legitimate and has no records to lose. - SemanticNetworkYAMLExporter.export() serializes before creating the output directory, so a rejected export leaves nothing behind. The two rejections keep distinct exception types, following what the codebase already does: a payload of the wrong *type* cannot be exported at all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph; a mapping whose *contents* are unusable raises ValidationError, matching normalize_graph_payload. _require_mapping therefore runs first at every entry point, so a non-mapping never reaches the resolver. Docstring Raises sections, export_usage.md and docs/reference/export.md record the accepted input shapes and both failures. Closes #953. * fix(export): reject payloads whose records resolve to nothing Addresses the Qodo findings on #958. Presence-only recognition (finding 1): checking that a recognized key is present answered "did the caller use our vocabulary" when the question that matters is "did anything the caller supplied survive". A payload like {"entities": [], "data": [...records...]} cleared the check, resolved to empty, and dropped every record under 'data' -- the silent-empty export by a narrower route. - utils/helpers.py — split the check in two. _require_recognized_keys keeps the presence rule; _require_nothing_dropped runs after resolution and refuses a payload that resolved to nothing while an unread key still holds records. Only a non-empty list counts as evidence: ContextGraph.to_dict() always carries a populated 'statistics' dict, and an empty graph must stay exportable, so 'metadata', 'statistics' and 'count' are named as context rather than records. - export/yaml_exporter.py — the schema path had the same hole and now runs both checks through the shared helpers rather than its own copy, so the two vocabularies cannot drift apart in what counts as a silent-empty export. Progress reported success on a failed write (finding 3): export_semantic_ network stops its tracking as completed once serialization returns, but export() then creates the directory and writes the file. A failure there left the tracker showing a completed export with no output. - export/yaml_exporter.py — the serialization span now says it serialized, not that it exported, and export() opens its own span around the filesystem work that stops as failed on error. Nothing reports a completed export until the bytes are on disk. Finding 2 (export_yaml no longer accepts List[Dict]) is the intended resolution of #952 rather than a regression: wrapping a bare list under a guessed key is what would mislabel the records. The signature, docstring and PR description already record the narrowed contract. Tests cover both directions of each fix, including that an empty ContextGraph still exports and that a failing write is not reported as completed. * fix(export): validate collection values and make Neo4j mappings strict Two gaps at the boundary the shared normalizer is supposed to own. _resolve_collection() resolved on truthiness alone, so a recognized key could still hold something that is not a collection of records: {"entities": "abc"} normalized to three single-character "records", and {"entities": 42} surfaced as a raw TypeError from list() inside whichever exporter happened to read it, naming the exporter rather than the payload key at fault. Collection values are now validated before conversion -- strings, bytes, mappings, and non-iterable scalars are rejected by key name, and each element must be a mapping or an attribute-carrying object, the two record shapes the exporters actually read. None stays legal as an absent collection, the spelling a JSON round-trip produces for []; it cannot hide dropped records, since _require_nothing_dropped() still runs. Every spelling present is validated, not just the one that wins, so a malformed alias is not excused by a well-formed canonical key. Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check for mappings, which left it able to turn {"data": [...]} into header-only CSVs indistinguishable from a genuinely empty graph -- the exact failure the rest of the change exists to prevent. Mapping payloads now go through normalize_graph_payload() on its default terms. The attribute path for graph objects is untouched. With no caller left opting out, the require_recognized flag is removed rather than kept as a way back into the silent-empty export. Regression tests cover the malformed values end to end through every export path that reads the normalizer, and assert the rejected Neo4j export writes no CSV files. * fix(export): close YAML schema and record validation gaps Fix 1 -- _require_usable_schema silent data loss (P1): _require_usable_schema() passed all values from _SCHEMA_KEYS into _require_nothing_dropped() as evidence that records survived. Scalar metadata fields such as version='1.0' and uri='http://...' are truthy strings, so any one of them caused _require_nothing_dropped() to return early and silently discard records stored under an unread key alongside them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}). Fixed by building the resolved list from only non-empty list/tuple values of recognised schema keys. Fix 2 -- _is_record accepts modules and type objects (P2): _is_record() accepted any object with __dict__, which includes Python modules and class objects. Elements that passed _coerce_records then reached exporters and raised AttributeError (e.g. module 'math' has no attribute 'get') rather than a ValidationError at the validation boundary. Fixed by excluding types.ModuleType and type from the __dict__ branch while preserving support for all user-defined attribute-bearing record objects. Tests: 101 tests pass across tests/utils/test_normalize_graph_payload.py tests/export/test_yaml_exporter_key_recognition.py tests/export/test_yaml_exporter_input_validation.py tests/export/test_neo4j_csv_exporter.py * fix(export): close exception-type and record-shape gaps in normalize_graph_payload LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no type guard, so non-mapping input raised ValidationError from inside the resolver while the YAML and Neo4j exporters raised ProcessingError for the identical mistake -- inconsistent with the exception-type contract this PR establishes. Both now use the shared _require_mapping() guard (moved from yaml_exporter.py into utils/helpers.py so all three can use it). Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a non-dict Mapping (MappingProxyType, ChainMap) fell through to the object-attribute branch and was rejected, even though the identical payload exported fine via the other three exporters. Now checks isinstance(graph, Mapping). normalize_graph_payload() accepts dataclass/attribute-bearing object records, but LPGExporter/ArangoAQLExporter call .get(...) directly on resolved entities -- an object-shaped record passed validation only to crash with a raw AttributeError once used, the exact failure this boundary exists to prevent. Records are now converted to plain dicts at the boundary (_coerce_records -> new _record_to_dict), so every consumer gets a uniform shape regardless of which reading the caller used. Two non-empty spellings of the same collection holding identical records in a different order were rejected as conflicting, since the check used plain list equality. Comparison is now an order-independent multiset of each record's canonical JSON form. * docs(changelog): add entry for #958 YAML export input hardening Documents the full arc of #958 -- the normalize_graph_payload() centralization, YAML input validation, both review rounds from @Sameer6305, and the exception-type/record-shape follow-up fixes -- plus closes #956, #952, #953. --------- Co-authored-by: Pravit Ampapathini Co-authored-by: Sameer Kadam Co-authored-by: KaifAhmad1 --- CHANGELOG.md | 16 + docs/reference/export.md | 13 + semantica/export/arango_aql_exporter.py | 24 +- semantica/export/export_usage.md | 68 +++ semantica/export/lpg_exporter.py | 40 +- semantica/export/methods.py | 22 +- semantica/export/neo4j_csv_exporter.py | 32 +- semantica/export/yaml_exporter.py | 195 ++++++- semantica/utils/__init__.py | 2 + semantica/utils/helpers.py | 373 +++++++++++++- tests/export/test_neo4j_csv_exporter.py | 57 ++ .../test_yaml_exporter_input_validation.py | 181 +++++++ .../test_yaml_exporter_key_recognition.py | 434 ++++++++++++++++ tests/utils/test_normalize_graph_payload.py | 487 ++++++++++++++++++ 14 files changed, 1897 insertions(+), 47 deletions(-) create mode 100644 tests/export/test_yaml_exporter_input_validation.py create mode 100644 tests/export/test_yaml_exporter_key_recognition.py create mode 100644 tests/utils/test_normalize_graph_payload.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 758fa407..a94fd882 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -62,6 +62,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`export_yaml` raised a raw `AttributeError` on list input, silently wrote empty exports for unrecognized dict keys, and graph payloads were reconciled differently by every exporter** (#958, closes #956, #952, #953) by @pravit-amp, reviewed by @Sameer6305 + - Graph payloads circulate under two vocabularies, `entities`/`relationships` and `nodes`/`edges`, and each exporter reconciled them locally with a different idiom — `LPGExporter` in particular dropped every entity whenever `nodes` was present but empty, the exact shape `JSONExporter` emits. A new `normalize_graph_payload()` in `utils/helpers.py` centralizes that decision once, adopted by `LPGExporter`, `ArangoAQLExporter`, `Neo4jCSVExporter`, and both YAML exporters; `ContextGraph.to_dict()` now round-trips through YAML correctly as a result + - `export_yaml(records, path)` on a bare list previously failed with `AttributeError` from inside the exporter; it and the other YAML methods now reject non-mapping input with an actionable `ProcessingError` naming the expected keys, since these formats distinguish entities/relationships/triplets and guessing which one a list represents would mislabel the records + - `export_yaml({"data": [...]}, path)` previously wrote a structurally valid file with every collection empty, no exception, no warning, and the progress log reporting a completed export. `export_semantic_network`, `export_for_pipeline`, and `export_ontology_schema` now raise `ValidationError` when the payload shares no recognized key with what the method reads, or resolves to nothing while an unread key still holds records — an empty mapping is still accepted, since a genuinely empty graph has no records to lose + - **Breaking**: the two cases above, plus a bare list, now raise instead of returning cleanly with data silently dropped or a raw `AttributeError` from exporter internals. Migration: pass records under a recognized key (`{"entities": [...]}` / `{"nodes": [...]}` for `semantic_network`, `{"classes": [...]}` for `schema`) + - **Fixed during review** (Qodo): progress tracking could report a completed export before the output directory existed or the file was written; `export()` now creates the directory and serializes before starting tracking, so a rejected export leaves nothing behind + - **Fixed during review** (@Sameer6305, round 1): `normalize_graph_payload()`'s collection resolver treated any truthy value as a collection — `{"entities": "abc"}` silently became three single-character records, `{"entities": 42}` leaked a raw `TypeError` from inside `list()`. Collection values are now validated before conversion, rejecting strings/bytes/mappings/non-iterable scalars by name. Separately, `Neo4jCSVExporter._normalize_graph` called the shared resolver with `require_recognized=False`, so it alone kept accepting an unrecognized mapping as a silent empty export; the opt-out (introduced earlier in this same PR, with no other caller) was removed + - **Fixed during review** (@Sameer6305, round 2): `YAMLSchemaExporter`'s usable-schema check could treat scalar schema metadata (`version`, `uri`, `title`, `description`) as evidence records had been exported, letting records under an unread key drop silently; and `_is_record()` accepted modules and class/type objects through the generic `__dict__` path, which would have reached exporter internals instead of failing at the boundary. Both closed, with regression coverage + - **Fixed during final maintainer review** (before merge): four more gaps in the shared boundary that the earlier rounds didn't reach + - `LPGExporter`/`ArangoAQLExporter` called `normalize_graph_payload()` with no type guard, so non-mapping input raised `ValidationError` from inside the resolver — while YAML and `Neo4jCSVExporter` raised `ProcessingError` for the identical mistake, per this PR's own stated contract. The `_require_mapping()` guard that already existed in `yaml_exporter.py` is now shared from `utils/helpers.py` and used by all three + - `Neo4jCSVExporter._normalize_graph` checked `isinstance(graph, dict)`, so a non-dict `Mapping` (`MappingProxyType`, `ChainMap`) fell through to the object-attribute branch and was rejected, even though the identical payload exported fine via `LPGExporter`/`ArangoAQLExporter`/YAML. Now checks `isinstance(graph, Mapping)` + - `normalize_graph_payload()` accepts dataclass and attribute-bearing object records (`Neo4jCSVExporter._record_to_dict` reads them), but `LPGExporter`/`ArangoAQLExporter` call `.get(...)` directly on resolved entities — an object-shaped record passed validation only to crash with a raw `AttributeError` once used, the exact failure this PR's boundary exists to prevent. Records are now converted to plain dicts at the boundary (`_coerce_records` → new `_record_to_dict`), so every consumer gets a uniform shape regardless of which reading the caller used + - Two non-empty spellings of the same collection (e.g. `entities` and `nodes`) holding identical records in a different order were rejected as conflicting, since the check used plain list equality; a caller round-tripping through a dict-keyed cache or a set has no reason to preserve order. Comparison is now an order-independent multiset of each record's canonical JSON form + - New regression coverage in `tests/utils/test_normalize_graph_payload.py`: exception-type parity for non-mapping input across `export_lpg`/`export_arango`/`export_neo4j_csv`, dataclass-record conversion verified end-to-end through the same three exporters, `Neo4jCSVExporter` accepting a `MappingProxyType` payload, and reordered-alias equality (plus a duplicate-count case confirming the multiset check still catches real conflicts); 4 existing tests updated to assert the corrected dict-conversion behavior instead of the previous object passthrough + - `pytest tests/export tests/utils tests/context tests/test_export_module.py tests/test_export_methods_wrapper.py tests/test_notebooks_simulation.py`: 718 passed, 4 skipped (up from 641 passed, 62 subtests at PR submission); `black`/`isort`/`flake8 --max-line-length=88` clean on every line this PR touches; `python -m build`: succeeds + - **`ContextGraph.add_edge` had no dedupe — identical edges were stored repeatedly under one shared edge ID, and re-ingest doubled the edge set** (#926, closes #922) by @pravit-amp - `_add_internal_edge` appended to `self.edges`, `edge_type_index`, and `_adjacency` unconditionally, with no check for an edge already present. Edge identity is content-derived (`_resolve_edge_identity` builds `edge_id` from `source_id`/`target_id`/`edge_type`/`weight`/`metadata`/`valid_from`/`valid_until`), so two identical `add_edge` calls produced two edge objects sharing one `edge_id` — the graph already considered them the same edge, it just kept both copies. `self.nodes` already deduped by ID; edges did not, so `stats()["edge_count"]` inflated, `density()` could exceed its mathematical maximum of `1.0`, and a refresh/restore job calling `build_from_entities_and_relationships()` (or reloading a saved graph) doubled the edge set on every cycle - Added an `edge_id -> ContextEdge` index (`_edge_index`), mirroring how `self.nodes` dedupes by node ID. `_add_internal_edge` now returns `False` when the `edge_id` already exists, checked before touching `edges`/`edge_type_index`/`_adjacency` and before firing the mutation callback, so a repeat `add_edge` is a silent no-op with no phantom `ADD_EDGE` audit event diff --git a/docs/reference/export.md b/docs/reference/export.md index 49df715e..eef19e94 100644 --- a/docs/reference/export.md +++ b/docs/reference/export.md @@ -203,6 +203,13 @@ export_lpg(graph, "import.cypher", method="cypher") exporter = SemanticNetworkYAMLExporter() exporter.export(graph, "graph.yaml") ``` + + The YAML exporters read `entities`/`relationships`/`triplets` (with + `nodes`/`edges` accepted as aliases, so `ContextGraph.to_dict()` exports + directly). A non-empty mapping supplying none of them raises + `ValidationError` rather than writing a file with every collection empty, + as does one whose collection value is not a list of records + (`{"entities": "abc"}`). **LPGExporter** writes Cypher `CREATE` statements for Neo4j and Memgraph: @@ -236,6 +243,12 @@ export_lpg(graph, "import.cypher", method="cypher") Both exporters write to a file and return `None`. + `LPGExporter`, `ArangoAQLExporter`, and `Neo4jCSVExporter` resolve mapping + payloads on the same terms as the YAML exporters above, so an unrecognized + or malformed mapping is rejected instead of exported as an empty graph. + `Neo4jCSVExporter` still reads graph *objects* off their + `nodes`/`entities` and `edges`/`relationships` attributes. + **`ArangoAQLExporter.export()` and `LPGExporter.export()` write to a file and return `None`.** They do not return the AQL/Cypher string. Write to a file and read it back if you need the string. diff --git a/semantica/export/arango_aql_exporter.py b/semantica/export/arango_aql_exporter.py index 69140481..d5f9a571 100644 --- a/semantica/export/arango_aql_exporter.py +++ b/semantica/export/arango_aql_exporter.py @@ -28,7 +28,7 @@ import json from pathlib import Path from typing import Any, Dict, List, Optional, Union -from ..utils.helpers import ensure_directory +from ..utils.helpers import _require_mapping, ensure_directory, normalize_graph_payload from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker @@ -204,17 +204,19 @@ class ArangoAQLExporter: self._generate_collection_creation(vertex_collection, edge_collection) ) - # Extract entities and relationships - entities = knowledge_graph.get("entities", []) - relationships = knowledge_graph.get("relationships", []) - nodes = knowledge_graph.get("nodes", entities) - edges = knowledge_graph.get("edges", relationships) + # A non-mapping payload cannot reach normalize_graph_payload(): it + # raises ValidationError for that case, which would leave this + # exporter alone in raising a different exception type than the YAML + # and Neo4j exporters raise for the identical mistake. + _require_mapping( + knowledge_graph, ("entities", "relationships", "nodes", "edges") + ) - # Use nodes/edges if entities/relationships are empty - if not entities and nodes: - entities = nodes - if not relationships and edges: - relationships = edges + # Accept either vocabulary; resolution is centralized so every + # exporter agrees on what a given payload means. + normalized = normalize_graph_payload(knowledge_graph) + entities = normalized["entities"] + relationships = normalized["relationships"] # Generate vertex INSERT statements vertex_statements = self._generate_vertex_inserts(entities, vertex_collection) diff --git a/semantica/export/export_usage.md b/semantica/export/export_usage.md index 140c5a01..7c8881e2 100644 --- a/semantica/export/export_usage.md +++ b/semantica/export/export_usage.md @@ -297,6 +297,57 @@ export_yaml(semantic_network, "network.yaml", method="semantic_network") export_yaml(schema, "schema.yaml", method="schema") ``` +### Accepted Input + +Both YAML exporters read their payload by key, so the input must be a mapping; +anything else raises `ProcessingError`. A bare list is rejected rather than +wrapped, since these formats distinguish entities from relationships from +triplets and guessing which one a list holds would mislabel the records. + +Each exporter then reads a fixed set of keys, and raises `ValidationError` on a +non-empty mapping that supplies none of them — such a payload would otherwise +serialize to a valid file with every collection empty. Naming a recognized key +is not enough on its own: `{"entities": [], "data": [...]}` also raises, since +nothing resolves while the records sit under a key the exporter never reads. + +| Method | Recognized keys | +| :--- | :--- | +| `"semantic_network"` | `entities` (alias `nodes`), `relationships` (alias `edges`), `triplets` | +| `"schema"` | `classes`, `properties`, `namespaces`, `uri`, `title`, `description`, `version` | + +`metadata` is carried through on both, but does not by itself make a payload +recognized — an `export_json` envelope (`{"data": [...], "count": N, +"metadata": {...}}`) carries one and is rejected. + +```python +# ContextGraph.to_dict() exports directly via the nodes/edges aliases +export_yaml(context_graph.to_dict(), "graph.yaml") + +# A bare list has no unambiguous meaning here +export_yaml(records, "out.yaml") # ProcessingError + +# An export_json payload is refused rather than written out empty +export_yaml({"data": records}, "out.yaml") # ValidationError + +# ...and so is one that names a recognized key but leaves it empty +export_yaml({"entities": [], "data": records}, "out.yaml") # ValidationError +``` + +The value under a recognized key must be a collection of records — a list or +tuple of mappings or objects. A string, a bare mapping, or a scalar raises +`ValidationError` naming the key, rather than being iterated into +character-sized "records" or surfacing as a `TypeError` from inside the +exporter. `None` is read as an absent collection, the same as `[]`. + +```python +export_yaml({"entities": "abc"}, "out.yaml") # ValidationError +export_yaml({"entities": 42}, "out.yaml") # ValidationError +export_yaml({"nodes": {"id": "n1"}}, "out.yaml") # ValidationError — wrap it in a list +``` + +An empty mapping is still accepted: an empty graph is a legitimate export and +has no records to lose. + ## OWL Export ### OWL/XML Format @@ -479,6 +530,23 @@ Pass `validate=True` to run a post-export integrity check before returning: export_neo4j_csv(kg, "neo4j_import/", validate=True) ``` +#### Accepted Input + +Mapping payloads are read on the same terms as the YAML exporters (see [Accepted +Input](#accepted-input) above): `entities`/`relationships`, with `nodes`/`edges` +accepted as aliases. A non-empty mapping that supplies neither — or that supplies +a malformed collection value — raises `ValidationError` rather than writing +header-only CSVs indistinguishable from a genuinely exported empty graph. The +payload is normalized before any file is opened, so a rejected export writes +nothing. + +Graph *objects* are unaffected: they are still read off `nodes`/`entities` and +`edges`/`relationships` attributes. + +```python +export_neo4j_csv({"data": [{"id": "e1"}]}, "neo4j_import/") # ValidationError +``` + #### Importing into Neo4j Once the CSV files are generated, they can be imported into a new Neo4j database using the `neo4j-admin database import` command: diff --git a/semantica/export/lpg_exporter.py b/semantica/export/lpg_exporter.py index 40f688a5..06ac4554 100644 --- a/semantica/export/lpg_exporter.py +++ b/semantica/export/lpg_exporter.py @@ -24,7 +24,7 @@ from pathlib import Path from typing import Any, Dict, List, Optional, Union from ..utils.exceptions import ProcessingError, ValidationError -from ..utils.helpers import ensure_directory +from ..utils.helpers import _require_mapping, ensure_directory, normalize_graph_payload from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker @@ -154,15 +154,26 @@ class LPGExporter: """ queries = [] - # Generate indexes if requested - if self.include_indexes: - queries.extend(self._generate_indexes(knowledge_graph)) + # A non-mapping payload cannot reach normalize_graph_payload(): it + # raises ValidationError for that case, which would leave this + # exporter alone in raising a different exception type than the YAML + # and Neo4j exporters raise for the identical mistake. + _require_mapping( + knowledge_graph, ("entities", "relationships", "nodes", "edges") + ) - # Extract entities and relationships - entities = knowledge_graph.get("entities", []) - relationships = knowledge_graph.get("relationships", []) - nodes = knowledge_graph.get("nodes", entities) - edges = knowledge_graph.get("edges", relationships) + # Accept either vocabulary. Reading 'nodes' with 'entities' as the + # default dropped every entity when 'nodes' was present but empty -- + # the shape JSONExporter emits -- so resolution is centralized. + normalized = normalize_graph_payload(knowledge_graph) + nodes = normalized["entities"] + edges = normalized["relationships"] + + # Generate indexes if requested. Fed the normalized entities so index + # generation sees the same records as node generation; reading + # 'entities' directly here skipped indexes for nodes/edges payloads. + if self.include_indexes: + queries.extend(self._generate_indexes(nodes)) # Generate node creation queries node_queries = self._generate_node_queries(nodes) @@ -174,13 +185,18 @@ class LPGExporter: return queries - def _generate_indexes(self, knowledge_graph: Dict[str, Any]) -> List[str]: - """Generate Cypher index and constraint creation queries.""" + def _generate_indexes(self, entities: List[Dict[str, Any]]) -> List[str]: + """Generate Cypher index and constraint creation queries. + + Args: + entities: Entity records, already resolved from whichever + vocabulary the caller supplied. + """ indexes = [] # Get unique entity types for labels entity_types = set() - for entity in knowledge_graph.get("entities", []): + for entity in entities: entity_type = entity.get("type") or entity.get("entity_type") if entity_type: entity_types.add(entity_type) diff --git a/semantica/export/methods.py b/semantica/export/methods.py index 6cec22c4..d2af2581 100644 --- a/semantica/export/methods.py +++ b/semantica/export/methods.py @@ -494,7 +494,7 @@ def export_graph( def export_yaml( - data: Union[Dict[str, Any], List[Dict[str, Any]]], + data: Dict[str, Any], file_path: Union[str, Path], method: str = "semantic_network", **kwargs, @@ -504,14 +504,32 @@ def export_yaml( This is a user-friendly wrapper that exports data to YAML format. + Unlike :func:`export_json` and :func:`export_csv`, which treat a list as + opaque records, both YAML methods are keyed formats: they distinguish + entities from relationships from triplets (and classes from properties + for ``method="schema"``). A bare list is therefore rejected rather than + guessed at, since inferring which collection it represents would silently + mislabel the records. + Args: - data: Data to export (semantic network, entities, relationships) + data: Data to export, as a mapping. For ``method="semantic_network"``, + keyed by 'entities'/'relationships'/'triplets'; for + ``method="schema"``, by 'classes'/'properties'. file_path: Output YAML file path method: Export method (default: "semantic_network") - "semantic_network": Semantic network YAML export - "schema": Schema YAML export **kwargs: Additional options passed to YAML exporters + Raises: + ProcessingError: if ``data`` is not a mapping, or if ``method`` is not + a known YAML export method. + ValidationError: if ``data`` is a mapping whose keys the selected + exporter does not read -- an ``export_json`` envelope + (``{"data": [...], "count": N, "metadata": {...}}``) is the + common case. Such a payload used to be written out as a valid + YAML file with every collection empty. + Examples: >>> from semantica.export.methods import export_yaml >>> export_yaml(semantic_network, "network.yaml", method="semantic_network") diff --git a/semantica/export/neo4j_csv_exporter.py b/semantica/export/neo4j_csv_exporter.py index 7a3461d1..8dbbd31b 100644 --- a/semantica/export/neo4j_csv_exporter.py +++ b/semantica/export/neo4j_csv_exporter.py @@ -32,12 +32,13 @@ from __future__ import annotations import csv import hashlib import json +from collections.abc import Mapping from dataclasses import asdict, is_dataclass from pathlib import Path from typing import Any, Dict, Iterable, List, Optional, Sequence, Union from ..utils.exceptions import ProcessingError, ValidationError -from ..utils.helpers import ensure_directory +from ..utils.helpers import ensure_directory, normalize_graph_payload from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker @@ -173,6 +174,19 @@ class Neo4jCSVExporter: Returns: Mapping with ``"nodes"`` and ``"relationships"`` output paths. + + Raises: + ValidationError: if a mapping payload carries no recognized graph + key, resolves to nothing while an unread key still holds + records, or holds something other than records under one -- + see + :func:`~semantica.utils.helpers.normalize_graph_payload`. + Each would otherwise be written out as header-only CSVs + indistinguishable from a genuinely empty graph. The payload is + normalized before any file is opened, so a rejected export + writes nothing. + ProcessingError: if a non-mapping payload exposes none of the + graph attributes. """ output_dir = Path(output_dir) ensure_directory(output_dir) @@ -494,9 +508,19 @@ class Neo4jCSVExporter: return prepared def _normalize_graph(self, graph: Any) -> Dict[str, List[Dict[str, Any]]]: - if isinstance(graph, dict): - nodes = graph.get("nodes") or graph.get("entities") or [] - relationships = graph.get("edges") or graph.get("relationships") or [] + if isinstance(graph, Mapping): + # Mapping payloads go through the shared resolver on its default + # terms, so this backend cannot drift from the others: an + # unrecognized mapping raises here rather than writing header-only + # CSVs that read as a successful export of an empty graph. Checked + # against Mapping rather than dict, so a non-dict Mapping (a + # MappingProxyType, a ChainMap) takes this path too, instead of + # falling through to the attribute branch below and being rejected + # as an unrecognized object -- the LPG, Arango, and YAML exporters + # already accept such payloads via the same resolver. + resolved = normalize_graph_payload(graph) + nodes = resolved["entities"] + relationships = resolved["relationships"] else: nodes = getattr(graph, "nodes", None) if nodes is None: diff --git a/semantica/export/yaml_exporter.py b/semantica/export/yaml_exporter.py index 0249ec08..ecbd55fa 100644 --- a/semantica/export/yaml_exporter.py +++ b/semantica/export/yaml_exporter.py @@ -21,15 +21,83 @@ Author: Semantica Contributors License: MIT """ +from collections.abc import Mapping from datetime import datetime from pathlib import Path from typing import Any, Dict, List, Optional, Union -from ..utils.exceptions import ProcessingError, ValidationError -from ..utils.helpers import ensure_directory +from ..utils.exceptions import ValidationError +from ..utils.helpers import ( + _require_mapping, + _require_nothing_dropped, + _require_recognized_keys, + ensure_directory, + normalize_graph_payload, +) from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker +# Keys YAMLSchemaExporter.export_ontology_schema reads. Graph payloads use the +# recognized set owned by normalize_graph_payload() instead; schemas are a +# separate vocabulary with no aliasing, so the set lives here. +_SCHEMA_KEYS = ( + "classes", + "properties", + "namespaces", + "uri", + "title", + "description", + "version", +) + + +def _require_usable_schema(ontology: Mapping) -> None: + """Reject a schema mapping this exporter cannot read. + + Two ways an ontology mapping produces an empty file: it shares no key with + the recognized set at all, or it names a recognized key that is empty + while the real records sit under a key this exporter does not read + (``{"classes": [], "nodes": [...]}``). Both are refused, using the same + checks the graph payloads go through, so the two vocabularies cannot drift + apart in what they consider a silent-empty export. + + An empty mapping is allowed through: it carries nothing that could be + lost, and an empty export is a legitimate result. + + Note the deliberate split in exception types, which the codebase already + makes: a wrong *type* cannot be exported at all and raises + ProcessingError, matching ``Neo4jCSVExporter._normalize_graph``; a mapping + whose *contents* are unusable raises ValidationError, matching + ``normalize_graph_payload``. + + Args: + ontology: Mapping already checked by :func:`_require_mapping`. + + Raises: + ValidationError: if the mapping shares no key with ``_SCHEMA_KEYS``, + or resolves to nothing while an unread key still holds records. + """ + _require_recognized_keys(ontology, _SCHEMA_KEYS, what="Ontology schema") + # Only non-empty list/tuple values from recognized schema keys count as + # evidence that records survived export. Scalar metadata fields such as + # 'uri', 'title', 'description', and 'version' are truthy strings, but + # their presence does not mean the caller's record collections were + # exported -- passing them as ``resolved`` would let any scalar value + # short-circuit the dropped-records check and silently discard a list + # under an unread key alongside e.g. {"version": "1.0", "nodes": [...]}. + resolved = [ + v + for key in _SCHEMA_KEYS + for v in (ontology.get(key),) + if isinstance(v, (list, tuple)) and v + ] + _require_nothing_dropped( + ontology, + _SCHEMA_KEYS, + resolved, + what="Ontology schema", + ) + class SemanticNetworkYAMLExporter: """ @@ -90,15 +158,39 @@ class SemanticNetworkYAMLExporter: Args: semantic_network: Semantic network dictionary containing: - - entities: List of entity dictionaries + - entities: List of entity dictionaries (alias: 'nodes') - relationships: List of relationship dictionaries + (alias: 'edges') - triplets: List of triplet dictionaries (optional) - metadata: Metadata dictionary (optional) + + Key resolution is delegated to + :func:`~semantica.utils.helpers.normalize_graph_payload`, so + ``ContextGraph.to_dict()`` output ('nodes'/'edges') exports + directly. **options: Additional export options (unused) Returns: String containing YAML representation of semantic network + Raises: + ProcessingError: if ``semantic_network`` is not a mapping. A bare + list of records cannot be exported here because this format + distinguishes entities, relationships, and triplets, and + guessing which one a list represents would silently mislabel + it. + ValidationError: if the mapping carries both spellings of a + collection with different contents; if it is non-empty and + shares no key with the recognized set; or if it resolves to + nothing while an unread key still holds records + (``{"entities": [], "data": [...]}``). Each previously + serialized to a file with every collection empty while the log + reported success. An empty mapping is still accepted -- it has + no records to lose. Note that 'metadata' alone is not a + recognized key: an ``export_json`` envelope carries one, and + accepting it would readmit the silent-empty export it is the + most likely source of. + Example: >>> network = { ... "entities": [...], @@ -107,6 +199,8 @@ class SemanticNetworkYAMLExporter: ... } >>> yaml_str = exporter.export_semantic_network(network) """ + _require_mapping(semantic_network, ("entities", "relationships", "triplets")) + # Track YAML export tracking_id = self.progress_tracker.start_tracking( file=None, @@ -119,15 +213,14 @@ class SemanticNetworkYAMLExporter: self.progress_tracker.update_tracking( tracking_id, message="Preparing YAML data..." ) + records = normalize_graph_payload(semantic_network) yaml_data = { "metadata": { "exported_at": datetime.now().isoformat(), "version": "1.0", **semantic_network.get("metadata", {}), }, - "entities": semantic_network.get("entities", []), - "relationships": semantic_network.get("relationships", []), - "triplets": semantic_network.get("triplets", []), + **records, } self.progress_tracker.update_tracking( @@ -140,7 +233,7 @@ class SemanticNetworkYAMLExporter: self.progress_tracker.stop_tracking( tracking_id, status="completed", - message="Exported semantic network to YAML", + message="Serialized semantic network to YAML", ) return result @@ -160,16 +253,46 @@ class SemanticNetworkYAMLExporter: data: Data to export file_path: Output file path **options: Additional options + + Raises: + ProcessingError: if ``data`` is not a mapping. + ValidationError: on the mappings :meth:`export_semantic_network` + rejects. Serialization runs before the output directory is + created, so a rejected export leaves nothing behind. + OSError: if the file cannot be written. The write is tracked + separately from serialization, so no progress entry reports a + completed export until the bytes are on disk. """ file_path = Path(file_path) - ensure_directory(file_path.parent) - yaml_content = self.export_semantic_network(data, **options) - with open(file_path, "w", encoding="utf-8") as f: - f.write(yaml_content) + # Serialization reports its own completion, but it says nothing about + # the file: without this second span, a failing write would leave the + # tracker showing a completed export and no output. + tracking_id = self.progress_tracker.start_tracking( + file=str(file_path), + module="export", + submodule="SemanticNetworkYAMLExporter", + message=f"Writing YAML to {file_path}", + ) - self.logger.info(f"Exported YAML to: {file_path}") + try: + ensure_directory(file_path.parent) + with open(file_path, "w", encoding="utf-8") as f: + f.write(yaml_content) + + self.logger.info(f"Exported YAML to: {file_path}") + self.progress_tracker.stop_tracking( + tracking_id, + status="completed", + message=f"Exported YAML to: {file_path}", + ) + + except Exception as e: + self.progress_tracker.stop_tracking( + tracking_id, status="failed", message=str(e) + ) + raise def export_entities( self, entities: List[Dict[str, Any]], include_metadata: bool = True, **options @@ -263,18 +386,34 @@ class SemanticNetworkYAMLExporter: • Structure for definition generation • Include extraction metadata • Return pipeline-ready YAML + + Args: + extracted_data: Semantic network mapping, read through + :func:`~semantica.utils.helpers.normalize_graph_payload` on + the same terms as :meth:`export_semantic_network`. + pipeline_stage: Stage number recorded in the output. + **options: Additional export options (unused) + + Returns: + Pipeline-ready YAML string. + + Raises: + ProcessingError: if ``extracted_data`` is not a mapping. + ValidationError: on the same mappings as + :meth:`export_semantic_network` -- this method built its + nested semantic network from the same defaulted lookups and + so had the same silent-empty failure. """ + _require_mapping(extracted_data, ("entities", "relationships", "triplets")) + + semantic_network = normalize_graph_payload(extracted_data) yaml_data = { "pipeline_stage": pipeline_stage, "metadata": { "extracted_at": datetime.now().isoformat(), **extracted_data.get("metadata", {}), }, - "semantic_network": { - "entities": extracted_data.get("entities", []), - "relationships": extracted_data.get("relationships", []), - "triplets": extracted_data.get("triplets", []), - }, + "semantic_network": semantic_network, } return self.yaml.dump(yaml_data, default_flow_style=False, sort_keys=False) @@ -308,7 +447,29 @@ class YAMLSchemaExporter: • Include hierarchies and constraints • Structure for easy editing • Return YAML schema + + Args: + ontology: Ontology mapping keyed by any of 'classes', + 'properties', 'namespaces', 'uri', 'title', 'description', + 'version'. + **options: Additional export options (unused) + + Returns: + YAML schema string. + + Raises: + ProcessingError: if ``ontology`` is not a mapping. + ValidationError: if ``ontology`` is a non-empty mapping sharing + no key with the recognized set, or resolves to nothing while + an unread key still holds records + (``{"classes": [], "nodes": [...]}``) -- each previously + produced a file with empty 'classes', 'properties' and + 'namespaces' and no indication anything was dropped. An empty + mapping is still accepted. """ + _require_mapping(ontology, ("classes", "properties")) + _require_usable_schema(ontology) + yaml_data = { "ontology": { "uri": ontology.get("uri", ""), diff --git a/semantica/utils/__init__.py b/semantica/utils/__init__.py index bd0e7e2d..002a139d 100644 --- a/semantica/utils/__init__.py +++ b/semantica/utils/__init__.py @@ -80,6 +80,7 @@ from .helpers import ( hash_data, merge_dicts, normalize_entities, + normalize_graph_payload, parse_timestamp, read_json_file, retry_on_error, @@ -183,6 +184,7 @@ __all__ = [ "format_data", "clean_text", "normalize_entities", + "normalize_graph_payload", "hash_data", "safe_filename", "ensure_directory", diff --git a/semantica/utils/helpers.py b/semantica/utils/helpers.py index 21f6cfc2..7462f6db 100644 --- a/semantica/utils/helpers.py +++ b/semantica/utils/helpers.py @@ -63,9 +63,16 @@ import importlib import json import os import re +import types +from collections import Counter +from collections.abc import Iterable as IterableABC +from collections.abc import Mapping +from dataclasses import asdict, is_dataclass from datetime import datetime, timezone from pathlib import Path -from typing import Any, Dict, List, Optional, Tuple, Type, Union +from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple, Type, Union + +from .exceptions import ProcessingError, ValidationError def format_data(data: Any, format_type: str = "json") -> str: @@ -584,3 +591,367 @@ def classify_path_distance(hop_count: int) -> str: if hop_count <= 6: return "mid-range" return "distant" + + +# Graph payloads circulate under two vocabularies: 'entities'/'relationships' +# (kg builders, most exporters) and 'nodes'/'edges' (ContextGraph.to_dict, +# Neo4jCSVExporter, the Explorer routes). Consumers each reconciled them +# locally, with at least three competing idioms, so the same payload could be +# exported, silently dropped, or rejected depending on which consumer read it. +# This is the single place that decision is made. +_ENTITY_KEYS = ("entities", "nodes") +_RELATIONSHIP_KEYS = ("relationships", "edges") +_TRIPLET_KEYS = ("triplets",) + +# Keys that legitimately travel alongside the collections without being +# records themselves, so their presence is never evidence that records were +# dropped: ContextGraph.to_dict() carries 'statistics', JSON envelopes carry +# 'metadata' and 'count'. +_CONTEXT_KEYS = ("metadata", "statistics", "count") + + +def _require_recognized_keys( + payload: Mapping, recognized_keys: Sequence[str], *, what: str +) -> None: + """Reject a mapping that shares no key with the recognized set. + + A consumer that reads a fixed set of keys turns an unrecognized mapping + into an empty result that looks like a legitimate one. An empty mapping is + allowed through -- it carries nothing that could be lost. + + Args: + payload: Mapping to check. + recognized_keys: Keys the consumer reads. + what: Noun for the error message, e.g. ``"Graph payload"``. + + Raises: + ValidationError: if ``payload`` is non-empty and shares no key with + ``recognized_keys``. + """ + if not payload or any(key in payload for key in recognized_keys): + return + + supplied = ", ".join(f"'{key}'" for key in sorted(map(str, payload))) + expected = ", ".join(f"'{key}'" for key in recognized_keys) + raise ValidationError( + f"{what} has no recognized key. Supplied: {supplied}. " + f"Expected at least one of: {expected}." + ) + + +def _require_nothing_dropped( + payload: Mapping, + recognized_keys: Sequence[str], + resolved: Iterable[Any], + *, + what: str, +) -> None: + """Reject a mapping that resolved to nothing while still holding records. + + Checking that a recognized key is *present* is not enough: + ``{"entities": [], "data": [...]}`` clears that bar and still resolves to + empty, dropping every record under 'data'. Presence answers "did the + caller use our vocabulary"; this answers the question that actually + matters, "did anything the caller supplied survive". + + Only non-empty lists count as evidence of dropped records. A payload can + carry scalars and dicts that are not collections -- ContextGraph.to_dict() + always includes 'statistics' -- and an empty graph must stay exportable. + + Args: + payload: Mapping to check. + recognized_keys: Keys the consumer reads. + resolved: The collections the consumer resolved from ``payload``. + what: Noun for the error message, e.g. ``"Graph payload"``. + + Raises: + ValidationError: if nothing resolved and an unread key holds a + non-empty list. + """ + if any(resolved): + return + + dropped = sorted( + str(key) + for key, value in payload.items() + if key not in recognized_keys + and key not in _CONTEXT_KEYS + and isinstance(value, (list, tuple)) + and value + ) + if not dropped: + return + + named = ", ".join(f"'{key}'" for key in dropped) + expected = ", ".join(f"'{key}'" for key in recognized_keys) + raise ValidationError( + f"{what} resolved to nothing, but {named} still holds records. " + f"Exporting it would drop them silently. Supply the records under " + f"one of: {expected}." + ) + + +def _is_record(value: Any) -> bool: + """Report whether a value can stand in for a graph record. + + Consumers read records either as mappings (``entity.get("type")`` in the + LPG and Arango exporters) or as objects with attributes + (``Neo4jCSVExporter._record_to_dict`` accepts dataclasses and anything + carrying a ``__dict__``). Both are legitimate, so both are accepted here; + strings, numbers, and nested sequences are not records under either + reading. + + Modules and class/type objects are excluded even though they carry + ``__dict__``: they are not graph records under any supported reading, and + passing them through the boundary would produce ``AttributeError`` inside + exporters rather than a ``ValidationError`` at the boundary where the + problem is visible. + """ + return isinstance(value, Mapping) or is_dataclass(value) or ( + hasattr(value, "__dict__") + and not isinstance(value, (types.ModuleType, type)) + ) + + +def _record_to_dict(record: Any) -> Dict[str, Any]: + """Convert an accepted record to a plain dict. + + :func:`_is_record` accepts mappings, dataclasses, and objects carrying + ``__dict__`` as legitimate record shapes, but consumers of + :func:`normalize_graph_payload` -- YAML serialization, ``entity.get(...)`` + in the LPG and Arango exporters -- read records as dicts. Converting here, + at the boundary, means every exporter gets the same shape regardless of + which reading the caller used; previously only ``Neo4jCSVExporter`` + converted object-shaped records locally, so a dataclass record passed + validation for the other exporters only to crash with a raw + ``AttributeError`` once used. + """ + if isinstance(record, Mapping): + return dict(record) + if is_dataclass(record): + return asdict(record) + return { + key: value for key, value in vars(record).items() if not key.startswith("_") + } + + +def _coerce_records(key: str, value: Any) -> List[Any]: + """Validate one collection value and materialize it as a list of records. + + This runs before any truthiness or ``list()`` call, because both mislead + on malformed input: ``list("abc")`` quietly turns a string into three + single-character "records", and ``list(42)`` raises a bare ``TypeError`` + from deep inside the exporter that named the exporter rather than the + offending payload key. Neither reaches the caller as an actionable + message, so the shapes that produce them are rejected by name instead. + + ``None`` is deliberately not rejected: JSON round-trips an absent + collection to null, and treating that as "no records under this key" is + the same answer an explicit ``[]`` gets. It is not silent data loss -- + a null collection alongside records under an unread key is still caught + by :func:`_require_nothing_dropped`. + + Args: + key: Payload key the value came from, for the error message. + value: The raw value stored under ``key``. + + Returns: + The records as a new list, so the result never aliases the input. + + Raises: + ValidationError: if ``value`` is a string, bytes, a mapping, or any + non-iterable scalar; or if any element is not a record. + """ + if value is None: + return [] + + if isinstance(value, (str, bytes, bytearray)): + raise ValidationError( + f"Graph payload key '{key}' holds a {type(value).__name__}, not a " + f"collection of records. Iterating it would yield characters, not " + f"records. Supply a list of records." + ) + + if isinstance(value, Mapping): + raise ValidationError( + f"Graph payload key '{key}' holds a mapping, not a collection of " + f"records. If it is a single record, wrap it in a list; if it is " + f"keyed by ID, supply its values as a list." + ) + + if not isinstance(value, IterableABC): + raise ValidationError( + f"Graph payload key '{key}' holds a " + f"{type(value).__name__}, not a collection of records. Supply a " + f"list of records." + ) + + records = list(value) + for index, record in enumerate(records): + if not _is_record(record): + raise ValidationError( + f"Graph payload key '{key}' holds a " + f"{type(record).__name__} at index {index}, not a record. " + f"Records must be mappings or objects with attributes." + ) + return [_record_to_dict(record) for record in records] + + +def _canonical_record_multiset(records: List[Dict[str, Any]]) -> "Counter[str]": + """Represent records as an order-independent multiset for equality checks. + + Two spellings of the same collection (``entities`` and ``nodes``) can + legitimately list identical records in a different order -- a caller + round-tripping through a dict-keyed cache or a set has no reason to + preserve list order. Comparing with plain list equality would treat that + as a conflict and reject a payload that carries no real data loss, so + records are compared as a multiset of their canonical JSON form instead. + """ + return Counter( + json.dumps(record, sort_keys=True, default=str) for record in records + ) + + +def _resolve_collection( + payload: Dict[str, Any], keys: Tuple[str, ...] +) -> List[Dict[str, Any]]: + """Pick one collection from a payload that may use either vocabulary. + + Both spellings may legitimately be present: ``JSONExporter`` writes + 'entities' and 'nodes' side by side, so a round-trip of its output carries + both, one of them empty. Where only one holds records, that one wins. + + Two non-empty, unequal spellings are a different matter -- there is no + basis for preferring either, and picking one would silently discard the + other -- so that is refused rather than guessed at. + + Every spelling present is validated, not just the one that wins: a + malformed 'nodes' alongside a well-formed 'entities' is a payload the + caller should hear about, and validating only the winner would let it + through on the strength of the other key. + + Args: + payload: Mapping to read from. + keys: Accepted spellings, most canonical first. + + Returns: + The resolved collection, or an empty list if no spelling is present. + + Raises: + ValidationError: if a spelling holds something other than a collection + of records; or if two spellings are both present, both non-empty, + and hold different records, order ignored. + """ + present = { + key: _coerce_records(key, payload[key]) for key in keys if key in payload + } + populated = {key: value for key, value in present.items() if value} + + if len(populated) > 1: + values = list(populated.values()) + canonical = [_canonical_record_multiset(value) for value in values] + if any(entry != canonical[0] for entry in canonical[1:]): + named = " and ".join(f"'{key}'" for key in populated) + raise ValidationError( + f"Graph payload carries {named} with different contents; " + f"cannot determine which to export. Supply one, or make them " + f"identical." + ) + + for key in keys: + value = present.get(key) + if value: + # Already a fresh list from _coerce_records, so the result cannot + # alias the caller's collection. + return value + + # Every spelling present is empty (or none is): an explicit empty + # collection is a legitimate answer, distinct from "unrecognized". + return [] + + +def _require_mapping(data: Any, expected_keys: Sequence[str]) -> None: + """Reject non-mapping export input with an actionable error. + + Shared by every consumer of :func:`normalize_graph_payload` so that a + wrong *type* fails the same way everywhere. Handed a sequence (or any + other non-mapping), every downstream key lookup would fail with a bare + ``AttributeError: 'list' object has no attribute 'get'``, which tells the + caller nothing about the shape expected -- and ``normalize_graph_payload`` + itself raises ``ValidationError`` for this case, which would leave + exporters that skip this guard raising a different exception type than + the ones that call it, for the identical mistake. + + A list is rejected rather than wrapped: these formats distinguish + entities from relationships from triplets (or nodes/edges), so inferring + which one a bare list represents would silently mislabel the records. + + Args: + data: Candidate export payload. + expected_keys: Key names the caller reads, named in the error so the + caller learns the expected shape. + + Raises: + ProcessingError: if ``data`` is not a mapping. + """ + if not isinstance(data, Mapping): + keys = "/".join(f"'{key}'" for key in expected_keys) + raise ProcessingError( + f"Cannot export object of type '{type(data).__name__}': " + f"expected a dict with {keys}." + ) + + +def normalize_graph_payload( + payload: Dict[str, Any], +) -> Dict[str, List[Dict[str, Any]]]: + """Reduce a graph payload to one canonical vocabulary. + + Accepts either 'entities'/'relationships' or 'nodes'/'edges' (or a mix) + and returns the canonical spelling, so consumers read one shape instead of + reimplementing the reconciliation. + + This is the validation boundary for graph payloads: it either returns + collections of records or raises. Nothing that reaches an exporter through + it needs re-checking, and nothing malformed passes through it as a + valid-looking empty graph. + + Args: + payload: Graph payload mapping. + + Returns: + ``{"entities": [...], "relationships": [...], "triplets": [...]}``. + + Raises: + ValidationError: if ``payload`` is not a mapping; if a recognized key + holds something other than a collection of records; if two + spellings of the same collection are both non-empty and differ; if + a non-empty mapping contains no recognized key; or if it resolves + to nothing while an unread key still holds records. The last two + would otherwise hand the caller a valid-looking result with their + records silently dropped. + + Example: + >>> normalize_graph_payload({"nodes": [{"id": "n1"}], "edges": []}) + {'entities': [{'id': 'n1'}], 'relationships': [], 'triplets': []} + """ + if not isinstance(payload, Mapping): + raise ValidationError( + f"Cannot normalize graph payload of type " + f"'{type(payload).__name__}': expected a mapping." + ) + + recognized = _ENTITY_KEYS + _RELATIONSHIP_KEYS + _TRIPLET_KEYS + _require_recognized_keys(payload, recognized, what="Graph payload") + + resolved = { + "entities": _resolve_collection(payload, _ENTITY_KEYS), + "relationships": _resolve_collection(payload, _RELATIONSHIP_KEYS), + "triplets": _resolve_collection(payload, _TRIPLET_KEYS), + } + + _require_nothing_dropped( + payload, recognized, resolved.values(), what="Graph payload" + ) + + return resolved diff --git a/tests/export/test_neo4j_csv_exporter.py b/tests/export/test_neo4j_csv_exporter.py index 529f7bd4..e7f4d0c9 100644 --- a/tests/export/test_neo4j_csv_exporter.py +++ b/tests/export/test_neo4j_csv_exporter.py @@ -301,3 +301,60 @@ def test_nested_properties_are_json_serialized(tmp_path): by_id = {row[0]: row for row in rows[1:]} assert by_id["node1"][2] == '{"k":"v"}' assert by_id["node1"][3] == "[1,2,3]" + + +def test_unrecognized_mapping_is_refused_rather_than_exported_empty(tmp_path): + """The Neo4j path reads mappings on the shared normalizer's default terms. + + An ``export_json`` envelope names no graph key, so it resolves to nothing. + Written out, that is a pair of header-only CSVs indistinguishable from a + genuinely empty graph -- the silent-empty export the shared contract + exists to prevent. + """ + exporter = Neo4jCSVExporter() + + with pytest.raises(ValidationError) as excinfo: + exporter.export({"data": [{"id": "e1"}]}, tmp_path) + + message = str(excinfo.value) + assert "data" in message + assert "entities" in message + + assert not (tmp_path / "nodes.csv").exists() + assert not (tmp_path / "relationships.csv").exists() + + +def test_records_under_an_unread_key_are_not_dropped_silently(tmp_path): + """Naming a recognized key is not enough if nothing resolves from it.""" + exporter = Neo4jCSVExporter() + + with pytest.raises(ValidationError): + exporter.export({"nodes": [], "data": [{"id": "e1"}]}, tmp_path) + + assert not (tmp_path / "nodes.csv").exists() + + +def test_malformed_collection_value_is_refused(tmp_path): + """``list("abc")`` would otherwise export one node per character.""" + exporter = Neo4jCSVExporter() + + for value in ("abc", 42, {"id": "n1"}): + with pytest.raises(ValidationError) as excinfo: + exporter.export({"nodes": value}, tmp_path) + assert "nodes" in str(excinfo.value) + + assert not (tmp_path / "nodes.csv").exists() + + +def test_graph_objects_still_use_the_attribute_path(tmp_path): + """Only mappings changed; objects are not mappings and are unaffected.""" + + class Graph: + def __init__(self): + self.nodes = [{"id": "e1", "type": "Person", "name": "Acme"}] + self.edges = [] + + exporter = Neo4jCSVExporter() + exporter.export(Graph(), tmp_path) + + assert "Acme" in (tmp_path / "nodes.csv").read_text(encoding="utf-8") diff --git a/tests/export/test_yaml_exporter_input_validation.py b/tests/export/test_yaml_exporter_input_validation.py new file mode 100644 index 00000000..13a375d7 --- /dev/null +++ b/tests/export/test_yaml_exporter_input_validation.py @@ -0,0 +1,181 @@ +"""Regression tests for YAML export input validation (issue #952). + +``export_yaml`` declared ``Union[Dict[str, Any], List[Dict[str, Any]]]`` but +both YAML exporters read their payload by key, so a list reached +``semantic_network.get(...)`` and surfaced as a bare +``AttributeError: 'list' object has no attribute 'get'`` from inside the +exporter — an error that names neither the offending argument nor the shape +expected. + +A list is rejected rather than wrapped. These formats distinguish entities +from relationships from triplets, so inferring which collection a bare list +represents would silently mislabel the records; and wrapping it under an +unrecognised key would write a structurally valid file with every collection +empty, trading a loud failure for silent data loss. + +Both directions are pinned: non-mappings raise ``ProcessingError`` with an +actionable message, and every mapping that worked before still exports. +""" + +import os +import shutil +import tempfile +import unittest +from collections import OrderedDict, defaultdict + +import yaml + +from semantica.export.methods import export_yaml +from semantica.export.yaml_exporter import ( + SemanticNetworkYAMLExporter, + YAMLSchemaExporter, +) +from semantica.utils.exceptions import ProcessingError + +# Non-mapping payloads that must be rejected. A list of dicts is the shape +# from #952; the rest guard the same path against other sequence/scalar types. +NON_MAPPINGS = { + "list_of_dicts": [{"id": "1", "name": "Acme"}], + "empty_list": [], + "tuple_of_dicts": ({"id": "1"},), + "list_of_scalars": ["a", "b"], + "string": "entities", + "bytes": b"entities", + "int": 42, + "none": None, + "set": {"a"}, +} + +# Both YAML methods, with a minimal valid payload and the key names the +# corresponding error message must mention. +METHODS = { + "semantic_network": { + "valid": { + "entities": [{"id": "1", "name": "Acme"}], + "relationships": [], + "triplets": [], + }, + "expected_key": "entities", + "top_level_key": "entities", + }, + "schema": { + "valid": {"classes": [{"name": "Thing"}], "properties": []}, + "expected_key": "classes", + "top_level_key": "classes", + }, +} + + +class TestExportYamlRejectsNonMappings(unittest.TestCase): + """Non-mapping input fails loudly, through the public wrapper.""" + + def setUp(self): + self.tmpdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True) + + def _path(self, name="out.yaml"): + return os.path.join(self.tmpdir, name) + + def test_fixture_tables_are_populated(self): + """Guard against a vacuous suite. + + Every test below iterates a table; emptying or renaming one would let + those loops pass without asserting anything. + """ + self.assertGreaterEqual(len(NON_MAPPINGS), 9) + self.assertEqual(set(METHODS), {"semantic_network", "schema"}) + + def test_non_mapping_raises_processing_error(self): + for method in METHODS: + for label, payload in NON_MAPPINGS.items(): + with self.subTest(method=method, case=label): + with self.assertRaises(ProcessingError): + export_yaml(payload, self._path(), method=method) + + def test_error_names_the_offending_type_and_expected_keys(self): + """The message must be actionable, not just the right exception type.""" + for method, spec in METHODS.items(): + with self.subTest(method=method): + with self.assertRaises(ProcessingError) as ctx: + export_yaml([{"id": "1"}], self._path(), method=method) + message = str(ctx.exception) + self.assertIn("list", message) + self.assertIn(spec["expected_key"], message) + + def test_no_file_is_written_when_input_is_rejected(self): + """A rejected export must not leave a partial or empty artefact.""" + for method in METHODS: + with self.subTest(method=method): + path = self._path(f"{method}_rejected.yaml") + with self.assertRaises(ProcessingError): + export_yaml([{"id": "1"}], path, method=method) + self.assertFalse(os.path.exists(path)) + + def test_exporter_classes_reject_non_mappings_directly(self): + """Validation lives in the exporters, not only the convenience wrapper. + + Callers using the classes directly get the same contract. + """ + for label, payload in NON_MAPPINGS.items(): + with self.subTest(exporter="SemanticNetworkYAMLExporter", case=label): + with self.assertRaises(ProcessingError): + SemanticNetworkYAMLExporter().export_semantic_network(payload) + with self.subTest(exporter="YAMLSchemaExporter", case=label): + with self.assertRaises(ProcessingError): + YAMLSchemaExporter().export_ontology_schema(payload) + + +class TestExportYamlStillAcceptsMappings(unittest.TestCase): + """Everything that exported before must still export.""" + + def setUp(self): + self.tmpdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True) + + def _path(self, name="out.yaml"): + return os.path.join(self.tmpdir, name) + + def _load(self, path): + with open(path, encoding="utf-8") as handle: + return yaml.safe_load(handle) + + def test_valid_mapping_exports_for_each_method(self): + for method, spec in METHODS.items(): + with self.subTest(method=method): + path = self._path(f"{method}.yaml") + export_yaml(spec["valid"], path, method=method) + self.assertTrue(os.path.exists(path)) + loaded = self._load(path) + self.assertIn(spec["top_level_key"], loaded) + + def test_semantic_network_records_survive_the_round_trip(self): + path = self._path("network.yaml") + export_yaml(METHODS["semantic_network"]["valid"], path) + loaded = self._load(path) + self.assertEqual(loaded["entities"], [{"id": "1", "name": "Acme"}]) + + def test_empty_mapping_is_still_accepted(self): + """An empty dict is a mapping; rejecting it would be a behaviour change.""" + for method in METHODS: + with self.subTest(method=method): + path = self._path(f"{method}_empty.yaml") + export_yaml({}, path, method=method) + self.assertTrue(os.path.exists(path)) + + def test_mapping_subclasses_are_accepted(self): + """Validation is by Mapping, not dict, so these must keep working.""" + valid = METHODS["semantic_network"]["valid"] + subclasses = { + "OrderedDict": OrderedDict(valid), + "defaultdict": defaultdict(list, valid), + } + for label, payload in subclasses.items(): + with self.subTest(case=label): + path = self._path(f"{label}.yaml") + export_yaml(payload, path) + loaded = self._load(path) + self.assertEqual(loaded["entities"], valid["entities"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/export/test_yaml_exporter_key_recognition.py b/tests/export/test_yaml_exporter_key_recognition.py new file mode 100644 index 00000000..d7dc2839 --- /dev/null +++ b/tests/export/test_yaml_exporter_key_recognition.py @@ -0,0 +1,434 @@ +"""Tests for YAML export key recognition (issue #953). + +``SemanticNetworkYAMLExporter`` built its output from ``.get(key, [])`` +lookups, so a mapping keyed by anything it did not read -- an ``export_json`` +envelope, a typo'd 'entitys', ``ContextGraph.to_dict()``'s 'nodes'/'edges' -- +serialized to a structurally valid file with every collection empty. Nothing +signalled the loss: no exception, no warning, and the progress log reported a +completed export. ``YAMLSchemaExporter`` had the same defect over a different +key set. + +The exporters are run for real rather than mocked, and the written files are +parsed back, since the behaviour under test is what actually lands on disk. +""" + +from pathlib import Path + +import pytest +import yaml + +from semantica.context.context_graph import ContextGraph +from semantica.export.methods import export_json, export_yaml +from semantica.export.yaml_exporter import ( + SemanticNetworkYAMLExporter, + YAMLSchemaExporter, +) +from semantica.utils.exceptions import ProcessingError, ValidationError + +ENTITIES = [{"id": "e1", "name": "Acme"}, {"id": "e2", "name": "Beta"}] +RELATIONSHIPS = [{"id": "r1", "source": "e1", "target": "e2", "type": "PARTNER"}] +TRIPLETS = [{"subject": "e1", "predicate": "partner_of", "object": "e2"}] + + +def _load(path): + with open(path, "r", encoding="utf-8") as handle: + return yaml.safe_load(handle) + + +class TestSemanticNetworkKeyRecognition: + """An unrecognized mapping is refused instead of silently emptied.""" + + def test_export_json_envelope_is_rejected(self, tmp_path): + """The realistic trigger: re-exporting an export_json payload. + + ``export_json`` wraps records as ``{"data": [...], "count": N, + "metadata": {...}}``. Feeding that straight to ``export_yaml`` used to + write a file with every record gone. Note the envelope's 'metadata' + key is deliberately not enough to make the payload recognized -- + treating it as sufficient would readmit exactly this case. + """ + json_path = tmp_path / "records.json" + export_json(ENTITIES, json_path) + envelope = yaml.safe_load(json_path.read_text(encoding="utf-8")) + assert "data" in envelope and "metadata" in envelope + + yaml_path = tmp_path / "records.yaml" + with pytest.raises(ValidationError) as excinfo: + export_yaml(envelope, yaml_path) + + message = str(excinfo.value) + assert "'data'" in message, "error should name the supplied keys" + assert "'entities'" in message, "error should name the expected keys" + assert not yaml_path.exists(), "a rejected export must write nothing" + + @pytest.mark.parametrize( + "payload", + [ + {"records": ENTITIES}, + {"entitys": ENTITIES}, + {"data": ENTITIES}, + {"metadata": {"source": "test"}}, + ], + ids=["records", "typo", "data", "metadata-only"], + ) + def test_unrecognized_mappings_are_rejected(self, payload): + exporter = SemanticNetworkYAMLExporter() + with pytest.raises(ValidationError): + exporter.export_semantic_network(payload) + + @pytest.mark.parametrize( + "payload", + [ + {"entities": [], "data": ENTITIES}, + {"nodes": [], "edges": [], "records": ENTITIES}, + {"triplets": [], "data": ENTITIES, "metadata": {"source": "test"}}, + ], + ids=["entities-empty", "nodes-edges-empty", "triplets-empty"], + ) + def test_recognized_but_empty_with_records_elsewhere_is_rejected(self, payload): + """Presence of a recognized key is not proof the records survived. + + ``{"entities": [], "data": [...]}`` clears a presence-only check and + still resolves to empty, dropping everything under 'data' -- the same + silent-empty export by a narrower route. + """ + exporter = SemanticNetworkYAMLExporter() + with pytest.raises(ValidationError) as excinfo: + exporter.export_semantic_network(payload) + + message = str(excinfo.value) + assert "holds records" in message + assert "'entities'" in message, "error should name where records belong" + + def test_empty_graph_with_non_record_keys_still_exports(self, tmp_path): + """The rejection must key on dropped *records*, not on unread keys. + + ``ContextGraph.to_dict()`` always carries a populated 'statistics' + dict, so an empty graph would be refused if any unread key counted. + """ + graph = ContextGraph() + path = tmp_path / "empty_graph.yaml" + export_yaml(graph.to_dict(), path) + + written = _load(path) + assert written["entities"] == [] + assert written["relationships"] == [] + + def test_empty_mapping_still_exports(self, tmp_path): + """An empty graph is legitimate and carries nothing that could be lost.""" + path = tmp_path / "empty.yaml" + export_yaml({}, path) + + written = _load(path) + assert written["entities"] == [] + assert written["relationships"] == [] + assert written["triplets"] == [] + + def test_recognized_keys_still_export(self, tmp_path): + path = tmp_path / "network.yaml" + export_yaml( + { + "entities": ENTITIES, + "relationships": RELATIONSHIPS, + "triplets": TRIPLETS, + "metadata": {"source": "test"}, + }, + path, + ) + + written = _load(path) + assert written["entities"] == ENTITIES + assert written["relationships"] == RELATIONSHIPS + assert written["triplets"] == TRIPLETS + assert written["metadata"]["source"] == "test" + + def test_nodes_edges_alias_exports_records(self, tmp_path): + path = tmp_path / "aliased.yaml" + export_yaml({"nodes": ENTITIES, "edges": RELATIONSHIPS}, path) + + written = _load(path) + assert written["entities"] == ENTITIES + assert written["relationships"] == RELATIONSHIPS + + def test_context_graph_to_dict_round_trips(self, tmp_path): + """The most direct path from this library's own graph type to YAML. + + Built from a real ``ContextGraph`` rather than a hand-written + 'nodes'/'edges' dict, so the test breaks if ``to_dict()`` changes + vocabulary. + """ + graph = ContextGraph() + graph.add_node("n1", node_type="Person", content="Alice") + graph.add_node("n2", node_type="Org", content="Acme") + graph.add_edge("n1", "n2", "WORKS_FOR") + + path = tmp_path / "context.yaml" + export_yaml(graph.to_dict(), path) + + written = _load(path) + assert len(written["entities"]) == 2 + assert len(written["relationships"]) == 1 + + def test_conflicting_spellings_are_refused(self): + """Two populated spellings of one collection: no basis to pick either.""" + exporter = SemanticNetworkYAMLExporter() + with pytest.raises(ValidationError): + exporter.export_semantic_network( + {"entities": ENTITIES, "nodes": [{"id": "other"}]} + ) + + def test_non_mapping_raises_processing_error(self): + """A wrong type is a different failure from a wrong-keyed mapping. + + ProcessingError says the object cannot be exported at all; + ValidationError says the mapping's contents are unusable. Pinned here + so the two do not quietly converge. + """ + exporter = SemanticNetworkYAMLExporter() + with pytest.raises(ProcessingError): + exporter.export_semantic_network(ENTITIES) + + def test_rejected_export_creates_no_output_directory(self, tmp_path): + """Validation runs before the output directory is created.""" + target = tmp_path / "nested" / "out.yaml" + exporter = SemanticNetworkYAMLExporter() + + with pytest.raises(ValidationError): + exporter.export({"data": ENTITIES}, target) + + assert not target.parent.exists() + + +class TestPipelineExportKeyRecognition: + """export_for_pipeline read the same defaulted lookups, so it had the bug too.""" + + def test_unrecognized_mapping_is_rejected(self): + exporter = SemanticNetworkYAMLExporter() + with pytest.raises(ValidationError): + exporter.export_for_pipeline({"data": ENTITIES}) + + def test_non_mapping_raises_processing_error(self): + exporter = SemanticNetworkYAMLExporter() + with pytest.raises(ProcessingError): + exporter.export_for_pipeline(ENTITIES) + + def test_aliases_resolve_into_the_semantic_network(self): + exporter = SemanticNetworkYAMLExporter() + written = yaml.safe_load( + exporter.export_for_pipeline({"nodes": ENTITIES, "edges": RELATIONSHIPS}) + ) + + assert written["semantic_network"]["entities"] == ENTITIES + assert written["semantic_network"]["relationships"] == RELATIONSHIPS + + def test_metadata_is_preserved(self): + exporter = SemanticNetworkYAMLExporter() + written = yaml.safe_load( + exporter.export_for_pipeline( + {"entities": ENTITIES, "metadata": {"source": "test"}} + ) + ) + + assert written["metadata"]["source"] == "test" + assert written["semantic_network"]["entities"] == ENTITIES + + +class TestSchemaKeyRecognition: + """method="schema" emitted empty classes/properties/namespaces the same way.""" + + def test_unrecognized_mapping_is_rejected(self, tmp_path): + path = tmp_path / "schema.yaml" + with pytest.raises(ValidationError) as excinfo: + export_yaml({"nodes": [{"id": "1"}]}, path, method="schema") + + message = str(excinfo.value) + assert "'nodes'" in message + assert "'classes'" in message + assert not path.exists() + + def test_non_mapping_raises_processing_error(self): + exporter = YAMLSchemaExporter() + with pytest.raises(ProcessingError): + exporter.export_ontology_schema([{"id": "1"}]) + + def test_recognized_but_empty_with_records_elsewhere_is_rejected(self): + """The schema path had the same presence-only hole.""" + exporter = YAMLSchemaExporter() + with pytest.raises(ValidationError) as excinfo: + exporter.export_ontology_schema({"classes": [], "nodes": [{"id": "1"}]}) + + assert "holds records" in str(excinfo.value) + + def test_ontology_metadata_without_records_still_exports(self): + """A schema described only by its identity is not a dropped export.""" + exporter = YAMLSchemaExporter() + written = yaml.safe_load( + exporter.export_ontology_schema( + {"uri": "http://example.org/o", "classes": []} + ) + ) + + assert written["ontology"]["uri"] == "http://example.org/o" + assert written["classes"] == [] + + def test_empty_mapping_still_exports(self, tmp_path): + path = tmp_path / "schema.yaml" + export_yaml({}, path, method="schema") + + written = _load(path) + assert written["classes"] == [] + assert written["properties"] == [] + assert written["namespaces"] == {} + + @pytest.mark.parametrize( + "payload", + [ + {"classes": ["Person"], "properties": ["WORKS_FOR"]}, + {"namespaces": {"ex": "http://example.org/"}}, + {"uri": "http://example.org/ontology"}, + ], + ids=["classes-properties", "namespaces-only", "uri-only"], + ) + def test_recognized_keys_still_export(self, payload, tmp_path): + path = tmp_path / "schema.yaml" + export_yaml(payload, path, method="schema") + + written = _load(path) + assert written["classes"] == payload.get("classes", []) + assert written["properties"] == payload.get("properties", []) + assert written["ontology"]["uri"] == payload.get("uri", "") + + # ── Fix regression: scalar recognized keys must not short-circuit the ── + # ── dropped-records check (version, uri, title, description). ────────── + + @pytest.mark.parametrize( + "scalar_key, scalar_value", + [ + ("version", "1.0"), + ("uri", "http://example.org/ontology"), + ("title", "My Ontology"), + ("description", "A test ontology"), + ], + ids=["version", "uri", "title", "description"], + ) + def test_scalar_recognized_key_does_not_excuse_records_under_unread_key( + self, scalar_key, scalar_value + ): + """A truthy scalar such as version='1.0' must not silence the dropped- + records check. Before the fix, any truthy value from _SCHEMA_KEYS + would make _require_nothing_dropped believe something resolved and + return early, silently discarding a list under an unread key. + """ + exporter = YAMLSchemaExporter() + with pytest.raises(ValidationError) as excinfo: + exporter.export_ontology_schema( + {scalar_key: scalar_value, "nodes": [{"id": "c1"}]} + ) + assert "holds records" in str(excinfo.value), str(excinfo.value) + + def test_valid_classes_with_scalar_metadata_is_accepted(self): + """classes/properties populated alongside version/uri must still work.""" + exporter = YAMLSchemaExporter() + written = yaml.safe_load( + exporter.export_ontology_schema( + { + "classes": [{"id": "Person"}], + "properties": [{"id": "name"}], + "version": "2.0", + "uri": "http://example.org/o", + } + ) + ) + assert written["classes"] == [{"id": "Person"}] + assert written["properties"] == [{"id": "name"}] + assert written["ontology"]["version"] == "2.0" + assert written["ontology"]["uri"] == "http://example.org/o" + + +class TestFailureIsObservable: + """The complaint in #953 was that the logs affirmatively reported success.""" + + def test_no_success_is_logged_for_a_rejected_export(self, tmp_path, caplog): + path = tmp_path / "out.yaml" + + with caplog.at_level("DEBUG"): + with pytest.raises(ValidationError): + export_yaml({"data": ENTITIES}, path) + + assert "Exported YAML to" not in caplog.text + assert any( + record.levelname in ("WARNING", "ERROR", "CRITICAL") + for record in caplog.records + ), "a rejected export should leave something at warning or above" + + +class _RecordingTracker: + """Records the exporter's own progress calls, which are what is under test.""" + + def __init__(self): + self.stopped = [] + self._next_id = 0 + + def start_tracking(self, **kwargs): + self._next_id += 1 + return str(self._next_id) + + def update_tracking(self, tracking_id, **kwargs): + pass + + def stop_tracking(self, tracking_id, status=None, message=None): + self.stopped.append((status, message)) + + +class TestProgressReflectsTheWrite: + """Serialization completing is not the same as the file landing on disk.""" + + def test_failed_write_is_not_reported_as_completed(self, tmp_path): + """A write failure after serialization must not leave a clean tracker. + + The path's parent is an existing *file*, so directory creation fails + after `export_semantic_network` has already reported its own + completion. + """ + blocker = tmp_path / "blocker" + blocker.write_text("not a directory", encoding="utf-8") + target = blocker / "nested" / "out.yaml" + + exporter = SemanticNetworkYAMLExporter() + tracker = _RecordingTracker() + exporter.progress_tracker = tracker + + with pytest.raises(OSError): + exporter.export({"entities": ENTITIES}, target) + + assert not target.exists() + statuses = [status for status, _ in tracker.stopped] + assert "failed" in statuses, f"write failure went unreported: {tracker.stopped}" + assert not any( + status == "completed" and "Exported YAML" in (message or "") + for status, message in tracker.stopped + ), "no span may claim a completed export when nothing was written" + + def test_successful_write_is_reported_as_completed(self, tmp_path): + target = tmp_path / "out.yaml" + exporter = SemanticNetworkYAMLExporter() + tracker = _RecordingTracker() + exporter.progress_tracker = tracker + + exporter.export({"entities": ENTITIES}, target) + + assert target.exists() + assert all(status == "completed" for status, _ in tracker.stopped) + assert any( + "Exported YAML" in (message or "") for _, message in tracker.stopped + ), "the write should report its own completion, not just serialization" + + +class TestUnaffectedExporters: + """export_json's own behaviour is untouched -- only the YAML path changed.""" + + def test_export_json_still_accepts_a_bare_list(self, tmp_path): + path = tmp_path / "records.json" + export_json(ENTITIES, path) + + assert Path(path).exists() diff --git a/tests/utils/test_normalize_graph_payload.py b/tests/utils/test_normalize_graph_payload.py new file mode 100644 index 00000000..02b53219 --- /dev/null +++ b/tests/utils/test_normalize_graph_payload.py @@ -0,0 +1,487 @@ +"""Tests for the shared graph-payload normalizer (issue #956). + +Graph payloads circulate under two vocabularies -- 'entities'/'relationships' +and 'nodes'/'edges' -- and consumers each reconciled them locally with at +least three competing idioms. The same payload could therefore be exported, +silently dropped, or rejected depending on which consumer read it: +``export_lpg`` dropped every entity when 'nodes' was present but empty, which +is precisely the shape ``JSONExporter`` emits. + +The end-to-end assertions run the real exporters rather than mocking them, +since the behaviour under test is that the exporters now agree. +""" + +import os +import shutil +import tempfile +import unittest +from dataclasses import dataclass + +from semantica.export import methods as export_methods +from semantica.utils import normalize_graph_payload +from semantica.utils.exceptions import ValidationError + +ENTITY = {"id": "e1", "name": "Acme"} +RELATIONSHIP = {"id": "r1", "source": "e1", "target": "e2"} + + +class TestVocabularyResolution(unittest.TestCase): + def test_canonical_keys_pass_through(self): + result = normalize_graph_payload( + {"entities": [ENTITY], "relationships": [RELATIONSHIP]} + ) + self.assertEqual(result["entities"], [ENTITY]) + self.assertEqual(result["relationships"], [RELATIONSHIP]) + self.assertEqual(result["triplets"], []) + + def test_aliases_are_mapped_to_canonical_keys(self): + result = normalize_graph_payload({"nodes": [ENTITY], "edges": [RELATIONSHIP]}) + self.assertEqual(result["entities"], [ENTITY]) + self.assertEqual(result["relationships"], [RELATIONSHIP]) + + def test_empty_alias_does_not_mask_a_populated_canonical_key(self): + """The JSONExporter round-trip shape, and the #956 data-loss case.""" + result = normalize_graph_payload( + {"entities": [ENTITY], "nodes": [], "relationships": [], "edges": []} + ) + self.assertEqual(result["entities"], [ENTITY]) + + def test_empty_canonical_key_does_not_mask_a_populated_alias(self): + result = normalize_graph_payload({"entities": [], "nodes": [ENTITY]}) + self.assertEqual(result["entities"], [ENTITY]) + + def test_identical_spellings_are_accepted(self): + result = normalize_graph_payload({"entities": [ENTITY], "nodes": [ENTITY]}) + self.assertEqual(result["entities"], [ENTITY]) + + def test_conflicting_spellings_are_refused(self): + """No basis to prefer either, and picking one would lose the other.""" + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload( + {"entities": [ENTITY], "nodes": [{"id": "different"}]} + ) + message = str(ctx.exception) + self.assertIn("entities", message) + self.assertIn("nodes", message) + + def test_reordered_identical_spellings_are_accepted(self): + """Same records, different order, is not a conflict. + + A caller round-tripping through a dict-keyed cache or a set has no + reason to preserve list order; comparing spellings with plain list + equality rejected this as if the records differed. + """ + other = {"id": "e2", "name": "Beta"} + result = normalize_graph_payload( + {"entities": [ENTITY, other], "nodes": [other, ENTITY]} + ) + self.assertCountEqual(result["entities"], [ENTITY, other]) + + def test_reordered_spellings_with_duplicate_records_still_conflict(self): + """Multiset comparison must still catch a real count mismatch.""" + with self.assertRaises(ValidationError): + normalize_graph_payload({"entities": [ENTITY, ENTITY], "nodes": [ENTITY]}) + + def test_triplets_are_carried_through(self): + result = normalize_graph_payload({"triplets": [{"s": "a", "p": "b", "o": "c"}]}) + self.assertEqual(result["triplets"], [{"s": "a", "p": "b", "o": "c"}]) + + def test_missing_collections_default_to_empty_lists(self): + result = normalize_graph_payload({"entities": [ENTITY]}) + self.assertEqual(result["relationships"], []) + self.assertEqual(result["triplets"], []) + + def test_result_does_not_alias_the_input_collections(self): + payload = {"entities": [ENTITY]} + result = normalize_graph_payload(payload) + result["entities"].append({"id": "e2"}) + self.assertEqual(len(payload["entities"]), 1) + + +class TestUnrecognizedInput(unittest.TestCase): + def test_unrecognized_keys_raise_by_default(self): + for payload in ({"data": [ENTITY]}, {"records": [ENTITY]}, {"foo": "bar"}): + with self.subTest(payload=payload): + with self.assertRaises(ValidationError): + normalize_graph_payload(payload) + + def test_error_names_supplied_and_expected_keys(self): + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({"data": [ENTITY]}) + message = str(ctx.exception) + self.assertIn("data", message) + self.assertIn("entities", message) + self.assertIn("nodes", message) + + def test_empty_mapping_is_accepted(self): + """An empty graph is legitimate and carries nothing that could be lost.""" + result = normalize_graph_payload({}) + self.assertEqual(result, {"entities": [], "relationships": [], "triplets": []}) + + def test_non_mapping_input_raises(self): + for payload in ([ENTITY], (ENTITY,), "entities", 42, None): + with self.subTest(payload=repr(payload)): + with self.assertRaises(ValidationError): + normalize_graph_payload(payload) + + +class TestExportersAgree(unittest.TestCase): + """The divergence from #956, run against the real exporters.""" + + # export_csv is excluded: it writes entities/relationships/nodes/edges to + # four separate files by design, so it is not resolving two spellings of + # one collection and is out of scope for this change. + EXPORTERS = ("export_json", "export_arango", "export_neo4j_csv", "export_lpg") + + def setUp(self): + self.tmpdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True) + + def _export_and_read(self, name, payload): + outdir = os.path.join(self.tmpdir, name) + os.makedirs(outdir, exist_ok=True) + getattr(export_methods, name)(payload, os.path.join(outdir, "out")) + blob = "" + for root, _, files in os.walk(outdir): + for filename in files: + with open(os.path.join(root, filename), errors="ignore") as handle: + blob += handle.read() + return blob + + def test_exporter_list_is_populated(self): + """Guard against a vacuous suite if the list is emptied.""" + self.assertGreaterEqual(len(self.EXPORTERS), 4) + + def test_every_exporter_keeps_records_when_an_alias_is_empty(self): + payload = { + "entities": [ENTITY], + "nodes": [], + "relationships": [], + "edges": [], + } + for name in self.EXPORTERS: + with self.subTest(exporter=name): + self.assertIn( + "Acme", + self._export_and_read(name, payload), + f"{name} dropped the entity when 'nodes' was present but empty", + ) + + def test_every_exporter_accepts_the_alias_vocabulary(self): + payload = {"nodes": [ENTITY], "edges": []} + for name in self.EXPORTERS: + with self.subTest(exporter=name): + self.assertIn( + "Acme", + self._export_and_read(name, payload), + f"{name} dropped the entity supplied as 'nodes'", + ) + + def test_every_exporter_raises_processing_error_for_non_mapping_input(self): + """A wrong-type payload is rejected the same way everywhere. + + export_yaml and export_neo4j_csv raised ProcessingError for a bare + list; export_lpg and export_arango called normalize_graph_payload() + directly with no type guard, so they alone raised ValidationError + (from inside the resolver) for the identical mistake. + """ + from semantica.utils.exceptions import ProcessingError + + for name in ("export_arango", "export_neo4j_csv", "export_lpg"): + with self.subTest(exporter=name): + outdir = os.path.join(self.tmpdir, name + "_bad_type") + os.makedirs(outdir, exist_ok=True) + with self.assertRaises(ProcessingError): + getattr(export_methods, name)([ENTITY], os.path.join(outdir, "out")) + + def test_every_exporter_converts_object_shaped_records(self): + """A dataclass record must not merely pass validation. + + normalize_graph_payload() accepts dataclass/attribute-bearing + records (Neo4jCSVExporter reads them off attributes), but + export_lpg and export_arango read records with ``.get(...)``. A + record that passed validation unconverted crashed with a raw + AttributeError once used -- the exact failure the boundary exists + to prevent. + """ + + @dataclass + class Node: + id: str + name: str + + payload = {"entities": [Node(id="e1", name="Acme")], "relationships": []} + for name in ("export_arango", "export_neo4j_csv", "export_lpg"): + with self.subTest(exporter=name): + self.assertIn("Acme", self._export_and_read(name, payload)) + + def test_neo4j_accepts_non_dict_mappings(self): + """Neo4jCSVExporter's mapping path must not be narrower than the rest. + + _normalize_graph checked isinstance(graph, dict), so a non-dict + Mapping (a MappingProxyType, a ChainMap) fell into the + object-attribute branch and was rejected as an unrecognized object, + even though the identical payload exports fine via LPG/Arango/YAML. + """ + import types + + payload = types.MappingProxyType({"entities": [ENTITY], "relationships": []}) + self.assertIn("Acme", self._export_and_read("export_neo4j_csv", payload)) + + +class TestRecordsCannotBeDroppedSilently(unittest.TestCase): + """Presence of a recognized key is not proof the records survived. + + ``{"entities": [], "data": [...]}`` clears a presence-only check and still + resolves to empty, so the records under 'data' would be dropped with no + signal -- the same failure the recognition check exists to prevent. + """ + + def test_empty_recognized_key_does_not_excuse_records_elsewhere(self): + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({"entities": [], "data": [ENTITY]}) + + message = str(ctx.exception) + self.assertIn("'data'", message) + self.assertIn("holds records", message) + + def test_check_applies_to_every_recognized_spelling(self): + for key in ("entities", "nodes", "relationships", "edges", "triplets"): + with self.subTest(key=key): + with self.assertRaises(ValidationError): + normalize_graph_payload({key: [], "records": [ENTITY]}) + + def test_non_record_keys_are_not_mistaken_for_dropped_records(self): + """ContextGraph.to_dict() always carries 'statistics'. + + An empty graph must stay exportable, so only a non-empty list counts + as evidence that records were dropped. + """ + result = normalize_graph_payload( + {"nodes": [], "edges": [], "statistics": {"node_count": 0}} + ) + + self.assertEqual(result["entities"], []) + self.assertEqual(result["relationships"], []) + + def test_records_alongside_a_populated_collection_are_not_refused(self): + """Something resolved, so the export is not silently empty.""" + result = normalize_graph_payload({"entities": [ENTITY], "statistics": {"n": 1}}) + + self.assertEqual(result["entities"], [ENTITY]) + + +class TestCollectionValuesAreValidated(unittest.TestCase): + """A recognized key is not proof its value is a collection of records. + + Resolving on truthiness alone let ``{"entities": "abc"}`` through as three + single-character "records" and let ``{"entities": 42}`` surface as a raw + ``TypeError`` from ``list()`` inside an exporter, naming the exporter + rather than the payload key at fault. Both are rejected here, at the + boundary that owns the question. + """ + + COLLECTION_KEYS = ("entities", "nodes", "relationships", "edges", "triplets") + + # Every public export path that reads its payload through the normalizer. + # export_json is excluded: it treats the payload as opaque records rather + # than resolving graph collections, so it never calls the normalizer. + NORMALIZING_EXPORTERS = ( + "export_arango", + "export_neo4j_csv", + "export_lpg", + "export_yaml", + ) + + def test_string_value_is_not_treated_as_a_collection(self): + for key in self.COLLECTION_KEYS: + with self.subTest(key=key): + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({key: "abc"}) + message = str(ctx.exception) + self.assertIn(f"'{key}'", message) + self.assertIn("str", message) + + def test_bytes_value_is_not_treated_as_a_collection(self): + for value in (b"abc", bytearray(b"abc")): + with self.subTest(value=repr(value)): + with self.assertRaises(ValidationError): + normalize_graph_payload({"entities": value}) + + def test_scalar_value_raises_validation_error_not_type_error(self): + for key in self.COLLECTION_KEYS: + for value in (42, 3.5, True, object()): + with self.subTest(key=key, value=repr(value)): + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({key: value}) + self.assertIn(f"'{key}'", str(ctx.exception)) + + def test_mapping_value_is_not_treated_as_a_collection(self): + """``{"nodes": {"id": "n1"}}`` -- a single record, or an ID index.""" + for payload in ( + {"nodes": {"id": "n1"}}, + {"entities": {"e1": ENTITY}}, + {"edges": {"id": "r1"}}, + ): + with self.subTest(payload=payload): + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload(payload) + self.assertIn("mapping", str(ctx.exception)) + + def test_non_record_elements_are_rejected(self): + for value in (["Acme"], [ENTITY, "Acme"], [42], [None], [[ENTITY]]): + with self.subTest(value=repr(value)): + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({"entities": value}) + self.assertIn("'entities'", str(ctx.exception)) + + def test_error_names_the_offending_index(self): + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({"entities": [ENTITY, ENTITY, "Acme"]}) + self.assertIn("index 2", str(ctx.exception)) + + def test_object_records_are_accepted(self): + """Attribute-bearing objects are accepted and converted to dicts. + + LPGExporter and ArangoAQLExporter read records with ``.get(...)``, so + an object record that merely passed validation unconverted would + still crash with AttributeError once used; the boundary converts it. + """ + + class Node: + def __init__(self): + self.id = "e1" + self.name = "Acme" + + node = Node() + result = normalize_graph_payload({"entities": [node]}) + self.assertEqual(result["entities"], [{"id": "e1", "name": "Acme"}]) + + def test_dataclass_records_are_accepted(self): + @dataclass + class Node: + id: str + + node = Node(id="e1") + result = normalize_graph_payload({"entities": [node]}) + self.assertEqual(result["entities"], [{"id": "e1"}]) + + def test_tuple_collections_are_accepted_and_materialized(self): + result = normalize_graph_payload({"entities": (ENTITY,)}) + self.assertEqual(result["entities"], [ENTITY]) + + def test_none_is_read_as_an_absent_collection(self): + """JSON round-trips an absent collection to null.""" + result = normalize_graph_payload( + {"entities": None, "relationships": [RELATIONSHIP]} + ) + self.assertEqual(result["entities"], []) + self.assertEqual(result["relationships"], [RELATIONSHIP]) + + def test_null_collection_still_cannot_hide_dropped_records(self): + with self.assertRaises(ValidationError): + normalize_graph_payload({"entities": None, "data": [ENTITY]}) + + def test_every_spelling_is_validated_not_just_the_winner(self): + """A malformed alias is a defect even when the canonical key resolves.""" + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({"entities": [ENTITY], "nodes": "abc"}) + self.assertIn("'nodes'", str(ctx.exception)) + + def test_malformed_value_reaches_no_exporter(self): + """The end-to-end half: no exporter sees a TypeError from list().""" + tmpdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmpdir, ignore_errors=True) + + for name in self.NORMALIZING_EXPORTERS: + for value in ("abc", 42, {"id": "n1"}): + with self.subTest(exporter=name, value=repr(value)): + outdir = os.path.join(tmpdir, f"{name}_{type(value).__name__}") + os.makedirs(outdir, exist_ok=True) + with self.assertRaises(ValidationError): + getattr(export_methods, name)( + {"entities": value}, os.path.join(outdir, "out") + ) + + +class TestIsRecordBoundary(unittest.TestCase): + """_is_record gates the validation boundary introduced by this PR. + + Modules and class/type objects carry ``__dict__`` but are not graph + records. Passing them through previously produced ``AttributeError`` + inside exporters rather than a ``ValidationError`` at the boundary. + """ + + def test_python_module_in_entities_raises_validation_error(self): + """import math; {"entities": [math]} must be rejected at the boundary.""" + import math + + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({"entities": [math]}) + self.assertIn("'entities'", str(ctx.exception)) + + def test_class_object_in_entities_raises_validation_error(self): + """A class (type object) is not a graph record.""" + + class MyNode: + pass + + with self.assertRaises(ValidationError) as ctx: + normalize_graph_payload({"entities": [MyNode]}) + self.assertIn("'entities'", str(ctx.exception)) + + def test_user_defined_instance_with_attributes_is_accepted(self): + """Attribute-bearing instances are the legitimate use-case, converted + to a dict so every exporter -- not just Neo4jCSVExporter -- can read + it with ``.get(...)``.""" + + class Node: + def __init__(self): + self.id = "n1" + self.name = "Alice" + + node = Node() + result = normalize_graph_payload({"entities": [node]}) + self.assertEqual(result["entities"], [{"id": "n1", "name": "Alice"}]) + + def test_dataclass_instance_is_accepted(self): + """Dataclasses are a common record type used by Neo4jCSVExporter, + converted to a dict at the boundary so LPGExporter and + ArangoAQLExporter can read it too.""" + node = dataclass_node() + result = normalize_graph_payload({"entities": [node]}) + self.assertEqual(result["entities"], [{"id": "dc1"}]) + + def test_mapping_record_is_accepted(self): + """Plain dicts are the canonical record shape.""" + result = normalize_graph_payload({"entities": [ENTITY]}) + self.assertEqual(result["entities"], [ENTITY]) + + def test_module_rejected_through_normalizing_exporter(self): + """End-to-end: a module element must not reach an exporter's internals.""" + import math + + tmpdir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, tmpdir, ignore_errors=True) + + for name in ("export_arango", "export_neo4j_csv", "export_lpg"): + with self.subTest(exporter=name): + outdir = os.path.join(tmpdir, name) + os.makedirs(outdir, exist_ok=True) + with self.assertRaises(ValidationError): + getattr(export_methods, name)( + {"entities": [math]}, os.path.join(outdir, "out") + ) + + +@dataclass +class _DataclassNode: + id: str + + +def dataclass_node(): + return _DataclassNode(id="dc1") + + +if __name__ == "__main__": + unittest.main() From 2f04bc01a32552a2310363cc03edc66651572ba8 Mon Sep 17 00:00:00 2001 From: Accute9 Date: Sat, 15 Aug 2026 20:37:56 -0400 Subject: [PATCH 02/19] route spaCy model loads through process cache --- semantica/split/methods.py | 3 ++- semantica/split/semantic_chunker.py | 5 ++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/semantica/split/methods.py b/semantica/split/methods.py index 8c338dc6..f2b3f525 100644 --- a/semantica/split/methods.py +++ b/semantica/split/methods.py @@ -93,6 +93,7 @@ from ..utils.exceptions import ProcessingError from ..utils.helpers import safe_import from ..utils.logging import get_logger from .semantic_chunker import Chunk +from ..semantic_extract.methods import load_spacy_model logger = get_logger("split_methods") @@ -336,7 +337,7 @@ def split_by_sentences( # Try spaCy first if SPACY_AVAILABLE and kwargs.get("use_spacy", True): try: - nlp = spacy.load("en_core_web_sm") + nlp = load_spacy_model("en_core_web_sm") doc = nlp(text) sentences = [sent.text for sent in doc.sents] except Exception: diff --git a/semantica/split/semantic_chunker.py b/semantica/split/semantic_chunker.py index 079ba976..5d712627 100644 --- a/semantica/split/semantic_chunker.py +++ b/semantica/split/semantic_chunker.py @@ -35,6 +35,8 @@ from ..utils.exceptions import ProcessingError from ..utils.helpers import safe_import from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker +from ..semantic_extract.methods import load_spacy_model + spacy, SPACY_AVAILABLE = safe_import("spacy") @@ -79,7 +81,8 @@ class SemanticChunker: if SPACY_AVAILABLE: model_name = config.get("model", "en_core_web_sm") try: - self.nlp = spacy.load(model_name) + # self.nlp = spacy.load(model_name) + self.nlp = load_spacy_model(model_name) except OSError: self.logger.warning( f"spaCy model {model_name} not found. Using fallback chunking." From 83649f68219ef069f1a12f58424a99abb4f2639d Mon Sep 17 00:00:00 2001 From: Accute9 Date: Sat, 15 Aug 2026 20:56:39 -0400 Subject: [PATCH 03/19] forgot to remove comment --- semantica/split/semantic_chunker.py | 1 - 1 file changed, 1 deletion(-) diff --git a/semantica/split/semantic_chunker.py b/semantica/split/semantic_chunker.py index 5d712627..d7f72726 100644 --- a/semantica/split/semantic_chunker.py +++ b/semantica/split/semantic_chunker.py @@ -81,7 +81,6 @@ class SemanticChunker: if SPACY_AVAILABLE: model_name = config.get("model", "en_core_web_sm") try: - # self.nlp = spacy.load(model_name) self.nlp = load_spacy_model(model_name) except OSError: self.logger.warning( From d94d8f6ab83cbfd6efb2782224bda0574a3d9433 Mon Sep 17 00:00:00 2001 From: Shinde vinayak rao patil <119512435+Shindevrp@users.noreply.github.com> Date: Sun, 16 Aug 2026 11:45:43 +0530 Subject: [PATCH 04/19] Feat/crewai integration (#988) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(crewai): add first-class CrewAI integration (#962) Add native CrewAI support so Crew agents can share a ContextGraph and AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching the existing agno integration pattern. - SemanticaKGTool: 5 KG actions (extract_entities, extract_relations, add_to_graph, query_graph, find_related) with sync run()/async arun() - SemanticaDecisionTool: 5 decision-intelligence actions (record_decision, find_precedents, trace_causal_chain, analyze_impact, check_policy) over AgentContext - SemanticaKnowledgeSource: serializes a ContextGraph into crew knowledge storage; bridges legacy load_content() and current validate_content()/aadd() contracts for crewai>=0.80.0 - All classes degrade gracefully when crewai is absent - New pip extra crewai=... included in the all bundle - 70 new tests (stub-based present-case + subprocess degradation path) - Docs: integrations/crewai.md, docs.json nav, README matrix updates * fix(crewai): harden tools against real Semantica dataclass shapes (#962) Bugs found during live testing with crewai 1.15.16: - SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses ('str' object has no attribute 'end_char'): string names were passed to extract_relations(entities=...), which requires Entity objects, and the tool read .name/.source/.target instead of Entity's .text/.label and Relation's .subject/.object. Add shape-agnostic field helpers. - SemanticaDecisionTool() created an AgentContext without a knowledge_graph, so _decision_backend was never set and record_decision raised 'Decision tracking is not enabled'. Wire in a ContextGraph. - record_decision hard-failed when the agent omitted optional fields; fall back to category='general', reasoning='agent decision', outcome='recorded'. Add tests covering real Entity/Relation dataclass shapes and the live auto-created AgentContext path (now 77 crewai tests, 212 total). * fix(crewai): make find_related traverse edges undirected (#962) ContextGraph.get_neighbors only follows outgoing edges, so a node whose only edge is incoming (A -> B) reported no related concepts. Rebuild a bidirectional adjacency from find_edges() in SemanticaKGTool._find_related so 'related' honors both directions. * fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962) - Exclude live graph/context/extractor state from JSON serialization (model_dump(mode="json")) so CrewAI checkpointing no longer raises PydanticSerializationError; model_post_init self-heals defaults on restore - query_graph now searches node content via graph.query() plus id/type - trace_causal_chain returns an explicit error when causal tracing is unavailable instead of substituting similarity precedents; call trace_decision_causality(..., max_depth=...) with the correct kwarg name - find_precedents propagates max_precedents/limit to the backend instead of being silently capped at 10 - Serialize add_to_graph batches under a module lock to prevent concurrent double-counting; skip nameless entities instead of creating repr()-junk nodes - aadd() runs CPU-bound serialization in a thread executor - Mirror crewai args_schema serialize/restore in the conftest stub and add serialization regression tests (crewai: 92 tests) * fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962) - _eval_rule now coerces rule values type-aware: bool("false") was truthy, so 'enabled == false' reported a violation for enabled=false, and string datums like "0.90" were compared lexicographically instead of numerically - _trace_causal_chain no longer raises AttributeError (which escaped _run) when the decision context lacks knowledge_graph; returns honest error JSON - SemanticaKnowledgeSource storage failures log an actionable ERROR; without a configured crew embedder agents previously retrieved nothing silently - add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a process-global one: independent graphs no longer serialize each other and re-entrant extractor callbacks cannot deadlock - entity/relation confidence=None normalizes to 1.0 instead of failing the whole extraction with float(None) - add subprocess integration test against real crewai covering Crew-level serialization round-trip and checkpoint restore (stub tests cannot see it) - docs: embedder requirement for SemanticaKnowledgeSource; resume contract note * fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962) Re-verification against real crewai showed the embedder-missing failure raises ValueError even though storage IS wired, so the old except-ValueError branch mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure. Distinguish by storage presence instead of exception type: storage is None -> DEBUG keep-in-memory (legitimate standalone use); storage wired but save() raises -> actionable ERROR. Add regression test mirroring real crewai's ValueError-on-missing-embedder behavior. * fix(crewai): expose run()/arun() entry points in degraded mode (#962) The public crewai contract is run()/arun(); without crewai installed they were missing (only the private _run existed), so the documented 'usable without crewai' path raised AttributeError at the entry point. Define them in degraded mode only, leaving crewai's BaseTool implementations untouched when present. Extend the degradation subprocess test to exercise run() and arun(). * fix(crewai): standardize query shape, field-name rules, and restore-state flag - _query_graph: id/type matches now return the same schema as content matches (id/type/label/content/score) instead of a bare list - _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys (e.g. "risk-score >= 0.9") are addressable in policy rules - add had_live_state/reconstructed_state so checkpoint-restored tools and knowledge sources signal that their live graph/context was lost and an empty one reconstructed; knowledge source no longer hides the loss by eagerly rebuilding its graph inside __init__ (pydantic calls __init__ during model_validate) * fix(crewai): address Qodo review — confidence errors, string trim, holistic availability - record_decision: stop calling float() in _run, so malformed confidence values surface as JSON errors (via _record_decision's handling) instead of crashing the tool - _coerce_value: return the stripped string for non-numeric literals so whitespace-padded decision_data fields match policy rules - centralize crewai availability in _availability.py so the exported CREWAI_AVAILABLE flag is holistic across tools and knowledge source (previously each module probed crewai independently and the package flag came from decision_tool only) * ci: regenerate requirements-ci.txt for the crewai extra The crewai extra in pyproject.toml brings in crewai, crewai-tools and transitive deps (chromadb, lancedb, ...). Recompile with uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes. * ci: keep crewai out of the locked CI dependency set crewai (all versions) hard-requires chromadb~=1.1.0, which carries a pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c) with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in the 'all' extra failed pip-audit and the safety check on requirements-ci.txt. - drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is unchanged and still installs crewai) - stop listing crewai-tools in the extra: the integration only uses crewai core (BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps - regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0 vulnerabilities, staleness check matches * docs(crewai): document crewai extra scope and chromadb CVE-2026-45829 - CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not part of the 'all' bundle, with the chromadb CVE-2026-45829 reason - integrations/crewai/README.md: add a security warning that installing the extra pulls chromadb~=1.1.0, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 --------- --- CHANGELOG.md | 11 + README.md | 20 +- docs/docs.json | 1 + docs/integrations/crewai.md | 147 +++++ integrations/crewai/README.md | 108 ++++ integrations/crewai/__init__.py | 44 ++ integrations/crewai/_availability.py | 24 + integrations/crewai/decision_tool.py | 555 +++++++++++++++++ integrations/crewai/kg_tool.py | 573 ++++++++++++++++++ integrations/crewai/knowledge_source.py | 331 ++++++++++ pyproject.toml | 8 + requirements-ci.txt | 8 +- tests/integrations/crewai/conftest.py | 151 +++++ .../integrations/crewai/test_decision_tool.py | 562 +++++++++++++++++ tests/integrations/crewai/test_degradation.py | 103 ++++ tests/integrations/crewai/test_kg_tool.py | 453 ++++++++++++++ .../crewai/test_knowledge_source.py | 228 +++++++ .../crewai/test_real_crewai_integration.py | 123 ++++ 18 files changed, 3434 insertions(+), 16 deletions(-) create mode 100644 docs/integrations/crewai.md create mode 100644 integrations/crewai/README.md create mode 100644 integrations/crewai/__init__.py create mode 100644 integrations/crewai/_availability.py create mode 100644 integrations/crewai/decision_tool.py create mode 100644 integrations/crewai/kg_tool.py create mode 100644 integrations/crewai/knowledge_source.py create mode 100644 tests/integrations/crewai/conftest.py create mode 100644 tests/integrations/crewai/test_decision_tool.py create mode 100644 tests/integrations/crewai/test_degradation.py create mode 100644 tests/integrations/crewai/test_kg_tool.py create mode 100644 tests/integrations/crewai/test_knowledge_source.py create mode 100644 tests/integrations/crewai/test_real_crewai_integration.py diff --git a/CHANGELOG.md b/CHANGELOG.md index a94fd882..78212679 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **First-class CrewAI integration** (#962) + - New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`) + - `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()` + - `integrations/crewai/SemanticaDecisionTool` — a CrewAI `BaseTool` wrapping `AgentContext` with 5 decision-intelligence actions (`record_decision`, `find_precedents`, `trace_causal_chain`, `analyze_impact`, `check_policy`) + - `integrations/crewai/SemanticaKnowledgeSource` — a CrewAI `BaseKnowledgeSource` that serializes a `ContextGraph` into crew knowledge storage; implements both the legacy `load_content()` and current `validate_content()`/`aadd()` contracts so it works across `crewai>=0.80.0` + - All three classes degrade gracefully when `crewai` is not installed (still importable, full Semantica API available) + - New `tests/integrations/crewai/`: 70 tests covering stub-based present-case behavior (Pydantic/BaseTool subclassing, every action, knowledge-source chunking/storage) plus a subprocess isolation test for the crewai-absent degradation path + - Docs: `docs/integrations/crewai.md` page, `docs.json` Integrations nav entry, and README integration-matrix/install updates + - **Hardened during code review**: live `graph`/`context`/extractor state is excluded from CrewAI JSON serialization (`model_dump(mode="json")`) with `model_post_init` self-healing defaults, so checkpoint/resume no longer raises `PydanticSerializationError`; `query_graph` now searches node content (not just ids/types); `trace_causal_chain` returns an explicit error instead of substituting similarity precedents when causal tracing is unavailable, and calls `trace_decision_causality(..., max_depth=...)` with the correct argument name; `find_precedents` propagates `max_precedents` as the backend `limit`; `add_to_graph` writes are serialized under a module lock so concurrent agents can't double-count duplicate adds; nameless entities are skipped instead of creating `repr()`-junk nodes + - **Hardened during second code review**: `check_policy` rules are now coerced type-aware — `bool("false")` was truthy, so `enabled == false` reported a violation for `enabled: false`, and string datums like `"0.90"` were compared lexicographically instead of numerically; `trace_causal_chain` no longer raises `AttributeError` (which escaped the tool) when the decision context has no `knowledge_graph`, returning honest error JSON instead; knowledge-source storage failures log an actionable ERROR (a missing crew embedder otherwise silently left agents with empty retrieval); `add_to_graph` uses a per-graph re-entrant lock instead of a process-global one (independent graphs no longer serialize each other, and re-entrant extractors can't deadlock); entity/relation `confidence=None` normalizes to `1.0` instead of failing the whole extraction; added a subprocess integration test against the real `crewai` package covering `Crew`-level serialization round-trip and restore + - **`ContextGraph` gains retraction and purge — the graph previously had no way to remove a node or edge without discarding everything via `clear()`** (#957, closes #955) by @pravit-amp, reviewed by @KaifAhmad1 - `retract_node()`/`retract_edge()` close an entity's validity window rather than deleting it, reusing the existing `valid_from`/`valid_until`/`state_at()` machinery: the entity drops out of `find_active_nodes()` and future `state_at()` queries going forward, but `state_at()` calls before the retraction time still return it, so decisions recorded against it stay explainable. A `("kind", id)`-keyed retraction record captures who/why/when, retrievable via `get_retraction()`/`list_retractions()` - `purge_node()`/`purge_edge()` are the destructive counterpart: the entity is removed outright, from history as well as the active view, for erasure obligations retraction alone cannot satisfy (e.g. GDPR Article 17). Only a tombstone remains — that a purge happened, when, and why — deliberately never the purged content, via `get_tombstone()`/`list_tombstones()`. Purge is graph-scope only: `AgentMemory` and any bound vector store are not reached, so it is one step of an erasure workflow rather than the whole of it diff --git a/README.md b/README.md index fbc20781..fe3272ea 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter - **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built - **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code - **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench -- **Drop-in Integrations:** Native Agno support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors +- **Drop-in Integrations:** Native Agno and CrewAI support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors --- @@ -1189,7 +1189,7 @@ Start with `semantica`, verify with `doctor`, build a graph, and explore the com ## Integrations -Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno support for multi-agent shared context. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more. +Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno and CrewAI support for agentic frameworks. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more. MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below. @@ -1303,6 +1303,11 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below. Agno
First-class · pip install semantica[agno] + +CrewAI
+CrewAI
+First-class · pip install semantica[crewai] + Already Supported via REST API & MCP @@ -1319,11 +1324,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below. REST API · MCP -CrewAI
-CrewAI
-REST API · MCP - - LlamaIndex
LlamaIndex
REST API · MCP @@ -1354,11 +1354,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below. Dedicated toolkit -CrewAI
-CrewAI
-Dedicated toolkit - - LlamaIndex
LlamaIndex
Dedicated toolkit @@ -1514,6 +1509,7 @@ pip install semantica[all] # everything ```bash pip install semantica[agno] # Agno multi-agent integration +pip install semantica[crewai] # CrewAI integration pip install semantica[llm-litellm] # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more pip install semantica[graph-neo4j] # Neo4j graph store (LPG) pip install semantica[graph-falkordb] # FalkorDB graph store (LPG) diff --git a/docs/docs.json b/docs/docs.json index f52cdbd6..d2ad5da2 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -102,6 +102,7 @@ "group": "Integrations", "pages": [ "integrations/agno", + "integrations/crewai", "integrations/docling", "integrations/snowflake", "integrations/databricks" diff --git a/docs/integrations/crewai.md b/docs/integrations/crewai.md new file mode 100644 index 00000000..fb555cda --- /dev/null +++ b/docs/integrations/crewai.md @@ -0,0 +1,147 @@ +--- +title: "CrewAI Integration" +description: "Give CrewAI crews a shared semantic knowledge graph, decision intelligence, and graph-based retrieval via three drop-in components." +icon: "users" +--- + +> Three drop-in components that bring Semantica's knowledge graph and decision intelligence into any CrewAI crew. + +## Installation + +```bash +pip install "semantica[crewai]" +``` + +Requires `crewai >= 0.80.0`. If `crewai` is not installed, the integration still imports — every class carries the full Semantica API and degrades gracefully, but cannot be passed to a `Crew`. + +## Components at a Glance + +- **SemanticaKGTool** — `Agent(tools=[…])`: 5 KG construction/query actions: extract entities, extract relations, add to graph, query graph, find related. +- **SemanticaDecisionTool** — `Agent(tools=[…])`: 5 decision intelligence actions: record decisions, find precedents, trace causal chains, analyze impact, check policies. +- **SemanticaKnowledgeSource** — `Crew(knowledge_sources=[…])`: Serializes a `ContextGraph` into CrewAI knowledge storage so every agent gets retrieval access to the graph. + +## Component Details + + + + Lets agents actively **build and query** a shared `ContextGraph` mid-reasoning. + + ```python + from crewai import Agent, Crew, Task + from semantica.context import ContextGraph + from integrations.crewai import SemanticaKGTool + + graph = ContextGraph() + + analyst = Agent( + role="Knowledge Analyst", + goal="Build and explore a knowledge graph from documents", + backstory="You map entities and relationships into a shared graph.", + tools=[SemanticaKGTool(graph=graph)], + ) + + crew = Crew( + agents=[analyst], + tasks=[Task( + description="Extract and link key entities from the brief", + expected_output="JSON", + agent=analyst, + )], + ) + crew.kickoff() + ``` + + | Tool | Description | + | :------ | :------------- | + | `extract_entities` | Extract named entities from `text` | + | `extract_relations` | Extract relationships between entities in `text` | + | `add_to_graph` | Extract entities/relations from `text` and add them to the shared graph | + | `query_graph` | Keyword-search the graph by node id, type, and content using `query` | + | `find_related` | Find concepts related to `entity` within `hops` hops | + + All actions return JSON so agents get parseable results. + + **Sharing a graph:** the tool reads/writes whatever `graph` you pass in. When no `graph` is given, a fresh in-memory `ContextGraph()` is created (and a warning is logged) — two tool instances that each auto-create their own graph do **not** share knowledge. Pass the same `ContextGraph` to every agent that must share state. + + + Exposes Semantica's decision intelligence as a native CrewAI tool, backed by `AgentContext`. + + ```python + from crewai import Agent, Crew, Task + from integrations.crewai import SemanticaDecisionTool + + planner = Agent( + role="Decision Planner", + goal="Make grounded, precedented decisions", + backstory="You record decisions and validate them against policy.", + tools=[SemanticaDecisionTool()], + ) + + crew = Crew(agents=[planner], tasks=[...]) + ``` + + When no `AgentContext` is passed, one is created in-memory with `decision_tracking=True` and its own `ContextGraph`, so decision actions work out of the box (a warning is logged — pass the same `AgentContext` to every agent that must share decision state). Missing optional fields in `record_decision` fall back to `category="general"`, `reasoning="agent decision"`, and `outcome="recorded"`. `find_precedents` returns up to `max_precedents` results. If a knowledge graph cannot trace causality, `trace_causal_chain` returns an explicit error rather than substituting similarity-based results. + + | Tool | Description | + | :------ | :------------- | + | `record_decision` | Record a decision with reasoning, outcome, and confidence | + | `find_precedents` | Search for similar past decisions | + | `trace_causal_chain` | Trace the causal chain from a decision | + | `analyze_impact` | Assess downstream influence of a decision | + | `check_policy` | Validate a proposed decision against policy rules | + + + Gives **every agent in the crew** retrieval access to a `ContextGraph`. + + ```python + from crewai import Agent, Crew, Task + from semantica.context import ContextGraph + from integrations.crewai import SemanticaKnowledgeSource + + graph = ContextGraph() + graph.add_node(node_id="privacy", node_type="policy", content="...") + + researcher = Agent( + role="Policy Researcher", + goal="Answer questions from the knowledge base", + backstory="You retrieve from graph knowledge to answer accurately.", + ) + + crew = Crew( + agents=[researcher], + tasks=[...], + knowledge_sources=[SemanticaKnowledgeSource(graph=graph)], + ) + ``` + + On kickoff the graph's nodes and edges are serialized, chunked, and stored through CrewAI's knowledge pipeline. + + > **Embedder required:** storing chunks goes through CrewAI's knowledge pipeline, which needs an embedder to be configured. Set `Crew(embedder=...)` (or provide the default credentials CrewAI falls back to, e.g. `OPENAI_API_KEY`). If no working embedder is configured, storage fails, an ERROR is logged, and agents will retrieve **nothing** — the crew still runs, but its knowledge queries return empty. + + **Compatibility:** CrewAI's `BaseKnowledgeSource` contract changed between `0.80.x` and current releases (`load_content()` → `validate_content()`/`aadd()`). `SemanticaKnowledgeSource` implements both legacy and current methods, so it works across `crewai>=0.80.0`. + + + +## Checkpoints & Serialization + +CrewAI serializes tools and knowledge sources to JSON for checkpointing/resume. Live Semantica state (`ContextGraph`, `AgentContext`, extractors) is **excluded from that serialization** — a restored tool/source comes back with a fresh in-memory `ContextGraph` and logs a warning. Until you re-attach the live graph/context, the restored objects answer queries against an **empty** graph, so re-wire them after resuming (e.g. `restored_tool.graph = live_graph`) before agents continue. + +## API Reference + +```python +from integrations.crewai import ( + SemanticaKGTool, # BaseTool: KG construction/query actions + SemanticaDecisionTool, # BaseTool: decision intelligence actions + SemanticaKnowledgeSource, # BaseKnowledgeSource: graph → crew knowledge + CREWAI_AVAILABLE, # bool: True if crewai is installed +) +``` + +All three classes are usable without `crewai` installed: they carry the full Semantica API and degrade gracefully. + +## See Also + +- [Context Module](../reference/context) — AgentContext and ContextGraph backing the integration. +- [Semantic Extraction](../reference/semantic_extract) — NERExtractor / RelationExtractor used by SemanticaKGTool. +- [LLMs](../reference/llms) — Configure LLM providers for your crew's agents. +- [Vector Store](../reference/vector_store) — Vector backend used by SemanticaDecisionTool. diff --git a/integrations/crewai/README.md b/integrations/crewai/README.md new file mode 100644 index 00000000..e083dd7e --- /dev/null +++ b/integrations/crewai/README.md @@ -0,0 +1,108 @@ +# Semantica × CrewAI + +First-class integration between Semantica and [CrewAI](https://github.com/crewAIInc/crewAI) — give your crews a shared semantic knowledge graph, decision intelligence, and graph-based retrieval. + +## Installation + +```bash +pip install semantica[crewai] +``` + +Requires `crewai >= 0.80.0`. If `crewai` is not installed, the integration still imports (classes degrade gracefully), but you can't pass the objects to a `Crew`. + +> **⚠️ Security note:** crewai hard-requires `chromadb~=1.1.0`, which is currently affected by the unpatched pre-authentication code-injection advisory **CVE-2026-45829** (no fixed release — even the latest chromadb 1.5.9 is affected). Installing `semantica[crewai]` pulls that dependency into your environment. The `crewai` extra is intentionally **not** part of `semantica[all]` for this reason — only install it where you actually use CrewAI, and follow chromadb for a patched release. + +## 1. SemanticaKGTool + +A `BaseTool` that lets agents **build and query** a shared `ContextGraph` mid-reasoning: + +- `extract_entities` — extract named entities from `text` +- `extract_relations` — extract relationships from `text` +- `add_to_graph` — extract entities/relations from `text` and add them to the shared graph +- `query_graph` — keyword-search the graph using `query` +- `find_related` — find concepts related to `entity` within `hops` + +```python +from crewai import Agent, Crew, Task +from semantica.context import ContextGraph +from integrations.crewai import SemanticaKGTool + +graph = ContextGraph() + +analyst = Agent( + role="Knowledge Analyst", + goal="Build and explore a knowledge graph from documents", + backstory="You map entities and relationships into a shared graph.", + tools=[SemanticaKGTool(graph=graph)], +) + +crew = Crew( + agents=[analyst], + tasks=[Task(description="Extract and link key entities from the brief", expected_output="JSON", agent=analyst)], +) +result = crew.kickoff() +``` + +All actions return JSON, so agents get parseable results. + +## 2. SemanticaDecisionTool + +A `BaseTool` that wraps `AgentContext` and exposes decision intelligence: + +- `record_decision` — record a decision with reasoning and outcome +- `find_precedents` — retrieve past decisions similar to a scenario +- `trace_causal_chain` — trace the causal chain from a decision +- `analyze_impact` — assess downstream influence using graph centrality +- `check_policy` — validate a proposed decision against rule-based policies + +```python +from crewai import Agent, Crew, Task +from integrations.crewai import SemanticaDecisionTool + +planner = Agent( + role="Decision Planner", + goal="Make grounded, precedented decisions", + backstory="You record decisions and validate them against policy.", + tools=[SemanticaDecisionTool()], +) + +crew = Crew(agents=[planner], tasks=[...]) +``` + +When no `AgentContext` is passed, one is created in-memory with `decision_tracking=True`. + +## 3. SemanticaKnowledgeSource + +A `BaseKnowledgeSource` that serializes the current state of a `ContextGraph` (nodes, edges, metadata) into CrewAI's knowledge storage, giving **every agent in the crew** retrieval access to the graph: + +```python +from crewai import Agent, Crew, Task +from semantica.context import ContextGraph +from integrations.crewai import SemanticaKnowledgeSource + +graph = ContextGraph() +graph.add_node(node_id="privacy", node_type="policy", content="...") + +researcher = Agent( + role="Policy Researcher", + goal="Answer questions from the knowledge base", + backstory="You retrieve from graph knowledge to answer accurately.", +) + +crew = Crew( + agents=[researcher], + tasks=[...], + knowledge_sources=[SemanticaKnowledgeSource(graph=graph)], +) +``` + +> **Embedder required:** storing chunks goes through CrewAI's knowledge pipeline, which needs an embedder. Set `Crew(embedder=...)` (or provide CrewAI's default credentials, e.g. `OPENAI_API_KEY`). Without a working embedder, storage fails, an ERROR is logged, and agents retrieve nothing — the crew still runs with empty knowledge queries. + +### Compatibility note + +CrewAI's `BaseKnowledgeSource` contract changed between `0.80.x` and current releases (`load_content()` → `validate_content()`/`aadd()`). `SemanticaKnowledgeSource` implements both the legacy and current methods, so it works across `crewai>=0.80.0`. + +### Sharing state & checkpoints + +- Each tool/source holds whatever `graph`/`context` you pass it. When omitted, a fresh in-memory object is created and a warning is logged — instances that auto-create their own state do **not** share knowledge, so pass the same object to every agent that must share. +- Live state (`ContextGraph`, `AgentContext`, extractors) is excluded from CrewAI's JSON serialization. After restoring from a checkpoint, re-attach the live graph/context to the restored objects. diff --git a/integrations/crewai/__init__.py b/integrations/crewai/__init__.py new file mode 100644 index 00000000..96c027c7 --- /dev/null +++ b/integrations/crewai/__init__.py @@ -0,0 +1,44 @@ +""" +Semantica × CrewAI Integration +============================== + +First-class integration between the Semantica semantic intelligence stack and +the `CrewAI `_ agentic framework. + +Public surface +-------------- +SemanticaKGTool — CrewAI ``BaseTool`` exposing KG construction/query actions +SemanticaDecisionTool — CrewAI ``BaseTool`` exposing decision-intelligence actions +SemanticaKnowledgeSource— CrewAI ``BaseKnowledgeSource`` giving crews graph knowledge + +Quick start +----------- + pip install semantica[crewai] + + >>> from integrations.crewai import ( + ... SemanticaKGTool, + ... SemanticaDecisionTool, + ... SemanticaKnowledgeSource, + ... ) + +Compatibility +------------- +Requires ``crewai >= 0.80.0``. All three classes degrade gracefully when +``crewai`` is not installed — they are still importable and carry the full +Semantica API, but cannot be passed to ``Crew`` / ``Agent`` constructors. +""" + +from ._availability import CREWAI_AVAILABLE, CREWAI_IMPORT_ERROR +from .decision_tool import SemanticaDecisionTool +from .kg_tool import SemanticaKGTool +from .knowledge_source import SemanticaKnowledgeSource + +__all__ = [ + "SemanticaKGTool", + "SemanticaDecisionTool", + "SemanticaKnowledgeSource", + "CREWAI_AVAILABLE", + "CREWAI_IMPORT_ERROR", +] + +__version__ = "0.1.0" diff --git a/integrations/crewai/_availability.py b/integrations/crewai/_availability.py new file mode 100644 index 00000000..6c871628 --- /dev/null +++ b/integrations/crewai/_availability.py @@ -0,0 +1,24 @@ +""" +Shared CrewAI availability probe. + +Every integration module needs to know whether the real ``crewai`` package is +installed. Probing once here (instead of once per module) guarantees the +exported ``CREWAI_AVAILABLE`` flag means the *whole* integration is ready — a +caller gating on it will never see tools using CrewAI while a knowledge source +silently degrades (or vice versa). +""" + +from typing import Optional + +CREWAI_AVAILABLE = False +CREWAI_IMPORT_ERROR: Optional[str] = None + +try: + from crewai.knowledge.source.base_knowledge_source import ( # noqa: F401 + BaseKnowledgeSource, + ) + from crewai.tools import BaseTool # noqa: F401 + + CREWAI_AVAILABLE = True +except ImportError as exc: + CREWAI_IMPORT_ERROR = str(exc) diff --git a/integrations/crewai/decision_tool.py b/integrations/crewai/decision_tool.py new file mode 100644 index 00000000..bf3552bf --- /dev/null +++ b/integrations/crewai/decision_tool.py @@ -0,0 +1,555 @@ +""" +SemanticaDecisionTool — a CrewAI ``BaseTool`` exposing Semantica's decision +intelligence (``AgentContext``) to agents. + +Lets agents record decisions with reasoning, retrieve past precedents, trace +causal chains, analyse downstream impact, and validate proposed decisions +against policy rules. + +Install +------- + pip install semantica[crewai] + +Example +------- + >>> from integrations.crewai import SemanticaDecisionTool + >>> from crewai import Agent, Crew, Task + >>> tool = SemanticaDecisionTool() + >>> crew = Crew( + ... agents=[Agent(role="...", goal="...", backstory="...", tools=[tool])], + ... tasks=[...], + ... ) + +Tools exposed +------------- +record_decision — Record a decision with reasoning and outcome +find_precedents — Search past decisions similar to a scenario +trace_causal_chain— Trace the causal chain from a decision node +analyze_impact — Assess downstream influence of a decision +check_policy — Validate a proposed decision against policy rules +""" + +from __future__ import annotations + +import json +import re +from typing import Any, Dict, List, Literal, Optional, Type + +from pydantic import BaseModel, Field + +from semantica.utils.logging import get_logger + +from ._availability import CREWAI_AVAILABLE + +logger = get_logger(__name__) + +# --------------------------------------------------------------------------- +# Optional: CrewAI BaseTool base class +# --------------------------------------------------------------------------- +_BaseTool: Any = object + +if CREWAI_AVAILABLE: + from crewai.tools import BaseTool as _BaseTool # type: ignore + + +# --------------------------------------------------------------------------- +# Input schema +# --------------------------------------------------------------------------- +class SemanticaDecisionToolInput(BaseModel): + """ + Input schema for ``SemanticaDecisionTool``. + + Exactly one action is dispatched per call; the remaining fields are only + used by the actions that need them. + """ + + action: Literal[ + "record_decision", + "find_precedents", + "trace_causal_chain", + "analyze_impact", + "check_policy", + ] = Field( + ..., + description=( + "Which decision-intelligence operation to run. One of: " + "'record_decision', 'find_precedents', 'trace_causal_chain', " + "'analyze_impact', 'check_policy'." + ), + ) + category: Optional[str] = Field( + None, + description="Domain category, e.g. 'loan_approval'. Used by 'record_decision'.", + ) + scenario: Optional[str] = Field( + None, + description=( + "Short description of the situation. Used by 'record_decision' and " + "'find_precedents'." + ), + ) + reasoning: Optional[str] = Field( + None, description="Why this outcome was chosen. Used by 'record_decision'." + ) + outcome: Optional[str] = Field( + None, description="The decision result. Used by 'record_decision'." + ) + confidence: float = Field( + 0.8, + ge=0.0, + le=1.0, + description="Confidence score in [0, 1]. Used by 'record_decision'.", + ) + entities: Optional[str] = Field( + None, + description="Comma-separated entity names. Used by 'record_decision'.", + ) + decision_id: Optional[str] = Field( + None, + description=( + "Identifier of a decision. Used by 'trace_causal_chain' and " + "'analyze_impact'." + ), + ) + depth: int = Field( + 3, + ge=1, + le=20, + description="Maximum chain depth. Used by 'trace_causal_chain'.", + ) + decision_data: Optional[str] = Field( + None, + description=( + "JSON object describing a proposed decision. Used by 'check_policy'." + ), + ) + policy_rules: Optional[str] = Field( + None, + description=( + "JSON list of rule strings like 'confidence >= 0.7'. Used by " + "'check_policy'." + ), + ) + + +# --------------------------------------------------------------------------- +# SemanticaDecisionTool +# --------------------------------------------------------------------------- +class SemanticaDecisionTool(_BaseTool): # type: ignore[misc] + """ + CrewAI tool that surfaces Semantica's decision intelligence as agent actions. + + Parameters + ---------- + context: + A ``semantica.context.AgentContext`` (or compatible object exposing + ``record_decision``, ``find_precedents_advanced``, + ``analyze_decision_influence``). A fresh in-memory context is created + when ``None``. + max_precedents: + Default number of precedents returned by ``find_precedents``. + causal_depth: + Default chain depth used by ``trace_causal_chain``. + """ + + name: str = "semantica_decision" + description: str = ( + "Decision intelligence toolkit. Actions: 'record_decision' (record a " + "decision with category, scenario, reasoning, outcome, confidence), " + "'find_precedents' (search past decisions similar to 'scenario'), " + "'trace_causal_chain' (trace the causal chain from 'decision_id'), " + "'analyze_impact' (assess downstream influence of 'decision_id'), " + "'check_policy' (validate 'decision_data' JSON against 'policy_rules' " + "rules like 'confidence >= 0.7'). Returns JSON." + ) + args_schema: Type[BaseModel] = SemanticaDecisionToolInput + context: Any = Field(default=None, exclude=True) + max_precedents: int = 5 + causal_depth: int = 3 + had_live_state: bool = False + reconstructed_state: bool = Field(default=False, exclude=True) + + def __init__( + self, + context: Any = None, + max_precedents: int = 5, + causal_depth: int = 3, + **kwargs: Any, + ) -> None: + if CREWAI_AVAILABLE: + super().__init__( + context=context, + max_precedents=max_precedents, + causal_depth=causal_depth, + **kwargs, + ) + else: + super().__init__() + self.context = context + self.max_precedents = max_precedents + self.causal_depth = causal_depth + # Degraded mode is a plain class — no model_post_init lifecycle. + self._ensure_defaults() + + logger.info("SemanticaDecisionTool initialised (crewai=%s)", CREWAI_AVAILABLE) + + def model_post_init(self, __context: Any) -> None: + """Re-create default state after validation/deserialisation. + + ``context`` is excluded from JSON serialisation (CrewAI checkpoints + serialise every tool via ``model_dump(mode="json")``), so a tool + restored from a checkpoint has ``None`` state until this runs. + """ + self._ensure_defaults() + super().model_post_init(__context) + + def _ensure_defaults(self) -> None: + """Lazy-import and build a real AgentContext when none is wired.""" + if self.context is None: + from semantica.context import AgentContext, ContextGraph + from semantica.vector_store import VectorStore + + self.context = AgentContext( + vector_store=VectorStore(backend="faiss"), + decision_tracking=True, + knowledge_graph=ContextGraph(), + ) + if self.had_live_state: + self.reconstructed_state = True + logger.warning( + "SemanticaDecisionTool: the live decision context was lost " + "during serialization/checkpoint restore — an EMPTY " + "context was reconstructed; re-attach the original context " + "before continuing" + ) + else: + logger.warning( + "SemanticaDecisionTool created a fresh in-memory " + "AgentContext — agents sharing decision state must be " + "wired to the same context" + ) + self.had_live_state = True + + # ------------------------------------------------------------------ + # CrewAI entry points + # ------------------------------------------------------------------ + + def _run( + self, + action: str, + category: Optional[str] = None, + scenario: Optional[str] = None, + reasoning: Optional[str] = None, + outcome: Optional[str] = None, + confidence: float = 0.8, + entities: Optional[str] = None, + decision_id: Optional[str] = None, + depth: int = 3, + decision_data: Optional[str] = None, + policy_rules: Optional[str] = None, + **kwargs: Any, + ) -> str: + valid = { + "record_decision", + "find_precedents", + "trace_causal_chain", + "analyze_impact", + "check_policy", + } + if action not in valid: + return json.dumps( + { + "error": f"Unknown action '{action}'. Valid actions: " + + ", ".join(sorted(valid)) + } + ) + + if action == "record_decision": + return self._record_decision( + category=category or "general", + scenario=scenario or "decision recorded", + reasoning=reasoning or "agent decision", + outcome=outcome or "recorded", + confidence=confidence, + entities=entities, + ) + if action == "find_precedents": + return self._find_precedents(scenario=scenario or "", category=category) + if action == "trace_causal_chain": + return self._trace_causal_chain(decision_id or "", depth=depth) + if action == "analyze_impact": + return self._analyze_impact(decision_id or "") + return self._check_policy(decision_data or "", policy_rules) + + async def _arun(self, action: str, **kwargs: Any) -> str: + """Async variant of ``_run`` for CrewAI's async tool path.""" + return self._run(action=action, **kwargs) + + # ------------------------------------------------------------------ + # Actions + # ------------------------------------------------------------------ + + def _record_decision( + self, + category: str, + scenario: str, + reasoning: str, + outcome: str, + confidence: float = 0.8, + entities: Optional[str] = None, + ) -> str: + entity_list: Optional[List[str]] = None + if entities: + entity_list = [e.strip() for e in entities.split(",") if e.strip()] + + try: + decision_id = self.context.record_decision( + category=category, + scenario=scenario, + reasoning=reasoning, + outcome=outcome, + confidence=float(confidence), + entities=entity_list, + ) + result = {"decision_id": str(decision_id), "status": "recorded"} + logger.info("record_decision → %s", decision_id) + except Exception as exc: + result = {"error": str(exc), "status": "failed"} + logger.warning("record_decision failed: %s", exc) + + return json.dumps(result) + + def _find_precedents( + self, + scenario: str, + category: Optional[str] = None, + limit: Optional[int] = None, + ) -> str: + k = limit if limit is not None else self.max_precedents + try: + precedents = self.context.find_precedents_advanced( + scenario=scenario, + category=category, + limit=k, + ) + out: List[Dict[str, Any]] = [] + for p in (precedents or [])[:k]: + if isinstance(p, dict): + out.append(p) + else: + out.append( + { + "scenario": getattr(p, "scenario", str(p)), + "outcome": getattr(p, "outcome", ""), + "confidence": getattr(p, "confidence", 0.0), + "category": getattr(p, "category", ""), + } + ) + logger.info("find_precedents('%s') → %d results", scenario, len(out)) + return json.dumps({"precedents": out, "count": len(out)}) + except Exception as exc: + logger.warning("find_precedents failed: %s", exc) + return json.dumps({"precedents": [], "count": 0, "error": str(exc)}) + + def _trace_causal_chain(self, decision_id: str, depth: Optional[int] = None) -> str: + if not decision_id: + return json.dumps( + { + "error": "decision_id is required for trace_causal_chain", + "causal_chain": [], + "decision_id": "", + } + ) + max_depth = depth or self.causal_depth + try: + graph = getattr(self.context, "knowledge_graph", None) + if graph is None: + return json.dumps( + { + "error": ( + "causal tracing is not available on this knowledge " + "graph (the decision context has no knowledge_graph)" + ), + "causal_chain": [], + "decision_id": decision_id, + } + ) + trace = getattr(graph, "trace_decision_causality", None) + if trace is None: + return json.dumps( + { + "error": ( + "causal tracing is not available on this knowledge graph " + "(graph.trace_decision_causality is not implemented)" + ), + "causal_chain": [], + "decision_id": decision_id, + } + ) + chain = trace(decision_id, max_depth=max_depth) + return json.dumps({"causal_chain": chain, "decision_id": decision_id}) + except Exception as exc: + logger.warning("trace_causal_chain failed: %s", exc) + return json.dumps( + {"error": str(exc), "causal_chain": [], "decision_id": decision_id} + ) + + def _analyze_impact(self, decision_id: str) -> str: + try: + influence = self.context.analyze_decision_influence(decision_id) + if not isinstance(influence, dict): + influence = {"influence": str(influence)} + influence["decision_id"] = decision_id + return json.dumps(influence) + except Exception as exc: + logger.warning("analyze_impact failed: %s", exc) + return json.dumps({"error": str(exc), "decision_id": decision_id}) + + def _check_policy( + self, + decision_data: str, + policy_rules: Optional[str] = None, + ) -> str: + try: + data = ( + json.loads(decision_data) + if isinstance(decision_data, str) + else decision_data + ) + except json.JSONDecodeError as exc: + return json.dumps( + { + "compliant": False, + "violations": [f"Invalid decision_data JSON: {exc}"], + "warnings": [], + } + ) + + if not isinstance(data, dict): + return json.dumps( + { + "compliant": False, + "violations": [ + f"decision_data must decode to a JSON object, " + f"got {type(data).__name__}: {data!r}" + ], + "warnings": [], + } + ) + + violations: List[str] = [] + warnings: List[str] = [] + + rules: List[str] = [] + if policy_rules: + try: + parsed_rules = json.loads(policy_rules) + except json.JSONDecodeError: + rules = [r.strip() for r in policy_rules.split(",") if r.strip()] + else: + if isinstance(parsed_rules, str): + rules = [parsed_rules] + elif isinstance(parsed_rules, list): + for item in parsed_rules: + if isinstance(item, str): + rules.append(item) + else: + warnings.append( + f"Ignoring non-string policy rule entry: {item!r}" + ) + else: + warnings.append( + f"policy_rules must decode to a JSON list of rule strings, " + f"got {type(parsed_rules).__name__}: {parsed_rules!r}" + ) + + for rule in rules: + try: + if not self._eval_rule(rule, data): + violations.append(f"Rule violated: {rule}") + except Exception as exc: + warnings.append(f"Could not evaluate rule '{rule}': {exc}") + + compliant = len(violations) == 0 + logger.debug( + "check_policy: compliant=%s, violations=%d", compliant, len(violations) + ) + return json.dumps( + { + "compliant": compliant, + "violations": violations, + "warnings": warnings, + } + ) + + def _eval_rule(self, rule: str, data: Dict[str, Any]) -> bool: + """Evaluate a simple comparison rule (``field op value``) against data. + + This is a small standalone evaluator for the tool's ``check_policy`` + action — it is intentionally independent of Semantica's policy engine + so agents get a bounded, side-effect-free rule check. Rules are + `` `` comparisons only; there is no expression + evaluation (no ``eval``), so untrusted rule strings are safe to pass. + + Values are coerced type-aware: ``true``/``false`` (and ``1``/``0``) + become booleans, numeric literals become numbers, and string values + that parse as numbers are compared numerically, so ``score == 0.9`` + holds for ``score: "0.90"`` and ``enabled == false`` holds for + ``enabled: false``. Field names may contain hyphens, dots and spaces + (e.g. ``risk-score >= 0.9``); they are matched against ``data`` keys + as-is. + """ + m = re.match(r"(.+?)\s*(>=|<=|!=|==|>|<)\s*(.+)$", rule.strip()) + if not m: + raise ValueError(f"unrecognised rule format: {rule!r}") + field, op, val_str = m.group(1), m.group(2), m.group(3).strip().strip("\"'") + if field not in data: + raise ValueError(f"rule references undefined field {field!r}") + actual = data[field] + if actual is None: + raise ValueError(f"field {field!r} is null — cannot evaluate rule") + val = self._coerce_value(val_str) + if isinstance(actual, str): + actual = self._coerce_value(actual) + ops = { + ">=": lambda a, b: a >= b, + "<=": lambda a, b: a <= b, + "!=": lambda a, b: a != b, + "==": lambda a, b: a == b, + ">": lambda a, b: a > b, + "<": lambda a, b: a < b, + } + return ops[op](actual, val) + + @staticmethod + def _coerce_value(value: str) -> Any: + """Parse a rule literal into its most specific Python type.""" + text = value.strip() + lowered = text.lower() + if lowered in ("true", "1"): + return True + if lowered in ("false", "0"): + return False + try: + return int(text) + except ValueError: + pass + try: + return float(text) + except ValueError: + pass + return text + + # When crewai is absent there is no BaseTool to provide the public + # ``run``/``arun`` entry points, so expose them directly. With crewai + # installed these are left untouched so crewai's own implementations + # (usage tracking, ``result_as_answer``) win. + if not CREWAI_AVAILABLE: + + def run(self, *args: Any, **kwargs: Any) -> str: + """Run the tool synchronously (degraded mode, no crewai).""" + return self._run(*args, **kwargs) + + async def arun(self, *args: Any, **kwargs: Any) -> str: + """Run the tool asynchronously (degraded mode, no crewai).""" + return self._run(*args, **kwargs) diff --git a/integrations/crewai/kg_tool.py b/integrations/crewai/kg_tool.py new file mode 100644 index 00000000..7740e3fa --- /dev/null +++ b/integrations/crewai/kg_tool.py @@ -0,0 +1,573 @@ +""" +SemanticaKGTool — a CrewAI ``BaseTool`` exposing Semantica's knowledge-graph +pipeline (``NERExtractor``, ``RelationExtractor``, ``ContextGraph``) to agents. + +Lets agents build and query a shared ``ContextGraph`` as part of their +reasoning loop. + +Install +------- + pip install semantica[crewai] + +Example +------- + >>> from integrations.crewai import SemanticaKGTool + >>> from semantica.context import ContextGraph + >>> from crewai import Agent, Crew, Task + >>> graph = ContextGraph() + >>> tool = SemanticaKGTool(graph=graph) + >>> crew = Crew( + ... agents=[Agent(role="...", goal="...", backstory="...", tools=[tool])], + ... tasks=[...], + ... ) + +Tools exposed +------------- +extract_entities — Extract named entities from text +extract_relations — Extract relationships between entities +add_to_graph — Extract entities/relations from text and add them to the graph +query_graph — Query the graph by keyword +find_related — Find concepts related to a given entity within ``hops`` +""" + +from __future__ import annotations + +import json +import threading +import weakref +from typing import Any, Dict, List, Literal, Optional, Sequence, Type + +from pydantic import BaseModel, Field + +from semantica.utils.logging import get_logger + +from ._availability import CREWAI_AVAILABLE, CREWAI_IMPORT_ERROR # noqa: F401 + +logger = get_logger(__name__) + +# --------------------------------------------------------------------------- +# Optional: CrewAI BaseTool base class +# --------------------------------------------------------------------------- +_BaseTool: Any = object + +if CREWAI_AVAILABLE: + from crewai.tools import BaseTool as _BaseTool # type: ignore + +# One re-entrant lock per graph so concurrent tool invocations sharing a graph +# cannot double-count duplicate adds (check-then-act is not atomic), while +# independent graphs are never serialised against each other. An RLock also +# means an extractor callback that re-enters add_to_graph on the same graph +# cannot deadlock. +_graph_locks_guard = threading.Lock() +_graph_locks: "weakref.WeakKeyDictionary[Any, threading.RLock]" = ( + weakref.WeakKeyDictionary() +) + + +# --------------------------------------------------------------------------- +# Input schema +# --------------------------------------------------------------------------- +class SemanticaKGToolInput(BaseModel): + """ + Input schema for ``SemanticaKGTool``. + + Exactly one action is dispatched per call; the remaining fields are only + used by the actions that need them. + """ + + action: Literal[ + "extract_entities", + "extract_relations", + "add_to_graph", + "query_graph", + "find_related", + ] = Field( + ..., + description=( + "Which graph operation to run. One of: 'extract_entities', " + "'extract_relations', 'add_to_graph', 'query_graph', 'find_related'." + ), + ) + text: Optional[str] = Field( + None, + description=( + "Input text. Used by 'extract_entities', 'extract_relations' and " + "'add_to_graph'." + ), + ) + query: Optional[str] = Field( + None, description="Search query. Used by 'query_graph'." + ) + entity: Optional[str] = Field( + None, + description="Root entity name. Used by 'find_related'.", + ) + hops: int = Field( + 1, + ge=1, + le=10, + description="Maximum relationship hops. Used by 'find_related'.", + ) + + +# --------------------------------------------------------------------------- +# SemanticaKGTool +# --------------------------------------------------------------------------- +class SemanticaKGTool(_BaseTool): # type: ignore[misc] + """ + CrewAI tool that surfaces Semantica's KG pipeline as agent actions. + + Parameters + ---------- + graph: + A ``semantica.context.ContextGraph`` to read/write. A fresh in-memory + graph is used when ``None``. + ner_extractor: + A ``semantica.semantic_extract.NERExtractor`` instance; auto-created + when ``None``. + relation_extractor: + A ``semantica.semantic_extract.RelationExtractor`` instance; auto- + created when ``None``. + """ + + name: str = "semantica_knowledge_graph" + description: str = ( + "Build and query a semantic knowledge graph. Actions: " + "'extract_entities' (extract named entities from 'text'), " + "'extract_relations' (extract relationships from 'text'), " + "'add_to_graph' (extract entities/relations from 'text' and add them " + "to the shared graph), 'query_graph' (keyword search using 'query'), " + "'find_related' (find concepts related to 'entity' within 'hops' " + "hops). Returns JSON." + ) + args_schema: Type[BaseModel] = SemanticaKGToolInput + graph: Any = Field(default=None, exclude=True) + ner_extractor: Any = Field(default=None, exclude=True) + relation_extractor: Any = Field(default=None, exclude=True) + had_live_state: bool = False + reconstructed_state: bool = Field(default=False, exclude=True) + + def __init__( + self, + graph: Any = None, + ner_extractor: Any = None, + relation_extractor: Any = None, + **kwargs: Any, + ) -> None: + if CREWAI_AVAILABLE: + super().__init__( + graph=graph, + ner_extractor=ner_extractor, + relation_extractor=relation_extractor, + **kwargs, + ) + else: + super().__init__() + self.graph = graph + self.ner_extractor = ner_extractor + self.relation_extractor = relation_extractor + # Degraded mode is a plain class — no model_post_init lifecycle. + self._ensure_defaults() + + logger.info("SemanticaKGTool initialised (crewai=%s)", CREWAI_AVAILABLE) + + def model_post_init(self, __context: Any) -> None: + """Re-create default state after validation/deserialisation. + + ``graph``/extractors are excluded from JSON serialisation (CrewAI + checkpoints serialise every tool via ``model_dump(mode="json")``), so a + tool restored from a checkpoint has ``None`` state until this runs. + """ + self._ensure_defaults() + super().model_post_init(__context) + + def _ensure_defaults(self) -> None: + """Lazy-import and build defaults for any missing shared state.""" + # Lazy imports keep the module importable without heavy deps + if self.graph is None: + from semantica.context import ContextGraph + + self.graph = ContextGraph() + if self.had_live_state: + self.reconstructed_state = True + logger.warning( + "SemanticaKGTool: the live graph was lost during " + "serialization/checkpoint restore — an EMPTY graph was " + "reconstructed; re-attach the original graph before " + "continuing" + ) + else: + logger.warning( + "SemanticaKGTool created a fresh in-memory ContextGraph — " + "agents sharing this tool's graph must be wired explicitly" + ) + self.had_live_state = True + if self.ner_extractor is None: + from semantica.semantic_extract import NERExtractor + + self.ner_extractor = NERExtractor() + if self.relation_extractor is None: + from semantica.semantic_extract import RelationExtractor + + self.relation_extractor = RelationExtractor() + + # ------------------------------------------------------------------ + # CrewAI entry points + # ------------------------------------------------------------------ + + def _run( + self, + action: str, + text: Optional[str] = None, + query: Optional[str] = None, + entity: Optional[str] = None, + hops: int = 1, + **kwargs: Any, + ) -> str: + """ + Dispatch a graph action. Always returns a JSON string so the agent + receives a structured, parseable result. + """ + valid = { + "extract_entities", + "extract_relations", + "add_to_graph", + "query_graph", + "find_related", + } + if action not in valid: + return json.dumps( + { + "error": f"Unknown action '{action}'. Valid actions: " + + ", ".join(sorted(valid)) + } + ) + + if action == "extract_entities": + return self._extract_entities(text or "") + if action == "extract_relations": + return self._extract_relations(text or "") + if action == "add_to_graph": + return self._add_from_text(text or "") + if action == "query_graph": + return self._query_graph(query or "") + return self._find_related(entity or "", hops=hops) + + async def _arun( + self, + action: str, + text: Optional[str] = None, + query: Optional[str] = None, + entity: Optional[str] = None, + hops: int = 1, + **kwargs: Any, + ) -> str: + """ + Async variant of ``_run`` for CrewAI's async tool path. + """ + return self._run( + action=action, text=text, query=query, entity=entity, hops=hops, **kwargs + ) + + # ------------------------------------------------------------------ + # Entity/relation field access (handles both Semantica dataclasses and + # third-party shapes like MagicMock/plain dicts in stubs) + # ------------------------------------------------------------------ + + @staticmethod + def _first_str(obj: Any, attrs: Sequence[str]) -> str: + """Return the first attribute value that is a non-empty string.""" + for attr in attrs: + value = getattr(obj, attr, None) + if isinstance(value, str) and value: + return value + if isinstance(obj, dict): + for key in attrs: + value = obj.get(key) + if isinstance(value, str) and value: + return value + return "" + + @classmethod + def _entity_name(cls, e: Any) -> str: + """Best-effort name for an entity-like object.""" + return cls._first_str(e, ("name", "text", "label", "node_id", "id")) + + @classmethod + def _entity_type(cls, e: Any) -> str: + """Best-effort type/label for an entity-like object.""" + return cls._first_str(e, ("type", "label")) or "Entity" + + @classmethod + def _relation_source(cls, r: Any) -> str: + """Best-effort source of a relation-like object.""" + src = cls._first_str(r, ("source",)) + if not src: + src = cls._entity_name(getattr(r, "subject", None)) + return src + + @classmethod + def _relation_target(cls, r: Any) -> str: + """Best-effort target of a relation-like object.""" + tgt = cls._first_str(r, ("target",)) + if not tgt: + tgt = cls._entity_name(getattr(r, "object", None)) + return tgt + + @classmethod + def _relation_type(cls, r: Any) -> str: + """Best-effort relation type of a relation-like object.""" + rtype = cls._first_str(r, ("type", "relation", "predicate")) + return rtype or "related_to" + + @classmethod + def _confidence(cls, e: Any) -> float: + """Normalise an entity/relation confidence value to a float.""" + try: + val = getattr(e, "confidence", None) + if val is None: + return 1.0 + return round(float(val), 4) + except (TypeError, ValueError): + return 1.0 + + @classmethod + def _graph_lock(cls, graph: Any) -> threading.RLock: + """Return the re-entrant lock guarding a specific graph.""" + with _graph_locks_guard: + lock = _graph_locks.get(graph) + if lock is None: + lock = threading.RLock() + _graph_locks[graph] = lock + return lock + + # ------------------------------------------------------------------ + # Actions + # ------------------------------------------------------------------ + + def _extract_entities(self, text: str) -> str: + """Extract named entities from ``text``.""" + try: + raw = self.ner_extractor.extract_entities(text) or [] + entities = [ + { + "name": self._entity_name(e), + "type": self._entity_type(e), + "confidence": self._confidence(e), + } + for e in raw + if self._entity_name(e) + ] + logger.debug("extract_entities → %d entities", len(entities)) + return json.dumps({"entities": entities, "count": len(entities)}) + except Exception as exc: + logger.warning("extract_entities failed: %s", exc) + return json.dumps({"entities": [], "count": 0, "error": str(exc)}) + + def _extract_relations(self, text: str) -> str: + """Extract relationships between entities in ``text``.""" + try: + raw = self.relation_extractor.extract_relations(text) or [] + relations = [ + { + "source": self._relation_source(r), + "relation": self._relation_type(r), + "target": self._relation_target(r), + "confidence": self._confidence(r), + } + for r in raw + ] + logger.debug("extract_relations → %d relations", len(relations)) + return json.dumps({"relations": relations, "count": len(relations)}) + except Exception as exc: + logger.warning("extract_relations failed: %s", exc) + return json.dumps({"relations": [], "count": 0, "error": str(exc)}) + + def _add_from_text(self, text: str) -> str: + """ + Extract entities and relations from ``text`` and add them to the graph. + + Duplicate nodes/edges (same id, or same source/type/target) are + skipped so repeated calls are idempotent. Returns JSON with the + number of nodes/edges added. + """ + nodes_added = 0 + edges_added = 0 + try: + with self._graph_lock(self.graph): + existing_nodes = { + n.get("id") or n.get("node_id") + for n in ( + self.graph.find_nodes() or [] # type: ignore[attr-defined] + ) + if n.get("id") or n.get("node_id") + } + existing_edges = { + (e.get("source"), e.get("type") or "related_to", e.get("target")) + for e in ( + self.graph.find_edges() or [] # type: ignore[attr-defined] + ) + if e.get("source") and e.get("target") + } + + raw_entities = self.ner_extractor.extract_entities(text) or [] + entities: List[Any] = [] + seen: set = set() + for e in raw_entities: + name = self._entity_name(e) + ntype = self._entity_type(e) + if not name or name in seen: + continue + seen.add(name) + entities.append(e) + if name in existing_nodes: + continue + try: + if self.graph.add_node(node_id=name, node_type=ntype): + nodes_added += 1 + existing_nodes.add(name) + except Exception as exc: + logger.debug("add_node(%r) failed: %s", name, exc) + + raw_relations = ( + self.relation_extractor.extract_relations(text, entities=entities) + or [] + ) + for r in raw_relations: + src = self._relation_source(r) + tgt = self._relation_target(r) + rtype = self._relation_type(r) + if not src or not tgt: + continue + key = (src, rtype, tgt) + if key in existing_edges: + continue + try: + if self.graph.add_edge( + source_id=src, target_id=tgt, edge_type=rtype + ): + edges_added += 1 + existing_edges.add(key) + except Exception as exc: + logger.debug("add_edge(%r) failed: %s", key, exc) + logger.debug("add_to_graph: +%d nodes, +%d edges", nodes_added, edges_added) + return json.dumps({"nodes_added": nodes_added, "edges_added": edges_added}) + except Exception as exc: + logger.warning("add_to_graph failed: %s", exc) + return json.dumps({"nodes_added": 0, "edges_added": 0, "error": str(exc)}) + + def _query_graph(self, query: str) -> str: + """Keyword-search graph nodes by id, type and content.""" + try: + q = (query or "").strip().lower() + out: List[dict] = [] + seen: set = set() + + query_method = getattr(self.graph, "query", None) + if query_method is not None: + for match in query_method(query) or []: + node = match.get("node") or {} + nid = node.get("id", "") or node.get("node_id", "") + if not nid or nid in seen: + continue + seen.add(nid) + content = match.get("content") or node.get("content", "") + out.append( + { + "id": nid, + "type": node.get("type", "") or node.get("node_type", ""), + "label": nid, + "content": str(content)[:500], + "score": round(float(match.get("score") or 0.0), 4), + } + ) + + if q: + for n in self.graph.find_nodes() or []: # type: ignore[attr-defined] + if isinstance(n, dict): + nid = n.get("id", "") or n.get("node_id", "") + ntype = n.get("type", "") or n.get("node_type", "") + content = str( + n.get("content") + or (n.get("properties") or {}).get("content", "") + or "" + ) + else: + nid = getattr(n, "id", getattr(n, "label", "")) + ntype = getattr(n, "node_type", "") + content = str(getattr(n, "content", "") or "") + if not nid or nid in seen: + continue + if q in str(nid).lower() or q in str(ntype).lower(): + seen.add(nid) + out.append( + { + "id": nid, + "type": ntype, + "label": nid, + "content": content[:500], + "score": 1.0, + } + ) + return json.dumps({"results": out, "count": len(out)}) + except Exception as exc: + logger.warning("query_graph failed: %s", exc) + return json.dumps({"results": [], "count": 0, "error": str(exc)}) + + def _find_related(self, entity: str, hops: int = 1) -> str: + """Find concepts related to ``entity`` within ``hops`` graph hops. + + Traversal is undirected — an edge counts as related regardless of + direction, so both outgoing and incoming edges are honored. + """ + try: + adjacency: Dict[str, List[str]] = {} + for edge in self.graph.find_edges() or []: # type: ignore[attr-defined] + if isinstance(edge, dict): + src = edge.get("source") + tgt = edge.get("target") + else: + src = getattr(edge, "source", None) + tgt = getattr(edge, "target", None) + if not src or not tgt: + continue + adjacency.setdefault(src, []).append(tgt) + adjacency.setdefault(tgt, []).append(src) + + related: List[str] = [] + frontier = [entity] + visited = {entity} + for _ in range(max(1, hops)): + next_frontier: List[str] = [] + for e in frontier: + for n in adjacency.get(e, []): + if n in visited: + continue + visited.add(n) + next_frontier.append(n) + related.append(n) + frontier = next_frontier + + logger.debug("find_related('%s', hops=%d) → %d", entity, hops, len(related)) + return json.dumps( + {"entity": entity, "related": related, "count": len(related)} + ) + except Exception as exc: + logger.warning("find_related failed: %s", exc) + return json.dumps( + {"entity": entity, "related": [], "count": 0, "error": str(exc)} + ) + + # When crewai is absent there is no BaseTool to provide the public + # ``run``/``arun`` entry points, so expose them directly. With crewai + # installed these are left untouched so crewai's own implementations + # (usage tracking, ``result_as_answer``) win. + if not CREWAI_AVAILABLE: + + def run(self, *args: Any, **kwargs: Any) -> str: + """Run the tool synchronously (degraded mode, no crewai).""" + return self._run(*args, **kwargs) + + async def arun(self, *args: Any, **kwargs: Any) -> str: + """Run the tool asynchronously (degraded mode, no crewai).""" + return self._run(*args, **kwargs) diff --git a/integrations/crewai/knowledge_source.py b/integrations/crewai/knowledge_source.py new file mode 100644 index 00000000..a61bbfce --- /dev/null +++ b/integrations/crewai/knowledge_source.py @@ -0,0 +1,331 @@ +""" +SemanticaKnowledgeSource — expose a Semantica ``ContextGraph`` as a CrewAI +knowledge source. + +Lets a ``Crew`` load the current state of a knowledge graph (nodes, edges, +metadata) into its knowledge storage, so every agent gets retrieval access to +graph knowledge during the kickoff. + +Install +------- + pip install semantica[crewai] + +Example +------- + >>> from integrations.crewai import SemanticaKnowledgeSource + >>> from semantica.context import ContextGraph + >>> from crewai import Agent, Crew, Task + >>> graph = ContextGraph() + >>> graph.add_node(node_id="privacy", node_type="policy") + >>> crew = Crew( + ... agents=[...], + ... tasks=[...], + ... knowledge_sources=[SemanticaKnowledgeSource(graph=graph)], + ... ) + +Compatibility +------------- +Works with ``crewai >= 0.80.0``. The ``BaseKnowledgeSource`` contract changed +between versions (``load_content`` → ``validate_content``/``aadd``), so this +source implements both legacy and current methods. It degrades gracefully +when ``crewai`` is not installed: the class is still importable and carries the +full Semantica API, but cannot be passed to a ``Crew``. +""" + +from __future__ import annotations + +import asyncio +from typing import Any, Dict, List, Optional + +from pydantic import Field + +from semantica.utils.logging import get_logger + +from ._availability import CREWAI_AVAILABLE + +logger = get_logger(__name__) + +# --------------------------------------------------------------------------- +# Optional: CrewAI BaseKnowledgeSource base class +# --------------------------------------------------------------------------- +_BaseKnowledgeSource: Any = object + +if CREWAI_AVAILABLE: + from crewai.knowledge.source.base_knowledge_source import ( + BaseKnowledgeSource as _BaseKnowledgeSource, # type: ignore + ) + + +def _chunk_text_manual(text: str, chunk_size: int, chunk_overlap: int) -> List[str]: + """Fallback plain-text chunker for when CrewAI helpers are unavailable.""" + if not text: + return [] + if int(chunk_size) <= 0: + return [text] + size = max(1, int(chunk_size)) + overlap = max(0, int(chunk_overlap)) + if len(text) <= size: + return [text] + step = max(1, size - overlap) + return [text[i : i + size] for i in range(0, len(text), step)] + + +class SemanticaKnowledgeSource(_BaseKnowledgeSource): # type: ignore[misc] + """ + CrewAI knowledge source backed by a Semantica ``ContextGraph``. + + On ``add()`` the graph's nodes and edges are serialised into readable text + and pushed through the standard CrewAI chunking / storage pipeline, making + graph knowledge retrievable by every agent in the crew. + + Parameters + ---------- + graph: + A ``semantica.context.ContextGraph`` to expose. A fresh in-memory + graph is created when ``None``. + name: + Source name. Defaults to ``"semantica_knowledge_graph"``. + chunk_size: + Max characters per chunk (default 4000). + chunk_overlap: + Character overlap between adjacent chunks (default 200). + """ + + name: str = "semantica_knowledge_graph" + graph: Any = Field(default=None, exclude=True) + chunk_size: int = 4000 + chunk_overlap: int = 200 + had_live_state: bool = False + reconstructed_state: bool = Field(default=False, exclude=True) + + def __init__( + self, + graph: Any = None, + name: Optional[str] = None, + chunk_size: int = 4000, + chunk_overlap: int = 200, + **kwargs: Any, + ) -> None: + if CREWAI_AVAILABLE: + # Do NOT eagerly build a graph here: pydantic calls this ``__init__`` + # during ``model_validate`` (checkpoint restore), and the eager + # build would hide that a live graph was lost. ``model_post_init`` + # rebuilds defaults and flags ``reconstructed_state`` instead. + super().__init__( + graph=graph, + name=name or "semantica_knowledge_graph", + chunk_size=int(chunk_size), + chunk_overlap=int(chunk_overlap), + **kwargs, + ) + else: + if graph is None: + from semantica.context import ContextGraph + + graph = ContextGraph() + super().__init__() + self.graph = graph + self.name = name or "semantica_knowledge_graph" + self.chunk_size = int(chunk_size) + self.chunk_overlap = int(chunk_overlap) + + logger.info( + "SemanticaKnowledgeSource initialised (crewai=%s, chunk_size=%d)", + CREWAI_AVAILABLE, + self.chunk_size, + ) + self.had_live_state = True + + def model_post_init(self, __context: Any) -> None: + """Re-create default state after validation/deserialisation. + + ``graph`` is excluded from JSON serialisation (CrewAI checkpoints + serialise their models via ``model_dump(mode="json")``), so a source + restored from a checkpoint has ``None`` state until this runs. + """ + if self.graph is None: + from semantica.context import ContextGraph + + self.graph = ContextGraph() + if self.had_live_state: + self.reconstructed_state = True + logger.warning( + "SemanticaKnowledgeSource: the live graph was lost during " + "serialization/checkpoint restore — an EMPTY graph was " + "reconstructed; re-attach the original graph before " + "continuing" + ) + else: + logger.warning( + "SemanticaKnowledgeSource created a fresh in-memory " + "ContextGraph — sources sharing knowledge must be wired to " + "the same graph explicitly" + ) + self.had_live_state = True + super().model_post_init(__context) + + # ------------------------------------------------------------------ + # Content extraction + # ------------------------------------------------------------------ + + def load_content(self) -> Dict[str, str]: + """ + Serialise the graph into ``{id: readable_text}`` pairs. + + Nodes are rendered with their type/content/metadata, edges with their + source, relation type and target. This satisfies the legacy CrewAI + ``BaseKnowledgeSource.load_content`` contract. + """ + content: Dict[str, str] = {} + graph = self.graph + if graph is None: + return content + + try: + for node in graph.find_nodes() or []: # type: ignore[attr-defined] + nid = node.get("id") or node.get("node_id") or "" + if not nid: + continue + parts = [ + "Entity", + str(nid), + "type: " + str(node.get("type", "entity")), + ] + if node.get("content"): + parts.append("content: " + str(node["content"])) + if node.get("metadata"): + try: + import json + + parts.append("metadata: " + json.dumps(node["metadata"])) + except Exception: + parts.append("metadata: " + str(node["metadata"])) + content[str(nid)] = " | ".join(parts) + except Exception as exc: + logger.warning( + "SemanticaKnowledgeSource.load_content (nodes) failed: %s", exc + ) + + try: + for idx, edge in enumerate( + graph.find_edges() or [] # type: ignore[attr-defined] + ): + src = edge.get("source") + tgt = edge.get("target") + if not src or not tgt: + continue + rel = edge.get("type") or edge.get("edge_type") or "related_to" + weight = edge.get("weight") + text = f"{src} -[{rel}]-> {tgt}" + if weight is not None: + text += f" (weight: {weight})" + content[f"edge-{idx}"] = text + except Exception as exc: + logger.warning( + "SemanticaKnowledgeSource.load_content (edges) failed: %s", exc + ) + + return content + + def validate_content(self) -> Any: + """ + Validate that a readable graph is attached. + + Satisfies the current CrewAI ``BaseKnowledgeSource.validate_content`` + contract. + """ + if self.graph is None: + raise ValueError("SemanticaKnowledgeSource requires a ContextGraph.") + return True + + # ------------------------------------------------------------------ + # Chunking + storage (abstract in both CrewAI generations) + # ------------------------------------------------------------------ + + def _chunk(self, text: str) -> List[str]: + """Chunk ``text`` using CrewAI's helper when available, else manual.""" + helper = getattr(self, "_chunk_text", None) + if helper is not None: + try: + return list(helper(text) or []) + except Exception as exc: + logger.debug( + "SemanticaKnowledgeSource._chunk_text failed, falling back: %s", exc + ) + return _chunk_text_manual(text, self.chunk_size, self.chunk_overlap) + + def add(self) -> None: + """ + Process the graph into chunks and store them via CrewAI storage. + + Sets both ``chunks`` (current CrewAI) and ``_chunks`` (legacy CrewAI) + so either ``_save_documents`` implementation picks them up. If no + storage has been wired (e.g. not yet attached to a ``Crew``), chunks + are kept in memory. + """ + content = self.load_content() + if not content: + logger.debug("SemanticaKnowledgeSource.add: empty graph — nothing to store") + return + + chunks: List[str] = [] + for _, text in content.items(): + if text: + chunks.extend(self._chunk(text)) + + self.chunks = chunks + self._chunks = chunks + + save = getattr(self, "_save_documents", None) + if save is not None: + if getattr(self, "storage", None) is None: + logger.debug( + "SemanticaKnowledgeSource.add: storage not wired — " + "keeping chunks in memory" + ) + else: + try: + save() + logger.info( + "SemanticaKnowledgeSource.add: stored %d chunks", len(chunks) + ) + return + except Exception as exc: + logger.error( + "SemanticaKnowledgeSource.add: storage save FAILED (%s) — " + "chunks are only kept in memory and agents will retrieve " + "nothing. Configure the Crew embedder (e.g. an OpenAI " + "embedder with OPENAI_API_KEY, or a local embedder) before " + "running the crew.", + exc, + ) + + logger.info( + "SemanticaKnowledgeSource.add: %d chunks ready in memory", len(chunks) + ) + + async def aadd(self) -> None: + """ + Asynchronous variant of ``add()`` (current CrewAI contract). + + The graph serialisation is CPU-bound, so it runs in a thread pool to + avoid blocking the event loop. + """ + loop = asyncio.get_running_loop() + await loop.run_in_executor(None, self.add) + + # ------------------------------------------------------------------ + # Inspection helpers + # ------------------------------------------------------------------ + + def get_content_summary(self) -> Dict[str, Any]: + """ + Summarise what the source exposes (helpful for debugging / testing). + """ + content = self.load_content() + return { + "name": self.name, + "source_count": len(content), + "chunks": len(getattr(self, "chunks", []) or []), + "crewai_available": CREWAI_AVAILABLE, + } diff --git a/pyproject.toml b/pyproject.toml index 03949d4e..341e57ed 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -201,6 +201,10 @@ gpu = [ # ---- Agentic Framework Integrations ---- agno = ["agno>=1.0.0"] +# crewai core provides BaseTool and BaseKnowledgeSource; crewai-tools is not +# needed (it pulls vulnerable transitive deps like chromadb) and would only +# duplicate the prebuilt tooling users can install separately. +crewai = ["crewai>=0.80.0"] # ---- File Watching ---- watch = ["watchdog>=6.0.0"] @@ -242,6 +246,10 @@ explorer-lite = [ ] # Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux) +# NOTE: the ``crewai`` extra is intentionally NOT in ``all``: crewai hard-requires +# ``chromadb~=1.1.0``, which carries a pre-authentication code-injection advisory +# (CVE-2026-45829) with no fixed release — including it here would fail the CI +# dependency-audit/security gates. Install it explicitly via ``semantica[crewai]``. all = [ "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]", "semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]" diff --git a/requirements-ci.txt b/requirements-ci.txt index d57fa744..17df46d4 100644 --- a/requirements-ci.txt +++ b/requirements-ci.txt @@ -1,5 +1,5 @@ # This file was autogenerated by uv via the following command: -# uv pip compile -p 3.11 --extra all --generate-hashes -o requirements-ci.txt pyproject.toml +# uv pip compile pyproject.toml --python-version 3.11 --extra all --generate-hashes -o requirements-ci.txt accelerate==1.14.0 \ --hash=sha256:41b9c4377a54e0b460a959b0defa1b736e4ca0a2373252d9a539964c2afe3c8d \ --hash=sha256:e94390c2863b873be18f623f9df48a0d8fe5eff13ea7f1a00092b0a7904888c6 @@ -4123,9 +4123,9 @@ pooch==1.9.0 \ --hash=sha256:de46729579b9857ffd3e741987a2f6d5e0e03219892c167c6578c0091fb511ed \ --hash=sha256:f265597baa9f760d25ceb29d0beb8186c243d6607b0f60b83ecf14078dbc703b # via librosa -portalocker==3.2.0 \ - --hash=sha256:1f3002956a54a8c3730586c5c77bf18fae4149e07eaf1c29fc3faf4d5a3f89ac \ - --hash=sha256:3cdc5f565312224bc570c49337bd21428bba0ef363bbcf58b9ef4a9f11779968 +portalocker==2.7.0 \ + --hash=sha256:032e81d534a88ec1736d03f780ba073f047a06c478b06e2937486f334e955c51 \ + --hash=sha256:a07c5b4f3985c3cf4798369631fb7011adb498e2a46d8440efc75a8f29a0f983 # via qdrant-client pre-commit==4.6.2 \ --hash=sha256:8f5d7bfb021ecdbcd9d49d89847082dd24172ccde534390081a679ad046e2441 \ diff --git a/tests/integrations/crewai/conftest.py b/tests/integrations/crewai/conftest.py new file mode 100644 index 00000000..68991e54 --- /dev/null +++ b/tests/integrations/crewai/conftest.py @@ -0,0 +1,151 @@ +""" +Shared pytest configuration for CrewAI integration tests. + +Installs comprehensive crewai stubs into sys.modules before any test in this +directory runs, so every test file can import the integration modules with +``CREWAI_AVAILABLE == True`` and exercise the real subclassing code paths +without a real crewai installation. + +The stubs mirror the current CrewAI contracts: +- ``crewai.tools.BaseTool`` — Pydantic ``BaseModel`` (arbitrary types allowed) +- ``crewai.knowledge.source.base_knowledge_source.BaseKnowledgeSource`` — + Pydantic model with ``validate_content``/``add``/``aadd`` abstract methods + and ``_chunk_text``/``_save_documents`` helpers. + +The graceful-degradation path (crewai genuinely absent) is covered separately +in ``test_degradation.py`` via a subprocess, so this stub never has to be torn +down mid-session. +""" + +from __future__ import annotations + +import sys +import types +from typing import Any, Optional + +from pydantic import BaseModel, ConfigDict, Field, field_serializer, field_validator + + +def _install_crewai_stubs() -> None: + """Install a full set of crewai stubs into sys.modules.""" + + # ----------------------------------------------------------------------- + # crewai.tools — BaseTool + # ----------------------------------------------------------------------- + class BaseTool(BaseModel): # noqa: D101 + """Stub mirroring crewai.tools.base_tool.BaseTool.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + name: str = "base_tool" + description: str = "" + args_schema: Any = None + result_as_answer: bool = False + + @field_serializer("args_schema", when_used="json") + def _ser_args_schema(self, schema): # noqa: D102 + if schema is None: + return None + return {"__schema__": f"{schema.__module__}.{schema.__qualname__}"} + + @field_validator("args_schema", mode="before") + @classmethod + def _restore_args_schema(cls, v): # noqa: D102 + if isinstance(v, dict) and "__schema__" in v: + import importlib + + mod_name, cls_name = v["__schema__"].rsplit(".", 1) + return getattr(importlib.import_module(mod_name), cls_name) + return v + + def run(self, *args: Any, **kwargs: Any) -> str: # noqa: D102 + return self._run(*args, **kwargs) + + async def arun(self, *args: Any, **kwargs: Any) -> str: # noqa: D102 + return await self._arun(*args, **kwargs) + + def _run(self, *args: Any, **kwargs: Any) -> str: # noqa: D102 + raise NotImplementedError + + async def _arun(self, *args: Any, **kwargs: Any) -> str: # noqa: D102 + raise NotImplementedError + + tools_mod = types.ModuleType("crewai.tools") + tools_mod.BaseTool = BaseTool # type: ignore[attr-defined] + + tools_base_mod = types.ModuleType("crewai.tools.base_tool") + tools_base_mod.BaseTool = BaseTool # type: ignore[attr-defined] + + # ----------------------------------------------------------------------- + # crewai.knowledge.source.base_knowledge_source — BaseKnowledgeSource + # ----------------------------------------------------------------------- + class BaseKnowledgeSource(BaseModel): # noqa: D101 + """Stub mirroring crewai.knowledge.source.base_knowledge_source.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + chunk_size: int = 4000 + chunk_overlap: int = 200 + chunks: list = Field(default_factory=list) + chunk_embeddings: list = Field(default_factory=list, exclude=True) + storage: Any = None + metadata: dict = Field(default_factory=dict) + collection_name: Optional[str] = None + + def _chunk_text(self, text: str) -> list: # noqa: D102 + return [ + text[i : i + self.chunk_size] + for i in range(0, len(text), self.chunk_size - self.chunk_overlap) + ] + + def _save_documents(self) -> None: # noqa: D102 + if self.storage is not None: + self.storage.save(self.chunks) + else: + raise ValueError("No storage found to save documents.") + + async def _asave_documents(self) -> None: # noqa: D102 + if self.storage is not None: + await self.storage.asave(self.chunks) + else: + raise ValueError("No storage found to save documents.") + + def validate_content(self) -> Any: # noqa: D102 + raise NotImplementedError + + def add(self) -> None: # noqa: D102 + raise NotImplementedError + + async def aadd(self) -> None: # noqa: D102 + raise NotImplementedError + + knowledge_pkg = types.ModuleType("crewai.knowledge") + source_pkg = types.ModuleType("crewai.knowledge.source") + source_base_mod = types.ModuleType("crewai.knowledge.source.base_knowledge_source") + source_base_mod.BaseKnowledgeSource = ( # type: ignore[attr-defined] + BaseKnowledgeSource + ) + source_pkg.BaseKnowledgeSource = BaseKnowledgeSource # type: ignore[attr-defined] + knowledge_pkg.source = source_pkg + + # ----------------------------------------------------------------------- + # Register everything + # ----------------------------------------------------------------------- + crewai = types.ModuleType("crewai") + crewai.tools = tools_mod # type: ignore[attr-defined] + crewai.knowledge = knowledge_pkg # type: ignore[attr-defined] + + _mods = { + "crewai": crewai, + "crewai.tools": tools_mod, + "crewai.tools.base_tool": tools_base_mod, + "crewai.knowledge": knowledge_pkg, + "crewai.knowledge.source": source_pkg, + "crewai.knowledge.source.base_knowledge_source": source_base_mod, + } + for name, mod in _mods.items(): + sys.modules[name] = mod + + +# Install once at import time (conftest is imported before any test file) +_install_crewai_stubs() diff --git a/tests/integrations/crewai/test_decision_tool.py b/tests/integrations/crewai/test_decision_tool.py new file mode 100644 index 00000000..c7d4a1e6 --- /dev/null +++ b/tests/integrations/crewai/test_decision_tool.py @@ -0,0 +1,562 @@ +""" +Tests for SemanticaDecisionTool — decision intelligence CrewAI tool. + +Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is +``True`` and the real Pydantic/BaseTool subclassing path is exercised. A +MagicMock ``AgentContext`` is used so no vector store / faiss is required. +""" + +from __future__ import annotations + +import json +import unittest +from unittest.mock import MagicMock + +from integrations.crewai import SemanticaDecisionTool +from integrations.crewai.decision_tool import ( + CREWAI_AVAILABLE, + SemanticaDecisionToolInput, +) + + +def _make_context() -> MagicMock: + ctx = MagicMock() + ctx.record_decision.return_value = "dec-test-001" + ctx.find_precedents_advanced.return_value = [ + { + "scenario": "past loan", + "outcome": "approved", + "confidence": 0.9, + "category": "loan", + } + ] + ctx.analyze_decision_influence.return_value = {"centrality": 0.75, "influenced": 3} + ctx.knowledge_graph = MagicMock() + ctx.knowledge_graph.trace_decision_causality = MagicMock( + return_value=["step1", "step2"] + ) + return ctx + + +class TestSemanticaDecisionToolInit(unittest.TestCase): + + def test_crewai_available_via_stub(self): + self.assertTrue(CREWAI_AVAILABLE) + + def test_is_base_tool_subclass(self): + from crewai.tools import BaseTool + + self.assertTrue(issubclass(SemanticaDecisionTool, BaseTool)) + + def test_creates_with_explicit_context(self): + ctx = _make_context() + tool = SemanticaDecisionTool(context=ctx) + self.assertIs(tool.context, ctx) + + def test_creates_context_when_none(self): + tool = SemanticaDecisionTool() + self.assertIsNotNone(tool.context) + + def test_default_metadata(self): + tool = SemanticaDecisionTool(context=_make_context()) + self.assertEqual(tool.name, "semantica_decision") + self.assertTrue(tool.description) + self.assertEqual(tool.args_schema, SemanticaDecisionToolInput) + + def test_input_schema_validates(self): + inp = SemanticaDecisionToolInput(action="record_decision", confidence=0.5) + self.assertEqual(inp.confidence, 0.5) + with self.assertRaises(Exception): + SemanticaDecisionToolInput(action="bogus") + + def test_max_precedents_and_causal_depth_defaults(self): + tool = SemanticaDecisionTool(context=_make_context()) + self.assertEqual(tool.max_precedents, 5) + self.assertEqual(tool.causal_depth, 3) + + +class TestSemanticaDecisionToolSerialization(unittest.TestCase): + """CrewAI checkpoints serialise tools via ``model_dump(mode="json")`` — the + live context must not break that (regression for PydanticSerializationError + on arbitrary state objects).""" + + def test_model_dump_json_excludes_context(self): + tool = SemanticaDecisionTool(context=_make_context()) + dumped = tool.model_dump(mode="json") + self.assertNotIn("context", dumped) + self.assertEqual(dumped["max_precedents"], 5) + self.assertEqual(dumped["causal_depth"], 3) + + def test_model_validate_restores_defaults(self): + tool = SemanticaDecisionTool(context=_make_context()) + restored = SemanticaDecisionTool.model_validate(tool.model_dump(mode="json")) + self.assertIsNotNone(restored.context) + self.assertEqual(restored.max_precedents, 5) + self.assertEqual(restored.causal_depth, 3) + + def test_restore_flags_lost_live_state(self): + """A tool restored from a checkpoint must signal that its live context + was excluded and an empty one reconstructed (``reconstructed_state``).""" + tool = SemanticaDecisionTool(context=_make_context()) + dumped = tool.model_dump(mode="json") + self.assertTrue(dumped["had_live_state"]) + self.assertNotIn("reconstructed_state", dumped) + restored = SemanticaDecisionTool.model_validate(dumped) + self.assertTrue(restored.reconstructed_state) + self.assertFalse(SemanticaDecisionTool().reconstructed_state) + + +class TestRecordDecision(unittest.TestCase): + + def setUp(self): + self.ctx = _make_context() + self.tool = SemanticaDecisionTool(context=self.ctx) + + def test_returns_json_with_decision_id(self): + result = json.loads( + self.tool._run( + action="record_decision", + category="loan", + scenario="Customer A loan application", + reasoning="Good credit score 740", + outcome="approved", + confidence=0.95, + ) + ) + self.assertEqual(result["decision_id"], "dec-test-001") + self.assertEqual(result["status"], "recorded") + + def test_delegates_to_context(self): + self.tool._run( + action="record_decision", + category="content", + scenario="Moderation check", + reasoning="No violations", + outcome="allowed", + confidence=0.88, + ) + self.ctx.record_decision.assert_called_once() + + def test_parses_entities_string(self): + self.tool._run( + action="record_decision", + category="hr", + scenario="Hire decision", + reasoning="Qualified", + outcome="hired", + confidence=0.9, + entities="Alice, ACME Corp, Senior Engineer", + ) + call_kwargs = self.ctx.record_decision.call_args[1] + self.assertIsInstance(call_kwargs["entities"], list) + self.assertEqual(len(call_kwargs["entities"]), 3) + + def test_returns_error_json_on_failure(self): + self.ctx.record_decision.side_effect = RuntimeError("DB unavailable") + result = json.loads( + self.tool._run( + action="record_decision", + category="x", + scenario="y", + reasoning="z", + outcome="failed", + ) + ) + self.assertEqual(result["status"], "failed") + self.assertIn("error", result) + + def test_default_confidence_used(self): + self.tool._run( + action="record_decision", + category="test", + scenario="Default confidence test", + reasoning="N/A", + outcome="pass", + ) + call_kwargs = self.ctx.record_decision.call_args[1] + self.assertEqual(call_kwargs["confidence"], 0.8) + + def test_malformed_confidence_returns_error_json(self): + """A non-numeric confidence must not crash the tool — it is coerced + inside ``_record_decision``'s error handling and reported as JSON.""" + for bad in ("high", None, "0.9"): + result = json.loads( + self.tool._run( + action="record_decision", + category="x", + scenario="y", + reasoning="z", + outcome="failed", + confidence=bad, + ) + ) + if bad == "0.9": + self.assertEqual(result["status"], "recorded") + else: + self.assertEqual(result["status"], "failed") + self.assertIn("error", result) + + def test_missing_fields_get_sane_defaults(self): + """record_decision must not hard-fail when the agent omits optional + fields — category/reasoning/outcome get defaults.""" + result = json.loads(self.tool._run(action="record_decision")) + self.assertEqual(result["status"], "recorded") + call_kwargs = self.ctx.record_decision.call_args[1] + self.assertEqual(call_kwargs["category"], "general") + self.assertEqual(call_kwargs["scenario"], "decision recorded") + self.assertEqual(call_kwargs["reasoning"], "agent decision") + self.assertEqual(call_kwargs["outcome"], "recorded") + + +class TestRealAutoCreatedContext(unittest.TestCase): + """The no-context path builds a real AgentContext with a knowledge graph so + decision tracking is actually enabled (regression for the live + 'Decision tracking is not enabled' failure).""" + + def setUp(self): + self.tool = SemanticaDecisionTool() + + def test_context_is_real_agent_context(self): + from semantica.context import AgentContext + + self.assertIsInstance(self.tool.context, AgentContext) + self.assertIsNotNone(self.tool.context.knowledge_graph) + + def test_record_decision_actually_records(self): + result = json.loads( + self.tool.run( + action="record_decision", + scenario="ship v2", + reasoning="user demand", + confidence=0.9, + ) + ) + self.assertEqual(result["status"], "recorded") + self.assertTrue(result["decision_id"]) + + def test_find_precedents_runs_against_real_context(self): + result = json.loads(self.tool.run(action="find_precedents", scenario="ship v2")) + self.assertIn("precedents", result) + + def test_trace_causal_chain_runs_against_real_context(self): + """Regression: trace_decision_causality takes ``max_depth``, not + ``depth`` — must not raise against a real ContextGraph.""" + rec = json.loads( + self.tool.run( + action="record_decision", + scenario="ship v2", + reasoning="user demand", + confidence=0.9, + ) + ) + trace = json.loads( + self.tool.run(action="trace_causal_chain", decision_id=rec["decision_id"]) + ) + self.assertIn("causal_chain", trace) + self.assertEqual(trace["decision_id"], rec["decision_id"]) + + +class TestFindPrecedents(unittest.TestCase): + + def setUp(self): + self.ctx = _make_context() + self.tool = SemanticaDecisionTool(context=self.ctx) + + def test_returns_json_with_precedents(self): + result = json.loads( + self.tool._run(action="find_precedents", scenario="new loan application") + ) + self.assertIn("precedents", result) + self.assertIsInstance(result["precedents"], list) + + def test_count_in_result(self): + result = json.loads( + self.tool._run(action="find_precedents", scenario="test scenario") + ) + self.assertEqual(result["count"], len(result["precedents"])) + + def test_category_filter_passed(self): + self.tool._run( + action="find_precedents", scenario="scenario", category="finance" + ) + call_kwargs = self.ctx.find_precedents_advanced.call_args[1] + self.assertEqual(call_kwargs.get("category"), "finance") + + def test_limit_propagated_to_backend(self): + self.tool.max_precedents = 20 + self.tool._run(action="find_precedents", scenario="scenario") + call_kwargs = self.ctx.find_precedents_advanced.call_args[1] + self.assertEqual(call_kwargs.get("limit"), 20) + + def test_handles_exception_gracefully(self): + self.ctx.find_precedents_advanced.side_effect = RuntimeError("fail") + result = json.loads(self.tool._run(action="find_precedents", scenario="broken")) + self.assertEqual(result["precedents"], []) + self.assertIn("error", result) + + +class TestTraceCausalChain(unittest.TestCase): + + def setUp(self): + self.ctx = _make_context() + self.tool = SemanticaDecisionTool(context=self.ctx) + + def test_returns_json_with_causal_chain(self): + result = json.loads( + self.tool._run(action="trace_causal_chain", decision_id="dec-001") + ) + self.assertIn("causal_chain", result) + self.assertEqual(result["decision_id"], "dec-001") + + def test_honest_error_when_causal_trace_unavailable(self): + """When the graph cannot trace causality, the tool must say so — it + must NOT substitute similarity-based precedents as a causal chain.""" + del self.ctx.knowledge_graph.trace_decision_causality + result = json.loads( + self.tool._run(action="trace_causal_chain", decision_id="dec-002") + ) + self.assertEqual(result["causal_chain"], []) + self.assertIn("error", result) + self.ctx.knowledge_graph.find_precedents.assert_not_called() + + def test_missing_decision_id_reports_error(self): + result = json.loads(self.tool._run(action="trace_causal_chain")) + self.assertIn("error", result) + self.assertEqual(result["causal_chain"], []) + + def test_depth_used(self): + self.tool._run(action="trace_causal_chain", decision_id="dec-001", depth=5) + self.ctx.knowledge_graph.trace_decision_causality.assert_called_once_with( + "dec-001", max_depth=5 + ) + + def test_graceful_error_when_context_has_no_knowledge_graph(self): + """Regression: an unguarded ``self.context.knowledge_graph`` read raised + AttributeError out of ``_run`` and could hard-fail a crew task. It must + return honest error JSON instead.""" + del self.ctx.knowledge_graph + result = json.loads( + self.tool._run(action="trace_causal_chain", decision_id="dec-003") + ) + self.assertEqual(result["causal_chain"], []) + self.assertIn("error", result) + + +class TestAnalyzeImpact(unittest.TestCase): + + def setUp(self): + self.ctx = _make_context() + self.tool = SemanticaDecisionTool(context=self.ctx) + + def test_returns_json_with_decision_id(self): + result = json.loads( + self.tool._run(action="analyze_impact", decision_id="dec-001") + ) + self.assertEqual(result["decision_id"], "dec-001") + + def test_includes_influence_metrics(self): + result = json.loads( + self.tool._run(action="analyze_impact", decision_id="dec-001") + ) + self.assertIn("centrality", result) + + +class TestCheckPolicy(unittest.TestCase): + + def setUp(self): + self.ctx = _make_context() + self.tool = SemanticaDecisionTool(context=self.ctx) + + def test_returns_json_with_compliant_key(self): + decision = json.dumps( + {"category": "loan", "outcome": "approved", "confidence": 0.9} + ) + result = json.loads( + self.tool._run(action="check_policy", decision_data=decision) + ) + self.assertIn("compliant", result) + + def test_invalid_json_returns_error(self): + result = json.loads( + self.tool._run(action="check_policy", decision_data="{not valid json}") + ) + self.assertFalse(result["compliant"]) + self.assertGreater(len(result["violations"]), 0) + + def test_rule_violation_detected(self): + decision = json.dumps({"confidence": 0.5}) + rules = json.dumps(["confidence >= 0.9"]) + result = json.loads( + self.tool._run( + action="check_policy", decision_data=decision, policy_rules=rules + ) + ) + self.assertFalse(result["compliant"]) + self.assertEqual(len(result["violations"]), 1) + + def test_bool_false_rule_is_compliant(self): + """Regression: ``enabled == false`` with ``enabled: false`` must be + compliant — bool("false") is truthy, so the old coercion inverted it.""" + decision = json.dumps({"enabled": False, "confidence": 0.95}) + rules = json.dumps(["enabled == false"]) + result = json.loads( + self.tool._run( + action="check_policy", decision_data=decision, policy_rules=rules + ) + ) + self.assertTrue(result["compliant"]) + self.assertEqual(result["violations"], []) + + def test_bool_true_rule_is_compliant(self): + decision = json.dumps({"enabled": True}) + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=decision, + policy_rules=json.dumps(["enabled == true"]), + ) + ) + self.assertTrue(result["compliant"]) + + def test_whitespace_padded_strings_are_trimmed(self): + """Regression: ``_coerce_value`` must return the *stripped* string for + non-numeric literals, or padded decision_data fields never match.""" + decision = json.dumps({"status": " approved "}) + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=decision, + policy_rules=json.dumps(["status == approved"]), + ) + ) + self.assertTrue(result["compliant"]) + self.assertEqual(result["violations"], []) + + def test_bool_false_rule_violated_when_true(self): + decision = json.dumps({"enabled": True}) + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=decision, + policy_rules=json.dumps(["enabled == false"]), + ) + ) + self.assertFalse(result["compliant"]) + self.assertEqual(len(result["violations"]), 1) + + def test_zero_one_flag_parsed_as_bool(self): + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=json.dumps({"flag": 1}), + policy_rules=json.dumps(["flag != 0"]), + ) + ) + self.assertTrue(result["compliant"]) + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=json.dumps({"flag": 0}), + policy_rules=json.dumps(["flag != 0"]), + ) + ) + self.assertFalse(result["compliant"]) + + def test_numeric_string_value_compared_numerically(self): + """Regression: a string datum like "0.90" must compare numerically to + rule literal 0.9, not lexicographically.""" + decision = json.dumps({"score": "0.90"}) + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=decision, + policy_rules=json.dumps(["score == 0.9"]), + ) + ) + self.assertTrue(result["compliant"]) + + def test_numeric_string_ordering(self): + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=json.dumps({"pct": "0.95"}), + policy_rules=json.dumps(["pct >= 0.9"]), + ) + ) + self.assertTrue(result["compliant"]) + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=json.dumps({"pct": "0.85"}), + policy_rules=json.dumps(["pct >= 0.9"]), + ) + ) + self.assertFalse(result["compliant"]) + + def test_field_names_with_hyphens_dots_spaces(self): + """Rule field names are not limited to ``\\w+`` — hyphenated/dotted + (and space-containing) JSON keys must be addressable.""" + decision = json.dumps({"risk-score": 0.95, "max.risk": 0.2, "min score": 0.4}) + compliant = json.loads( + self.tool._run( + action="check_policy", + decision_data=decision, + policy_rules=json.dumps( + ["risk-score >= 0.9", "max.risk <= 0.5", "min score >= 0.3"] + ), + ) + ) + self.assertTrue(compliant["compliant"]) + self.assertEqual(compliant["violations"], []) + violated = json.loads( + self.tool._run( + action="check_policy", + decision_data=decision, + policy_rules=json.dumps(["max.risk >= 0.5"]), + ) + ) + self.assertFalse(violated["compliant"]) + self.assertEqual(len(violated["violations"]), 1) + + def test_rule_missing_field_warns_not_silently_compliant(self): + decision = json.dumps({"confidence": 0.95}) + rules = json.dumps(["minimum_score >= 0.9"]) + result = json.loads( + self.tool._run( + action="check_policy", decision_data=decision, policy_rules=rules + ) + ) + self.assertTrue(result["compliant"]) + self.assertEqual(result["violations"], []) + self.assertEqual(len(result["warnings"]), 1) + self.assertIn("minimum_score", result["warnings"][0]) + + def test_decision_data_non_object_rejected(self): + result = json.loads( + self.tool._run( + action="check_policy", + decision_data=json.dumps(["confidence", 0.95]), + policy_rules=json.dumps(["confidence >= 0.9"]), + ) + ) + self.assertFalse(result["compliant"]) + self.assertEqual(len(result["violations"]), 1) + self.assertIn("JSON object", result["violations"][0]) + + def test_unknown_action_returns_error(self): + result = json.loads(self.tool._run(action="nope")) + self.assertIn("error", result) + + def test_run_entrypoint(self): + result = json.loads( + self.tool.run( + action="check_policy", + decision_data=json.dumps({"confidence": 0.95}), + policy_rules=json.dumps(["confidence >= 0.9"]), + ) + ) + self.assertTrue(result["compliant"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/integrations/crewai/test_degradation.py b/tests/integrations/crewai/test_degradation.py new file mode 100644 index 00000000..7986f783 --- /dev/null +++ b/tests/integrations/crewai/test_degradation.py @@ -0,0 +1,103 @@ +""" +Graceful-degradation tests for the CrewAI integration. + +These run the integration modules in a fresh subprocess (no conftest crewai +stubs, no real crewai) to prove that every public class remains importable and +functional when ``crewai`` is absent. A subprocess is used because the other +test files in this directory install crewai stubs into ``sys.modules`` for the +whole pytest session; a subprocess keeps the two scenarios isolated. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import unittest + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))) + +_SCRIPT = r""" +import json +import sys + +try: + import crewai # noqa: F401 + real_crewai = True +except ImportError: + real_crewai = False + +from integrations.crewai import ( + CREWAI_AVAILABLE, + SemanticaKGTool, + SemanticaDecisionTool, + SemanticaKnowledgeSource, +) +from semantica.context import ContextGraph + +assert CREWAI_AVAILABLE == real_crewai, ( + f"CREWAI_AVAILABLE={CREWAI_AVAILABLE} but real crewai={real_crewai}" +) + +# --- SemanticaKGTool: importable + functional without crewai ----------------- +graph = ContextGraph() +graph.add_node(node_id="privacy", node_type="policy", content="privacy policy doc") + +tool = SemanticaKGTool(graph=graph) +assert tool.name == "semantica_knowledge_graph" +assert tool.args_schema is not None + +res = json.loads(tool._run(action="query_graph", query="privacy")) +assert res["count"] == 1, res +res = json.loads(tool._run(action="find_related", entity="ghost", hops=1)) +assert res["count"] == 0, res + +# The public run()/arun() entry points must exist without crewai too. +res = json.loads(tool.run(action="query_graph", query="privacy")) +assert res["count"] == 1, res +import asyncio +res = json.loads(asyncio.run(tool.arun(action="query_graph", query="privacy"))) +assert res["count"] == 1, res + +# --- SemanticaKnowledgeSource: importable + functional without crewai -------- +src = SemanticaKnowledgeSource(graph=graph, chunk_size=40, chunk_overlap=5) +assert src.load_content() != {} +assert src.validate_content() is True +src.add() # must not raise; chunks kept in memory +assert len(src.chunks) > 0 + +# --- SemanticaDecisionTool: importable, builds its own context -------------- +dt = SemanticaDecisionTool() +assert dt.name == "semantica_decision" +res = json.loads(dt.run(action="find_precedents", scenario="x")) +assert "precedents" in res, res +res = json.loads(asyncio.run(dt.arun(action="find_precedents", scenario="x"))) +assert "precedents" in res, res + +print("DEGRADATION_OK") +""" + + +class TestDegradation(unittest.TestCase): + + def test_importable_and_functional_without_crewai(self): + result = subprocess.run( + [sys.executable, "-c", _SCRIPT], + cwd=REPO_ROOT, + capture_output=True, + text=True, + timeout=180, + ) + self.assertEqual( + result.returncode, + 0, + msg=( + f"subprocess failed:\nSTDOUT:\n{result.stdout}\n" + f"STDERR:\n{result.stderr}" + ), + ) + self.assertIn("DEGRADATION_OK", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/integrations/crewai/test_kg_tool.py b/tests/integrations/crewai/test_kg_tool.py new file mode 100644 index 00000000..18ee1eda --- /dev/null +++ b/tests/integrations/crewai/test_kg_tool.py @@ -0,0 +1,453 @@ +""" +Tests for SemanticaKGTool — knowledge graph CrewAI tool. + +Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is +``True`` and the real Pydantic/BaseTool subclassing path is exercised. +""" + +from __future__ import annotations + +import asyncio +import json +import unittest +from unittest.mock import MagicMock + +from integrations.crewai import SemanticaKGTool as ImportedSemanticaKGTool +from integrations.crewai.kg_tool import ( + CREWAI_AVAILABLE, + CREWAI_IMPORT_ERROR, + SemanticaKGTool, + SemanticaKGToolInput, +) +from semantica.context import ContextGraph + + +# --------------------------------------------------------------------------- +# Fakes +# --------------------------------------------------------------------------- +def _fake_entity(name="Tesla", etype="ORG", conf=0.9): + e = MagicMock() + e.name = name + e.type = etype + e.confidence = conf + return e + + +def _fake_relation(src="Tesla", rel="FOUNDED_BY", tgt="Elon Musk", conf=0.85): + r = MagicMock() + r.source = src + r.type = rel + r.target = tgt + r.confidence = conf + return r + + +class _FakeNER: + def extract_entities(self, text): + return [_fake_entity("Tesla"), _fake_entity("Elon Musk", "PERSON")] + + +class _FakeRelExtractor: + def extract_relations(self, text, entities=None): + return [_fake_relation()] + + +class _DataclassNER: + """Returns Semantica's real ``Entity`` dataclass shape (text/label, no name).""" + + def extract_entities(self, text): + from semantica.semantic_extract.types import Entity + + return [ + Entity(text="Tesla", label="ORG", start_char=0, end_char=5), + Entity(text="Elon Musk", label="PERSON", start_char=17, end_char=26), + ] + + +class _DataclassRelExtractor: + """Returns Semantica's real ``Relation`` dataclass shape (subject/object).""" + + def __init__(self): + self.received_entities = None + + def extract_relations(self, text, entities=None): + from semantica.semantic_extract.types import Entity, Relation + + self.received_entities = entities + return [ + Relation( + subject=Entity(text="Tesla", label="ORG", start_char=0, end_char=5), + predicate="FOUNDED_BY", + object=Entity( + text="Elon Musk", label="PERSON", start_char=17, end_char=26 + ), + ) + ] + + +class TestSemanticaKGToolInit(unittest.TestCase): + + def test_crewai_available_via_stub(self): + self.assertTrue(CREWAI_AVAILABLE) + self.assertIsNone(CREWAI_IMPORT_ERROR) + + def test_is_base_tool_subclass(self): + from crewai.tools import BaseTool + + self.assertTrue(issubclass(SemanticaKGTool, BaseTool)) + + def test_exposed_from_package_init(self): + self.assertIs(ImportedSemanticaKGTool, SemanticaKGTool) + + def test_creates_with_explicit_graph(self): + graph = ContextGraph() + tool = SemanticaKGTool(graph=graph) + self.assertIs(tool.graph, graph) + + def test_creates_fresh_graph_when_none(self): + tool = SemanticaKGTool( + ner_extractor=_FakeNER(), relation_extractor=_FakeRelExtractor() + ) + self.assertIsNotNone(tool.graph) + self.assertIsInstance(tool.graph, ContextGraph) + + def test_default_metadata(self): + tool = SemanticaKGTool( + ner_extractor=_FakeNER(), relation_extractor=_FakeRelExtractor() + ) + self.assertEqual(tool.name, "semantica_knowledge_graph") + self.assertTrue(tool.description) + self.assertEqual(tool.args_schema, SemanticaKGToolInput) + + def test_input_schema_validates(self): + inp = SemanticaKGToolInput(action="query_graph", query="privacy", hops=2) + self.assertEqual(inp.hops, 2) + with self.assertRaises(Exception): + SemanticaKGToolInput(action="bogus") + + def test_custom_kwargs_forwarded(self): + tool = SemanticaKGTool( + ner_extractor=_FakeNER(), + relation_extractor=_FakeRelExtractor(), + result_as_answer=True, + ) + self.assertTrue(tool.result_as_answer) + + +class TestSemanticaKGToolSerialization(unittest.TestCase): + """CrewAI checkpoints serialise tools via ``model_dump(mode="json")`` — the + live graph/extractors must not break that (regression for + PydanticSerializationError on arbitrary state objects).""" + + def setUp(self): + self.tool = SemanticaKGTool( + graph=ContextGraph(), + ner_extractor=_FakeNER(), + relation_extractor=_FakeRelExtractor(), + ) + + def test_model_dump_json_excludes_shared_state(self): + dumped = self.tool.model_dump(mode="json") + self.assertNotIn("graph", dumped) + self.assertNotIn("ner_extractor", dumped) + self.assertNotIn("relation_extractor", dumped) + self.assertEqual(dumped["name"], "semantica_knowledge_graph") + + def test_model_validate_restores_defaults(self): + restored = SemanticaKGTool.model_validate(self.tool.model_dump(mode="json")) + self.assertIsInstance(restored.graph, ContextGraph) + self.assertIs(restored.args_schema, SemanticaKGToolInput) + self.assertEqual(restored.name, "semantica_knowledge_graph") + + def test_model_validate_restored_tool_still_runs(self): + restored = SemanticaKGTool.model_validate(self.tool.model_dump(mode="json")) + restored.graph.add_node(node_id="privacy", node_type="policy") + result = json.loads(restored._run(action="query_graph", query="privacy")) + self.assertEqual(result["count"], 1) + + def test_restore_flags_lost_live_state(self): + """A tool restored from a checkpoint must signal that its live graph + was excluded and an empty one reconstructed (``reconstructed_state``).""" + dumped = self.tool.model_dump(mode="json") + self.assertTrue(dumped["had_live_state"]) + self.assertNotIn("reconstructed_state", dumped) + restored = SemanticaKGTool.model_validate(dumped) + self.assertTrue(restored.reconstructed_state) + self.assertFalse(SemanticaKGTool().reconstructed_state) + + +class TestSemanticaKGToolActions(unittest.TestCase): + + def setUp(self): + self.graph = ContextGraph() + self.tool = SemanticaKGTool( + graph=self.graph, + ner_extractor=_FakeNER(), + relation_extractor=_FakeRelExtractor(), + ) + + def test_extract_entities(self): + result = json.loads( + self.tool._run( + action="extract_entities", text="Tesla was founded by Elon Musk" + ) + ) + self.assertEqual(result["count"], 2) + self.assertEqual(result["entities"][0]["name"], "Tesla") + self.assertEqual(result["entities"][0]["type"], "ORG") + + def test_extract_relations(self): + result = json.loads( + self.tool._run( + action="extract_relations", text="Tesla was founded by Elon Musk" + ) + ) + self.assertEqual(result["count"], 1) + self.assertEqual(result["relations"][0]["source"], "Tesla") + self.assertEqual(result["relations"][0]["target"], "Elon Musk") + + def test_add_to_graph_populates_graph(self): + result = json.loads( + self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk") + ) + self.assertGreaterEqual(result["nodes_added"], 2) + self.assertGreaterEqual(result["edges_added"], 1) + nodes = self.graph.find_nodes() + node_ids = {n["id"] for n in nodes} + self.assertIn("Tesla", node_ids) + self.assertIn("Elon Musk", node_ids) + + def test_add_to_graph_is_idempotent(self): + self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk") + second = json.loads( + self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk") + ) + self.assertEqual(second["nodes_added"], 0) + self.assertEqual(second["edges_added"], 0) + + def test_query_graph_finds_matching_node(self): + self.graph.add_node( + node_id="privacy", node_type="policy", content="privacy policy doc" + ) + result = json.loads(self.tool._run(action="query_graph", query="privacy")) + self.assertEqual(result["count"], 1) + self.assertEqual(result["results"][0]["id"], "privacy") + + def test_query_graph_no_match(self): + result = json.loads( + self.tool._run(action="query_graph", query="nothing-matches") + ) + self.assertEqual(result["count"], 0) + self.assertEqual(result["results"], []) + + def test_query_graph_searches_node_content(self): + """query_graph must match node content, not just ids/types.""" + self.graph.add_node( + node_id="n1", + node_type="policy", + content="all refunds must be processed within 30 days", + ) + result = json.loads(self.tool._run(action="query_graph", query="refunds")) + self.assertEqual(result["count"], 1) + self.assertEqual(result["results"][0]["id"], "n1") + + def test_query_graph_matches_type(self): + self.graph.add_node(node_id="n2", node_type="risk") + result = json.loads(self.tool._run(action="query_graph", query="risk")) + self.assertEqual(result["count"], 1) + self.assertEqual(result["results"][0]["id"], "n2") + + def test_query_graph_result_shape_is_consistent(self): + """Every result — content match or id/type match — must carry the same + keys (id, type, label, content, score) so agents get one schema.""" + self.graph.add_node( + node_id="n1", + node_type="policy", + content="all refunds within 30 days", + ) + by_content = json.loads(self.tool._run(action="query_graph", query="refunds"))[ + "results" + ][0] + expected_keys = {"id", "type", "label", "content", "score"} + self.assertEqual(set(by_content.keys()), expected_keys) + + by_id = json.loads(self.tool._run(action="query_graph", query="n1"))["results"][ + 0 + ] + self.assertEqual(set(by_id.keys()), expected_keys) + self.assertEqual(by_id["content"], "all refunds within 30 days") + self.assertEqual(by_id["score"], 1.0) + + def test_extract_entities_skips_nameless_entities(self): + class _NamelessNER: + def extract_entities(self, text): + e = MagicMock() + e.name = None + e.type = "MISC" + e.confidence = 0.5 + return [e] + + tool = SemanticaKGTool( + graph=self.graph, + ner_extractor=_NamelessNER(), + relation_extractor=_FakeRelExtractor(), + ) + result = json.loads(tool._run(action="extract_entities", text="text")) + self.assertEqual(result["count"], 0) + self.assertEqual(result["entities"], []) + + def test_find_related_multi_hop(self): + self.graph.add_node(node_id="A", node_type="concept") + self.graph.add_node(node_id="B", node_type="concept") + self.graph.add_node(node_id="C", node_type="concept") + self.graph.add_edge(source_id="A", target_id="B", edge_type="related_to") + self.graph.add_edge(source_id="B", target_id="C", edge_type="related_to") + result = json.loads(self.tool._run(action="find_related", entity="A", hops=2)) + self.assertEqual(result["count"], 2) + self.assertIn("B", result["related"]) + self.assertIn("C", result["related"]) + + def test_find_related_unknown_entity(self): + result = json.loads( + self.tool._run(action="find_related", entity="Ghost", hops=1) + ) + self.assertEqual(result["count"], 0) + self.assertEqual(result["related"], []) + + def test_find_related_honors_incoming_edges(self): + """find_related must be undirected: a node whose only edge is + incoming (A -> B) is still related to A.""" + self.graph.add_node(node_id="OpenAI", node_type="ORG") + self.graph.add_node(node_id="Google", node_type="ORG") + self.graph.add_edge( + source_id="OpenAI", target_id="Google", edge_type="related_to" + ) + result = json.loads(self.tool._run(action="find_related", entity="Google")) + self.assertEqual(result["related"], ["OpenAI"]) + result_out = json.loads(self.tool._run(action="find_related", entity="OpenAI")) + self.assertEqual(result_out["related"], ["Google"]) + + def test_unknown_action_returns_error(self): + result = json.loads(self.tool._run(action="do_something_else")) + self.assertIn("error", result) + self.assertIn("do_something_else", result["error"]) + + def test_extract_entities_empty_text_is_graceful(self): + result = json.loads(self.tool._run(action="extract_entities", text="")) + self.assertIn("entities", result) + + def test_extract_entities_confidence_none_defaults_to_one(self): + """A single entity with ``confidence=None`` must not nuke the whole + extract result — it normalises to 1.0 instead of raising float(None).""" + + class _NoneConfNER: + def extract_entities(self, text): + e = MagicMock() + e.name = "X" + e.type = "MISC" + e.confidence = None + return [e] + + tool = SemanticaKGTool( + graph=self.graph, + ner_extractor=_NoneConfNER(), + relation_extractor=_FakeRelExtractor(), + ) + result = json.loads(tool._run(action="extract_entities", text="text")) + self.assertEqual(result["count"], 1) + self.assertEqual(result["entities"][0]["name"], "X") + self.assertEqual(result["entities"][0]["confidence"], 1.0) + self.assertNotIn("error", result) + + def test_graph_lock_is_per_graph(self): + """Independent graphs must not share a batch lock.""" + g2 = ContextGraph() + lock_a = self.tool._graph_lock(self.graph) + lock_a_again = self.tool._graph_lock(self.graph) + lock_b = self.tool._graph_lock(g2) + self.assertIs(lock_a, lock_a_again) + self.assertIsNot(lock_a, lock_b) + + +class TestSemanticaKGToolDataclassShapes(unittest.TestCase): + """Real Semantica ``Entity``/``Relation`` dataclasses (text/label, + subject/object) instead of MagicMock-shaped fakes.""" + + def setUp(self): + self.ner = _DataclassNER() + self.rel = _DataclassRelExtractor() + self.graph = ContextGraph() + self.tool = SemanticaKGTool( + graph=self.graph, ner_extractor=self.ner, relation_extractor=self.rel + ) + + def test_extract_entities_reads_text_label(self): + result = json.loads( + self.tool._run(action="extract_entities", text="Tesla founded by Elon Musk") + ) + self.assertEqual(result["count"], 2) + self.assertEqual(result["entities"][0]["name"], "Tesla") + self.assertEqual(result["entities"][0]["type"], "ORG") + self.assertEqual(result["entities"][1]["name"], "Elon Musk") + self.assertEqual(result["entities"][1]["type"], "PERSON") + + def test_extract_relations_reads_subject_object(self): + result = json.loads( + self.tool._run( + action="extract_relations", text="Tesla founded by Elon Musk" + ) + ) + self.assertEqual(result["count"], 1) + self.assertEqual(result["relations"][0]["source"], "Tesla") + self.assertEqual(result["relations"][0]["relation"], "FOUNDED_BY") + self.assertEqual(result["relations"][0]["target"], "Elon Musk") + + def test_add_to_graph_passes_entity_objects_to_relation_extractor(self): + result = json.loads( + self.tool._run(action="add_to_graph", text="Tesla founded by Elon Musk") + ) + self.assertEqual(result["nodes_added"], 2) + self.assertEqual(result["edges_added"], 1) + from semantica.semantic_extract.types import Entity + + self.assertIsNotNone(self.rel.received_entities) + for e in self.rel.received_entities: + self.assertIsInstance(e, Entity) + node_ids = {n["id"] for n in self.graph.find_nodes()} + self.assertIn("Tesla", node_ids) + self.assertIn("Elon Musk", node_ids) + edge_keys = { + (e["source"], e["type"], e["target"]) for e in self.graph.find_edges() + } + self.assertIn(("Tesla", "FOUNDED_BY", "Elon Musk"), edge_keys) + + +class TestSemanticaKGToolCrewAIEntrypoints(unittest.TestCase): + + def setUp(self): + self.tool = SemanticaKGTool( + graph=ContextGraph(), + ner_extractor=_FakeNER(), + relation_extractor=_FakeRelExtractor(), + ) + + def test_run_delegates_to_run(self): + result = json.loads( + self.tool.run(action="extract_entities", text="Tesla led by Elon Musk") + ) + self.assertEqual(result["count"], 2) + + def test_arun_async(self): + async def _call(): + return await self.tool.arun(action="query_graph", query="x") + + result = json.loads(asyncio.run(_call())) + self.assertIn("results", result) + + def test_run_returns_string(self): + out = self.tool.run(action="extract_entities", text="hello world") + self.assertIsInstance(out, str) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/integrations/crewai/test_knowledge_source.py b/tests/integrations/crewai/test_knowledge_source.py new file mode 100644 index 00000000..f76cc815 --- /dev/null +++ b/tests/integrations/crewai/test_knowledge_source.py @@ -0,0 +1,228 @@ +""" +Tests for SemanticaKnowledgeSource — CrewAI knowledge source backed by a +Semantica ContextGraph. + +Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is +``True`` and the real Pydantic/BaseKnowledgeSource subclassing path (including +the current ``validate_content`` / ``add`` / ``aadd`` contract) is exercised. +""" + +from __future__ import annotations + +import asyncio +import unittest + +from integrations.crewai import SemanticaKnowledgeSource +from integrations.crewai.knowledge_source import CREWAI_AVAILABLE, _chunk_text_manual +from semantica.context import ContextGraph + + +class _FakeStorage: + def __init__(self): + self.saved_chunks: list = [] + + def save(self, chunks: list) -> None: + self.saved_chunks.extend(chunks) + + async def asave(self, chunks: list) -> None: + self.saved_chunks.extend(chunks) + + +class _RaisingStorage(_FakeStorage): + """Mirrors real crewai: storage is wired but ``save`` raises ``ValueError`` + (e.g. the embedder has no credentials configured).""" + + def save(self, chunks: list) -> None: + raise ValueError("The OPENAI_API_KEY environment variable is not set.") + + async def asave(self, chunks: list) -> None: + raise ValueError("The OPENAI_API_KEY environment variable is not set.") + + +def _build_graph() -> ContextGraph: + graph = ContextGraph() + graph.add_node(node_id="privacy", node_type="policy", content="privacy policy doc") + graph.add_node(node_id="fraud", node_type="risk", content="fraud detection rules") + graph.add_edge(source_id="privacy", target_id="fraud", edge_type="constrains") + return graph + + +class TestSemanticaKnowledgeSourceInit(unittest.TestCase): + + def test_crewai_available_via_stub(self): + self.assertTrue(CREWAI_AVAILABLE) + + def test_is_base_knowledge_source_subclass(self): + from crewai.knowledge.source import BaseKnowledgeSource + + self.assertTrue(issubclass(SemanticaKnowledgeSource, BaseKnowledgeSource)) + + def test_creates_with_explicit_graph(self): + graph = _build_graph() + src = SemanticaKnowledgeSource(graph=graph) + self.assertIs(src.graph, graph) + + def test_creates_fresh_graph_when_none(self): + src = SemanticaKnowledgeSource() + self.assertIsNotNone(src.graph) + self.assertIsInstance(src.graph, ContextGraph) + + def test_default_metadata(self): + src = SemanticaKnowledgeSource(graph=_build_graph()) + self.assertEqual(src.name, "semantica_knowledge_graph") + self.assertEqual(src.chunk_size, 4000) + self.assertEqual(src.chunk_overlap, 200) + + def test_custom_chunking_params(self): + src = SemanticaKnowledgeSource( + graph=_build_graph(), chunk_size=50, chunk_overlap=10 + ) + self.assertEqual(src.chunk_size, 50) + self.assertEqual(src.chunk_overlap, 10) + + +class TestLoadContent(unittest.TestCase): + + def setUp(self): + self.graph = _build_graph() + self.src = SemanticaKnowledgeSource(graph=self.graph) + + def test_nodes_serialized(self): + content = self.src.load_content() + text = "\n".join(content.values()) + self.assertIn("privacy", text) + self.assertIn("fraud", text) + self.assertIn("policy", text) + + def test_edges_serialized(self): + content = self.src.load_content() + text = "\n".join(content.values()) + self.assertIn("-[" + "constrains" + "]->", text) + + def test_empty_graph_returns_empty(self): + src = SemanticaKnowledgeSource(graph=ContextGraph()) + self.assertEqual(src.load_content(), {}) + + def test_validate_content_passes(self): + self.assertTrue(self.src.validate_content()) + + def test_validate_content_raises_without_graph(self): + self.src.graph = None + with self.assertRaises(ValueError): + self.src.validate_content() + + +class TestAdd(unittest.TestCase): + + def setUp(self): + self.graph = _build_graph() + self.src = SemanticaKnowledgeSource( + graph=self.graph, chunk_size=40, chunk_overlap=5 + ) + + def test_add_saves_chunks_to_storage(self): + storage = _FakeStorage() + self.src.storage = storage + self.src.add() + self.assertGreater(len(storage.saved_chunks), 0) + self.assertTrue(all(isinstance(c, str) and c for c in storage.saved_chunks)) + + def test_add_without_storage_keeps_chunks_in_memory(self): + self.src.add() + self.assertGreater(len(self.src.chunks), 0) + self.assertGreater(len(self.src._chunks), 0) + + def test_add_wired_storage_failure_logs_error_not_debug(self): + """Regression: real crewai raises ``ValueError`` for a missing embedder + even though storage IS wired. That used to fall into the "storage not + wired" DEBUG branch, silently hiding the failure — it must log an + actionable ERROR instead.""" + self.src.storage = _RaisingStorage() + with self.assertLogs( + f"semantica.{SemanticaKnowledgeSource.__module__}", level="ERROR" + ) as caught: + self.src.add() + joined = "\n".join(caught.output) + self.assertIn("storage save FAILED", joined) + self.assertIn("OPENAI_API_KEY", joined) + self.assertGreater(len(self.src.chunks), 0) + + def test_add_empty_graph_no_chunks(self): + src = SemanticaKnowledgeSource( + graph=ContextGraph(), chunk_size=40, chunk_overlap=5 + ) + src.add() + self.assertEqual(src.chunks, []) + + def test_aadd_async(self): + storage = _FakeStorage() + self.src.storage = storage + asyncio.run(self.src.aadd()) + self.assertGreater(len(storage.saved_chunks), 0) + + def test_content_summary(self): + summary = self.src.get_content_summary() + self.assertEqual(summary["name"], "semantica_knowledge_graph") + self.assertGreater(summary["source_count"], 0) + self.assertTrue(summary["crewai_available"]) + + +class TestSemanticaKnowledgeSourceSerialization(unittest.TestCase): + """CrewAI checkpoints serialise their models via ``model_dump(mode="json")`` + — the live graph must not break that (regression for + PydanticSerializationError on arbitrary state objects).""" + + def test_model_dump_json_excludes_graph(self): + src = SemanticaKnowledgeSource(graph=_build_graph()) + dumped = src.model_dump(mode="json") + self.assertNotIn("graph", dumped) + self.assertEqual(dumped["name"], "semantica_knowledge_graph") + + def test_model_validate_restores_graph(self): + src = SemanticaKnowledgeSource(graph=_build_graph()) + restored = SemanticaKnowledgeSource.model_validate(src.model_dump(mode="json")) + self.assertIsInstance(restored.graph, ContextGraph) + + def test_restored_source_still_loads_content(self): + """A checkpoint-restored source gets a fresh graph (the live graph is + excluded from serialisation); once a graph is attached it works.""" + src = SemanticaKnowledgeSource(graph=_build_graph()) + restored = SemanticaKnowledgeSource.model_validate(src.model_dump(mode="json")) + restored.graph = _build_graph() + self.assertNotEqual(restored.load_content(), {}) + + def test_restore_flags_lost_live_state(self): + """A source restored from a checkpoint must signal that its live graph + was excluded and an empty one reconstructed (``reconstructed_state``). + Regression: an eager graph build in ``__init__`` used to hide this.""" + src = SemanticaKnowledgeSource(graph=_build_graph()) + dumped = src.model_dump(mode="json") + self.assertTrue(dumped["had_live_state"]) + self.assertNotIn("reconstructed_state", dumped) + restored = SemanticaKnowledgeSource.model_validate(dumped) + self.assertTrue(restored.reconstructed_state) + self.assertFalse(SemanticaKnowledgeSource().reconstructed_state) + self.assertIsInstance(SemanticaKnowledgeSource().graph, ContextGraph) + + +class TestManualChunker(unittest.TestCase): + + def test_short_text_single_chunk(self): + self.assertEqual(_chunk_text_manual("hello", 40, 5), ["hello"]) + + def test_empty_text(self): + self.assertEqual(_chunk_text_manual("", 40, 5), []) + + def test_long_text_overlaps(self): + chunks = _chunk_text_manual("a" * 100, 40, 10) + self.assertGreater(len(chunks), 1) + self.assertTrue(all(len(c) <= 40 for c in chunks)) + # Overlap means consecutive chunks share tail/head content + self.assertIn("a" * 10, chunks[0][-10:] + chunks[1][:10]) + + def test_zero_chunk_size_guarded(self): + self.assertEqual(_chunk_text_manual("hello world", 0, 5), ["hello world"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/integrations/crewai/test_real_crewai_integration.py b/tests/integrations/crewai/test_real_crewai_integration.py new file mode 100644 index 00000000..b9b9d619 --- /dev/null +++ b/tests/integrations/crewai/test_real_crewai_integration.py @@ -0,0 +1,123 @@ +""" +End-to-end integration tests against the REAL crewai package. + +These run in a subprocess because the stubs in ``conftest.py`` install a fake +``crewai`` module into ``sys.modules`` for the whole pytest session — the same +interpreter can never see both. Each test launches a fresh interpreter; if +crewai is genuinely not installed there, the test is skipped. + +This covers the failure class the stubs cannot: ``Crew``-level serialization +(list[BaseTool] inside Agent.tools), checkpoint restore via ``model_validate``, +and knowledge-source behaviour with a real ``Crew``. +""" + +from __future__ import annotations + +import subprocess +import sys +import textwrap +import unittest +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[3] + +_SCRIPT = textwrap.dedent( + """ + import os + import json + import sys + + sys.path.insert(0, os.getcwd()) + + try: + import crewai + except ImportError: + print("CREWAI_IMPORT_FAILED") + sys.exit(2) + + import crewai as crewai_mod + from crewai import Agent, Task, Crew + + from semantica.context import ContextGraph + from integrations.crewai import ( + SemanticaKGTool, + SemanticaDecisionTool, + SemanticaKnowledgeSource, + ) + + os.environ["CREWAI_DESERIALIZE_CALLBACKS"] = "1" + + # --- 1. Crew-level serialization round-trip ------------------------------ + graph = ContextGraph() + graph.add_node(node_id="privacy", node_type="policy", + content="privacy policy: no data sharing") + tool = SemanticaKGTool(graph=graph) + + decision_ctx = SemanticaDecisionTool() + decision_tool = SemanticaDecisionTool(context=decision_ctx.context) + + agent = Agent(role="researcher", goal="answer questions", + backstory="retrieves from a knowledge graph", + tools=[tool, decision_tool]) + task = Task(description="answer", expected_output="an answer", agent=agent) + crew = Crew(agents=[agent], tasks=[task]) + + dump = crew.model_dump(mode="json") + agents = dump["agents"] + assert len(agents) == 1, f"expected 1 agent, got {len(agents)}" + dumped_tools = agents[0]["tools"] + assert len(dumped_tools) == 2, f"expected 2 tools, got {len(dumped_tools)}" + for t in dumped_tools: + assert isinstance(t, dict), f"tool not serialized to dict: {type(t)}" + assert "graph" not in t, "live graph leaked into serialized tool" + assert "context" not in t, "live context leaked into serialized tool" + assert "ner_extractor" not in t, "extractor leaked into serialized tool" + + # --- 2. Restore a tool from the crew dump -------------------------------- + kg_dump = dumped_tools[0] + assert kg_dump["name"] == "semantica_knowledge_graph", kg_dump["name"] + restored = SemanticaKGTool.model_validate(kg_dump) + assert restored.graph is not None, "restored tool did not self-heal a graph" + q = json.loads(restored._run(action="query_graph", query="privacy")) + assert "results" in q, f"restored tool query_graph failed: {q}" + + # --- 3. Knowledge source with no embedder must not crash a Crew ---------- + ks = SemanticaKnowledgeSource(graph=graph) + agent2 = Agent(role="researcher2", goal="answer", + backstory="retrieves from knowledge") + task2 = Task(description="q", expected_output="a", agent=agent2) + crew2 = Crew(agents=[agent2], tasks=[task2], + knowledge_sources=[ks]) + assert ks.chunks, "knowledge source retained no chunks in memory" + assert crew2.knowledge is not None, "crew.knowledge not created" + + print("REAL_CREWAI_OK") + """ +) + + +class TestRealCrewAIIntegration(unittest.TestCase): + + def _run(self) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, "-c", _SCRIPT], + cwd=str(REPO_ROOT), + capture_output=True, + text=True, + timeout=240, + ) + + def test_crew_level_round_trip_with_real_crewai(self): + proc = self._run() + if proc.returncode == 2: + self.skipTest("real crewai is not installed in this environment") + self.assertEqual( + proc.returncode, + 0, + msg=f"subprocess failed:\n{proc.stdout}\n{proc.stderr}", + ) + self.assertIn("REAL_CREWAI_OK", proc.stdout) + + +if __name__ == "__main__": + unittest.main() From 8177d887538560d137edc705c7d60abcb9e23faa Mon Sep 17 00:00:00 2001 From: "Guofang.Tang" <136770748@qq.com> Date: Sun, 16 Aug 2026 14:23:24 +0800 Subject: [PATCH 05/19] fix(kg): preserve isolated nodes in graph analytics (#1011) * fix(kg): preserve isolated nodes in graph analytics * fix(kg): support node fallbacks and community payloads --------- --- semantica/kg/_graph_view.py | 206 ++++++++++++++++++++++++++ semantica/kg/centrality_calculator.py | 72 +-------- semantica/kg/community_detector.py | 127 ++++++---------- semantica/kg/connectivity_analyzer.py | 49 +----- tests/kg/test_analytics_node_scope.py | 112 ++++++++++++++ 5 files changed, 374 insertions(+), 192 deletions(-) create mode 100644 semantica/kg/_graph_view.py create mode 100644 tests/kg/test_analytics_node_scope.py diff --git a/semantica/kg/_graph_view.py b/semantica/kg/_graph_view.py new file mode 100644 index 00000000..f467125b --- /dev/null +++ b/semantica/kg/_graph_view.py @@ -0,0 +1,206 @@ +"""Internal graph view helpers shared by KG analytics modules.""" + +from dataclasses import dataclass +from typing import Any, Dict, Iterable, List, Optional, Set, Tuple + + +@dataclass +class GraphView: + """Normalized node and edge view used by graph analytics.""" + + nodes: List[Any] + edges: List[Tuple[Any, Any]] + + +def build_graph_view(graph: Any) -> GraphView: + """Build a graph view without dropping explicitly declared nodes. + + Graph analytics accepts graph dictionaries, ContextGraph-like objects, and + NetworkX graphs. Nodes declared without an incident edge remain in the + returned view so callers can choose how to handle isolated nodes. + """ + nodes: List[Any] = [] + edges: List[Tuple[Any, Any]] = [] + seen_nodes: Set[Any] = set() + seen_edges: Set[Tuple[Any, Any]] = set() + + def add_node(value: Any) -> Optional[Any]: + node_id = _node_id(value) + if node_id is None or node_id == "": + return None + if node_id not in seen_nodes: + seen_nodes.add(node_id) + nodes.append(node_id) + return node_id + + for node in _extract_nodes(graph): + add_node(node) + + for raw_edge in _extract_edges(graph): + edge = _edge_endpoints(raw_edge) + if edge is None: + continue + source, target = edge + source = add_node(source) + target = add_node(target) + if source is None or target is None: + continue + if (source, target) not in seen_edges: + seen_edges.add((source, target)) + edges.append((source, target)) + + return GraphView(nodes=nodes, edges=edges) + + +def build_adjacency(graph: Any, directed: bool = False) -> Dict[Any, List[Any]]: + """Build an adjacency list while preserving isolated graph nodes.""" + view = build_graph_view(graph) + adjacency: Dict[Any, List[Any]] = {node: [] for node in view.nodes} + + for source, target in view.edges: + if target not in adjacency[source]: + adjacency[source].append(target) + if not directed and source not in adjacency[target]: + adjacency[target].append(source) + + return adjacency + + +def _extract_nodes(graph: Any) -> Iterable[Any]: + if isinstance(graph, dict): + raw_nodes: List[Any] = [] + for key in ("entities", "nodes"): + values = graph.get(key, []) + if isinstance(values, dict): + raw_nodes.extend(values.keys()) + elif values: + raw_nodes.extend(values) + return raw_nodes + + raw_nodes = getattr(graph, "nodes", None) + if callable(raw_nodes): + return raw_nodes() + if isinstance(raw_nodes, dict): + return raw_nodes.keys() + if raw_nodes is not None: + return raw_nodes + + get_nodes = getattr(graph, "get_nodes", None) + if callable(get_nodes): + return get_nodes() + return [] + + +def _extract_edges(graph: Any) -> Iterable[Any]: + if isinstance(graph, dict): + raw_edges: List[Any] = [] + for key in ("relationships", "edges"): + values = graph.get(key, []) + if values: + raw_edges.extend(values) + return raw_edges + + raw_edges: List[Any] = [] + relationships = getattr(graph, "relationships", None) + if relationships is not None: + raw_edges.extend(relationships) + edges = getattr(graph, "edges", None) + if callable(edges): + raw_edges.extend(edges()) + elif edges is not None: + raw_edges.extend(edges) + if raw_edges: + return raw_edges + + get_relationships = getattr(graph, "get_relationships", None) + if callable(get_relationships): + return get_relationships() + return [] + + +def _edge_endpoints(edge: Any) -> Optional[Tuple[Any, Any]]: + if isinstance(edge, (tuple, list)) and len(edge) >= 2: + return edge[0], edge[1] + + if isinstance(edge, dict): + source = _first_value( + edge, + "source", + "source_id", + "subject", + "start", + "start_id", + "from", + "src", + "START_ID", + ":START_ID", + ) + target = _first_value( + edge, + "target", + "target_id", + "object", + "end", + "end_id", + "to", + "dst", + "END_ID", + ":END_ID", + ) + else: + source = _first_attribute( + edge, + "source_id", + "source", + "subject", + "start", + "start_id", + "from_id", + ) + target = _first_attribute( + edge, + "target_id", + "target", + "object", + "end", + "end_id", + "to_id", + ) + + if source is None or target is None: + return None + return source, target + + +def _node_id(value: Any) -> Any: + if isinstance(value, dict): + value = _first_value( + value, "id", "node_id", "entity_id", "key", "name", "text" + ) + elif not isinstance(value, (str, int, float, bool, bytes, tuple)): + value = _first_attribute( + value, "node_id", "id", "entity_id", "key", "name", "text" + ) + + if value is None: + return None + try: + hash(value) + except TypeError: + return str(value) + return value + + +def _first_value(mapping: Dict[str, Any], *keys: str) -> Any: + for key in keys: + if key in mapping and mapping[key] not in (None, ""): + return mapping[key] + return None + + +def _first_attribute(value: Any, *names: str) -> Any: + for name in names: + attribute = getattr(value, name, None) + if attribute not in (None, ""): + return attribute + return None diff --git a/semantica/kg/centrality_calculator.py b/semantica/kg/centrality_calculator.py index 9fe9a956..ac59db6e 100644 --- a/semantica/kg/centrality_calculator.py +++ b/semantica/kg/centrality_calculator.py @@ -43,7 +43,7 @@ Author: Semantica Contributors License: MIT """ -from collections import defaultdict, deque +from collections import deque from typing import Any, Dict, List, Optional import numpy as np @@ -51,6 +51,7 @@ from scipy import sparse from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker +from ._graph_view import build_adjacency, build_graph_view class CentralityCalculator: @@ -518,76 +519,15 @@ class CentralityCalculator: def _build_adjacency(self, graph) -> Dict[str, List[str]]: """Build adjacency list from graph.""" - adjacency = defaultdict(list) - - # Extract relationships - relationships = [] - if hasattr(graph, "relationships"): - relationships = graph.relationships - elif hasattr(graph, "get_relationships"): - relationships = graph.get_relationships() - elif isinstance(graph, dict): - relationships = graph.get("relationships", graph.get("edges", [])) - elif hasattr(graph, "edges") and not callable(graph.edges): - # ContextGraph-style: edges is a list of dataclass objects with source_id/target_id - for edge in (graph.edges or []): - if isinstance(edge, dict): - src = edge.get("source") or edge.get("source_id") - tgt = edge.get("target") or edge.get("target_id") - else: - src = getattr(edge, "source_id", None) or getattr(edge, "source", None) - tgt = getattr(edge, "target_id", None) or getattr(edge, "target", None) - if src and tgt: - src, tgt = str(src), str(tgt) - if tgt not in adjacency[src]: - adjacency[src].append(tgt) - if src not in adjacency[tgt]: - adjacency[tgt].append(src) - return dict(adjacency) - - # Build adjacency - for rel in relationships: - # Handle tuple/list edges (e.g., from NetworkX) - if isinstance(rel, (tuple, list)) and len(rel) >= 2: - source, target = str(rel[0]), str(rel[1]) - if source and target: - if target not in adjacency[source]: - adjacency[source].append(target) - if source not in adjacency[target]: - adjacency[target].append(source) - continue - source = rel.get("source") or rel.get("subject") - target = rel.get("target") or rel.get("object") - - # Extract IDs if objects are passed - if source and not isinstance(source, (str, int, float)): - if isinstance(source, dict): - source = source.get("id") or source.get("entity_id") or source.get("text") or str(source) - else: - source = getattr(source, "id", getattr(source, "text", str(source))) - - if target and not isinstance(target, (str, int, float)): - if isinstance(target, dict): - target = target.get("id") or target.get("entity_id") or target.get("text") or str(target) - else: - target = getattr(target, "id", getattr(target, "text", str(target))) - - if source and target: - if target not in adjacency[source]: - adjacency[source].append(target) - if source not in adjacency[target]: - adjacency[target].append(source) - - return dict(adjacency) + return build_adjacency(graph) def _to_networkx(self, graph): """Convert graph to NetworkX format.""" - adjacency = self._build_adjacency(graph) + view = build_graph_view(graph) nx_graph = self.nx.Graph() - for source, targets in adjacency.items(): - for target in targets: - nx_graph.add_edge(source, target) + nx_graph.add_nodes_from(view.nodes) + nx_graph.add_edges_from(view.edges) return nx_graph diff --git a/semantica/kg/community_detector.py b/semantica/kg/community_detector.py index 8aaaa236..01fa6064 100644 --- a/semantica/kg/community_detector.py +++ b/semantica/kg/community_detector.py @@ -49,6 +49,16 @@ from typing import Any, Dict, List, Optional from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker +from ._graph_view import build_adjacency, build_graph_view + + +def _is_hashable(value: Any) -> bool: + """Return whether a community identifier can be used in a set.""" + try: + hash(value) + except TypeError: + return False + return True class CommunityDetector: @@ -157,17 +167,18 @@ class CommunityDetector: nx_graph = self._to_networkx(graph) - # Check if graph is empty or has no edges + # An empty graph has no communities. A graph with nodes but + # no edges still has singleton communities. num_nodes = nx_graph.number_of_nodes() num_edges = nx_graph.number_of_edges() self.logger.debug(f"Graph stats: nodes={num_nodes}, edges={num_edges}") - if num_nodes == 0 or num_edges == 0: - self.logger.warning("Graph is empty or has no edges, returning 0 communities") + if num_nodes == 0: + self.logger.warning("Graph is empty, returning 0 communities") self.progress_tracker.stop_tracking( tracking_id, status="completed", - message="Detected 0 communities (empty graph/no edges)", + message="Detected 0 communities (empty graph)", ) return { "communities": [], @@ -350,17 +361,7 @@ class CommunityDetector: adjacency = self._build_adjacency(graph) - # Extract community structure - if isinstance(communities, dict): - node_communities = communities - elif isinstance(communities, dict) and "node_assignments" in communities: - node_communities = communities["node_assignments"] - else: - # Convert list of communities to node assignments - node_communities = {} - for i, community in enumerate(communities): - for node in community: - node_communities[node] = i + node_communities = self._to_node_assignments(communities) # Calculate metrics num_communities = len(set(node_communities.values())) @@ -408,16 +409,7 @@ class CommunityDetector: metrics = self.calculate_community_metrics(graph, communities) - # Extract node assignments - if isinstance(communities, dict) and "node_assignments" in communities: - node_communities = communities["node_assignments"] - elif isinstance(communities, dict): - node_communities = communities - else: - node_communities = {} - for i, community in enumerate(communities): - for node in community: - node_communities[node] = i + node_communities = self._to_node_assignments(communities) # Analyze connectivity between communities adjacency = self._build_adjacency(graph) @@ -440,6 +432,32 @@ class CommunityDetector: "edge_ratio": intra_community_edges / (inter_community_edges + 1), } + @staticmethod + def _to_node_assignments(communities: Any) -> Dict[Any, Any]: + """Normalize community results to a node-to-community mapping.""" + if isinstance(communities, dict): + assignments = communities.get("node_assignments") + if isinstance(assignments, dict): + return assignments + + detected_communities = communities.get("communities") + if isinstance(detected_communities, (list, tuple)): + communities = detected_communities + elif "communities" in communities: + raise ValueError("Community results must contain a list of communities") + elif not all(_is_hashable(value) for value in communities.values()): + raise ValueError( + "Community assignments must map nodes to hashable community IDs" + ) + else: + return communities + + node_assignments: Dict[Any, Any] = {} + for community_id, community in enumerate(communities or []): + for node in community: + node_assignments[node] = community_id + return node_assignments + def detect_communities( self, graph: Any, algorithm: str = "louvain", method: str = None, **options ) -> Dict[str, Any]: @@ -478,57 +496,7 @@ class CommunityDetector: def _build_adjacency(self, graph) -> Dict[str, List[str]]: """Build adjacency list from graph.""" - from collections import defaultdict - - adjacency = defaultdict(list) - - # Extract relationships - relationships = [] - raw_edges = [] # flat (u, v) tuples - if hasattr(graph, "relationships"): - relationships = graph.relationships - elif hasattr(graph, "get_relationships"): - relationships = graph.get_relationships() - elif isinstance(graph, dict): - relationships = graph.get("relationships", []) - # Also handle 'edges' key (list of tuples or dicts) - for edge in graph.get("edges", []): - if isinstance(edge, (list, tuple)) and len(edge) >= 2: - raw_edges.append((str(edge[0]), str(edge[1]))) - elif isinstance(edge, dict): - relationships.append(edge) - - # Add raw (u, v) edges - for u, v in raw_edges: - if u and v: - adjacency[u].append(v) - adjacency[v].append(u) - - # Build adjacency - for rel in relationships: - source = rel.get("source") or rel.get("subject") - target = rel.get("target") or rel.get("object") - - # Extract IDs if objects are passed - if source and not isinstance(source, (str, int, float)): - if isinstance(source, dict): - source = source.get("id") or source.get("entity_id") or source.get("text") or str(source) - else: - source = getattr(source, "id", getattr(source, "text", str(source))) - - if target and not isinstance(target, (str, int, float)): - if isinstance(target, dict): - target = target.get("id") or target.get("entity_id") or target.get("text") or str(target) - else: - target = getattr(target, "id", getattr(target, "text", str(target))) - - if source and target: - if target not in adjacency[source]: - adjacency[source].append(target) - if source not in adjacency[target]: - adjacency[target].append(source) - - return dict(adjacency) + return build_adjacency(graph) def _to_networkx(self, graph): """Convert graph to NetworkX format.""" @@ -536,12 +504,11 @@ class CommunityDetector: if hasattr(graph, 'nodes') and hasattr(graph, 'edges') and hasattr(graph, 'number_of_nodes'): return graph - adjacency = self._build_adjacency(graph) + view = build_graph_view(graph) nx_graph = self.nx.Graph() - for source, targets in adjacency.items(): - for target in targets: - nx_graph.add_edge(source, target) + nx_graph.add_nodes_from(view.nodes) + nx_graph.add_edges_from(view.edges) return nx_graph diff --git a/semantica/kg/connectivity_analyzer.py b/semantica/kg/connectivity_analyzer.py index 00d2ad22..463c9ed5 100644 --- a/semantica/kg/connectivity_analyzer.py +++ b/semantica/kg/connectivity_analyzer.py @@ -48,11 +48,12 @@ Author: Semantica Contributors License: MIT """ -from collections import defaultdict, deque +from collections import deque from typing import Any, Dict, List, Optional, Set, Tuple from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker +from ._graph_view import build_adjacency class ConnectivityAnalyzer: @@ -385,51 +386,7 @@ class ConnectivityAnalyzer: def _build_adjacency(self, graph) -> Dict[str, List[str]]: """Build adjacency list from graph.""" - adjacency = defaultdict(list) - - # Extract relationships - relationships = [] - if hasattr(graph, "relationships"): - relationships = graph.relationships - elif hasattr(graph, "get_relationships"): - relationships = graph.get_relationships() - elif isinstance(graph, dict): - relationships = graph.get("relationships", graph.get("edges", [])) - - # Build adjacency - for rel in relationships: - # Handle tuple/list edges (e.g., from NetworkX) - if isinstance(rel, (tuple, list)) and len(rel) >= 2: - source, target = str(rel[0]), str(rel[1]) - if source and target: - if target not in adjacency[source]: - adjacency[source].append(target) - if source not in adjacency[target]: - adjacency[target].append(source) - continue - source = rel.get("source") or rel.get("subject") - target = rel.get("target") or rel.get("object") - - # Extract IDs if objects are passed - if source and not isinstance(source, (str, int, float)): - if isinstance(source, dict): - source = source.get("id") or source.get("entity_id") or source.get("text") or str(source) - else: - source = getattr(source, "id", getattr(source, "text", str(source))) - - if target and not isinstance(target, (str, int, float)): - if isinstance(target, dict): - target = target.get("id") or target.get("entity_id") or target.get("text") or str(target) - else: - target = getattr(target, "id", getattr(target, "text", str(target))) - - if source and target: - if target not in adjacency[source]: - adjacency[source].append(target) - if source not in adjacency[target]: - adjacency[target].append(source) - - return dict(adjacency) + return build_adjacency(graph) def _bfs_shortest_path( self, adjacency: Dict[str, List[str]], source: str, target: str diff --git a/tests/kg/test_analytics_node_scope.py b/tests/kg/test_analytics_node_scope.py new file mode 100644 index 00000000..fda5d4f8 --- /dev/null +++ b/tests/kg/test_analytics_node_scope.py @@ -0,0 +1,112 @@ +"""Regression tests for KG analytics node scope handling.""" + +import networkx as nx + +from semantica.kg.centrality_calculator import CentralityCalculator +from semantica.kg.community_detector import CommunityDetector +from semantica.kg.connectivity_analyzer import ConnectivityAnalyzer + + +def _graph_with_isolated_node(): + return { + "entities": [{"id": "A"}, {"id": "B"}, {"id": "C"}], + "relationships": [{"source": "A", "target": "B"}], + } + + +def test_centrality_keeps_declared_isolated_nodes(): + result = CentralityCalculator().calculate_degree_centrality( + _graph_with_isolated_node() + ) + + assert result["total_nodes"] == 3 + assert result["centrality"]["C"] == 0.0 + + +def test_connectivity_reports_declared_isolated_nodes(): + result = ConnectivityAnalyzer().analyze_connectivity( + _graph_with_isolated_node() + ) + + assert result["num_nodes"] == 3 + assert result["num_components"] == 2 + assert ["C"] in result["components"] + assert result["is_connected"] is False + + +def test_community_detection_keeps_declared_isolated_nodes(): + detector = CommunityDetector() + result = detector.detect_communities(_graph_with_isolated_node()) + + assert set(result["node_assignments"]) == {"A", "B", "C"} + metrics = detector.calculate_community_metrics( + _graph_with_isolated_node(), result + ) + assert metrics["num_communities"] == 2 + structure = detector.analyze_community_structure( + _graph_with_isolated_node(), result + ) + assert structure["num_communities"] == 2 + + +def test_community_detection_returns_singletons_for_edgeless_graph(): + graph = {"entities": [{"id": "A"}, {"id": "B"}], "relationships": []} + + result = CommunityDetector().detect_communities(graph) + + assert {frozenset(community) for community in result["communities"]} == { + frozenset({"A"}), + frozenset({"B"}), + } + + +def test_networkx_graph_keeps_isolated_nodes_for_analytics(): + graph = nx.Graph() + graph.add_nodes_from(["A", "B", "C"]) + graph.add_edge("A", "B") + + centrality = CentralityCalculator().calculate_degree_centrality(graph) + connectivity = ConnectivityAnalyzer().analyze_connectivity(graph) + + assert centrality["total_nodes"] == 3 + assert centrality["centrality"]["C"] == 0.0 + assert connectivity["num_nodes"] == 3 + assert connectivity["num_components"] == 2 + + +def test_nodes_edges_payload_keeps_declared_isolated_nodes(): + graph = { + "nodes": [{"id": "A"}, {"id": "B"}, {"id": "C"}], + "edges": [("A", "B")], + } + + result = CentralityCalculator().calculate_degree_centrality(graph) + + assert result["total_nodes"] == 3 + assert result["centrality"]["C"] == 0.0 + + +def test_name_and_text_nodes_are_kept_when_ids_are_missing(): + graph = { + "entities": [{"name": "Alice"}, {"text": "Bob"}], + "relationships": [], + } + + result = CentralityCalculator().calculate_degree_centrality(graph) + + assert result["total_nodes"] == 2 + assert set(result["centrality"]) == {"Alice", "Bob"} + + +def test_community_metrics_accepts_communities_payload(): + detector = CommunityDetector() + graph = { + "entities": [{"id": "A"}, {"id": "B"}, {"id": "C"}], + "relationships": [{"source": "A", "target": "B"}], + } + result = {"communities": [["A", "B"], ["C"]]} + + metrics = detector.calculate_community_metrics(graph, result) + + assert metrics["num_communities"] == 2 + assert metrics["community_sizes"] == {0: 2, 1: 1} From 15171fd31a61a488391ffac97efcfd0ef97ea553 Mon Sep 17 00:00:00 2001 From: pravit-amp <43916793+pravit-amp@users.noreply.github.com> Date: Sun, 16 Aug 2026 02:07:10 -0700 Subject: [PATCH 06/19] fix(parse): import get_progress_tracker in ExcelParser (#1016) ExcelParser.__init__ called get_progress_tracker() without importing it, so every instantiation raised NameError and the class was unusable. The existing test imported ExcelParser but never constructed it, so nothing caught it. Same defect as #530 in SimilarityCalculator, which was fixed without sweeping the rest of the codebase. Add construction coverage for every parser exported from semantica.parse, driven off __all__ so later additions are covered automatically. These live outside test_parse_comprehensive.py, whose setUp patches get_progress_tracker into each parse module and would mock away the interaction under test. Closes #1014 Co-authored-by: Pravit Ampapathini --- semantica/parse/excel_parser.py | 1 + tests/parse/test_parser_construction.py | 73 +++++++++++++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 tests/parse/test_parser_construction.py diff --git a/semantica/parse/excel_parser.py b/semantica/parse/excel_parser.py index efd84aa5..b8943939 100644 --- a/semantica/parse/excel_parser.py +++ b/semantica/parse/excel_parser.py @@ -37,6 +37,7 @@ from openpyxl import load_workbook from ..utils.exceptions import ProcessingError, ValidationError from ..utils.logging import get_logger +from ..utils.progress_tracker import get_progress_tracker @dataclass diff --git a/tests/parse/test_parser_construction.py b/tests/parse/test_parser_construction.py new file mode 100644 index 00000000..b649954b --- /dev/null +++ b/tests/parse/test_parser_construction.py @@ -0,0 +1,73 @@ +"""Construction coverage for the parse module's public parser classes. + +Regression tests for #1014: ``ExcelParser.__init__`` called ``get_progress_tracker()`` +without importing it, so every instantiation raised ``NameError``. The class was +covered by an import-only test, which passes regardless of whether ``__init__`` +works, so nothing caught it. #530 was the same bug in ``SimilarityCalculator``. + +These tests deliberately do **not** patch ``get_logger``/``get_progress_tracker``. +``tests/parse/test_parse_comprehensive.py`` patches both into every parse module +that exposes them, which would mock away the exact interaction under test here and +let the regression back in silently. +""" + +import unittest + +import semantica.parse as parse_module +from semantica.parse.excel_parser import ExcelParser + + +def _exported_parser_classes(): + """Public parser classes, taken from the package's own ``__all__``. + + Driven off ``__all__`` rather than a hand-written list so a parser added later + is covered without anyone remembering to update this file. + """ + return [ + (name, getattr(parse_module, name)) + for name in parse_module.__all__ + if name.endswith("Parser") + ] + + +class TestExcelParserConstruction(unittest.TestCase): + """ExcelParser must be constructible -- see #1014.""" + + def test_excel_parser_constructs(self): + parser = ExcelParser() + self.assertIsNotNone(parser) + + def test_excel_parser_wires_progress_tracker(self): + """The missing import was for the tracker, so assert it is actually set. + + A bare construction check would pass against a version that dropped the + tracker call entirely; this pins the attribute the import exists to provide. + """ + parser = ExcelParser() + self.assertIsNotNone(parser.progress_tracker) + + +class TestExportedParsersConstruct(unittest.TestCase): + """Every parser the package exports must survive ``__init__``.""" + + def test_all_exported_parsers_construct(self): + classes = _exported_parser_classes() + self.assertGreater(len(classes), 0, "no exported parser classes found") + + for name, cls in classes: + with self.subTest(parser=name): + try: + self.assertIsNotNone(cls()) + except ImportError as exc: + # Parsers backed by an optional dependency raise a deliberate, + # actionable ImportError when it is absent (e.g. DoclingParser + # without `docling`). That is correct behavior, not a defect. + self.assertIn( + "install", + str(exc).lower(), + f"{name} raised ImportError without install guidance: {exc}", + ) + + +if __name__ == "__main__": + unittest.main() From c53ca4e84bd183fcf3aa708b56dadb86ffa88901 Mon Sep 17 00:00:00 2001 From: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Date: Sun, 16 Aug 2026 15:18:55 +0530 Subject: [PATCH 07/19] docs: formalize issue assignment and duplicate-PR triage workflow (#1030) * docs(contributing): formalize issue assignment and duplicate-PR triage workflow Comments are no longer required before an issue can be assigned - maintainers may assign directly based on recent activity. Also documents the duplicate-PR priority order for triage (contributor PR, claimed issue, activity tiebreak, late duplicates, overlapping scope). * docs(contributing): clarify assignment precedence and define activity tiebreak Addresses Qodo review feedback on PR #1030: the duplicate-PR priority list now states these rules apply on top of the assignment workflow (opening a PR pre-assignment doesn't grant priority), and the "most active" tiebreak now specifies a concrete 60-day window and signals instead of being subjective. --- .github/pull_request_template.md | 2 +- CONTRIBUTING.md | 20 +++++++++++++++++--- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 66f38cd9..b4d34e46 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,4 +1,4 @@ -> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — comment on the issue and wait for assignment before opening a PR, to avoid duplicate work. +> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — wait for the issue to be assigned to you before opening a PR, to avoid duplicate work. ## Description diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3edcc239..8a6c5c44 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,9 +25,9 @@ If you want to work on an open GitHub issue, please follow these steps to keep t 1. **Check the issue.** Look at the issue's assignees and recent comments. If someone is already actively working on it, consider a different issue or ask in the comments whether help is welcome. -2. **Comment before you start.** Leave a comment on the issue saying you'd like to work on it — something like *"I'd like to take this on"* is enough. This gives maintainers the context they need to assign the issue appropriately. +2. **Comment if you'd like the issue reserved.** Leaving a comment like *"I'd like to take this on"* is the fastest way to get assigned, but it isn't required — maintainers can also assign an issue directly to a contributor (e.g., based on recent activity in the repo) without waiting for a comment first. -3. **Wait for assignment.** A maintainer will review the request and assign the issue when appropriate. Please wait for this before investing significant time in implementation, as priorities and approaches can shift. +3. **Wait for assignment.** A maintainer will assign the issue when appropriate, whether or not a comment was left. Please wait for this before investing significant time in implementation, as priorities and approaches can shift. 4. **Create a branch and implement.** Once assigned, fork the repository (if you haven't already), create a dedicated branch, and begin your work. @@ -37,12 +37,26 @@ If you want to work on an open GitHub issue, please follow these steps to keep t 5. **Open a focused PR and link the issue.** When you're ready, open a pull request and reference the issue in the description (e.g., `Closes #123`). Keep the PR scoped to the work described in the issue. -> **Why this matters:** Commenting before opening a PR helps maintainers track who is working on what, assign issues correctly, and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code. +> **Why this matters:** Assignment (with or without a comment) helps maintainers track who is working on what and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code. Not sure where to start? Try a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue) or ask in [Discord](https://discord.gg/sV34vps5hH). --- +## 🔀 Duplicate PRs & Issue Priority + +When more than one pull request targets the same issue, maintainers triage using this order of priority. These rules decide between PRs that are otherwise following the [assignment workflow above](#-working-on-an-existing-issue) — opening a PR before being assigned doesn't grant priority on its own, and an unassigned PR can still be closed as a duplicate once someone else is assigned to the issue. + +1. **Contributor-raised issue with an existing PR.** If the person who opened the issue has also opened a PR for it, that PR is prioritized (they still need to be assigned before it's merged). +2. **Maintainer-raised issue with a claim comment.** If we opened the issue and someone has commented asking to work on it, we assign it to them and check their PR before picking up any other PR for the same issue. +3. **No prior assignment or comment.** If multiple PRs exist and no one was assigned or claimed the issue first, priority goes to whichever contributor has the most consistent activity in the repo over the last 60 days (e.g., merged PRs, substantive reviews, or issue triage participation) — not just PR volume. +4. **Late duplicate PRs.** If a PR is opened after another contributor has already been assigned to the issue, we close the duplicate early rather than let it sit open, and point the author to another open issue (or ask them to check `main` for newly opened ones). This avoids contributors spending time updating a PR that won't be merged. +5. **Overlapping scope.** If a PR covers multiple issues, or there's genuine overlap between competing PRs, maintainers discuss it on [Discord](https://discord.gg/sV34vps5hH) before deciding rather than resolving it unilaterally. + +**Why this matters:** it keeps triage predictable, avoids wasted contributor effort on PRs that won't merge, and helps retain active contributors. + +--- + ## 🎯 Ways to Contribute ### 💻 Code From 70aa9d01bf6cf9dac735b344f4a7354fad700b8b Mon Sep 17 00:00:00 2001 From: hari Date: Sun, 16 Aug 2026 15:24:34 +0530 Subject: [PATCH 08/19] fix(normalize): validate symbol currencies (#940) * fix(normalize): validate symbol currencies Signed-off-by: Mr-Neutr0n * fix(normalize): match currency codes by token boundaries Signed-off-by: Mr-Neutr0n --------- Signed-off-by: Mr-Neutr0n Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com> --- semantica/normalize/number_normalizer.py | 17 ++++++++++++----- tests/normalize/test_number_normalizer.py | 18 ++++++++++++++++++ 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/semantica/normalize/number_normalizer.py b/semantica/normalize/number_normalizer.py index f7876b7c..29911cb9 100644 --- a/semantica/normalize/number_normalizer.py +++ b/semantica/normalize/number_normalizer.py @@ -562,6 +562,11 @@ class CurrencyNormalizer: "SEK", "NOK", "DKK", + "RUB", + "KRW", + "ILS", + "NGN", + "PKR", ] self.logger.debug("Currency normalizer initialized") @@ -606,13 +611,15 @@ class CurrencyNormalizer: # Check for currency code if not currency_code: for code in self.currency_codes: - if code in currency_input.upper(): + match = re.search( + rf"(? Date: Sun, 16 Aug 2026 17:44:02 +0530 Subject: [PATCH 09/19] docs: clarify explainability is system-level, not foundation-model internal (#1033) Adds a consistent scope note to README and docs (concepts, FAQ, index) stating Semantica does not expose or reconstruct an LLM's internal reasoning/chain-of-thought. It explains and audits the AI system around the model: context, provenance, policies, decisions, and execution history. --- README.md | 2 ++ docs/concepts.md | 3 +++ docs/faq.md | 10 ++++++++++ docs/index.md | 6 +++++- 4 files changed, 20 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index fe3272ea..58afaf3f 100644 --- a/README.md +++ b/README.md @@ -1498,6 +1498,8 @@ Semantica is designed for environments where AI outputs must be explainable, aud - **Cybersecurity:** Threat attribution, incident response timelines, and IOC provenance tracking - **Autonomous Systems:** Decision logs, safety validation, and explainable AI for certification +> ⚠️ **This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits what the AI system did, not the LLM's private internal reasoning. + --- ## Installation diff --git a/docs/concepts.md b/docs/concepts.md index 698ef7d9..e05564f5 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -16,6 +16,9 @@ At its core, Semantica adds a **context and accountability layer** on top of you - **Accountability Layer** — Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable. - **Extension Layer** — `PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code. + + **This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits *what the AI system did*, not the foundation model's private internal reasoning. + ## Knowledge Graphs diff --git a/docs/faq.md b/docs/faq.md index e050df4c..05c708c1 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -52,6 +52,16 @@ Semantica works alongside these frameworks, not against them. + + +No. This is **system-level explainability, not foundation-model explainability**. Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. + +What Semantica explains is *outside* the model: what context and data were used, what decision was produced, the provenance behind it, the relevant relationships, the policies applied, and the resulting decision trail. + +In short: Semantica explains and audits *what the AI system did* — not the foundation model's private internal reasoning. + + + Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source. diff --git a/docs/index.md b/docs/index.md index 80ce7b12..a16d0107 100644 --- a/docs/index.md +++ b/docs/index.md @@ -192,7 +192,11 @@ decision_id = context.record_decision( ## Built for Where Mistakes Have Consequences -Semantica was designed for domains where every decision must be explainable and every fact must be traceable: +Semantica was designed for domains where every decision must be explainable and every fact must be traceable. + + + **This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. See [Core Concepts](concepts) for the full scope note. + **Healthcare & Life Sciences** - Clinical decision support with full audit trails From 4d37920007e75289fe80a40651250cb6aa27cc10 Mon Sep 17 00:00:00 2001 From: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Date: Sun, 16 Aug 2026 17:51:23 +0530 Subject: [PATCH 10/19] docs: surface explainability scope note near the top of the README (#1034) Moves a concise version of the system-level vs. foundation-model explainability clarification up next to the opening pitch, so it's visible before readers scroll to the high-stakes-domains section. --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 58afaf3f..8b89fd5f 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,8 @@ Most AI agents act without a trail. They store embeddings, not meaning: context Semantica sits underneath your LLM, vector store, and agent framework as a deterministic infrastructure layer: no LLM required for graph construction, reasoning, or provenance. +> ⚠️ **System-level explainability, not foundation-model explainability.** Semantica does not expose or reconstruct what happens *inside* the LLM — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. Semantica explains what's *outside* the model: the context and data fed in, the decision produced, its provenance, relevant relationships, applied policies, and the full execution trail. + **Who it's for:** - **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index From 893b6db3c3d2abf2c0656baeb3549fb24a840f13 Mon Sep 17 00:00:00 2001 From: Accute9 Date: Sun, 16 Aug 2026 16:07:45 -0400 Subject: [PATCH 11/19] regression tests added and tested for routing spaCy model loads through cache --- tests/split/test_spacy_model_cache.py | 169 ++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 tests/split/test_spacy_model_cache.py diff --git a/tests/split/test_spacy_model_cache.py b/tests/split/test_spacy_model_cache.py new file mode 100644 index 00000000..d3b27148 --- /dev/null +++ b/tests/split/test_spacy_model_cache.py @@ -0,0 +1,169 @@ +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +from semantica.semantic_extract import methods as se_methods +from semantica.split import methods as split_methods +from semantica.split import semantic_chunker + + +@pytest.fixture(autouse=True) +def clear_cache(): + se_methods.clear_spacy_model_cache() + yield + se_methods.clear_spacy_model_cache() + + +@pytest.fixture(autouse=True) +def force_spacy_available(monkeypatch): + # split.methods and split.semantic_chunker each compute their own + # SPACY_AVAILABLE flag from the real environment at import time; force + # both true so these tests exercise the spaCy branch regardless of + # whether spaCy is actually installed where they run. + monkeypatch.setattr(split_methods, "SPACY_AVAILABLE", True) + monkeypatch.setattr(semantic_chunker, "SPACY_AVAILABLE", True) + + +def _fake_spacy(load): + return SimpleNamespace(load=load, util=SimpleNamespace(is_package=lambda _name: True)) + + +def _nlp_mock(sentences=("Hello world.",)): + """A stand-in spaCy Language object: callable, returns a doc with .sents.""" + nlp = MagicMock() + nlp.return_value = SimpleNamespace( + sents=[SimpleNamespace(text=s) for s in sentences] + ) + return nlp + + +class TestSpacyModelCache: + """split.methods and split.semantic_chunker must share the cached model + defined in semantic_extract.methods instead of each calling spacy.load() + independently. + """ + + def test_split_by_sentences_reuses_cached_model(self, monkeypatch): + calls = [] + + def fake_load(name, **kwargs): + calls.append((name, kwargs)) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + split_methods.split_by_sentences("Hello world. Bye world.") + split_methods.split_by_sentences("Another sentence here.") + split_methods.split_by_sentences("A third call.") + + assert len(calls) == 1, "spacy.load should run once, not once per call" + assert calls[0][0] == "en_core_web_sm" + + def test_semantic_chunker_reuses_cached_model_across_instances(self, monkeypatch): + calls = [] + + def fake_load(name, **kwargs): + calls.append((name, kwargs)) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + chunker1 = semantic_chunker.SemanticChunker() + chunker2 = semantic_chunker.SemanticChunker() + + assert len(calls) == 1, "each new SemanticChunker should not reload the model" + assert chunker1.nlp is chunker2.nlp + + def test_split_methods_and_semantic_chunker_share_the_cache(self, monkeypatch): + calls = [] + + def fake_load(name, **kwargs): + calls.append((name, kwargs)) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + split_methods.split_by_sentences("Test sentence for split.methods.") + semantic_chunker.SemanticChunker() + + assert len(calls) == 1, ( + "split.methods and split.semantic_chunker must share one cached " + "model instead of each loading their own" + ) + + def test_distinct_model_names_load_separately(self, monkeypatch): + calls = [] + + def fake_load(name, **kwargs): + calls.append((name, kwargs)) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + sm_chunker = semantic_chunker.SemanticChunker(model="en_core_web_sm") + lg_chunker = semantic_chunker.SemanticChunker(model="en_core_web_lg") + sm_chunker_again = semantic_chunker.SemanticChunker(model="en_core_web_sm") + + assert [name for name, _ in calls] == ["en_core_web_sm", "en_core_web_lg"] + assert sm_chunker.nlp is sm_chunker_again.nlp + assert sm_chunker.nlp is not lg_chunker.nlp + + def test_no_disable_kwarg_requested(self, monkeypatch): + """split.methods and split.semantic_chunker both want the full + pipeline (they need .sents, which requires the parser/senter). If + either one later starts requesting a trimmed pipeline (e.g. + disable=["ner"]), the name-only cache key in load_spacy_model would + silently hand back a cached model built for a different config -- + this test should catch that the moment it happens. + """ + calls = [] + + def fake_load(_name, **kwargs): + calls.append(kwargs) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + split_methods.split_by_sentences("Hello world.") + se_methods.clear_spacy_model_cache() + semantic_chunker.SemanticChunker() + + assert calls == [{}, {}], "neither caller should request a partial pipeline" + + def test_missing_model_falls_back_without_poisoning_cache(self, monkeypatch): + attempts = [] + + def failing_load(name, **_kwargs): + attempts.append(name) + raise OSError(f"Can't find model '{name}'") + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load)) + + # split_by_sentences should fall back to regex splitting, not raise + chunks = split_methods.split_by_sentences("Hello world. Bye world.") + assert chunks, "fallback splitting should still produce chunks" + + # SemanticChunker should leave .nlp as None rather than propagate + chunker = semantic_chunker.SemanticChunker() + assert chunker.nlp is None + + assert len(attempts) == 2, "a failed load must not be cached" + + # Once the model is available, both callers should now get it, and + # share a single successful load. + def working_load(name, **_kwargs): + attempts.append(name) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load)) + + chunker2 = semantic_chunker.SemanticChunker() + split_methods.split_by_sentences("One more sentence.") + + assert len(attempts) == 3, "the model should load once after it becomes available" + assert chunker2.nlp is not None + + +if __name__ == "__main__": + pytest.main([__file__]) From 0b77e5fe9476fab239f5ec2705f939456bb1d2cb Mon Sep 17 00:00:00 2001 From: Aneesh Mandapati <93799543+Accute9@users.noreply.github.com> Date: Sun, 16 Aug 2026 20:31:06 -0400 Subject: [PATCH 12/19] Refactor for flake8 max line length (88) issue Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- tests/split/test_spacy_model_cache.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/split/test_spacy_model_cache.py b/tests/split/test_spacy_model_cache.py index d3b27148..551b82f9 100644 --- a/tests/split/test_spacy_model_cache.py +++ b/tests/split/test_spacy_model_cache.py @@ -26,7 +26,10 @@ def force_spacy_available(monkeypatch): def _fake_spacy(load): - return SimpleNamespace(load=load, util=SimpleNamespace(is_package=lambda _name: True)) + return SimpleNamespace( + load=load, + util=SimpleNamespace(is_package=lambda _name: True), + ) def _nlp_mock(sentences=("Hello world.",)): From 0f252ab355b60df015937cb75b51484c2f90bc34 Mon Sep 17 00:00:00 2001 From: Accute9 Date: Sun, 16 Aug 2026 21:04:57 -0400 Subject: [PATCH 13/19] Fixed max line length (88) issues and eager imports --- semantica/split/methods.py | 4 ++-- semantica/split/semantic_chunker.py | 4 ++-- tests/split/test_spacy_model_cache.py | 16 +++++++++++++--- 3 files changed, 17 insertions(+), 7 deletions(-) diff --git a/semantica/split/methods.py b/semantica/split/methods.py index f2b3f525..61b67ee0 100644 --- a/semantica/split/methods.py +++ b/semantica/split/methods.py @@ -93,12 +93,11 @@ from ..utils.exceptions import ProcessingError from ..utils.helpers import safe_import from ..utils.logging import get_logger from .semantic_chunker import Chunk -from ..semantic_extract.methods import load_spacy_model logger = get_logger("split_methods") # Try to import optional dependencies -spacy, SPACY_AVAILABLE = safe_import("spacy") +_, SPACY_AVAILABLE = safe_import("spacy") nltk, NLTK_AVAILABLE = safe_import("nltk") tiktoken, TIKTOKEN_AVAILABLE = safe_import("tiktoken") @@ -337,6 +336,7 @@ def split_by_sentences( # Try spaCy first if SPACY_AVAILABLE and kwargs.get("use_spacy", True): try: + from ..semantic_extract.methods import load_spacy_model nlp = load_spacy_model("en_core_web_sm") doc = nlp(text) sentences = [sent.text for sent in doc.sents] diff --git a/semantica/split/semantic_chunker.py b/semantica/split/semantic_chunker.py index d7f72726..2945bbd5 100644 --- a/semantica/split/semantic_chunker.py +++ b/semantica/split/semantic_chunker.py @@ -35,10 +35,9 @@ from ..utils.exceptions import ProcessingError from ..utils.helpers import safe_import from ..utils.logging import get_logger from ..utils.progress_tracker import get_progress_tracker -from ..semantic_extract.methods import load_spacy_model -spacy, SPACY_AVAILABLE = safe_import("spacy") +_, SPACY_AVAILABLE = safe_import("spacy") @dataclass @@ -81,6 +80,7 @@ class SemanticChunker: if SPACY_AVAILABLE: model_name = config.get("model", "en_core_web_sm") try: + from ..semantic_extract.methods import load_spacy_model self.nlp = load_spacy_model(model_name) except OSError: self.logger.warning( diff --git a/tests/split/test_spacy_model_cache.py b/tests/split/test_spacy_model_cache.py index d3b27148..97a4ad87 100644 --- a/tests/split/test_spacy_model_cache.py +++ b/tests/split/test_spacy_model_cache.py @@ -26,7 +26,12 @@ def force_spacy_available(monkeypatch): def _fake_spacy(load): - return SimpleNamespace(load=load, util=SimpleNamespace(is_package=lambda _name: True)) + return SimpleNamespace( + load=load, + util=SimpleNamespace( + is_package=lambda _name: True + ), + ) def _nlp_mock(sentences=("Hello world.",)): @@ -129,7 +134,10 @@ class TestSpacyModelCache: se_methods.clear_spacy_model_cache() semantic_chunker.SemanticChunker() - assert calls == [{}, {}], "neither caller should request a partial pipeline" + assert len(calls) == 2 + assert all("disable" not in kwargs for kwargs in calls), ( + "neither caller should request a partial pipeline" + ) def test_missing_model_falls_back_without_poisoning_cache(self, monkeypatch): attempts = [] @@ -161,7 +169,9 @@ class TestSpacyModelCache: chunker2 = semantic_chunker.SemanticChunker() split_methods.split_by_sentences("One more sentence.") - assert len(attempts) == 3, "the model should load once after it becomes available" + assert len(attempts) == 3, ( + "the model should load once after it becomes available" + ) assert chunker2.nlp is not None From c7415f2e92434c65246d564184292064f9c42224 Mon Sep 17 00:00:00 2001 From: Sameer6305 Date: Mon, 17 Aug 2026 12:40:10 +0530 Subject: [PATCH 14/19] fix: complete spaCy model cache integration --- semantica/semantic_extract/ner_extractor.py | 7 +- tests/split/test_spacy_model_cache.py | 138 +++++++++++++++++++- tests/split/test_splitter.py | 16 +-- tests/test_ner_configurations.py | 29 ++-- 4 files changed, 166 insertions(+), 24 deletions(-) diff --git a/semantica/semantic_extract/ner_extractor.py b/semantica/semantic_extract/ner_extractor.py index e8b57bcd..a920efe1 100644 --- a/semantica/semantic_extract/ner_extractor.py +++ b/semantica/semantic_extract/ner_extractor.py @@ -144,7 +144,12 @@ class NERExtractor: self._ml_runtime_usable = True if "ml" in self.method and SPACY_AVAILABLE: try: - self.nlp = spacy.load(self.model_name) + # Deferred import: keeps semantic_extract.methods out of the + # module-level import graph and routes loading through the + # process-level cache so repeated NERExtractor constructions + # never pay the ~120 ms spacy.load() cost more than once. + from .methods import load_spacy_model + self.nlp = load_spacy_model(self.model_name) except OSError: self.logger.warning( f"spaCy model {self.model_name} not found. ML method will fallback." diff --git a/tests/split/test_spacy_model_cache.py b/tests/split/test_spacy_model_cache.py index d8f38117..de21e433 100644 --- a/tests/split/test_spacy_model_cache.py +++ b/tests/split/test_spacy_model_cache.py @@ -6,6 +6,8 @@ import pytest from semantica.semantic_extract import methods as se_methods from semantica.split import methods as split_methods from semantica.split import semantic_chunker +from semantica.semantic_extract import ner_extractor as ner_extractor_module +from semantica.semantic_extract.ner_extractor import NERExtractor @pytest.fixture(autouse=True) @@ -17,12 +19,13 @@ def clear_cache(): @pytest.fixture(autouse=True) def force_spacy_available(monkeypatch): - # split.methods and split.semantic_chunker each compute their own - # SPACY_AVAILABLE flag from the real environment at import time; force - # both true so these tests exercise the spaCy branch regardless of + # split.methods, split.semantic_chunker, and ner_extractor each compute + # their own SPACY_AVAILABLE flag from the real environment at import time; + # force all true so these tests exercise the spaCy branch regardless of # whether spaCy is actually installed where they run. monkeypatch.setattr(split_methods, "SPACY_AVAILABLE", True) monkeypatch.setattr(semantic_chunker, "SPACY_AVAILABLE", True) + monkeypatch.setattr(ner_extractor_module, "SPACY_AVAILABLE", True) def _fake_spacy(load): @@ -133,9 +136,11 @@ class TestSpacyModelCache: semantic_chunker.SemanticChunker() assert len(calls) == 2 - assert all("disable" not in kwargs for kwargs in calls), ( - "neither caller should request a partial pipeline" - ) + assert all(kwargs == {} for kwargs in calls), ( + "neither caller should pass any pipeline-configuration kwargs; " + "the name-only cache key in load_spacy_model cannot distinguish " + "models loaded with different component configs" + ) def test_missing_model_falls_back_without_poisoning_cache(self, monkeypatch): attempts = [] @@ -173,5 +178,126 @@ class TestSpacyModelCache: assert chunker2.nlp is not None +class TestNERExtractorSpacyModelCache: + """NERExtractor(method="ml") must reuse the centralized cache in + semantic_extract.methods, not call spacy.load() on every construction. + + These tests mirror TestSpacyModelCache but focus on the NERExtractor path, + confirming that all three callers (split_by_sentences, SemanticChunker, and + NERExtractor) draw from the same process-level cache. + """ + + def test_ner_extractor_reuses_cached_model_across_instances(self, monkeypatch): + """Two NERExtractor(method='ml') constructions with the same model name + must cause exactly one underlying spacy.load() call.""" + calls = [] + + def fake_load(name, **kwargs): + calls.append(name) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + e1 = NERExtractor(method="ml") + e2 = NERExtractor(method="ml") + e3 = NERExtractor(method="ml", model="en_core_web_sm") + + assert len(calls) == 1, ( + "repeated NERExtractor constructions should not reload the model" + ) + assert e1.nlp is e2.nlp is e3.nlp + + def test_ner_extractor_and_split_callers_share_one_cached_model(self, monkeypatch): + """NERExtractor, SemanticChunker, and split_by_sentences must all use + the same cached Language object for the same model name.""" + calls = [] + + def fake_load(name, **kwargs): + calls.append(name) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + split_methods.split_by_sentences("First sentence.") + semantic_chunker.SemanticChunker() + NERExtractor(method="ml") + + assert len(calls) == 1, ( + "split_by_sentences, SemanticChunker, and NERExtractor must share " + "one cached model instead of each loading their own" + ) + + def test_ner_extractor_distinct_model_names_load_separately(self, monkeypatch): + """Different model names must produce separate cache entries.""" + calls = [] + + def fake_load(name, **kwargs): + calls.append(name) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + sm = NERExtractor(method="ml", model="en_core_web_sm") + lg = NERExtractor(method="ml", model="en_core_web_lg") + sm_again = NERExtractor(method="ml", model="en_core_web_sm") + + assert calls == ["en_core_web_sm", "en_core_web_lg"] + assert sm.nlp is sm_again.nlp + assert sm.nlp is not lg.nlp + + def test_ner_extractor_failed_load_not_cached_and_retried(self, monkeypatch): + """A missing model must not poison the cache. A subsequent construction + after the model becomes available must succeed and share the loaded model.""" + attempts = [] + + def failing_load(name, **_kwargs): + attempts.append(name) + raise OSError(f"Can't find model '{name}'") + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(failing_load)) + + # Construction with missing model: nlp must remain None, no crash + extractor1 = NERExtractor(method="ml") + assert extractor1.nlp is None + assert len(attempts) == 1, "one load attempt expected for the missing model" + + # Second construction: must retry (cache must not hold the failure) + extractor2 = NERExtractor(method="ml") + assert extractor2.nlp is None + assert len(attempts) == 2, "a failed load must not be cached" + + # Now install a working model and verify recovery + def working_load(name, **_kwargs): + attempts.append(name) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(working_load)) + + extractor3 = NERExtractor(method="ml") + extractor4 = NERExtractor(method="ml") + + assert extractor3.nlp is not None + assert extractor3.nlp is extractor4.nlp + assert len(attempts) == 3, ( + "exactly one successful load expected after the model becomes available" + ) + + def test_ner_extractor_non_ml_method_does_not_load_model(self, monkeypatch): + """NERExtractor with a non-ml method must not touch the spaCy cache.""" + calls = [] + + def fake_load(name, **kwargs): + calls.append(name) + return _nlp_mock() + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(fake_load)) + + NERExtractor(method="pattern") + NERExtractor(method="llm") + NERExtractor(method="regex") + + assert calls == [], "non-ml methods must not trigger any spacy.load()" + + if __name__ == "__main__": pytest.main([__file__]) diff --git a/tests/split/test_splitter.py b/tests/split/test_splitter.py index 76cc872f..725b959a 100644 --- a/tests/split/test_splitter.py +++ b/tests/split/test_splitter.py @@ -30,18 +30,16 @@ class TestSplitter(unittest.TestCase): splitter = TextSplitter(method=["recursive", "token"]) self.assertEqual(splitter.methods, ["recursive", "token"]) - @patch('semantica.split.semantic_chunker.spacy') + @patch('semantica.semantic_extract.methods.spacy') def test_semantic_chunker_initialization(self, mock_spacy): - # Mock spacy.load to return a mock nlp object + # SemanticChunker now loads spaCy through the centralized + # load_spacy_model() in semantic_extract.methods, so we patch + # methods.spacy rather than the removed semantic_chunker.spacy binding. mock_nlp = MagicMock() mock_spacy.load.return_value = mock_nlp - - # We need to ensure SPACY_AVAILABLE is True for this test context if possible, - # but it is imported at module level. - # If spacy is not installed, it sets SPACY_AVAILABLE = False. - # We might need to patch the module attribute or just test fallback if spacy missing. - - chunker = SemanticChunker(chunk_size=100) + + with patch('semantica.split.semantic_chunker.SPACY_AVAILABLE', True): + chunker = SemanticChunker(chunk_size=100) self.assertEqual(chunker.chunk_size, 100) def test_chunk_dataclass(self): diff --git a/tests/test_ner_configurations.py b/tests/test_ner_configurations.py index 2fead463..15c2568a 100644 --- a/tests/test_ner_configurations.py +++ b/tests/test_ner_configurations.py @@ -101,9 +101,15 @@ class TestNERConfigurations(unittest.TestCase): self.assertEqual(entities[0].metadata["extraction_method"], "ml") self.assertEqual(entities[0].metadata["model"], "en_core_web_trf") - @patch('semantica.semantic_extract.ner_extractor.spacy') + @patch('semantica.semantic_extract.methods.spacy') def test_ner_ml_init_falls_back_when_spacy_runtime_is_broken(self, mock_spacy): - """Test NER init does not crash when spaCy is installed but unusable at runtime.""" + """Test NER init does not crash when spaCy is installed but unusable at runtime. + + The model load now goes through load_spacy_model() in semantic_extract.methods, + so we patch methods.spacy (not ner_extractor.spacy) to inject the failure. + """ + from semantica.semantic_extract.methods import clear_spacy_model_cache + clear_spacy_model_cache() mock_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined") with patch('semantica.semantic_extract.ner_extractor.SPACY_AVAILABLE', True): @@ -112,17 +118,23 @@ class TestNERConfigurations(unittest.TestCase): self.assertIsNone(extractor.nlp) self.assertFalse(extractor._ml_runtime_usable) - @patch('semantica.semantic_extract.ner_extractor.spacy') @patch('semantica.semantic_extract.methods.get_entity_method') @patch('semantica.semantic_extract.methods.spacy') def test_ner_ml_runtime_failure_disables_repeated_ml_load_attempts( self, mock_methods_spacy, mock_get_method, - mock_init_spacy, ): - """Test degraded ML mode skips repeated spaCy load attempts after init failure.""" - mock_init_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined") + """Test degraded ML mode skips repeated spaCy load attempts after init failure. + + The model load at construction time now goes through load_spacy_model() in + semantic_extract.methods, so methods.spacy is the single mock target for the + init-time failure. After the RuntimeError is raised, _ml_runtime_usable is + False and no further spacy.load (or extract_entities_ml) calls are made. + """ + from semantica.semantic_extract.methods import clear_spacy_model_cache + clear_spacy_model_cache() + mock_methods_spacy.load.side_effect = RuntimeError("ConfigSchemaNlp is not fully defined") mock_ml_method = MagicMock(return_value=[]) mock_get_method.side_effect = lambda name: mock_ml_method if name == "ml" else (lambda *_args, **_kwargs: []) @@ -132,8 +144,9 @@ class TestNERConfigurations(unittest.TestCase): entities = extractor.extract_entities(self.text) self.assertFalse(extractor._ml_runtime_usable) - self.assertEqual(mock_init_spacy.load.call_count, 1) - self.assertEqual(mock_methods_spacy.load.call_count, 0) + # methods.spacy.load called once during __init__ (the RuntimeError); not again + # during extract_entities because _filter_unusable_methods removes "ml". + self.assertEqual(mock_methods_spacy.load.call_count, 1) self.assertEqual(mock_ml_method.call_count, 0) self.assertIsInstance(entities, list) From a8194dfc60a17de99e926f153bc7c8fa3f3a8598 Mon Sep 17 00:00:00 2001 From: KaifAhmad1 Date: Mon, 17 Aug 2026 13:16:17 +0530 Subject: [PATCH 15/19] fix(split): catch broken-runtime spaCy failures in SemanticChunker SemanticChunker.__init__ only caught OSError around load_spacy_model(), while NERExtractor's identical call (fixed earlier in this PR) also catches generic Exception for a model that is installed but fails at runtime. Bring SemanticChunker in line so a broken spaCy config degrades to fallback chunking instead of crashing __init__. Adds a regression test mirroring the existing NERExtractor case, and a CHANGELOG entry for #998/#1042. --- CHANGELOG.md | 8 ++++++++ semantica/split/semantic_chunker.py | 7 +++++++ tests/split/test_spacy_model_cache.py | 18 ++++++++++++++++++ 3 files changed, 33 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 78212679..f670cb22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -73,6 +73,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`split`/chunking paths bypassed the centralized spaCy model cache, reloading the model on every call** (#1042, closes #998) by @Accute9, reviewed by @Sameer6305 + - `semantica/split/methods.py`'s `split_by_sentences()` and `semantica/split/semantic_chunker.py`'s `SemanticChunker.__init__` each called `spacy.load()` directly instead of reusing the process-level cache added in #889/`semantic_extract/methods.py`'s `load_spacy_model()` — every call/construction re-paid the ~120ms model-load cost independently of `NERExtractor`, which already used the cache + - Both now route through `load_spacy_model()`, sharing one cached `Language` instance per model name across `split_by_sentences()`, `SemanticChunker`, and `NERExtractor`; a missing model still falls back to regex/paragraph chunking without poisoning the cache for a later successful load + - **Fixed during review** (@Sameer6305): `NERExtractor.__init__()` still had a direct `spacy.load()` call site with the same cache-bypass issue, outside the two files named in #998 but sharing the same root cause; routed through the cache alongside stale test patch targets and a strengthened cache-configuration assertion + - **Fixed during review** (@KaifAhmad1): `SemanticChunker.__init__` only caught `OSError` around `load_spacy_model()`, while the sibling fix to `NERExtractor` in this same PR added a broader `except Exception` for a model that is installed but fails at runtime (e.g. a config incompatible with the installed spaCy version). A broken-but-present model crashed `SemanticChunker()` outright instead of degrading to fallback chunking like every other path in this PR. Added the matching `except Exception` branch, leaving `self.nlp` as `None`; new `test_semantic_chunker_falls_back_when_spacy_runtime_is_broken` mirrors the existing `NERExtractor` regression test for the same scenario + - New `tests/split/test_spacy_model_cache.py`: cache reuse across repeated calls/instances, shared cache between `split_by_sentences()`/`SemanticChunker`/`NERExtractor`, distinct model names loading separately, missing-model fallback without poisoning the cache, and the broken-runtime fallback added above + - `pytest tests/split/test_spacy_model_cache.py tests/split/test_splitter.py tests/split/test_chunkers.py`: all passing (3 pre-existing, unrelated `tests/test_ner_configurations.py` failures confirmed present on `main` before this PR) + - **`export_yaml` raised a raw `AttributeError` on list input, silently wrote empty exports for unrecognized dict keys, and graph payloads were reconciled differently by every exporter** (#958, closes #956, #952, #953) by @pravit-amp, reviewed by @Sameer6305 - Graph payloads circulate under two vocabularies, `entities`/`relationships` and `nodes`/`edges`, and each exporter reconciled them locally with a different idiom — `LPGExporter` in particular dropped every entity whenever `nodes` was present but empty, the exact shape `JSONExporter` emits. A new `normalize_graph_payload()` in `utils/helpers.py` centralizes that decision once, adopted by `LPGExporter`, `ArangoAQLExporter`, `Neo4jCSVExporter`, and both YAML exporters; `ContextGraph.to_dict()` now round-trips through YAML correctly as a result - `export_yaml(records, path)` on a bare list previously failed with `AttributeError` from inside the exporter; it and the other YAML methods now reject non-mapping input with an actionable `ProcessingError` naming the expected keys, since these formats distinguish entities/relationships/triplets and guessing which one a list represents would mislabel the records diff --git a/semantica/split/semantic_chunker.py b/semantica/split/semantic_chunker.py index 2945bbd5..fc6fa6aa 100644 --- a/semantica/split/semantic_chunker.py +++ b/semantica/split/semantic_chunker.py @@ -86,6 +86,13 @@ class SemanticChunker: self.logger.warning( f"spaCy model {model_name} not found. Using fallback chunking." ) + except Exception: + self.logger.warning( + "spaCy model %s failed to initialize and will be disabled " + "for this chunker instance. Using fallback chunking.", + model_name, + exc_info=True, + ) def chunk(self, text: str, **options) -> List[Chunk]: """ diff --git a/tests/split/test_spacy_model_cache.py b/tests/split/test_spacy_model_cache.py index de21e433..00012030 100644 --- a/tests/split/test_spacy_model_cache.py +++ b/tests/split/test_spacy_model_cache.py @@ -177,6 +177,24 @@ class TestSpacyModelCache: ) assert chunker2.nlp is not None + def test_semantic_chunker_falls_back_when_spacy_runtime_is_broken( + self, monkeypatch + ): + """A spaCy model that is installed but unusable at runtime (e.g. a + config incompatible with the installed spaCy version) must degrade + SemanticChunker to fallback chunking, not crash __init__ -- mirrors + TestNERExtractorSpacyModelCache's equivalent broken-runtime test. + """ + + def broken_load(name, **_kwargs): + raise RuntimeError("ConfigSchemaNlp is not fully defined") + + monkeypatch.setattr(se_methods, "spacy", _fake_spacy(broken_load)) + + chunker = semantic_chunker.SemanticChunker() + + assert chunker.nlp is None + class TestNERExtractorSpacyModelCache: """NERExtractor(method="ml") must reuse the centralized cache in From eedf1425cae948d84c5e0fb0a86497995661cffd Mon Sep 17 00:00:00 2001 From: Shahzaib Ahmad Date: Mon, 17 Aug 2026 14:30:12 +0500 Subject: [PATCH 16/19] Fix flatten_dict key collisions (#1062) * Fix flatten_dict key collisions * Fix flatten_dict formatting --------- Co-authored-by: Shahzaib Ahmad --- semantica/utils/helpers.py | 28 ++++++++++++++++++++-------- tests/utils/test_utils.py | 15 +++++++++++++++ 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/semantica/utils/helpers.py b/semantica/utils/helpers.py index 7462f6db..75031fe8 100644 --- a/semantica/utils/helpers.py +++ b/semantica/utils/helpers.py @@ -398,9 +398,7 @@ def chunk_list(items: List[Any], chunk_size: int) -> List[List[Any]]: Returns: List of chunks """ - return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)] - - + return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)] def flatten_dict( d: Dict[str, Any], parent_key: str = "", sep: str = "." ) -> Dict[str, Any]: @@ -414,18 +412,32 @@ def flatten_dict( Returns: Flattened dictionary + + Raises: + ValueError: If two input paths produce the same flattened key. """ - items = [] + result = {} for k, v in d.items(): new_key = f"{parent_key}{sep}{k}" if parent_key else k if isinstance(v, dict): - items.extend(flatten_dict(v, new_key, sep=sep).items()) - else: - items.append((new_key, v)) + nested = flatten_dict(v, new_key, sep=sep) - return dict(items) + for key, value in nested.items(): + if key in result: + raise ValueError( + f"Key collision while flattening dictionary: {key}" + ) + result[key] = value + else: + if new_key in result: + raise ValueError( + f"Key collision while flattening dictionary: {new_key}" + ) + result[new_key] = v + + return result def get_nested_value( diff --git a/tests/utils/test_utils.py b/tests/utils/test_utils.py index 479be1cf..5bbe3be3 100644 --- a/tests/utils/test_utils.py +++ b/tests/utils/test_utils.py @@ -31,6 +31,21 @@ class TestHelpers(unittest.TestCase): dict2 = {"b": {"d": 3}, "e": 4} merged = helpers.merge_dicts(dict1, dict2, deep=True) self.assertEqual(merged, {"a": 1, "b": {"c": 2, "d": 3}, "e": 4}) + def test_flatten_dict(self): + data = {"a": {"b": 1, "c": 2}} + result = helpers.flatten_dict(data) + self.assertEqual(result, {"a.b": 1, "a.c": 2}) + + def test_flatten_dict_key_collision(self): + data = { + "a.b": 1, + "a": { + "b": 2 + } + } + + with self.assertRaises(ValueError): + helpers.flatten_dict(data) def test_safe_import_returns_module_and_flag(self): module, available = helpers.safe_import("json") From 04602a0e0e35d7b353d535c5303d757541901823 Mon Sep 17 00:00:00 2001 From: Sameer Kadam Date: Mon, 17 Aug 2026 18:55:38 +0530 Subject: [PATCH 17/19] fix(security): prevent Authorization header leakage across redirects (#947) (#1067) * fix(security): prevent auth header leakage across redirects * fix(security): harden redirect credential handling Address Copilot and Qodo review findings for #947. - Remove unused variables, imports, and unnecessary pass statements from tests. - Harden cross-origin redirect handling for per-request auth credentials. - Strip session-level auth handlers before cross-origin redirect hops. - Prevent session.auth from regenerating Authorization headers. - Disable trust_env during cross-origin hops to prevent .netrc credential injection. - Restore session auth and trust_env state reliably with try/finally. - Add regression coverage for auth=, session.auth, trust_env, and multi-hop redirects. - Preserve existing security behavior and same-origin authentication semantics. Validated with 189/189 security and affected tests passing. * fix(security): scope allow_private_ips to same-host redirects, fix error handling gaps Follow-up to review findings on #1067: - MCPClient hardcoded allow_private_ips=True for every redirect hop, not just its operator-configured host, so a compromised/malicious MCP server could 302 into private address space (e.g. cloud metadata) unchecked. request_with_ssrf_guard() gains allow_private_ips_on_redirect: a redirect target inherits the original host's private-IP trust only when it matches that host; MCPClient now pins it to False. - detect_public_api() only caught requests.exceptions.RequestException, but the SSRF guard raises ValidationError for blocked hosts/redirects, unlike its sibling ingest_public_api(). Now catches and re-raises it the same way. - detect_public_api()/ingest_public_api() forwarded session/allow_private_ips through **options into request_with_ssrf_guard(), which already passes both explicitly -- a caller supplying either would hit a duplicate-kwarg TypeError. Both are now popped from request_options first. New regression coverage for all three in tests/ingest/, plus a CHANGELOG entry under Unreleased/Security. --------- Co-authored-by: KaifAhmad1 --- CHANGELOG.md | 10 + semantica/ingest/mcp_client.py | 49 +- semantica/ingest/public_api_ingestor.py | 38 +- semantica/ingest/ssrf.py | 227 +++- semantica/seed/seed_manager.py | 7 +- tests/ingest/conftest.py | 35 + .../test_auth_header_redirect_security.py | 1063 +++++++++++++++++ tests/ingest/test_cookbook_integration.py | 40 +- tests/ingest/test_public_api_ingestor.py | 85 +- tests/ingest/test_submodules.py | 135 +-- tests/test_seed_manager.py | 54 + 11 files changed, 1561 insertions(+), 182 deletions(-) create mode 100644 tests/ingest/conftest.py create mode 100644 tests/ingest/test_auth_header_redirect_security.py diff --git a/CHANGELOG.md b/CHANGELOG.md index f670cb22..0ebdc236 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -176,6 +176,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Security +- **`Authorization`/`Proxy-Authorization` credentials could leak to a different origin across HTTP redirects, and several ingest paths bypassed the shared SSRF/redirect guard entirely** (#1067, closes #947) by @Sameer6305, reviewed by @KaifAhmad1 + - `request_with_ssrf_guard()` previously only stripped sensitive headers from per-request `kwargs["headers"]` on a cross-origin redirect; session-level `Authorization`/`Proxy-Authorization` headers, `session.auth`, and `session.trust_env` (`.netrc` lookup) could all still resurrect credentials on the hop to a foreign origin. All five credential sources are now stripped case-insensitively, kept stripped for the remainder of a multi-hop redirect chain (no resurrection even if a later hop returns to the original host), and unconditionally restored via `finally` — including on exceptions and redirect-limit errors + - `MCPClient._send_request_http()` and `PublicAPIIngestor.detect_public_api()`/`ingest_public_api()` called `httpx.post()`/`requests.post()`/`session.request()` directly, bypassing `request_with_ssrf_guard()` entirely. Both now route through the shared guard, including when `validate_no_auth=False` + - `SeedDataManager.load_from_api()` mutated the caller-supplied `headers` dict in place when adding an API-key `Authorization` header, silently leaking the key back into a dict the caller might reuse elsewhere. Now copies before modifying + - **Fixed during review** (@KaifAhmad1): `allow_private_ips=True` (used to let MCP servers run on localhost/internal networks) was applied to every redirect hop, not just the operator-configured host — a compromised or malicious MCP server could 302-redirect to an internal address (e.g. `169.254.169.254` cloud metadata) and the guard would follow it unchecked, defeating the SSRF protection this PR otherwise adds. Added `allow_private_ips_on_redirect` to `request_with_ssrf_guard()`: a redirect target inherits the original host's private-IP trust only when it matches that host; any other host falls back to strict validation. `MCPClient` now pins `allow_private_ips_on_redirect=False`, so only same-host redirects on a trusted MCP server keep working — a cross-host hop into private address space is blocked + - **Fixed during review** (@KaifAhmad1): `detect_public_api()` only caught `requests.exceptions.RequestException`, but `request_with_ssrf_guard()` raises `ValidationError` (a disjoint hierarchy) for SSRF-blocked hosts, blocked redirect targets, missing `Location`, or exceeded redirect limits — unlike its sibling `ingest_public_api()`, which already caught it. Callers (including `is_public_api()`) got an undocumented raw `ValidationError` instead of `ProcessingError`, and the error-logging call was skipped. Now catches `(ValidationError, ProcessingError)` and re-raises, matching the sibling method + - **Fixed during review** (@KaifAhmad1): `detect_public_api()`/`ingest_public_api()` forwarded `**options` into `request_with_ssrf_guard(..., session=self.session, allow_private_ips=self.allow_private_ips, **request_options)` without stripping `session`/`allow_private_ips` from `request_options` first — a caller passing either through the per-call `**options` (a plausible mistake, since `allow_private_ips` is also a documented constructor-level knob) got a raw `TypeError: got multiple values for keyword argument`. Both are now popped from `request_options` before the call + - New regression coverage added during review: `TestAllowPrivateIpsOnRedirect` (cross-host redirect into private space blocked, same-host redirect trust preserved, default behavior unchanged for existing callers that don't pass the new kwarg) and `TestMCPClientAuthRedirect::test_redirect_to_private_ip_is_blocked`/`test_same_host_redirect_on_private_mcp_server_is_not_blocked` in `tests/ingest/test_auth_header_redirect_security.py`; `test_detect_public_api_propagates_ssrf_validation_error` and duplicate-kwarg regression tests for both methods in `tests/ingest/test_public_api_ingestor.py` + - `pytest tests/ingest/test_auth_header_redirect_security.py tests/ingest/test_public_api_ingestor.py tests/test_seed_manager.py tests/ingest/test_submodules.py tests/ingest/test_cookbook_integration.py`: 111 passed + - **`FeedIngestor`/`FeedMonitor` (RSS/Atom feed ingestion) had no SSRF protection, allowing requests to internal/private network targets** (#928, closes #927) by @ZohaibHassan16 - `FeedIngestor.ingest_feed()`, `discover_feeds()` (link-tag fetch, common-path HEAD probe, and feed-validation GET), and `FeedMonitor.check_updates()` all called `requests.get()`/`requests.head()` directly with default redirect-following and no scheme allowlist or private/loopback/link-local IP validation — despite `semantica/ingest/ssrf.py`'s `request_with_ssrf_guard()` already existing and being used by `web_ingestor.py`/`api_ingestor.py`. `ingest_feed()`'s own URL check only verified `urlparse(url).scheme`/`.netloc` were non-empty, never that the scheme was http/https or that the resolved target IP was safe. Reachable via the public `ingest_feed()`/`ingest()` entry points with any caller-supplied feed URL - All 5 call sites now route through `request_with_ssrf_guard()`, which validates scheme (http/https only) and resolved IP before the request, and re-validates every redirect `Location` before following it — closing both the direct-IP and redirect-chain SSRF paths. Added an `allow_private_ips` config option to both `FeedIngestor` and `FeedMonitor`, consistent with the other ingestors diff --git a/semantica/ingest/mcp_client.py b/semantica/ingest/mcp_client.py index 2b33dfe1..302e3365 100644 --- a/semantica/ingest/mcp_client.py +++ b/semantica/ingest/mcp_client.py @@ -41,6 +41,7 @@ from typing import Any, Dict, List, Optional, Union from ..utils.exceptions import ProcessingError, ValidationError from ..utils.logging import get_logger +from .ssrf import request_with_ssrf_guard @dataclass @@ -341,36 +342,38 @@ class MCPClient: raise def _send_request_http(self, request: Dict[str, Any]) -> Optional[Dict[str, Any]]: - """Send request via HTTP.""" - try: - import httpx + """Send request via HTTP, with redirect-safe credential handling. - response = httpx.post( + Uses ``request_with_ssrf_guard`` so that: + + * ``Authorization`` / ``Proxy-Authorization`` headers are **not** + forwarded to a different origin if the MCP server issues a redirect + (issue #947). + * The redirect chain is bounded (default 10 hops). + + ``allow_private_ips=True`` is set because MCP servers are explicitly + configured by the operator and frequently run on localhost or an + internal network — the same trust model as ``allow_private_ips`` opt-in + in the other ingestors. That trust covers only ``self.url`` itself: + ``allow_private_ips_on_redirect=False`` keeps redirect targets held to + the normal public-address check, so a compromised or malicious MCP + server cannot use a redirect to route the client into private/ + internal address space (e.g. cloud metadata) that the operator never + configured. Scheme validation (http/https only) and the + auth-stripping logic remain active regardless of these flags. + """ + try: + response = request_with_ssrf_guard( + "POST", self.url, - json=request, headers=self.headers, + json=request, timeout=self.config.get("timeout", 30.0), + allow_private_ips=True, + allow_private_ips_on_redirect=False, ) response.raise_for_status() return response.json() - except (ImportError, OSError): - # Fallback to requests if httpx not available - try: - import requests - - response = requests.post( - self.url, - json=request, - headers=self.headers, - timeout=self.config.get("timeout", 30.0), - ) - response.raise_for_status() - return response.json() - except (ImportError, OSError): - raise ProcessingError( - "HTTP transport requires 'httpx' or 'requests' package. " - "Install with: pip install httpx or pip install requests" - ) except Exception as e: self.logger.error(f"Failed to send HTTP request: {e}") raise diff --git a/semantica/ingest/public_api_ingestor.py b/semantica/ingest/public_api_ingestor.py index afefbe27..2ea15b20 100644 --- a/semantica/ingest/public_api_ingestor.py +++ b/semantica/ingest/public_api_ingestor.py @@ -45,6 +45,7 @@ except ModuleNotFoundError: # pragma: no cover - fallback for minimal installs from ..utils.exceptions import ProcessingError, ValidationError from ..utils.logging import get_logger from .api_ingestor import APIData, RESTIngestor +from .ssrf import request_with_ssrf_guard AUTH_HEADER_NAMES = { "authorization", @@ -359,18 +360,31 @@ class PublicAPIIngestor(RESTIngestor): request_options = options.copy() timeout = request_options.pop("timeout", self.config.get("timeout", 30)) rate_limit_delay = request_options.pop("rate_limit_delay", None) + # session and allow_private_ips are always supplied explicitly below; + # drop any caller-provided copies so request_with_ssrf_guard() does + # not receive duplicate keyword arguments. + request_options.pop("session", None) + request_options.pop("allow_private_ips", None) request_headers = self._merged_headers(headers) try: self._wait_if_needed(rate_limit_delay=rate_limit_delay) - response = self.session.request( - method=method, - url=endpoint, + # Route through the SSRF guard so that: + # * redirects to private/loopback IPs are blocked, and + # * Authorization / Proxy-Authorization are stripped on + # cross-origin redirects (issue #947). + response = request_with_ssrf_guard( + method, + endpoint, + session=self.session, headers=request_headers, params=params, timeout=timeout, + allow_private_ips=self.allow_private_ips, **request_options, ) + except (ValidationError, ProcessingError): + raise except requests.exceptions.RequestException as exc: self.logger.error(f"Failed to detect public API {endpoint}: {exc}") raise ProcessingError(f"Failed to detect public API: {exc}") from exc @@ -440,18 +454,30 @@ class PublicAPIIngestor(RESTIngestor): request_options = options.copy() timeout = request_options.pop("timeout", self.config.get("timeout", 30)) + # session and allow_private_ips are always supplied explicitly below; + # drop any caller-provided copies so request_with_ssrf_guard() does + # not receive duplicate keyword arguments. + request_options.pop("session", None) + request_options.pop("allow_private_ips", None) request_headers = self._merged_headers(headers) try: self._wait_if_needed(rate_limit_delay=rate_limit_delay) - response = self.session.request( - method=method, - url=endpoint, + # Route through the SSRF guard so that: + # * redirects to private/loopback IPs are blocked, and + # * Authorization / Proxy-Authorization are stripped on + # cross-origin redirects even when validate_no_auth=False + # (issue #947). + response = request_with_ssrf_guard( + method, + endpoint, + session=self.session, headers=request_headers, params=params, data=data, json=json_data, timeout=timeout, + allow_private_ips=self.allow_private_ips, **request_options, ) diff --git a/semantica/ingest/ssrf.py b/semantica/ingest/ssrf.py index 083fbcca..488ae3cf 100644 --- a/semantica/ingest/ssrf.py +++ b/semantica/ingest/ssrf.py @@ -268,6 +268,7 @@ def request_with_ssrf_guard( *, session: Optional[requests.Session] = None, allow_private_ips: bool = False, + allow_private_ips_on_redirect: Optional[bool] = None, max_redirects: int = _DEFAULT_MAX_REDIRECTS, **kwargs: Any, ) -> requests.Response: @@ -277,10 +278,65 @@ def request_with_ssrf_guard( public URL to bounce into private/loopback/link-local space. This helper disables automatic redirects and re-validates each ``Location`` target before issuing the next hop. + + ``allow_private_ips`` trusts the caller's own *url* (e.g. an + operator-configured internal endpoint). That trust follows a redirect + only when the redirect target's host matches the original host (e.g. a + same-host path redirect on a private/localhost server); a redirect to a + *different* host is validated with ``allow_private_ips_on_redirect`` + instead, which defaults to ``allow_private_ips`` for backward + compatibility but can be pinned to ``False`` by callers that want to + trust only the original host and never extend private-IP eligibility to + any other host a redirect chain might reach — otherwise a private-IP- + eligible endpoint could be tricked into redirecting into arbitrary + internal address space (e.g. cloud metadata) the caller never + configured. + + Authorization / credential-header handling (issue #947) + -------------------------------------------------------- + Credentials are stripped from **all** sources that ``requests`` can use to + attach an ``Authorization`` header whenever a redirect changes origin: + + 1. ``kwargs["headers"]`` — per-request header dict (already handled). + 2. ``session.headers`` — session-level headers that ``requests`` merges + automatically; cleared for the hop and restored via ``finally``. + 3. ``kwargs["auth"]`` — per-request auth tuple/callable; removed from the + local ``kwargs`` copy when stripping is required. This copy never + escapes to the caller, so there is nothing to restore. + 4. ``session.auth`` — session-level auth handler that ``requests`` merges + via ``merge_setting(auth, self.auth)`` inside ``prepare_request``; + cleared for the hop and restored via ``finally``. + 5. ``session.trust_env`` — when ``True``, ``requests`` reads ``~/.netrc`` + for the *redirect target* host and calls ``prepare_auth()`` with those + credentials even after sources 3 and 4 are cleared; disabled for + cross-origin hops and restored via ``finally``. + + Leaving any one of these intact allows ``requests`` to re-attach + credentials on the hop to the foreign origin, defeating the header-level + strip. + + Session state that was removed is unconditionally restored in a ``finally`` + block so the session is left in its original state after this call returns, + regardless of how it exits (normal return, exception, redirect cap). The + loop is sequential and single-threaded within one call, so the mutation is + safe as long as the caller does not share the session across concurrent + threads (the standard Semantica pattern: one session per ingestor instance). + + Once credentials have been stripped for a cross-origin hop they are NOT + re-added for subsequent hops in the same chain, even if a later hop + happens to point back to the original host. This prevents credential + resurrection via crafted multi-hop redirect chains. """ kwargs = dict(kwargs) kwargs.pop("allow_redirects", None) + redirect_allow_private_ips = ( + allow_private_ips + if allow_private_ips_on_redirect is None + else allow_private_ips_on_redirect + ) + _original_host = (urlparse(url).hostname or "").lower() + validate_url_for_request(url, allow_private_ips=allow_private_ips) requester = session.request if session is not None else requests.request @@ -288,56 +344,141 @@ def request_with_ssrf_guard( current_method = method.upper() redirects_followed = 0 - while True: - response = requester( - current_method, - current_url, - allow_redirects=False, - **kwargs, - ) + # -- issue #947: snapshot every session-level credential source so we can + # restore them unconditionally when this call exits. + _SENSITIVE = ("Authorization", "Proxy-Authorization") + _session_auth_backup: dict = {} + _session_auth_handler_backup: Any = None # session.auth backup + _session_trust_env_backup: bool = True # session.trust_env backup - if response.status_code not in _REDIRECT_STATUS_CODES: - return response + if session is not None: + for _h in _SENSITIVE: + # requests stores session headers in a case-insensitive dict; + # .get() matches regardless of the casing used at insertion time. + _val = session.headers.get(_h) + if _val is not None: + _session_auth_backup[_h] = _val + # Snapshot session.auth (HTTPBasicAuth, tuple, callable, or None). + _session_auth_handler_backup = session.auth + # Snapshot session.trust_env (controls .netrc / env proxy lookup). + _session_trust_env_backup = session.trust_env - if redirects_followed >= max_redirects: - response.close() - raise ValidationError( - f"Exceeded maximum redirects ({max_redirects}) while " - f"fetching '{url}'" + # Track whether credentials have been stripped for this redirect chain. + # Once stripped they must not reappear on any subsequent hop. + _auth_stripped = False + + try: + while True: + response = requester( + current_method, + current_url, + allow_redirects=False, + **kwargs, ) - location = response.headers.get("Location") - if not location or not str(location).strip(): - response.close() - raise ValidationError( - f"Redirect from '{current_url}' is missing a Location header" + if response.status_code not in _REDIRECT_STATUS_CODES: + return response + + if redirects_followed >= max_redirects: + response.close() + raise ValidationError( + f"Exceeded maximum redirects ({max_redirects}) while " + f"fetching '{url}'" + ) + + location = response.headers.get("Location") + if not location or not str(location).strip(): + response.close() + raise ValidationError( + f"Redirect from '{current_url}' is missing a Location header" + ) + + next_url = urljoin(current_url, str(location).strip()) + next_host = (urlparse(next_url).hostname or "").lower() + # A redirect back to the original host inherits the caller's + # trust in that host (e.g. a same-host path redirect on a + # private/localhost MCP server). A redirect to a *different* + # host must not inherit that trust, even if the original host + # was private/internal — otherwise a compromised or malicious + # endpoint could redirect into arbitrary private address space + # (e.g. cloud metadata) the caller never configured. + hop_allow_private_ips = ( + allow_private_ips + if next_host and next_host == _original_host + else redirect_allow_private_ips ) + validate_url_for_request(next_url, allow_private_ips=hop_allow_private_ips) - next_url = urljoin(current_url, str(location).strip()) - validate_url_for_request(next_url, allow_private_ips=allow_private_ips) + # Do not leak sensitive headers or auth handlers to a different + # origin on redirects. All four credential sources are cleared: + # • kwargs["headers"] — per-request header dict + # • session.headers — session-level header dict + # • kwargs["auth"] — per-request auth tuple/callable + # • session.auth — session-level auth handler + # + # Once stripped (_auth_stripped=True), credentials stay absent for + # the remainder of the chain — even if a later hop targets the + # original host — to prevent credential resurrection. + if _auth_stripped or _should_strip_auth(current_url, next_url): + _auth_stripped = True - # Do not leak sensitive headers to a different origin on redirects: - # reuse the caller's headers only while host, port, and scheme keep - # the credential safe, mirroring requests' should_strip_auth. - if _should_strip_auth(current_url, next_url): - kwargs = dict(kwargs) - headers = dict(kwargs.get("headers") or {}) - for sensitive in ("Authorization", "Proxy-Authorization"): - headers.pop(sensitive, None) - kwargs["headers"] = headers + # 1. Strip from per-request kwargs headers. + kwargs = dict(kwargs) + headers = dict(kwargs.get("headers") or {}) + for sensitive in _SENSITIVE: + headers.pop(sensitive, None) + # Also remove any case variant the caller may have used + # (e.g. "authorization" or "AUTHORIZATION"). + for key in list(headers): + if key.lower() == sensitive.lower(): + del headers[key] + kwargs["headers"] = headers - # Match requests' historical method rewriting for 301/302/303. - if ( - response.status_code in _STRIP_BODY_ON_REDIRECT - and current_method not in {"GET", "HEAD"} - ): - current_method = "GET" - for key in ("data", "json", "files"): - kwargs.pop(key, None) + # 2. Strip per-request auth kwarg so requests cannot call + # prepare_auth() with the caller's credential on this hop. + kwargs.pop("auth", None) - # Params apply to the original request URL only; Location is authoritative. - kwargs.pop("params", None) + # 3. Strip session-level headers so requests cannot re-inject + # them when merging session + per-request headers for this hop. + if session is not None: + for sensitive in _SENSITIVE: + # CaseInsensitiveDict.pop(key, None) handles any casing. + session.headers.pop(sensitive, None) - response.close() - current_url = next_url - redirects_followed += 1 + # 4. Clear session.auth so prepare_request's merge_setting() + # cannot fall back to the session-level auth handler and + # reattach credentials on the foreign-origin hop. + session.auth = None + + # 5. Disable .netrc / environment-proxy credential lookup so + # requests cannot inject credentials from ~/.netrc for the + # redirect target host on this hop. + session.trust_env = False + + # Match requests' historical method rewriting for 301/302/303. + if ( + response.status_code in _STRIP_BODY_ON_REDIRECT + and current_method not in {"GET", "HEAD"} + ): + current_method = "GET" + for key in ("data", "json", "files"): + kwargs.pop(key, None) + + # Params apply to the original request URL only; Location is authoritative. + kwargs.pop("params", None) + + response.close() + current_url = next_url + redirects_followed += 1 + + finally: + # Unconditionally restore every session credential source we touched, + # so the session is in its original state after this call returns or raises. + if session is not None: + if _session_auth_backup: + for _h, _v in _session_auth_backup.items(): + session.headers[_h] = _v + # Restore session.auth to whatever it was before this call. + session.auth = _session_auth_handler_backup + # Restore session.trust_env (.netrc / env-proxy lookup flag). + session.trust_env = _session_trust_env_backup diff --git a/semantica/seed/seed_manager.py b/semantica/seed/seed_manager.py index 16f21ea1..6e52c382 100644 --- a/semantica/seed/seed_manager.py +++ b/semantica/seed/seed_manager.py @@ -501,8 +501,11 @@ class SeedDataManager: else: full_url = api_url - # Prepare headers - request_headers = headers or {} + # Prepare headers — copy the caller's dict so we never mutate it in-place. + # Without the copy, adding "Authorization" here would silently modify the + # caller's original dict and potentially leak the key to subsequent calls + # that reuse the same dict without expecting it to contain credentials. + request_headers = dict(headers) if headers else {} if api_key: request_headers["Authorization"] = f"Bearer {api_key}" diff --git a/tests/ingest/conftest.py b/tests/ingest/conftest.py new file mode 100644 index 00000000..98f913a5 --- /dev/null +++ b/tests/ingest/conftest.py @@ -0,0 +1,35 @@ +""" +Shared pytest fixtures for the ingest test suite. + +The ``mock_dns`` fixture is applied to *every* test in this directory +(``autouse=True``). It stubs out ``socket.getaddrinfo`` inside the SSRF +guard module so that unit tests that mock ``requests.Session.request`` do not +accidentally hit the network for DNS resolution — which would fail in offline +CI environments and cause intermittent timeouts. + +Tests that explicitly need to exercise DNS-related behaviour (e.g. checking +that a hostname resolving to a private IP is blocked) override this fixture +by patching ``semantica.ingest.ssrf.socket.getaddrinfo`` with their own +``side_effect`` *inside* the test body; that inner patch wins because +``unittest.mock.patch`` applies patches in innermost-last order. +""" +from __future__ import annotations + +import socket +from unittest.mock import patch + +import pytest + +_PUBLIC_IP = "93.184.216.34" # example.com — a safe, routable public address + + +@pytest.fixture(autouse=True) +def mock_dns(): + """Map every hostname to a safe public IP for the duration of each test.""" + with patch( + "semantica.ingest.ssrf.socket.getaddrinfo", + return_value=[ + (socket.AF_INET, socket.SOCK_STREAM, 6, "", (_PUBLIC_IP, 0)) + ], + ): + yield diff --git a/tests/ingest/test_auth_header_redirect_security.py b/tests/ingest/test_auth_header_redirect_security.py new file mode 100644 index 00000000..882a9d59 --- /dev/null +++ b/tests/ingest/test_auth_header_redirect_security.py @@ -0,0 +1,1063 @@ +"""Security regression tests for issue #947. + +Prevents Authorization / Proxy-Authorization headers from leaking across +cross-origin redirects in request_with_ssrf_guard, MCPClient, and +PublicAPIIngestor. + +Each test is focused on a single, specific security property so that a future +regression immediately pinpoints the broken invariant. +""" +from __future__ import annotations + +import socket +from unittest.mock import MagicMock, patch + +import pytest +import requests + +from semantica.ingest.mcp_client import MCPClient +from semantica.ingest.public_api_ingestor import PublicAPIIngestor +from semantica.ingest.ssrf import request_with_ssrf_guard +from semantica.utils.exceptions import ValidationError + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +_PUBLIC_IP = "93.184.216.34" # example.com — public, safe + + +def _public_getaddrinfo(host, *args, **kwargs): + """DNS stub that maps every hostname to a safe public IP.""" + return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (_PUBLIC_IP, 0))] + + +def _make_session_with_auth(token: str = "Bearer secret") -> requests.Session: + """Return a real requests.Session with Authorization in session.headers.""" + sess = requests.Session() + sess.headers["Authorization"] = token + return sess + + +def _mock_redirect(location: str, status: int = 302) -> MagicMock: + r = MagicMock() + r.status_code = status + r.headers = {"Location": location} + r.close = MagicMock() + return r + + +def _mock_final(status: int = 200) -> MagicMock: + r = MagicMock() + r.status_code = status + r.headers = {} + r.close = MagicMock() + return r + + +# =========================================================================== +# Section 1 – request_with_ssrf_guard: session.headers stripping (#947) +# =========================================================================== + + +class TestSessionHeadersStripping: + """Authorization stored in session.headers must not reach a foreign origin.""" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_authorization_stripped_on_cross_origin_redirect(self, _): + """session.headers["Authorization"] must not appear in the hop to a new host.""" + sess = _make_session_with_auth() + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]) as mock_req: + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert mock_req.call_count == 2 + # The second call must not carry Authorization in kwargs["headers"]. + second_headers = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second_headers + # Also verify requests won't re-inject it via session (the guard must + # have cleared it from sess.headers before the second call). + assert "Authorization" not in sess.headers or sess.headers.get("Authorization") == "Bearer secret" + # Post-call restoration: session must be restored. + assert sess.headers.get("Authorization") == "Bearer secret" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_headers_cleared_before_second_hop_not_just_restored_after(self, _): + """Prove session.headers["Authorization"] is absent AT CALL TIME of the second hop. + + This test closes the gap where a mock-based test only checks kwargs["headers"] + but not whether session.headers was actually cleared before requests' internal + header-merge would re-inject the credential. + + Strategy: capture a snapshot of sess.headers at each call invocation so we + can assert it was empty during the second hop — not just after the guard returns. + """ + sess = _make_session_with_auth("Bearer proof-token") + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + snapshots: list = [] + + def capturing_side_effect(*args, **kwargs): + # Snapshot what session.headers contain at the exact moment of this call. + snapshots.append(dict(sess.headers)) + return [redirect, final][len(snapshots) - 1] + + with patch.object(sess, "request", side_effect=capturing_side_effect): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert len(snapshots) == 2 + + # Hop 1 (same origin, pre-redirect): Authorization PRESENT in session.headers. + assert snapshots[0].get("Authorization") == "Bearer proof-token", ( + "Authorization must be in session.headers for the first (same-origin) call" + ) + + # Hop 2 (cross-origin): Authorization ABSENT from session.headers. + # This is what prevents requests from re-injecting it via its header-merge step. + assert "Authorization" not in snapshots[1], ( + "Authorization must have been removed from session.headers BEFORE the " + "second (cross-origin) call — removing it only from kwargs is not enough " + "because requests.Session merges session.headers at call time." + ) + + # After the guard returns, session state is fully restored. + assert sess.headers.get("Authorization") == "Bearer proof-token", ( + "session.headers must be restored after the guard returns" + ) + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_authorization_preserved_on_same_origin_redirect(self, _): + """Same-origin redirect must keep Authorization in session.headers untouched.""" + sess = _make_session_with_auth() + redirect = _mock_redirect("https://example.com/page2") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]) as mock_req: + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert mock_req.call_count == 2 + # When no stripping occurred, kwargs["headers"] is unchanged from + # the caller (no headers kwarg was passed here, so it may be absent + # or empty — what matters is that the session header was NOT cleared). + assert sess.headers.get("Authorization") == "Bearer secret" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_credentials_restored_after_successful_request(self, _): + """Session headers must be restored after a redirect chain completes normally.""" + sess = _make_session_with_auth("Bearer my-token") + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert sess.headers.get("Authorization") == "Bearer my-token" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_credentials_restored_after_ssrf_exception(self, _): + """Session headers must be restored even when the guard raises ValidationError.""" + sess = _make_session_with_auth("Bearer my-token") + # Redirect to a loopback address — guard will raise. + redirect = _mock_redirect("http://127.0.0.1/secret") + + with patch.object(sess, "request", return_value=redirect): + with pytest.raises(ValidationError): + request_with_ssrf_guard( + "GET", "https://example.com/start", session=sess + ) + + assert sess.headers.get("Authorization") == "Bearer my-token" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_credentials_restored_after_max_redirects_exceeded(self, _): + """Session headers must be restored when the max-redirect cap is hit.""" + sess = _make_session_with_auth("Bearer loop-token") + hop = _mock_redirect("https://other.example/loop") + + # All hops redirect to the same foreign host → exceeds cap. + with patch.object(sess, "request", return_value=hop): + with pytest.raises(ValidationError, match="Exceeded maximum"): + request_with_ssrf_guard( + "GET", + "https://example.com/start", + session=sess, + max_redirects=2, + ) + + assert sess.headers.get("Authorization") == "Bearer loop-token" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_proxy_authorization_stripped_on_cross_origin_redirect(self, _): + """Proxy-Authorization must be stripped alongside Authorization.""" + sess = requests.Session() + sess.headers["Proxy-Authorization"] = "Basic cHJveHk6cGFzcw==" + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]) as mock_req: + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + second_headers = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Proxy-Authorization" not in second_headers + # Restored after call. + assert "Proxy-Authorization" in sess.headers + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_both_auth_headers_stripped_simultaneously(self, _): + """Both Authorization and Proxy-Authorization must be stripped together.""" + sess = requests.Session() + sess.headers["Authorization"] = "Bearer tok" + sess.headers["Proxy-Authorization"] = "Basic abc" + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]) as mock_req: + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + second_headers = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second_headers + assert "Proxy-Authorization" not in second_headers + # Restored after call. + assert sess.headers.get("Authorization") == "Bearer tok" + assert sess.headers.get("Proxy-Authorization") == "Basic abc" + + +# =========================================================================== +# Section 1b – request_with_ssrf_guard: auth= kwarg and session.auth stripping +# =========================================================================== + + +class TestAuthHandlerStripping: + """kwargs['auth'] and session.auth must not reach a foreign origin. + + requests uses two additional credential channels beyond header dicts: + • auth= kwarg → passed to PreparedRequest.prepare_auth() directly + • session.auth → merged by Session.prepare_request() via merge_setting() + and then calls prepare_auth() — so even if headers are + stripped, a live session.auth re-attaches Authorization. + + Both must be cleared on cross-origin redirect. + """ + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_kwargs_auth_stripped_on_cross_origin_redirect(self, _): + """auth= kwarg must not be forwarded to the second hop on a different host. + + Verifies that the second call to the underlying requester does NOT + receive an 'auth' kwarg, so requests cannot call prepare_auth() and + regenerate an Authorization header for the foreign origin. + """ + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + auth=("user", "secret-password"), + ) + + assert mock_req.call_count == 2 + + # First hop: auth= kwarg is present (same origin, no strip yet). + first_auth = mock_req.call_args_list[0].kwargs.get("auth") + assert first_auth == ("user", "secret-password"), ( + "auth= kwarg must be forwarded on the first (same-origin) hop" + ) + + # Second hop: auth= kwarg must be absent (cross-origin — stripped). + second_auth = mock_req.call_args_list[1].kwargs.get("auth") + assert second_auth is None, ( + "auth= kwarg must be removed before the cross-origin hop so " + "requests cannot call prepare_auth() and reattach Authorization" + ) + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_kwargs_auth_preserved_on_same_origin_redirect(self, _): + """auth= kwarg must survive a same-host redirect unchanged.""" + redirect = _mock_redirect("https://example.com/new-path") + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + auth=("user", "secret-password"), + ) + + assert mock_req.call_count == 2 + second_auth = mock_req.call_args_list[1].kwargs.get("auth") + assert second_auth == ("user", "secret-password"), ( + "auth= kwarg must be kept for same-origin redirects" + ) + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_auth_cleared_before_cross_origin_hop(self, _): + """session.auth must be None AT CALL TIME of the cross-origin hop. + + This test uses the same snapshot-at-invocation technique as the + session.headers equivalent: capture session.auth at the exact moment + each call is issued, so we can prove the handler was absent before + requests' merge_setting() could reattach it. + """ + sess = requests.Session() + sess.auth = ("user", "secret-password") + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + auth_snapshots: list = [] + + def capturing_side_effect(*args, **kwargs): + # Snapshot session.auth at the exact moment of this call. + auth_snapshots.append(sess.auth) + return [redirect, final][len(auth_snapshots) - 1] + + with patch.object(sess, "request", side_effect=capturing_side_effect): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert len(auth_snapshots) == 2 + + # Hop 1 (same origin): session.auth is PRESENT. + assert auth_snapshots[0] == ("user", "secret-password"), ( + "session.auth must be intact for the first (same-origin) call" + ) + + # Hop 2 (cross-origin): session.auth must be ABSENT (None). + assert auth_snapshots[1] is None, ( + "session.auth must have been cleared BEFORE the cross-origin call " + "so requests' merge_setting() cannot reattach the credential" + ) + + # After the guard returns, session.auth must be fully restored. + assert sess.auth == ("user", "secret-password"), ( + "session.auth must be restored after the guard returns" + ) + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_auth_preserved_on_same_origin_redirect(self, _): + """session.auth must not be touched for same-host redirects.""" + sess = requests.Session() + sess.auth = ("user", "secret-password") + redirect = _mock_redirect("https://example.com/page2") + final = _mock_final() + + auth_snapshots: list = [] + + def capturing_side_effect(*args, **kwargs): + auth_snapshots.append(sess.auth) + return [redirect, final][len(auth_snapshots) - 1] + + with patch.object(sess, "request", side_effect=capturing_side_effect): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert len(auth_snapshots) == 2 + # Both hops see session.auth intact. + assert auth_snapshots[0] == ("user", "secret-password") + assert auth_snapshots[1] == ("user", "secret-password") + assert sess.auth == ("user", "secret-password") + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_auth_restored_after_successful_request(self, _): + """session.auth must be restored to its original value after the call.""" + sess = requests.Session() + sess.auth = ("user", "secret-password") + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert sess.auth == ("user", "secret-password") + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_auth_restored_after_ssrf_exception(self, _): + """session.auth must be restored even when the guard raises.""" + sess = requests.Session() + sess.auth = ("user", "secret-password") + # Redirect to loopback — guard raises ValidationError. + redirect = _mock_redirect("http://127.0.0.1/secret") + + with patch.object(sess, "request", return_value=redirect): + with pytest.raises(ValidationError): + request_with_ssrf_guard( + "GET", "https://example.com/start", session=sess + ) + + assert sess.auth == ("user", "secret-password") + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_auth_none_by_default_remains_none(self, _): + """When session.auth is None (default), the finally block must not set it + to something unexpected — restoring None is a no-op, not a corruption.""" + sess = requests.Session() + assert sess.auth is None + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert sess.auth is None + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_kwargs_auth_does_not_reappear_in_multihop_chain(self, _): + """Once auth= is stripped at hop 2, it must not reappear at hop 3.""" + hop1 = _mock_redirect("https://other.example/step2") # cross-origin: strip + hop2 = _mock_redirect("https://other.example/final") # same host as hop1: stay stripped + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[hop1, hop2, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + auth=("user", "pass"), + ) + + assert mock_req.call_count == 3 + # Hop 1: auth present (same origin). + assert mock_req.call_args_list[0].kwargs.get("auth") == ("user", "pass") + # Hop 2: stripped. + assert mock_req.call_args_list[1].kwargs.get("auth") is None + # Hop 3: stays stripped — no resurrection. + assert mock_req.call_args_list[2].kwargs.get("auth") is None + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_trust_env_disabled_before_cross_origin_hop(self, _): + """session.trust_env must be False AT CALL TIME of the cross-origin hop. + + When trust_env=True, requests reads ~/.netrc for the redirect target host + and calls prepare_auth() with those credentials — even after session.auth + and kwargs['auth'] are cleared. Disabling trust_env before the hop closes + this bypass channel. + """ + sess = requests.Session() + sess.trust_env = True # explicit default + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + trust_env_snapshots: list = [] + + def capturing_side_effect(*args, **kwargs): + trust_env_snapshots.append(sess.trust_env) + return [redirect, final][len(trust_env_snapshots) - 1] + + with patch.object(sess, "request", side_effect=capturing_side_effect): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert len(trust_env_snapshots) == 2 + + # Hop 1 (same origin): trust_env is True (unchanged). + assert trust_env_snapshots[0] is True, ( + "trust_env must be unchanged for the first (same-origin) call" + ) + + # Hop 2 (cross-origin): trust_env must be False to block .netrc lookup. + assert trust_env_snapshots[1] is False, ( + "trust_env must be False BEFORE the cross-origin call to prevent " + "requests from looking up ~/.netrc credentials for the redirect target" + ) + + # After the guard returns, trust_env must be restored. + assert sess.trust_env is True, ( + "session.trust_env must be restored after the guard returns" + ) + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_trust_env_restored_after_exception(self, _): + """session.trust_env must be restored even when the guard raises.""" + sess = requests.Session() + sess.trust_env = True + redirect = _mock_redirect("http://127.0.0.1/secret") # will raise ValidationError + + with patch.object(sess, "request", return_value=redirect): + with pytest.raises(ValidationError): + request_with_ssrf_guard( + "GET", "https://example.com/start", session=sess + ) + + assert sess.trust_env is True + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_trust_env_false_stays_false_after_call(self, _): + """If trust_env was already False, it must stay False after the call.""" + sess = requests.Session() + sess.trust_env = False # caller explicitly disabled .netrc + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch.object(sess, "request", side_effect=[redirect, final]): + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + assert sess.trust_env is False # restored to the original False value + + +# =========================================================================== +# Section 2 – request_with_ssrf_guard: credential resurrection prevention +# =========================================================================== + + +class TestCredentialResurrection: + """Stripped credentials must not reappear for later hops in the same chain.""" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_credentials_do_not_reappear_after_cross_origin_hop(self, _): + """A subsequent same-origin-as-hop-2 redirect must not restore the credential.""" + # Chain: example.com → other.example (strip) → other.example/page2 (stay stripped) + hop1 = _mock_redirect("https://other.example/step2") + hop2 = _mock_redirect("https://other.example/final") # same host as hop1 target + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[hop1, hop2, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + headers={"Authorization": "Bearer secret"}, + ) + + assert mock_req.call_count == 3 + # Hop 1 (example.com): credential present + h1 = mock_req.call_args_list[0].kwargs.get("headers", {}) + assert h1.get("Authorization") == "Bearer secret" + # Hop 2 (other.example): stripped + h2 = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in h2 + # Hop 3 (still other.example): stays stripped — must NOT reappear + h3 = mock_req.call_args_list[2].kwargs.get("headers", {}) + assert "Authorization" not in h3 + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_auth_does_not_reappear_in_multihop_chain(self, _): + """session.headers auth stripped for hop 2 must stay absent for hop 3.""" + sess = _make_session_with_auth("Bearer multi") + hop1 = _mock_redirect("https://other.example/step2") # cross-origin: strip + hop2 = _mock_redirect("https://other.example/final") # same-as-hop1: stay stripped + final = _mock_final() + + with patch.object(sess, "request", side_effect=[hop1, hop2, final]) as mock_req: + request_with_ssrf_guard("GET", "https://example.com/start", session=sess) + + # After the call the session is restored. + assert sess.headers.get("Authorization") == "Bearer multi" + + h2 = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in h2 + h3 = mock_req.call_args_list[2].kwargs.get("headers", {}) + assert "Authorization" not in h3 + + +# =========================================================================== +# Section 3 – request_with_ssrf_guard: specific redirect-type coverage +# =========================================================================== + + +class TestRedirectTypesAndOriginChanges: + """Per-type and per-scenario auth-stripping rules.""" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_strips_on_307_cross_origin(self, _): + """307 Temporary Redirect to a different host must strip credentials.""" + redirect = MagicMock() + redirect.status_code = 307 + redirect.headers = {"Location": "https://other.example/final"} + redirect.close = MagicMock() + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + headers={"Authorization": "Bearer tok"}, + ) + + second = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_strips_on_308_cross_origin(self, _): + """308 Permanent Redirect to a different host must strip credentials.""" + redirect = MagicMock() + redirect.status_code = 308 + redirect.headers = {"Location": "https://other.example/final"} + redirect.close = MagicMock() + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + headers={"Authorization": "Bearer tok"}, + ) + + second = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_strips_on_port_change(self, _): + """Redirect that changes the port (non-default) must strip credentials.""" + redirect = _mock_redirect("https://example.com:8443/final") + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + headers={"Authorization": "Bearer tok"}, + ) + + second = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_strips_on_subdomain_change(self, _): + """Redirect from apex to subdomain (different hostname) must strip credentials.""" + redirect = _mock_redirect("https://api.example.com/final") + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + headers={"Authorization": "Bearer tok"}, + ) + + second = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_keeps_on_https_443_explicit_to_implicit(self, _): + """https://example.com:443 → https://example.com (same, just drop explicit port).""" + redirect = _mock_redirect("https://example.com/final") + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com:443/start", + headers={"Authorization": "Bearer tok"}, + ) + + second = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert second.get("Authorization") == "Bearer tok" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_case_insensitive_header_stripped(self, _): + """Lowercase/UPPERCASE variants of Authorization must also be stripped.""" + redirect = _mock_redirect("https://other.example/final") + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "https://example.com/start", + # Pass a lowercase variant to verify case-insensitive stripping. + headers={"authorization": "Bearer lower", "AUTHORIZATION": "Bearer upper"}, + ) + + second = mock_req.call_args_list[1].kwargs.get("headers", {}) + for key in second: + assert key.lower() != "authorization", ( + f"Authorization header variant {key!r} was not stripped" + ) + + +class TestAllowPrivateIpsOnRedirect: + """allow_private_ips must not extend to a redirect target on a different host.""" + + def test_cross_host_redirect_to_private_ip_is_blocked_when_pinned(self): + """allow_private_ips_on_redirect=False must block a cross-host hop into private space.""" + redirect = _mock_redirect("http://169.254.169.254/latest/meta-data/") + + with patch( + "semantica.ingest.ssrf.requests.request", + return_value=redirect, + ): + with pytest.raises(ValidationError, match="blocked"): + request_with_ssrf_guard( + "GET", + "https://trusted.example.com/start", + allow_private_ips=True, + allow_private_ips_on_redirect=False, + ) + + def test_same_host_redirect_keeps_private_ip_trust_when_pinned(self): + """A same-host redirect must still inherit the original host's trust.""" + redirect = _mock_redirect("http://localhost:8000/v2") + final = _mock_final() + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + request_with_ssrf_guard( + "GET", + "http://localhost:8000/start", + allow_private_ips=True, + allow_private_ips_on_redirect=False, + ) + + assert mock_req.call_count == 2 + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_default_behavior_unchanged_without_the_new_kwarg(self, _): + """Existing callers that never pass allow_private_ips_on_redirect keep old behavior.""" + redirect = _mock_redirect("http://169.254.169.254/latest/meta-data/") + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, _mock_final()], + ) as mock_req: + # allow_private_ips=True with no override: redirect target validation + # falls back to allow_private_ips, matching pre-fix behavior for the + # existing opt-in ingestors (web/feed/api/public-api/seed). + request_with_ssrf_guard( + "GET", + "https://trusted.example.com/start", + allow_private_ips=True, + ) + + assert mock_req.call_count == 2 + + +# =========================================================================== +# Section 4 – MCPClient: redirect auth-stripping (#947) +# =========================================================================== + + +class TestMCPClientAuthRedirect: + """MCPClient._send_request_http must not leak credentials on cross-origin redirect.""" + + def _mock_mcp_response(self, payload=None): + resp = MagicMock() + resp.status_code = 200 + resp.headers = {} + resp.raise_for_status = MagicMock() + resp.json.return_value = payload or { + "jsonrpc": "2.0", + "id": 1, + "result": {"serverInfo": {}, "capabilities": {}}, + } + return resp + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_cross_origin_redirect_strips_authorization(self, _): + """Authorization must not reach a different host after an MCP server redirect.""" + redirect = _mock_redirect("https://other.example/mcp") + redirect.status_code = 302 + final = self._mock_mcp_response() + + client = MCPClient( + url="https://mcp.example.com/mcp", + headers={"Authorization": "Bearer mcp-token"}, + ) + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + assert mock_req.call_count == 2 + second_headers = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second_headers + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_same_origin_redirect_preserves_authorization(self, _): + """Same-host redirect must keep Authorization intact.""" + redirect = _mock_redirect("https://mcp.example.com/mcp/v2") + redirect.status_code = 301 + final = self._mock_mcp_response() + + client = MCPClient( + url="https://mcp.example.com/mcp", + headers={"Authorization": "Bearer mcp-token"}, + ) + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + assert mock_req.call_count == 2 + second_headers = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert second_headers.get("Authorization") == "Bearer mcp-token" + + def test_localhost_mcp_server_is_not_blocked(self): + """localhost MCP endpoints must work (allow_private_ips=True).""" + final = self._mock_mcp_response() + client = MCPClient(url="http://localhost:8000/mcp") + + with patch( + "semantica.ingest.ssrf.requests.request", + return_value=final, + ) as mock_req: + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + mock_req.assert_called_once() + + def test_loopback_ip_mcp_server_is_not_blocked(self): + """127.0.0.1 MCP endpoints must work (allow_private_ips=True).""" + final = self._mock_mcp_response() + client = MCPClient(url="http://127.0.0.1:9000/mcp") + + with patch( + "semantica.ingest.ssrf.requests.request", + return_value=final, + ) as mock_req: + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + mock_req.assert_called_once() + + def test_same_host_redirect_on_private_mcp_server_is_not_blocked(self): + """A same-host redirect on a trusted private/localhost MCP server must still work.""" + redirect = _mock_redirect("http://localhost:8000/mcp/v2") + final = self._mock_mcp_response() + + client = MCPClient(url="http://localhost:8000/mcp") + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + assert mock_req.call_count == 2 + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_redirect_to_private_ip_is_blocked(self, _): + """A redirect from a public MCP server to a private/internal IP must be blocked. + + allow_private_ips=True trusts the operator-configured MCP host itself; + it must not let a compromised or malicious server redirect the client + into private address space (e.g. cloud metadata) via a cross-host hop. + """ + redirect = _mock_redirect("http://169.254.169.254/latest/meta-data/") + + client = MCPClient(url="https://mcp.example.com/mcp") + + with patch( + "semantica.ingest.ssrf.requests.request", + return_value=redirect, + ): + with pytest.raises(ValidationError, match="blocked"): + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_scheme_downgrade_strips_authorization(self, _): + """https MCP server that redirects to http must strip the credential.""" + redirect = _mock_redirect("http://mcp.example.com/mcp") + redirect.status_code = 302 + final = self._mock_mcp_response() + + client = MCPClient( + url="https://mcp.example.com/mcp", + headers={"Authorization": "Bearer downgrade-test"}, + ) + + with patch( + "semantica.ingest.ssrf.requests.request", + side_effect=[redirect, final], + ) as mock_req: + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + second_headers = mock_req.call_args_list[1].kwargs.get("headers", {}) + assert "Authorization" not in second_headers + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_max_redirect_cap_respected(self, _): + """Infinite redirect loop must raise ValidationError.""" + hop = _mock_redirect("https://mcp.example.com/mcp/loop") + + client = MCPClient(url="https://mcp.example.com/mcp") + + with patch( + "semantica.ingest.ssrf.requests.request", + return_value=hop, + ): + with pytest.raises((ValidationError, Exception), match="[Rr]edirect|[Ee]xceeded"): + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_allow_redirects_false_enforced(self, _): + """The guard must pass allow_redirects=False on every hop.""" + final = self._mock_mcp_response() + client = MCPClient( + url="https://mcp.example.com/mcp", + headers={"Authorization": "Bearer tok"}, + ) + + with patch( + "semantica.ingest.ssrf.requests.request", + return_value=final, + ) as mock_req: + client._send_request_http({"jsonrpc": "2.0", "method": "ping"}) + + assert mock_req.call_args.kwargs.get("allow_redirects") is False + + +# =========================================================================== +# Section 5 – PublicAPIIngestor: redirect auth-stripping (#947) +# =========================================================================== + + +def _mock_public_response(status: int = 200, json_payload=None) -> MagicMock: + resp = MagicMock() + resp.status_code = status + resp.headers = {"Content-Type": "application/json"} + resp.json.return_value = json_payload or [{"id": 1}] + resp.text = "" + if status >= 400: + resp.raise_for_status.side_effect = requests.exceptions.HTTPError( + f"{status} error" + ) + else: + resp.raise_for_status.return_value = None + resp.close = MagicMock() + return resp + + +class TestPublicAPIIngestorRedirectSecurity: + """PublicAPIIngestor must not leak credentials on redirect and must block SSRF.""" + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_redirect_to_private_ip_blocked_in_detect(self, _): + """detect_public_api() must reject a redirect that resolves to a private IP.""" + redirect = _mock_redirect("http://169.254.169.254/latest/meta-data/") + + with patch("requests.Session") as MockSession: + mock_session = MockSession.return_value + mock_session.headers = {} + mock_session.request.return_value = redirect + mock_session.request.return_value.close = MagicMock() + + ingestor = PublicAPIIngestor(rate_limit_delay=0) + with pytest.raises(ValidationError, match="blocked"): + ingestor.detect_public_api("https://example.com/api") + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_redirect_to_private_ip_blocked_in_ingest(self, _): + """ingest_public_api() must reject a redirect that resolves to a private IP.""" + redirect = _mock_redirect("http://10.0.0.1/internal") + + with patch("requests.Session") as MockSession: + mock_session = MockSession.return_value + mock_session.headers = {} + mock_session.request.return_value = redirect + mock_session.request.return_value.close = MagicMock() + + ingestor = PublicAPIIngestor(rate_limit_delay=0) + with pytest.raises(ValidationError, match="blocked"): + ingestor.ingest_public_api("https://example.com/api") + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_session_auth_not_leaked_on_cross_origin_redirect_ingest(self, _): + """Session-level auth header must not reach a foreign host via ingest_public_api.""" + redirect = _mock_redirect("https://other.example/api") + final = _mock_public_response(json_payload=[{"id": 1}]) + + # Simulate a session that somehow has Authorization (e.g. misconfiguration). + with patch("requests.Session") as MockSession: + mock_session = MockSession.return_value + mock_session.headers = {"Authorization": "Bearer leaked"} + mock_session.request.side_effect = [redirect, final] + + ingestor = PublicAPIIngestor( + rate_limit_delay=0, validate_no_auth=False + ) + # Inject the auth-bearing session directly. + ingestor.session = mock_session + + ingestor.ingest_public_api("https://example.com/api") + + assert mock_session.request.call_count == 2 + second_headers = mock_session.request.call_args_list[1].kwargs.get( + "headers", {} + ) + assert "Authorization" not in second_headers + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_allow_redirects_false_enforced_in_detect(self, _): + """detect_public_api() must pass allow_redirects=False to the underlying call.""" + final = _mock_public_response() + + with patch("requests.Session") as MockSession: + mock_session = MockSession.return_value + mock_session.headers = {} + mock_session.request.return_value = final + + ingestor = PublicAPIIngestor(rate_limit_delay=0) + ingestor.detect_public_api("https://example.com/api") + + assert mock_session.request.call_args.kwargs.get("allow_redirects") is False + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_allow_redirects_false_enforced_in_ingest(self, _): + """ingest_public_api() must pass allow_redirects=False to the underlying call.""" + final = _mock_public_response(json_payload=[{"id": 1}]) + + with patch("requests.Session") as MockSession: + mock_session = MockSession.return_value + mock_session.headers = {} + mock_session.request.return_value = final + + ingestor = PublicAPIIngestor(rate_limit_delay=0) + ingestor.ingest_public_api("https://example.com/api") + + assert mock_session.request.call_args.kwargs.get("allow_redirects") is False + + @patch("semantica.ingest.ssrf.socket.getaddrinfo", side_effect=_public_getaddrinfo) + def test_validate_no_auth_false_does_not_bypass_redirect_stripping(self, _): + """Even with validate_no_auth=False the guard strips auth on cross-origin redirect.""" + redirect = _mock_redirect("https://other.example/api") + final = _mock_public_response(json_payload=[{"id": 1}]) + + with patch("requests.Session") as MockSession: + mock_session = MockSession.return_value + mock_session.headers = {} + mock_session.request.side_effect = [redirect, final] + + ingestor = PublicAPIIngestor( + rate_limit_delay=0, validate_no_auth=False + ) + ingestor.ingest_public_api( + "https://example.com/api", + headers={"Authorization": "Bearer should-be-stripped"}, + ) + + second_headers = mock_session.request.call_args_list[1].kwargs.get( + "headers", {} + ) + assert "Authorization" not in second_headers diff --git a/tests/ingest/test_cookbook_integration.py b/tests/ingest/test_cookbook_integration.py index c5120d65..ad470558 100644 --- a/tests/ingest/test_cookbook_integration.py +++ b/tests/ingest/test_cookbook_integration.py @@ -10,19 +10,20 @@ class TestCookbookIntegration: @pytest.fixture def mock_mcp_server(self): - # We need to patch both httpx and requests because MCPClient tries httpx first - with patch("httpx.post") as mock_httpx_post, \ - patch("requests.post") as mock_requests_post: - - def side_effect(url, json=None, **kwargs): + # MCPClient._send_request_http now routes through request_with_ssrf_guard, + # which calls requests.request (not httpx.post / requests.post directly). + # Patch at the point where the guard issues the actual HTTP call. + with patch("semantica.ingest.ssrf.requests.request") as mock_request: + + def side_effect(method, url, json=None, **kwargs): if not json: return MagicMock() - - method = json.get("method") + + rpc_method = json.get("method") response_mock = MagicMock() response_mock.status_code = 200 - - if method == "initialize": + + if rpc_method == "initialize": response_mock.json.return_value = { "jsonrpc": "2.0", "id": json.get("id"), @@ -32,7 +33,7 @@ class TestCookbookIntegration: "serverInfo": {"name": "test_server", "version": "1.0"} } } - elif method == "resources/list": + elif rpc_method == "resources/list": response_mock.json.return_value = { "jsonrpc": "2.0", "id": json.get("id"), @@ -44,7 +45,7 @@ class TestCookbookIntegration: ] } } - elif method == "tools/list": + elif rpc_method == "tools/list": response_mock.json.return_value = { "jsonrpc": "2.0", "id": json.get("id"), @@ -56,7 +57,7 @@ class TestCookbookIntegration: ] } } - elif method == "resources/read": + elif rpc_method == "resources/read": response_mock.json.return_value = { "jsonrpc": "2.0", "id": json.get("id"), @@ -66,13 +67,13 @@ class TestCookbookIntegration: ] } } - elif method == "tools/call": + elif rpc_method == "tools/call": tool_name = json.get("params", {}).get("name") content = [{"type": "text", "text": "Tool Output"}] - + if tool_name == "query_inventory": content = [{"type": "text", "text": '{"warehouse_id": "WH001", "level": 100}'}] - + response_mock.json.return_value = { "jsonrpc": "2.0", "id": json.get("id"), @@ -86,12 +87,11 @@ class TestCookbookIntegration: "id": json.get("id"), "result": {} } - + return response_mock - - mock_httpx_post.side_effect = side_effect - mock_requests_post.side_effect = side_effect - yield mock_httpx_post + + mock_request.side_effect = side_effect + yield mock_request def test_financial_data_integration(self, mock_mcp_server): """ diff --git a/tests/ingest/test_public_api_ingestor.py b/tests/ingest/test_public_api_ingestor.py index 61119427..920a99c8 100644 --- a/tests/ingest/test_public_api_ingestor.py +++ b/tests/ingest/test_public_api_ingestor.py @@ -198,15 +198,82 @@ def test_public_api_detection_reports_auth_required() -> None: headers={"WWW-Authenticate": "Bearer"}, ) - detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api( - "https://api.example.com/private" - ) + with patch( + "semantica.ingest.ssrf.socket.getaddrinfo", + return_value=[(None, None, None, None, ("93.184.216.34", 0))], + ): + detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api( + "https://api.example.com/private" + ) assert detection.is_public is False assert detection.requires_auth is True assert detection.response_status == 401 +def test_detect_public_api_propagates_ssrf_validation_error() -> None: + """detect_public_api() must surface ValidationError, not swallow it. + + request_with_ssrf_guard() raises ValidationError (not + requests.exceptions.RequestException) for SSRF-blocked hosts, so + detect_public_api()'s error handling must catch it explicitly like its + sibling ingest_public_api() already does. + """ + with patch("requests.Session") as mock_session_class: + mock_session = mock_session_class.return_value + mock_session.headers = {} + + with patch( + "semantica.ingest.ssrf.socket.getaddrinfo", + return_value=[(None, None, None, None, ("127.0.0.1", 0))], + ): + with pytest.raises(ValidationError): + PublicAPIIngestor(rate_limit_delay=0).detect_public_api( + "https://blocked.example.com/data" + ) + + +def test_detect_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None: + """Passing session/allow_private_ips through **options must not crash. + + Both are always supplied explicitly to request_with_ssrf_guard(); caller + copies must be dropped from **options rather than causing a + 'got multiple values for keyword argument' TypeError. + """ + with patch("requests.Session") as mock_session_class: + mock_session = mock_session_class.return_value + mock_session.headers = {} + mock_session.request.return_value = _mock_response( + headers={"Content-Type": "application/json"} + ) + + detection = PublicAPIIngestor(rate_limit_delay=0).detect_public_api( + "https://jsonplaceholder.typicode.com/posts", + allow_private_ips=True, + session=object(), + ) + + assert detection.is_public is True + + +def test_ingest_public_api_rejects_duplicate_session_and_allow_private_ips_kwargs() -> None: + with patch("requests.Session") as mock_session_class: + mock_session = mock_session_class.return_value + mock_session.headers = {} + mock_session.request.return_value = _mock_response( + json_payload=[{"id": 1}], + headers={"Content-Type": "application/json"}, + ) + + result = PublicAPIIngestor(rate_limit_delay=0).ingest_public_api( + "https://jsonplaceholder.typicode.com/posts", + allow_private_ips=True, + session=object(), + ) + + assert result.response_status == 200 + + def test_public_api_ingestor_rejects_authentication_inputs() -> None: with patch("requests.Session") as mock_session_class: mock_session = mock_session_class.return_value @@ -238,10 +305,14 @@ def test_public_api_ingestor_parses_string_boolean_config() -> None: config={"validate_no_auth": "false"}, rate_limit_delay=0, ) - result = ingestor.ingest_public_api( - "https://api.example.com/data", - headers={"Authorization": "Bearer token"}, - ) + with patch( + "semantica.ingest.ssrf.socket.getaddrinfo", + return_value=[(None, None, None, None, ("93.184.216.34", 0))], + ): + result = ingestor.ingest_public_api( + "https://api.example.com/data", + headers={"Authorization": "Bearer token"}, + ) assert ingestor.validate_no_auth is False assert result.data == payload diff --git a/tests/ingest/test_submodules.py b/tests/ingest/test_submodules.py index e17cd330..cac9c092 100644 --- a/tests/ingest/test_submodules.py +++ b/tests/ingest/test_submodules.py @@ -195,89 +195,62 @@ class TestMCPIngestor: class TestMCPClient: def test_call_tool(self): - # Patch requests.post globally if requests is used, or httpx.post if httpx is used. - # The code tries importing httpx, then requests. - # We should patch both or ensure we catch the right one. - # Simpler to patch sys.modules to simulate httpx missing, then patch requests. - - with patch.dict(sys.modules, {'httpx': None}): - with patch("requests.post") as mock_post: - mock_response = MagicMock() - mock_response.status_code = 200 - - # Sequence of calls: - # 1. connect() calls _connect_http() -> calls _initialize() -> calls _send_request() - # _send_request() calls requests.post with method="initialize" - # 2. call_tool() calls _send_request() with method="tools/call" - - # Response for initialize - init_response = { - "jsonrpc": "2.0", - "result": {"serverInfo": {"name": "test", "version": "1.0"}}, - "id": 1 - } - - # Response for tool call - tool_response = { - "jsonrpc": "2.0", - "result": {"content": [{"type": "text", "text": "Tool Result"}]}, - "id": 2 - } - - mock_response.json.side_effect = [init_response, tool_response] - mock_post.return_value = mock_response - - client = MCPClient(url="http://localhost:8000") - client.connect() - - result = client.call_tool("my_tool", {"arg": "val"}) - - # result is the dict returned by tool call? - # call_tool returns dict? - # Check MCPClient.call_tool implementation - # It calls _send_request, which returns response.json(). - # But wait, call_tool might process the result. - # Let's check call_tool implementation in mcp_client.py (not read yet, but assumed). - # Wait, I read mcp_client.py but didn't check call_tool specifically. - # Assuming call_tool returns result part or whole response. - - # Actually, let's verify call_tool in mcp_client.py - pass + # MCPClient._send_request_http now routes through request_with_ssrf_guard, + # which calls requests.request (not requests.post) with allow_redirects=False. + # Patch the requests.request call inside ssrf.py. + with patch("semantica.ingest.ssrf.requests.request") as mock_request: + mock_response = MagicMock() + mock_response.status_code = 200 + + # Response for initialize + init_response = { + "jsonrpc": "2.0", + "result": {"serverInfo": {"name": "test", "version": "1.0"}}, + "id": 1, + } + + # Response for tool call + tool_response = { + "jsonrpc": "2.0", + "result": {"content": [{"type": "text", "text": "Tool Result"}]}, + "id": 2, + } + + mock_response.json.side_effect = [init_response, tool_response] + mock_request.return_value = mock_response + + client = MCPClient(url="http://localhost:8000") + client.connect() + + client.call_tool("my_tool", {"arg": "val"}) def test_call_tool_mock_check(self): - # Redoing the test with more specific mocking logic - with patch.dict(sys.modules, {'httpx': None}): - with patch("requests.post") as mock_post: - mock_response = MagicMock() - mock_response.status_code = 200 - - # initialize response - init_response = { - "jsonrpc": "2.0", - "result": {"serverInfo": {"name": "test", "version": "1.0"}}, - "id": 1 - } - - # tool call response - Assuming call_tool returns the 'result' part of JSON-RPC response - # If call_tool implementation wraps it, we need to know. - # Let's assume standard behavior for now. - tool_response = { - "jsonrpc": "2.0", - "result": {"content": [{"type": "text", "text": "Tool Result"}]}, - "id": 2 - } - - mock_response.json.side_effect = [init_response, tool_response] - mock_post.return_value = mock_response - - client = MCPClient(url="http://localhost:8000") - client.connect() - - result = client.call_tool("my_tool", {"arg": "val"}) - - # Verify result. - # If call_tool returns the 'result' dict from JSON-RPC: - assert result["content"] == [{"type": "text", "text": "Tool Result"}] + # Redo with the corrected patch target. + with patch("semantica.ingest.ssrf.requests.request") as mock_request: + mock_response = MagicMock() + mock_response.status_code = 200 + + init_response = { + "jsonrpc": "2.0", + "result": {"serverInfo": {"name": "test", "version": "1.0"}}, + "id": 1, + } + + tool_response = { + "jsonrpc": "2.0", + "result": {"content": [{"type": "text", "text": "Tool Result"}]}, + "id": 2, + } + + mock_response.json.side_effect = [init_response, tool_response] + mock_request.return_value = mock_response + + client = MCPClient(url="http://localhost:8000") + client.connect() + + result = client.call_tool("my_tool", {"arg": "val"}) + + assert result["content"] == [{"type": "text", "text": "Tool Result"}] class TestGDriveIngestor: def test_init_raises_if_no_google_libs(self): diff --git a/tests/test_seed_manager.py b/tests/test_seed_manager.py index c66149cc..95d490a1 100644 --- a/tests/test_seed_manager.py +++ b/tests/test_seed_manager.py @@ -209,6 +209,60 @@ def test_load_from_api_allows_private_when_configured(mock_guard, seed_manager): call_kwargs = mock_guard.call_args[1] assert call_kwargs["allow_private_ips"] is True + +@patch("semantica.seed.seed_manager.request_with_ssrf_guard") +def test_load_from_api_does_not_mutate_caller_headers_dict(mock_guard, seed_manager): + """Regression test for issue #947 audit: load_from_api must not mutate the + caller's headers dict in-place when api_key is provided. + + Before the fix, ``request_headers = headers or {}`` aliased the caller's dict. + Writing ``request_headers["Authorization"] = ...`` then silently modified the + caller's original dict, potentially leaking credentials to subsequent calls + that reused the same headers dict without expecting it to carry Authorization. + """ + mock_response = MagicMock() + mock_response.json.return_value = {"results": []} + mock_guard.return_value = mock_response + + # Caller owns this dict and expects it to be unchanged after the call. + original_headers = {"X-Custom-Header": "value"} + headers_before = dict(original_headers) # snapshot + + seed_manager.load_from_api( + api_url="http://api.example.com", + api_key="secret-key", + headers=original_headers, + ) + + # The caller's dict must be unchanged — Authorization must NOT have been added. + assert original_headers == headers_before, ( + "load_from_api must not mutate the caller's headers dict; " + f"expected {headers_before!r}, got {original_headers!r}" + ) + + # The guard must still have received Authorization (in its own copy). + call_kwargs = mock_guard.call_args[1] + guard_headers = call_kwargs.get("headers", {}) + assert guard_headers.get("Authorization") == "Bearer secret-key" + + +@patch("semantica.seed.seed_manager.request_with_ssrf_guard") +def test_load_from_api_does_not_mutate_empty_headers_dict(mock_guard, seed_manager): + """When headers=None, a fresh dict is created — no aliasing to a shared mutable default.""" + mock_response = MagicMock() + mock_response.json.return_value = {"results": []} + mock_guard.return_value = mock_response + + seed_manager.load_from_api( + api_url="http://api.example.com", + api_key="key", + headers=None, + ) + + call_kwargs = mock_guard.call_args[1] + guard_headers = call_kwargs.get("headers", {}) + assert guard_headers.get("Authorization") == "Bearer key" + def test_load_source(seed_manager, temp_data_dir): json_file = temp_data_dir / "source.json" with open(json_file, "w") as f: From baf8f01f85e0a5b1678a19dabe19216d66188368 Mon Sep 17 00:00:00 2001 From: Shubham Srivastava Date: Mon, 17 Aug 2026 22:18:39 +0100 Subject: [PATCH 18/19] test(export): guard Parquet tests on pyarrow itself, not the exporter import (#1056) * test(export): guard Parquet tests on pyarrow itself, not the exporter import Closes #1054 * test(export): guard on PARQUET_AVAILABLE so the skip matches the runtime check find_spec only proves pyarrow is discoverable, not importable. Addresses review feedback on #1056. --------- --- ...st_030_context_graph_realworld_extended.py | 56 +++++++++---------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/tests/test_030_context_graph_realworld_extended.py b/tests/test_030_context_graph_realworld_extended.py index 28d0f8f0..28ad1506 100644 --- a/tests/test_030_context_graph_realworld_extended.py +++ b/tests/test_030_context_graph_realworld_extended.py @@ -54,6 +54,11 @@ from semantica.context.decision_models import ( validate_decision, ) +# ── Export module ────────────────────────────────────────────────────────────── +# Set by the exporter's own `import pyarrow` attempt; False when pyarrow is +# missing or unimportable. +from semantica.export.parquet_exporter import PARQUET_AVAILABLE + # ── KG module ────────────────────────────────────────────────────────────────── from semantica.kg import ( CentralityCalculator, @@ -981,6 +986,17 @@ class TestParquetExportRealData: Requires: pyarrow (optional dep — tests skip if not installed). """ + # ParquetExporter imports fine without pyarrow and only raises ImportError + # when an export actually runs, so guarding on that import never skips + # anything. Guard on the exporter's own availability flag instead: it is set + # by the same `import pyarrow` / `import pyarrow.parquet` the exporter gates + # on, so the skip condition cannot drift from the runtime check — including + # when pyarrow is present on the path but fails to import. + pytestmark = pytest.mark.skipif( + not PARQUET_AVAILABLE, + reason="pyarrow not installed", + ) + @pytest.fixture def kg_data(self): return { @@ -1000,16 +1016,12 @@ class TestParquetExportRealData: } def test_parquet_exporter_importable(self): - try: - from semantica.export import ParquetExporter - except ImportError as e: - pytest.skip(f"ParquetExporter not available: {e}") + from semantica.export import ParquetExporter + + assert ParquetExporter is not None def test_parquet_export_entities_to_file(self, kg_data, tmp_path): - try: - from semantica.export import ParquetExporter - except ImportError: - pytest.skip("pyarrow not installed") + from semantica.export import ParquetExporter exporter = ParquetExporter(compression="snappy") out_path = tmp_path / "github_entities.parquet" @@ -1018,10 +1030,7 @@ class TestParquetExportRealData: assert out_path.stat().st_size > 0 def test_parquet_export_relationships_to_file(self, kg_data, tmp_path): - try: - from semantica.export import ParquetExporter - except ImportError: - pytest.skip("pyarrow not installed") + from semantica.export import ParquetExporter exporter = ParquetExporter(compression="gzip") out_path = tmp_path / "github_relationships.parquet" @@ -1030,10 +1039,7 @@ class TestParquetExportRealData: assert out_path.stat().st_size > 0 def test_parquet_export_knowledge_graph(self, kg_data, tmp_path): - try: - from semantica.export import ParquetExporter - except ImportError: - pytest.skip("pyarrow not installed") + from semantica.export import ParquetExporter exporter = ParquetExporter(compression="snappy") base_path = tmp_path / "github_kg" @@ -1043,30 +1049,24 @@ class TestParquetExportRealData: assert len(files) >= 1 def test_parquet_export_snappy_compression(self, kg_data, tmp_path): - try: - from semantica.export import ParquetExporter - except ImportError: - pytest.skip("pyarrow not installed") + from semantica.export import ParquetExporter + exporter = ParquetExporter(compression="snappy") out_path = tmp_path / "snappy_test.parquet" exporter.export_entities(kg_data["entities"], str(out_path)) assert out_path.exists() def test_parquet_export_none_compression(self, kg_data, tmp_path): - try: - from semantica.export import ParquetExporter - except ImportError: - pytest.skip("pyarrow not installed") + from semantica.export import ParquetExporter + exporter = ParquetExporter(compression="none") out_path = tmp_path / "uncompressed_test.parquet" exporter.export_entities(kg_data["entities"], str(out_path)) assert out_path.exists() def test_parquet_convenience_function(self, kg_data, tmp_path): - try: - from semantica.export.methods import export_parquet - except ImportError: - pytest.skip("pyarrow not installed") + from semantica.export.methods import export_parquet + out_path = tmp_path / "convenience_test.parquet" export_parquet(kg_data["entities"], str(out_path)) assert out_path.exists() From 5c2901ae27004a799e18cd3d6dfcdb9edcf524da Mon Sep 17 00:00:00 2001 From: pravit-amp <43916793+pravit-amp@users.noreply.github.com> Date: Mon, 17 Aug 2026 23:19:32 -0700 Subject: [PATCH 19/19] docs(context): fix unrunnable ContextGraph docstring example (#921) * docs(context): fix unrunnable ContextGraph docstring example The module docstring's Example Usage block called add_node/add_edge with keyword arguments they do not accept. add_node(node_id, node_type, ...) takes node_type positionally and has no properties parameter, so the documented call raised TypeError; add_edge's parameter is edge_type, so type= fell through to **properties and polluted edge metadata while appearing to work. Two of the three broken forms failed silently rather than raising, storing a nested properties dict or a stray type key instead of erroring. Add regression tests that execute the documented calls and assert the docstring itself does not reintroduce the invalid kwargs. Co-Authored-By: Claude Opus 5 * test(context): close two blind spots in the docstring regression guards The guards added in the previous commit could pass while checking nothing. _example_block() terminated the capture at the first "\n\n". The Example Usage block already contains ">>> " spacer lines, so any reformatting that turned one into a bare blank line would truncate the capture -- potentially to empty -- and the guards would then scan a block that no longer held the add_node/add_edge calls they exist to police. Both guards also iterated over re.findall() without asserting a match. Zero matches meant zero assertions and a green test, so the two failure modes compounded: a truncated block produced no matches, and no matches produced a pass. Terminate the block at the next top-level section header (^\S) or end of docstring instead, so blank lines inside the example are harmless, and assert the captured block, the parsed statement list, and each guard's match list are all non-empty. Extract statements with doctest.DocTestParser rather than a line regex. This also catches a call reformatted across "..." continuation lines, which the ">>> graph.add_node(.*" pattern silently skipped, and lets test_documented_calls_execute exec the docstring's own statements instead of a retyped copy that could drift from it. Full doctest.testmod isn't usable here: add_node/add_edge return True and the docs carry no expected-output lines, so it reports 4 spurious failures. Narrow the kwarg check to (? * fix(context): correct precedent lookup in docstring example --------- Co-authored-by: Pravit Ampapathini Co-authored-by: Claude Opus 5 Co-authored-by: Sameer Kadam Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> --- semantica/context/context_graph.py | 8 +- .../test_context_graph_docstring_example.py | 142 ++++++++++++++++++ 2 files changed, 146 insertions(+), 4 deletions(-) create mode 100644 tests/context/test_context_graph_docstring_example.py diff --git a/semantica/context/context_graph.py b/semantica/context/context_graph.py index 28b431ef..ad6ecf3f 100644 --- a/semantica/context/context_graph.py +++ b/semantica/context/context_graph.py @@ -72,9 +72,9 @@ Example Usage: ... node_embeddings=True) >>> >>> # Basic graph operations - >>> graph.add_node("Python", type="language", properties={"popularity": "high"}) - >>> graph.add_node("Programming", type="concept") - >>> graph.add_edge("Python", "Programming", type="related_to") + >>> graph.add_node("Python", "language", popularity="high") + >>> graph.add_node("Programming", "concept") + >>> graph.add_edge("Python", "Programming", "related_to") >>> centrality = graph.get_node_centrality("Python") >>> similar = graph.find_similar_nodes("Python", similarity_type="content") >>> analysis = graph.analyze_graph_with_kg() @@ -88,7 +88,7 @@ Example Usage: ... confidence=0.95, ... entities=["customer_123", "property_456"] ... ) - >>> precedents = graph.find_precedents("loan_approval", limit=5) + >>> precedents = graph.find_precedents(decision_id, limit=5) >>> influence = graph.analyze_decision_influence(decision_id) >>> insights = graph.get_decision_insights() >>> causality = graph.trace_decision_causality(decision_id) diff --git a/tests/context/test_context_graph_docstring_example.py b/tests/context/test_context_graph_docstring_example.py new file mode 100644 index 00000000..8dcd4039 --- /dev/null +++ b/tests/context/test_context_graph_docstring_example.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Regression tests for the ContextGraph module docstring example. + +The "Example Usage" block in ``semantica/context/context_graph.py`` previously +called ``add_node``/``add_edge`` with keyword arguments those methods do not +accept (``type=`` and ``properties=``), so the documented example raised +``TypeError`` -- and the near-miss variants silently nested the properties dict +instead of failing. + +These tests keep the documented example executable and pin the two behaviours +that made the original mistake easy to miss. +""" + +import doctest +import re +from typing import Dict, List + +import pytest + +import semantica.context.context_graph as context_graph_module +from semantica.context.context_graph import ContextGraph + +# The example block runs to the next top-level section header (a line starting +# in column 0, e.g. "Production Use Cases:") or the end of the docstring. +# Terminating on the next header rather than on a blank line keeps the capture +# intact when the example gains blank lines or extra paragraphs. +_EXAMPLE_BLOCK_RE = re.compile(r"^Example Usage:\n(.*?)(?=^\S|\Z)", re.DOTALL | re.MULTILINE) + +# ``type=`` as its own keyword, but not the legitimate ``node_type=``/``edge_type=``. +_BARE_TYPE_KWARG_RE = re.compile(r"(? str: + """Return the 'Example Usage' block from the module docstring.""" + doc = context_graph_module.__doc__ or "" + match = _EXAMPLE_BLOCK_RE.search(doc) + assert match, "module docstring no longer contains an 'Example Usage:' block" + block = match.group(1).strip() + assert block, "the 'Example Usage:' block in the module docstring is empty" + return block + + +def _example_statements() -> List[str]: + """Return the documented ``>>>`` statements, continuation lines included.""" + statements = [example.source for example in doctest.DocTestParser().get_examples(_example_block())] + assert statements, "the 'Example Usage:' block no longer contains any '>>>' statements" + return statements + + +def _statements_calling(method: str) -> List[str]: + """Return the documented statements that call ``graph.(``.""" + return [stmt for stmt in _example_statements() if "graph.{}(".format(method) in stmt] + + +def _run_example() -> Dict[str, object]: + """Execute the documented example verbatim and return its namespace.""" + source = "".join(_example_statements()) + namespace: Dict[str, object] = {} + exec(compile(source, "", "exec"), namespace) + return namespace + + +class TestDocstringExampleIsRunnable: + """The documented example must execute exactly as written.""" + + def test_documented_calls_execute(self): + # Run the docstring text itself so this test cannot drift from the docs. + ns = _run_example() + graph = ns["graph"] + + assert "Python" in graph.nodes + assert "Programming" in graph.nodes + assert graph.nodes["Python"].node_type == "language" + assert graph.nodes["Programming"].node_type == "concept" + + neighbors = graph.get_neighbors("Python", hops=1) + assert any(n["id"] == "Programming" for n in neighbors) + + # record_decision must return a non-empty string ID. + assert isinstance(ns["decision_id"], str) and ns["decision_id"] + # find_precedents must be called with that ID and return a list. + assert isinstance(ns["precedents"], list) + + def test_node_properties_are_stored_flat(self): + """``popularity`` must land as a top-level property, not nested. + + Passing the previously documented ``properties={...}`` does not raise -- + it stores a dict *inside* the properties dict, which is why the original + docs bug could reach a user's graph unnoticed. + """ + graph = ContextGraph(advanced_analytics=False) + graph.add_node("Python", "language", popularity="high") + + assert graph.nodes["Python"].properties == {"popularity": "high"} + assert graph.find_node("Python")["metadata"]["popularity"] == "high" + assert "properties" not in graph.nodes["Python"].properties + + def test_edge_type_is_positional_not_a_property(self): + """``related_to`` must be the edge type, not a stray metadata key.""" + graph = ContextGraph(advanced_analytics=False) + graph.add_node("Python", "language") + graph.add_node("Programming", "concept") + graph.add_edge("Python", "Programming", "related_to") + + edge = graph.edges[0] + assert edge.edge_type == "related_to" + assert "type" not in edge.metadata + + +class TestDocstringExampleDoesNotRegress: + """Guard the docstring text itself, not just equivalent code.""" + + def test_add_node_example_supplies_node_type_positionally(self): + calls = _statements_calling("add_node") + assert calls, "the 'Example Usage:' block no longer calls graph.add_node()" + for call in calls: + assert not _BARE_TYPE_KWARG_RE.search(call), ( + f"add_node example passes type= as a keyword: {call!r}. " + "node_type is positional-required; type= falls through to " + "**properties and the call raises TypeError." + ) + assert "properties=" not in call, ( + f"add_node example passes properties=: {call!r}. " + "add_node has no properties parameter; extra properties are " + "passed as **kwargs." + ) + + def test_add_edge_example_supplies_edge_type_positionally(self): + calls = _statements_calling("add_edge") + assert calls, "the 'Example Usage:' block no longer calls graph.add_edge()" + for call in calls: + assert not _BARE_TYPE_KWARG_RE.search(call), ( + f"add_edge example passes type= as a keyword: {call!r}. " + "The parameter is edge_type; type= is silently absorbed into " + "**properties and pollutes edge metadata." + ) + + def test_broken_form_still_raises(self): + """Pin the signature contract the example has to respect.""" + graph = ContextGraph(advanced_analytics=False) + with pytest.raises(TypeError, match="node_type"): + graph.add_node("Python", type="language", properties={"popularity": "high"})