Compare commits

..
Author SHA1 Message Date
KaifAhmad1 50927f99b5 docs: surface explainability scope note near the top of the README
Moves a concise version of the system-level vs. foundation-model
explainability clarification up next to the opening pitch, so it's
visible before readers scroll to the high-stakes-domains section.
2026-08-16 17:46:12 +05:30
KaifAhmad1 476237952d docs: clarify explainability is system-level, not foundation-model internal
Adds a consistent scope note to README and docs (concepts, FAQ, index)
stating Semantica does not expose or reconstruct an LLM's internal
reasoning/chain-of-thought. It explains and audits the AI system
around the model: context, provenance, policies, decisions, and
execution history.
2026-08-16 17:39:23 +05:30
hariandZohaib Hassnain 70aa9d01bf fix(normalize): validate symbol currencies (#940)
* fix(normalize): validate symbol currencies

Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com>

* fix(normalize): match currency codes by token boundaries

Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com>

---------

Signed-off-by: Mr-Neutr0n <harikp2002@gmail.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
2026-08-16 15:24:34 +05:30
Mohd Kaif c53ca4e84b docs: formalize issue assignment and duplicate-PR triage workflow (#1030)
* docs(contributing): formalize issue assignment and duplicate-PR triage workflow

Comments are no longer required before an issue can be assigned - maintainers
may assign directly based on recent activity. Also documents the duplicate-PR
priority order for triage (contributor PR, claimed issue, activity tiebreak,
late duplicates, overlapping scope).

* docs(contributing): clarify assignment precedence and define activity tiebreak

Addresses Qodo review feedback on PR #1030: the duplicate-PR priority list
now states these rules apply on top of the assignment workflow (opening a PR
pre-assignment doesn't grant priority), and the "most active" tiebreak now
specifies a concrete 60-day window and signals instead of being subjective.
2026-08-16 15:18:55 +05:30
pravit-ampandPravit Ampapathini 15171fd31a fix(parse): import get_progress_tracker in ExcelParser (#1016)
ExcelParser.__init__ called get_progress_tracker() without importing it,
so every instantiation raised NameError and the class was unusable. The
existing test imported ExcelParser but never constructed it, so nothing
caught it. Same defect as #530 in SimilarityCalculator, which was fixed
without sweeping the rest of the codebase.

Add construction coverage for every parser exported from semantica.parse,
driven off __all__ so later additions are covered automatically. These
live outside test_parse_comprehensive.py, whose setUp patches
get_progress_tracker into each parse module and would mock away the
interaction under test.

Closes #1014

Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
2026-08-16 14:07:10 +05:00
Guofang.Tang 8177d88753 fix(kg): preserve isolated nodes in graph analytics (#1011)
* fix(kg): preserve isolated nodes in graph analytics

* fix(kg): support node fallbacks and community payloads

---------
2026-08-16 11:23:24 +05:00
Shinde vinayak rao patil d94d8f6ab8 Feat/crewai integration (#988)
* feat(crewai): add first-class CrewAI integration (#962)

Add native CrewAI support so Crew agents can share a ContextGraph and
AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching
the existing agno integration pattern.

- SemanticaKGTool: 5 KG actions (extract_entities, extract_relations,
  add_to_graph, query_graph, find_related) with sync run()/async arun()
- SemanticaDecisionTool: 5 decision-intelligence actions
  (record_decision, find_precedents, trace_causal_chain,
  analyze_impact, check_policy) over AgentContext
- SemanticaKnowledgeSource: serializes a ContextGraph into crew
  knowledge storage; bridges legacy load_content() and current
  validate_content()/aadd() contracts for crewai>=0.80.0
- All classes degrade gracefully when crewai is absent
- New pip extra crewai=... included in the all bundle
- 70 new tests (stub-based present-case + subprocess degradation path)
- Docs: integrations/crewai.md, docs.json nav, README matrix updates

* fix(crewai): harden tools against real Semantica dataclass shapes (#962)

Bugs found during live testing with crewai 1.15.16:

- SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses
  ('str' object has no attribute 'end_char'): string names were passed to
  extract_relations(entities=...), which requires Entity objects, and the
  tool read .name/.source/.target instead of Entity's .text/.label and
  Relation's .subject/.object. Add shape-agnostic field helpers.
- SemanticaDecisionTool() created an AgentContext without a knowledge_graph,
  so _decision_backend was never set and record_decision raised 'Decision
  tracking is not enabled'. Wire in a ContextGraph.
- record_decision hard-failed when the agent omitted optional fields; fall
  back to category='general', reasoning='agent decision',
  outcome='recorded'.

Add tests covering real Entity/Relation dataclass shapes and the live
auto-created AgentContext path (now 77 crewai tests, 212 total).

* fix(crewai): make find_related traverse edges undirected (#962)

ContextGraph.get_neighbors only follows outgoing edges, so a node whose
only edge is incoming (A -> B) reported no related concepts. Rebuild a
bidirectional adjacency from find_edges() in SemanticaKGTool._find_related
so 'related' honors both directions.

* fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962)

- Exclude live graph/context/extractor state from JSON serialization
  (model_dump(mode="json")) so CrewAI checkpointing no longer raises
  PydanticSerializationError; model_post_init self-heals defaults on restore
- query_graph now searches node content via graph.query() plus id/type
- trace_causal_chain returns an explicit error when causal tracing is
  unavailable instead of substituting similarity precedents; call
  trace_decision_causality(..., max_depth=...) with the correct kwarg name
- find_precedents propagates max_precedents/limit to the backend instead of
  being silently capped at 10
- Serialize add_to_graph batches under a module lock to prevent concurrent
  double-counting; skip nameless entities instead of creating repr()-junk nodes
- aadd() runs CPU-bound serialization in a thread executor
- Mirror crewai args_schema serialize/restore in the conftest stub and add
  serialization regression tests (crewai: 92 tests)

* fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962)

- _eval_rule now coerces rule values type-aware: bool("false") was truthy, so
  'enabled == false' reported a violation for enabled=false, and string datums
  like "0.90" were compared lexicographically instead of numerically
- _trace_causal_chain no longer raises AttributeError (which escaped _run) when
  the decision context lacks knowledge_graph; returns honest error JSON
- SemanticaKnowledgeSource storage failures log an actionable ERROR; without a
  configured crew embedder agents previously retrieved nothing silently
- add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a
  process-global one: independent graphs no longer serialize each other and
  re-entrant extractor callbacks cannot deadlock
- entity/relation confidence=None normalizes to 1.0 instead of failing the
  whole extraction with float(None)
- add subprocess integration test against real crewai covering Crew-level
  serialization round-trip and checkpoint restore (stub tests cannot see it)
- docs: embedder requirement for SemanticaKnowledgeSource; resume contract note

* fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962)

Re-verification against real crewai showed the embedder-missing failure raises
ValueError even though storage IS wired, so the old except-ValueError branch
mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure.
Distinguish by storage presence instead of exception type: storage is None ->
DEBUG keep-in-memory (legitimate standalone use); storage wired but save()
raises -> actionable ERROR. Add regression test mirroring real crewai's
ValueError-on-missing-embedder behavior.

* fix(crewai): expose run()/arun() entry points in degraded mode (#962)

The public crewai contract is run()/arun(); without crewai installed they were
missing (only the private _run existed), so the documented 'usable without
crewai' path raised AttributeError at the entry point. Define them in degraded
mode only, leaving crewai's BaseTool implementations untouched when present.
Extend the degradation subprocess test to exercise run() and arun().

* fix(crewai): standardize query shape, field-name rules, and restore-state flag

- _query_graph: id/type matches now return the same schema as content
  matches (id/type/label/content/score) instead of a bare list
- _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys
  (e.g. "risk-score >= 0.9") are addressable in policy rules
- add had_live_state/reconstructed_state so checkpoint-restored tools
  and knowledge sources signal that their live graph/context was lost
  and an empty one reconstructed; knowledge source no longer hides the
  loss by eagerly rebuilding its graph inside __init__ (pydantic calls
  __init__ during model_validate)

* fix(crewai): address Qodo review — confidence errors, string trim, holistic availability

- record_decision: stop calling float() in _run, so malformed confidence
  values surface as JSON errors (via _record_decision's handling) instead
  of crashing the tool
- _coerce_value: return the stripped string for non-numeric literals so
  whitespace-padded decision_data fields match policy rules
- centralize crewai availability in _availability.py so the exported
  CREWAI_AVAILABLE flag is holistic across tools and knowledge source
  (previously each module probed crewai independently and the package
  flag came from decision_tool only)

* ci: regenerate requirements-ci.txt for the crewai extra

The crewai extra in pyproject.toml brings in crewai, crewai-tools and
transitive deps (chromadb, lancedb, ...). Recompile with
uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes.

* ci: keep crewai out of the locked CI dependency set

crewai (all versions) hard-requires chromadb~=1.1.0, which carries a
pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c)
with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in
the 'all' extra failed pip-audit and the safety check on requirements-ci.txt.

- drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is
  unchanged and still installs crewai)
- stop listing crewai-tools in the extra: the integration only uses crewai core
  (BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps
- regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0
  vulnerabilities, staleness check matches

* docs(crewai): document crewai extra scope and chromadb CVE-2026-45829

- CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not
  part of the 'all' bundle, with the chromadb CVE-2026-45829 reason
- integrations/crewai/README.md: add a security warning that installing
  the extra pulls chromadb~=1.1.0, which is affected by the unpatched
  pre-auth code-injection CVE-2026-45829

---------
2026-08-16 11:15:43 +05:00
5579851208 fix(export): harden YAML export input handling (#958)
* refactor(export): centralize graph-payload key normalization

Graph payloads circulate under two vocabularies, entities/relationships and nodes/edges, and consumers each reconciled them locally with competing idioms. The same payload could be exported, silently dropped, or rejected depending on which consumer read it.

Add normalize_graph_payload() to utils.helpers as the single place that decision is made. Both spellings present with one empty resolves to the populated one, which is the shape JSONExporter emits; both non-empty and different is refused, since there is no basis to prefer either and picking one would silently discard the other; a non-empty mapping with no recognized key raises rather than returning empty collections, with require_recognized=False for callers that should degrade.

Adopt it in the three exporters that genuinely alias. LPGExporter read nodes with entities as the default, so it dropped every entity when nodes was present but empty, losing everything on a JSON round-trip. ArangoAQLExporter had the same idiom plus a manual fallback. Neo4jCSVExporter routes its mapping branch through the shared resolver so the reference implementation cannot drift; its attribute branch stays local, since objects are not mappings.

Also feed LPGExporter._generate_indexes the resolved entities. It read entities directly, so a nodes/edges payload produced no indexes even once node generation was fixed.

CSVExporter and JSONExporter are deliberately excluded: they write entities, relationships, nodes and edges as separate outputs by design rather than reconciling two spellings of one collection, so normalizing there would rename output files.

* fix(export): reject non-mapping input to the YAML exporters

export_yaml declared Union[Dict[str, Any], List[Dict[str, Any]]], but both
YAML exporters read their payload by key, so a list reached .get() and
surfaced as a bare AttributeError from inside the exporter, naming neither
the offending argument nor the shape expected.

Reject rather than wrap. These formats distinguish entities from
relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records, and wrapping it under an
unrecognised key would write a structurally valid file with every
collection empty - trading a loud failure for silent data loss.

Validate in the exporters, matching the existing precedent in
Neo4jCSVExporter._normalize_graph, so direct users of the classes get the
same contract as callers of the convenience wrapper. Narrow the wrapper
type hint to Dict[str, Any] to match.

* fix(export): address YAML exporter review findings

- semantica/export/yaml_exporter.py — import Sequence from typing
  instead of collections.abc. `Sequence[str]` in _require_mapping's
  annotation is evaluated at function-definition time; collections.abc.Sequence
  only became subscriptable in Python 3.9, so on the 3.8 this project
  declares support for, importing this module raised TypeError.
  typing.Sequence has supported subscripting since 3.5.3. Mapping stays
  imported from collections.abc since it's only used for isinstance.
- tests/export/test_yaml_exporter_input_validation.py — clean up each
  test's tempfile.mkdtemp() dir via addCleanup instead of leaking it,
  and read exported YAML through a context manager instead of an
  unclosed yaml.safe_load(open(...)).

* fix(export): reject YAML export payloads with no recognized key

Both YAML exporters built their output from a fixed set of `.get(key, [])`
lookups, so a mapping keyed by anything else serialized to a structurally
valid file with every collection empty. Nothing signalled the loss: no
exception, no warning, and the progress log reported a completed export.
The only way to notice was to open the file. The realistic trigger is
re-exporting an `export_json` payload, whose `{"data", "count", "metadata"}`
envelope drops every record.

- SemanticNetworkYAMLExporter.export_semantic_network now resolves its
  collections through normalize_graph_payload(), which raises rather than
  returning empty collections for an unrecognized mapping. Adopting the
  shared resolver rather than repeating the check locally also brings the
  'nodes'/'edges' aliases, so ContextGraph.to_dict() — the most direct path
  from this library's own graph type to YAML, used in
  examples/capability_gap_context_graphs_example.py — exports its records
  instead of an empty file.
- export_for_pipeline built its nested semantic network from the same
  defaulted lookups and had the same defect; it goes through the resolver
  too.
- YAMLSchemaExporter.export_ontology_schema gets the equivalent check over
  its own key set. Schemas are a separate vocabulary with no aliasing, so
  _require_recognized_keys lives in this module rather than in the shared
  graph resolver.
- 'metadata' is deliberately not sufficient to make a payload recognized.
  An export_json envelope carries one, so accepting it would readmit the
  case this fix is most likely to be needed for.
- An empty mapping is still exported: an empty graph is legitimate and has
  no records to lose.
- SemanticNetworkYAMLExporter.export() serializes before creating the
  output directory, so a rejected export leaves nothing behind.

The two rejections keep distinct exception types, following what the
codebase already does: a payload of the wrong *type* cannot be exported at
all and raises ProcessingError, matching Neo4jCSVExporter._normalize_graph;
a mapping whose *contents* are unusable raises ValidationError, matching
normalize_graph_payload. _require_mapping therefore runs first at every
entry point, so a non-mapping never reaches the resolver.

Docstring Raises sections, export_usage.md and docs/reference/export.md
record the accepted input shapes and both failures.

Closes #953.

* fix(export): reject payloads whose records resolve to nothing

Addresses the Qodo findings on #958.

Presence-only recognition (finding 1): checking that a recognized key is
present answered "did the caller use our vocabulary" when the question that
matters is "did anything the caller supplied survive". A payload like
{"entities": [], "data": [...records...]} cleared the check, resolved to
empty, and dropped every record under 'data' -- the silent-empty export by a
narrower route.

- utils/helpers.py — split the check in two. _require_recognized_keys keeps
  the presence rule; _require_nothing_dropped runs after resolution and
  refuses a payload that resolved to nothing while an unread key still holds
  records. Only a non-empty list counts as evidence: ContextGraph.to_dict()
  always carries a populated 'statistics' dict, and an empty graph must stay
  exportable, so 'metadata', 'statistics' and 'count' are named as context
  rather than records.
- export/yaml_exporter.py — the schema path had the same hole and now runs
  both checks through the shared helpers rather than its own copy, so the
  two vocabularies cannot drift apart in what counts as a silent-empty
  export.

Progress reported success on a failed write (finding 3): export_semantic_
network stops its tracking as completed once serialization returns, but
export() then creates the directory and writes the file. A failure there
left the tracker showing a completed export with no output.

- export/yaml_exporter.py — the serialization span now says it serialized,
  not that it exported, and export() opens its own span around the
  filesystem work that stops as failed on error. Nothing reports a completed
  export until the bytes are on disk.

Finding 2 (export_yaml no longer accepts List[Dict]) is the intended
resolution of #952 rather than a regression: wrapping a bare list under a
guessed key is what would mislabel the records. The signature, docstring and
PR description already record the narrowed contract.

Tests cover both directions of each fix, including that an empty
ContextGraph still exports and that a failing write is not reported as
completed.

* fix(export): validate collection values and make Neo4j mappings strict

Two gaps at the boundary the shared normalizer is supposed to own.

_resolve_collection() resolved on truthiness alone, so a recognized key
could still hold something that is not a collection of records:
{"entities": "abc"} normalized to three single-character "records", and
{"entities": 42} surfaced as a raw TypeError from list() inside whichever
exporter happened to read it, naming the exporter rather than the payload
key at fault. Collection values are now validated before conversion --
strings, bytes, mappings, and non-iterable scalars are rejected by key
name, and each element must be a mapping or an attribute-carrying object,
the two record shapes the exporters actually read. None stays legal as an
absent collection, the spelling a JSON round-trip produces for []; it
cannot hide dropped records, since _require_nothing_dropped() still runs.
Every spelling present is validated, not just the one that wins, so a
malformed alias is not excused by a well-formed canonical key.

Neo4jCSVExporter._normalize_graph() opted out of the recognized-key check
for mappings, which left it able to turn {"data": [...]} into header-only
CSVs indistinguishable from a genuinely empty graph -- the exact failure
the rest of the change exists to prevent. Mapping payloads now go through
normalize_graph_payload() on its default terms. The attribute path for
graph objects is untouched. With no caller left opting out, the
require_recognized flag is removed rather than kept as a way back into
the silent-empty export.

Regression tests cover the malformed values end to end through every
export path that reads the normalizer, and assert the rejected Neo4j
export writes no CSV files.

* fix(export): close YAML schema and record validation gaps

Fix 1 -- _require_usable_schema silent data loss (P1):
_require_usable_schema() passed all values from _SCHEMA_KEYS into
_require_nothing_dropped() as evidence that records survived.  Scalar
metadata fields such as version='1.0' and uri='http://...' are truthy
strings, so any one of them caused _require_nothing_dropped() to return
early and silently discard records stored under an unread key alongside
them (e.g. {'version': '1.0', 'nodes': [{'id': 'c1'}]}).  Fixed by
building the resolved list from only non-empty list/tuple values of
recognised schema keys.

Fix 2 -- _is_record accepts modules and type objects (P2):
_is_record() accepted any object with __dict__, which includes Python
modules and class objects.  Elements that passed _coerce_records then
reached exporters and raised AttributeError (e.g. module 'math' has no
attribute 'get') rather than a ValidationError at the validation
boundary.  Fixed by excluding types.ModuleType and type from the
__dict__ branch while preserving support for all user-defined
attribute-bearing record objects.

Tests: 101 tests pass across
  tests/utils/test_normalize_graph_payload.py
  tests/export/test_yaml_exporter_key_recognition.py
  tests/export/test_yaml_exporter_input_validation.py
  tests/export/test_neo4j_csv_exporter.py

* fix(export): close exception-type and record-shape gaps in normalize_graph_payload

LPGExporter and ArangoAQLExporter called normalize_graph_payload() with no
type guard, so non-mapping input raised ValidationError from inside the
resolver while the YAML and Neo4j exporters raised ProcessingError for the
identical mistake -- inconsistent with the exception-type contract this PR
establishes. Both now use the shared _require_mapping() guard (moved from
yaml_exporter.py into utils/helpers.py so all three can use it).

Neo4jCSVExporter._normalize_graph checked isinstance(graph, dict), so a
non-dict Mapping (MappingProxyType, ChainMap) fell through to the
object-attribute branch and was rejected, even though the identical payload
exported fine via the other three exporters. Now checks isinstance(graph,
Mapping).

normalize_graph_payload() accepts dataclass/attribute-bearing object
records, but LPGExporter/ArangoAQLExporter call .get(...) directly on
resolved entities -- an object-shaped record passed validation only to
crash with a raw AttributeError once used, the exact failure this
boundary exists to prevent. Records are now converted to plain dicts at
the boundary (_coerce_records -> new _record_to_dict), so every consumer
gets a uniform shape regardless of which reading the caller used.

Two non-empty spellings of the same collection holding identical records
in a different order were rejected as conflicting, since the check used
plain list equality. Comparison is now an order-independent multiset of
each record's canonical JSON form.

* docs(changelog): add entry for #958 YAML export input hardening

Documents the full arc of #958 -- the normalize_graph_payload()
centralization, YAML input validation, both review rounds from
@Sameer6305, and the exception-type/record-shape follow-up fixes -- plus
closes #956, #952, #953.

---------

Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-08-15 22:10:04 +05:30
Lakshay Saini 115e7965cd fix(explorer): gate temporal requests on graph load (#1003)
Explorer was firing temporal requests before the graph even loaded.

When the backend is down, /api/graph/nodes fails but the temporal
bounds and snapshot effects didn't care , they fired anyway, off in
their own corner, ignoring whether the graph actually came up. Every
page load with no backend meant three failed requests instead of one,
and a scrubber that had nothing to scrub.

Added two small predicate functions and gated the temporal effects on
them. Basically: don't ask for time-based data until you know the
graph itself loaded. An empty graph still counts as loaded, so that
case isn't broken.

Confirmed with the backend down, before and after: three failing
requests down to one.

Fixes #982.
2026-08-15 17:47:35 +05:00
yzxcj797 8639cb9f16 fix(seed): pass connection string to DBIngestor and stop mislabeling OSError in load_from_database (#995)
Fix DBIngestor calls in load_from_database , it was never actually reaching the db.

execute_query/export_table need the connection string as their first arg,
but we were only passing it to the constructor's config dict, which
those methods don't read. Every call blew up with a TypeError before
connecting.

Also split the ImportError/OSError handling , they were caught together
so a real connection failure got reported as "module not available",
which sent people looking in the wrong place. OSError now surfaces as
an actual failure with the original exception chained via `from e`.

Fixes #973.
2026-08-15 17:18:36 +05:00
f1e7e64ad1 feat(context): add retraction and purge to ContextGraph (#957)
* feat(context): add retraction and purge to ContextGraph

ContextGraph had 56 public methods and none that removed anything: the only
option was clear(), which discards the whole graph. Removing one entity meant
exporting to a dict, filtering by hand and rebuilding, losing provenance.

Add two operations with deliberately different contracts.

retract_node/retract_edge close the entity's validity window. The entity stops
being active going forward, but state_at() before the retraction still returns
it, so decisions recorded against it remain explainable. This reuses the
valid_from/valid_until machinery already present rather than adding a new
subsystem.

purge_node/purge_edge remove the entity outright, from history as well as from
the active view, leaving a tombstone that records that a purge happened and why
but never the purged content. Scope is this graph only; copies in AgentMemory
or a bound vector store are not reached, so it is one step of an erasure
workflow rather than the whole of it.

Both record themselves through the existing mutation_callback path.
MutationRecord already documented REMOVE_NODE/REMOVE_EDGE in its operation
vocabulary, so retraction emits UPDATE_NODE and purge emits REMOVE_NODE with no
changes required to change_management.

Incident-edge lookup scans self.edges rather than _adjacency, which is keyed by
source only and would otherwise leave inbound edges pointing at a removed node.
Purge updates edges, edge_type_index and _adjacency together so the indexes
cannot drift, and clear() now resets the retraction and tombstone records.

* fix(context): address review findings on retraction and purge

* fix(context): close every duplicate when retracting/purging by edge_id

edge_id is content-derived and not yet guaranteed unique (#922, fix
pending in #926): two identical add_edge() calls produce two edge
objects sharing one id. retract_edge()/purge_edge() resolved "the
edge" via the first matching object only, so a duplicate was silently
left untouched (still live, still active) while the call returned
True and recorded a tombstone/retraction claiming it was fully
handled. Repeat purge_edge() calls also silently overwrote the
tombstone's reason/purged_at on each partial attempt instead of
no-op'ing once nothing remained to purge.

retract_node()'s cascade had the same root cause from the other
direction: it checked the live _retractions dict mid-loop, so the
first duplicate's just-written record made the second look already
handled and it was skipped outright, left permanently active.

retract_edge()/purge_edge() now act on every edge matching the id
under a single record; the cascade's dedup check is snapshotted
before the loop starts so within-call duplicates are still closed
rather than skipped.

Adds TestDuplicateEdgeId (5 tests) reproducing all three paths.

* docs(changelog): document retraction/purge feature

Adds an Unreleased/Added entry for #955/#957 covering retract_node,
retract_edge, purge_node, purge_edge and the get/list accessors, plus
the duplicate-edge_id fix caught and applied during review.

---------

Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-08-15 17:12:10 +05:30
pravit-amp 6df97cf0a0 fix(triplet_store): stop CONSTRUCT detection matching inside a leading comment (#951)
CONSTRUCT_QUERY_RE skipped comments with a bare \#[^\n]*, whose trailing *
backtracks. For '# CONSTRUCT ...\nSELECT ...' the engine gave back everything
after the '#', so the CONSTRUCT inside the comment satisfied the query-form
keyword and a SELECT/ASK was reported as a CONSTRUCT.

All four SPARQL backends delegate to this regex, so such a query took the
CONSTRUCT branch of execute_sparql, which sends Accept: text/turtle and parses
the body as Turtle — failing with a misleading 'Failed to parse CONSTRUCT
response as Turtle'.

Require a comment to reach a line terminator. Both LF and CR are accepted
because the SPARQL grammar ends a comment at either; matching only LF would
regress CR-terminated comments into false negatives.

Add regression tests covering both directions across all four backends.>
2026-08-15 16:20:49 +05:00
84ce3c5155 fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#926)
* fix(context): make ContextGraph.add_edge idempotent by deduping on edge_id (#922)

* docs(changelog): document add_edge dedupe fix

Adds an Unreleased/Fixed entry for #922/#926 so the ContextGraph
edge-dedupe bug and its fix are recorded per Keep a Changelog format.

---------

Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-08-15 16:09:33 +05:30
Guofang.Tang b8175ea801 fix(kg): make k-shortest path search side-effect free (#1000)
* fix(kg): make k-shortest path search side-effect free

* fix(kg): respect traversal direction for edge exclusion
2026-08-15 15:34:26 +05:00
557e29ee14 fix(explorer): repair /api/enrich/extract (always 503) and the /api/decisions routes (always 500) (#886)
* fix(explorer): repair /api/enrich/extract and the /api/decisions routes

Two Explorer API endpoints fail on every install.

/api/enrich/extract imported extract_entities and extract_relations from
semantic_extract.methods, where neither name is defined — that module ships
only the per-strategy variants (extract_entities_ml, extract_relations_regex,
...), and nothing re-exports a plain facade. The resulting ImportError was
caught and reported as "semantic_extract module not available. Ensure spacy
and transformers are installed.", so a wiring bug looked like a missing
dependency. The route now calls NamedEntityRecognizer and RelationExtractor
directly, the classes the README documents, and feeds the extracted entities
into relation extraction rather than re-deriving them. The 503 branch stays
for a genuinely absent module.

Every /api/decisions* route returned 500 once the graph held a decision:
record_decision() stores timestamp as datetime.now().timestamp(), a float,
while DecisionResponse types the field as str, so pydantic rejected the value
the library itself wrote. A before-mode field validator on DecisionResponse
normalizes float, int and datetime inputs to ISO-8601, covering every route
that builds the model instead of only the list endpoint.

The existing tests missed both: test_extract accepted 503 as a pass, and the
decision fixtures are hand-built nodes carrying no timestamp at all. Both are
tightened, and a TestRecordedDecisions class exercises the routes against
decisions created through record_decision().

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* perf(semantic_extract): cache spaCy models instead of loading one per call

extract_entities_ml(), extract_relations_similarity() and
extract_relations_dependency() called spacy.load() on every invocation, so the
model was re-read from disk and re-initialized per call. On a short sentence
that is ~120 ms of loading around ~2 ms of work, and successive calls never got
cheaper. The path is reachable from the CLI, the MCP extract_entities tool, the
pipeline ner_extract step and POST /api/enrich/extract, and process_batch()
multiplies it by the number of documents.

The module already had a cached loader for one code path — get_nlp_model() and
its _nlp_cache global — but the extraction functions bypassed it.

Adds load_spacy_model(), a process-level cache keyed by model name behind a
lock so concurrent callers do not each start a load, and routes the five call
sites through it. Errors are left uncached and propagate unchanged, so the
existing OSError fallbacks to pattern extraction still fire. get_nlp_model()
keeps its own entry: it loads with disable=["parser", "ner", "lemmatizer"] for
similarity work, so its model is not interchangeable with the NER one.

Cache entries record the spacy module object they came from. Several tests
patch methods.spacy with a mock and assert on load calls; without that guard a
name-keyed cache would hand a previous test's mock to a later one.

Measured on the same sentence, Python 3.12.13 / spacy 3.8.15 / en_core_web_sm:
extract_entities_ml() median 132 ms before, 2.1 ms after, identical entities.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(explorer): harden extraction and timestamp handling

* fix(explorer): catch OverflowError/OSError in decision timestamp validator

DecisionResponse._normalize_timestamp only guarded against NaN/inf via
math.isfinite(), but datetime.fromtimestamp() raises OverflowError or
OSError for finite epoch values outside the platform's representable
range (e.g. milliseconds stored where seconds were expected). Those
exceptions escaped the pydantic validator unhandled, reintroducing an
unhandled 500 on /api/decisions* for exactly the bug class this PR
closes. Also exclude bool from the numeric branch, since bool is an
int subclass and was being silently coerced to epoch 0/1.

* docs: add changelog entry for PR #886 (explorer extract/decisions fixes)

Documents the extraction 503, decisions timestamp 500, and folded-in
spaCy caching fixes, plus the review-round hardening from Sameer6305
and the timestamp overflow/bool fix from this follow-up commit.

---------

Co-authored-by: joseedson18jc <joseedson18jc@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com>
2026-08-15 13:48:26 +05:30
yzxcj797 c1be6dd7dc docs: fix dead allcontributors emoji-key link (#987) 2026-08-15 01:08:13 +05:30
Zohaib Hassnain 42afc06003 ci: refresh github/codeql-action pin to current v4 (#986)
The pin was 5595ccaf..., but upstream has since moved the v4 tag to
ff2f1c62.... The Verify Action Pins workflow flags this drift on every
PR that touches any workflow file, regardless of whether that PR
changed codeql.yml or defender-for-devops.yml.

Verified the new SHA against the GitHub API directly (not just the CI
error text) and confirmed .github/scripts/verify-action-pins.sh passes
clean locally (40/40 action references OK, exit 0).
2026-08-14 22:18:46 +05:00
64 changed files with 8149 additions and 433 deletions
+1 -1
View File
@@ -1,4 +1,4 @@
> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — comment on the issue and wait for assignment before opening a PR, to avoid duplicate work.
> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — wait for the issue to be assigned to you before opening a PR, to avoid duplicate work.
## Description
+54
View File
@@ -11,6 +11,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- **First-class CrewAI integration** (#962)
- New `pip install semantica[crewai]` extra (`crewai>=0.80.0`) — crewai core provides `BaseTool`/`BaseKnowledgeSource`, so `crewai-tools` is intentionally not included, and the extra is intentionally **not** part of the `all` bundle: crewai hard-requires `chromadb~=1.1.0`, which is affected by the unpatched pre-auth code-injection CVE-2026-45829 (see `integrations/crewai/README.md`)
- `integrations/crewai/SemanticaKGTool` — a CrewAI `BaseTool` exposing 5 KG actions (`extract_entities`, `extract_relations`, `add_to_graph`, `query_graph`, `find_related`) backed by `NERExtractor` / `RelationExtractor` / `ContextGraph`; supports both sync `run()` and async `arun()`
- `integrations/crewai/SemanticaDecisionTool` — a CrewAI `BaseTool` wrapping `AgentContext` with 5 decision-intelligence actions (`record_decision`, `find_precedents`, `trace_causal_chain`, `analyze_impact`, `check_policy`)
- `integrations/crewai/SemanticaKnowledgeSource` — a CrewAI `BaseKnowledgeSource` that serializes a `ContextGraph` into crew knowledge storage; implements both the legacy `load_content()` and current `validate_content()`/`aadd()` contracts so it works across `crewai>=0.80.0`
- All three classes degrade gracefully when `crewai` is not installed (still importable, full Semantica API available)
- New `tests/integrations/crewai/`: 70 tests covering stub-based present-case behavior (Pydantic/BaseTool subclassing, every action, knowledge-source chunking/storage) plus a subprocess isolation test for the crewai-absent degradation path
- Docs: `docs/integrations/crewai.md` page, `docs.json` Integrations nav entry, and README integration-matrix/install updates
- **Hardened during code review**: live `graph`/`context`/extractor state is excluded from CrewAI JSON serialization (`model_dump(mode="json")`) with `model_post_init` self-healing defaults, so checkpoint/resume no longer raises `PydanticSerializationError`; `query_graph` now searches node content (not just ids/types); `trace_causal_chain` returns an explicit error instead of substituting similarity precedents when causal tracing is unavailable, and calls `trace_decision_causality(..., max_depth=...)` with the correct argument name; `find_precedents` propagates `max_precedents` as the backend `limit`; `add_to_graph` writes are serialized under a module lock so concurrent agents can't double-count duplicate adds; nameless entities are skipped instead of creating `repr()`-junk nodes
- **Hardened during second code review**: `check_policy` rules are now coerced type-aware — `bool("false")` was truthy, so `enabled == false` reported a violation for `enabled: false`, and string datums like `"0.90"` were compared lexicographically instead of numerically; `trace_causal_chain` no longer raises `AttributeError` (which escaped the tool) when the decision context has no `knowledge_graph`, returning honest error JSON instead; knowledge-source storage failures log an actionable ERROR (a missing crew embedder otherwise silently left agents with empty retrieval); `add_to_graph` uses a per-graph re-entrant lock instead of a process-global one (independent graphs no longer serialize each other, and re-entrant extractors can't deadlock); entity/relation `confidence=None` normalizes to `1.0` instead of failing the whole extraction; added a subprocess integration test against the real `crewai` package covering `Crew`-level serialization round-trip and restore
- **`ContextGraph` gains retraction and purge — the graph previously had no way to remove a node or edge without discarding everything via `clear()`** (#957, closes #955) by @pravit-amp, reviewed by @KaifAhmad1
- `retract_node()`/`retract_edge()` close an entity's validity window rather than deleting it, reusing the existing `valid_from`/`valid_until`/`state_at()` machinery: the entity drops out of `find_active_nodes()` and future `state_at()` queries going forward, but `state_at()` calls before the retraction time still return it, so decisions recorded against it stay explainable. A `("kind", id)`-keyed retraction record captures who/why/when, retrievable via `get_retraction()`/`list_retractions()`
- `purge_node()`/`purge_edge()` are the destructive counterpart: the entity is removed outright, from history as well as the active view, for erasure obligations retraction alone cannot satisfy (e.g. GDPR Article 17). Only a tombstone remains — that a purge happened, when, and why — deliberately never the purged content, via `get_tombstone()`/`list_tombstones()`. Purge is graph-scope only: `AgentMemory` and any bound vector store are not reached, so it is one step of an erasure workflow rather than the whole of it
- Both operations default to `cascade=True` (also touching every incident edge, and for `purge_node`, the marker node of any cross-graph link the node exits through) since leaving edges active around an inactive/removed node produces an inconsistent active view or dangling endpoints; both accept `cascade=False` for callers that want to handle edges themselves
- Both are idempotent: retracting/purging an already-retracted/purged entity returns `False` rather than raising, and a repeat retraction preserves the original record's reason rather than overwriting it
- Retraction/purge closing a validity window never widens an existing one — a node or edge added with `valid_until` already in the past keeps that earlier bound rather than being pushed later by a subsequent retraction time
- Reuses the existing audit-trail path with no changes to `change_management`: `MutationRecord` already documented `REMOVE_NODE`/`REMOVE_EDGE` in its operation vocabulary; retraction now emits `UPDATE_NODE`/`UPDATE_EDGE`, purge emits `REMOVE_NODE`/`REMOVE_EDGE`, matching the documented contract. Mutation payloads are snapshotted inside the lock and the callback fires after it is released, so a callback that itself mutates the graph (e.g. `clear()`) can't observe or lose in-flight records
- **Fixed during review** (@KaifAhmad1): `retract_edge()`/`purge_edge()` resolved "the edge" for a given `edge_id` via the first matching object only. `edge_id` is content-derived and, prior to #926, was not guaranteed unique — a graph holding two identical `add_edge()` calls had two edge objects sharing one id. A direct `retract_edge()`/`purge_edge()` call would silently leave the second duplicate untouched (still live, still active) while returning `True` and recording a tombstone/retraction that claimed the edge was fully handled; repeat `purge_edge()` calls also silently overwrote the tombstone's `reason`/`purged_at` on each partial attempt instead of no-op'ing. The same gap let `retract_node()`'s cascade skip a duplicate outright, since it checked the live `_retractions` dict mid-loop and treated the first duplicate's just-written record as proof the second was already handled. `#926` (merged) stops *new* duplicates from being created, but any graph already holding one — loaded from a save made before that fix, or built during the window before it landed — could still trigger this. Now `retract_edge()`/`purge_edge()` act on every edge matching the id under one record, and the cascade's dedup check is snapshotted before the loop starts so within-call duplicates are still closed rather than skipped. 5 new regression tests in `TestDuplicateEdgeId`
- New `tests/context/test_context_graph_retraction.py`: 49 tests, covering retraction/purge semantics, cascade, idempotency, validity-window narrowing, id-keyspace collisions between node and edge ids, cross-graph link teardown, `clear()`/`load_from_file()` resetting retraction/tombstone state, audit-trail integration against a real `TemporalVersionManager`, mutation-emission ordering under a concurrent `clear()`, and concurrent purges
- Full `tests/context/` suite: 533 passed
- **`DistanceExporter.compute_pairs()` gains an opt-in `metric_errors` column to distinguish legitimate `None` results from computation failures** (#960, follow-up to #879) by @Karunasagar12
- Previously, a `None` in `hop_count`/`weighted_distance`/`semantic_similarity`/betweenness could mean either "no path exists" or "the underlying computation raised" — logged as a warning per #879, but not otherwise surfaced, so the two cases were indistinguishable in exported CSV/JSONL/DataFrame data. `include=["metric_errors"]` now adds a `metric_errors` field per row: `""` when all requested metrics succeeded, or a comma-separated list of metric names that raised (e.g. `"hop_count,weighted_distance"`)
- Opt-in only — default `compute_pairs()`/`to_csv()`/`to_dataframe()`/`to_jsonl()` schema is unchanged unless `"metric_errors"` is explicitly requested
@@ -52,6 +73,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **`export_yaml` raised a raw `AttributeError` on list input, silently wrote empty exports for unrecognized dict keys, and graph payloads were reconciled differently by every exporter** (#958, closes #956, #952, #953) by @pravit-amp, reviewed by @Sameer6305
- Graph payloads circulate under two vocabularies, `entities`/`relationships` and `nodes`/`edges`, and each exporter reconciled them locally with a different idiom — `LPGExporter` in particular dropped every entity whenever `nodes` was present but empty, the exact shape `JSONExporter` emits. A new `normalize_graph_payload()` in `utils/helpers.py` centralizes that decision once, adopted by `LPGExporter`, `ArangoAQLExporter`, `Neo4jCSVExporter`, and both YAML exporters; `ContextGraph.to_dict()` now round-trips through YAML correctly as a result
- `export_yaml(records, path)` on a bare list previously failed with `AttributeError` from inside the exporter; it and the other YAML methods now reject non-mapping input with an actionable `ProcessingError` naming the expected keys, since these formats distinguish entities/relationships/triplets and guessing which one a list represents would mislabel the records
- `export_yaml({"data": [...]}, path)` previously wrote a structurally valid file with every collection empty, no exception, no warning, and the progress log reporting a completed export. `export_semantic_network`, `export_for_pipeline`, and `export_ontology_schema` now raise `ValidationError` when the payload shares no recognized key with what the method reads, or resolves to nothing while an unread key still holds records — an empty mapping is still accepted, since a genuinely empty graph has no records to lose
- **Breaking**: the two cases above, plus a bare list, now raise instead of returning cleanly with data silently dropped or a raw `AttributeError` from exporter internals. Migration: pass records under a recognized key (`{"entities": [...]}` / `{"nodes": [...]}` for `semantic_network`, `{"classes": [...]}` for `schema`)
- **Fixed during review** (Qodo): progress tracking could report a completed export before the output directory existed or the file was written; `export()` now creates the directory and serializes before starting tracking, so a rejected export leaves nothing behind
- **Fixed during review** (@Sameer6305, round 1): `normalize_graph_payload()`'s collection resolver treated any truthy value as a collection — `{"entities": "abc"}` silently became three single-character records, `{"entities": 42}` leaked a raw `TypeError` from inside `list()`. Collection values are now validated before conversion, rejecting strings/bytes/mappings/non-iterable scalars by name. Separately, `Neo4jCSVExporter._normalize_graph` called the shared resolver with `require_recognized=False`, so it alone kept accepting an unrecognized mapping as a silent empty export; the opt-out (introduced earlier in this same PR, with no other caller) was removed
- **Fixed during review** (@Sameer6305, round 2): `YAMLSchemaExporter`'s usable-schema check could treat scalar schema metadata (`version`, `uri`, `title`, `description`) as evidence records had been exported, letting records under an unread key drop silently; and `_is_record()` accepted modules and class/type objects through the generic `__dict__` path, which would have reached exporter internals instead of failing at the boundary. Both closed, with regression coverage
- **Fixed during final maintainer review** (before merge): four more gaps in the shared boundary that the earlier rounds didn't reach
- `LPGExporter`/`ArangoAQLExporter` called `normalize_graph_payload()` with no type guard, so non-mapping input raised `ValidationError` from inside the resolver — while YAML and `Neo4jCSVExporter` raised `ProcessingError` for the identical mistake, per this PR's own stated contract. The `_require_mapping()` guard that already existed in `yaml_exporter.py` is now shared from `utils/helpers.py` and used by all three
- `Neo4jCSVExporter._normalize_graph` checked `isinstance(graph, dict)`, so a non-dict `Mapping` (`MappingProxyType`, `ChainMap`) fell through to the object-attribute branch and was rejected, even though the identical payload exported fine via `LPGExporter`/`ArangoAQLExporter`/YAML. Now checks `isinstance(graph, Mapping)`
- `normalize_graph_payload()` accepts dataclass and attribute-bearing object records (`Neo4jCSVExporter._record_to_dict` reads them), but `LPGExporter`/`ArangoAQLExporter` call `.get(...)` directly on resolved entities — an object-shaped record passed validation only to crash with a raw `AttributeError` once used, the exact failure this PR's boundary exists to prevent. Records are now converted to plain dicts at the boundary (`_coerce_records` → new `_record_to_dict`), so every consumer gets a uniform shape regardless of which reading the caller used
- Two non-empty spellings of the same collection (e.g. `entities` and `nodes`) holding identical records in a different order were rejected as conflicting, since the check used plain list equality; a caller round-tripping through a dict-keyed cache or a set has no reason to preserve order. Comparison is now an order-independent multiset of each record's canonical JSON form
- New regression coverage in `tests/utils/test_normalize_graph_payload.py`: exception-type parity for non-mapping input across `export_lpg`/`export_arango`/`export_neo4j_csv`, dataclass-record conversion verified end-to-end through the same three exporters, `Neo4jCSVExporter` accepting a `MappingProxyType` payload, and reordered-alias equality (plus a duplicate-count case confirming the multiset check still catches real conflicts); 4 existing tests updated to assert the corrected dict-conversion behavior instead of the previous object passthrough
- `pytest tests/export tests/utils tests/context tests/test_export_module.py tests/test_export_methods_wrapper.py tests/test_notebooks_simulation.py`: 718 passed, 4 skipped (up from 641 passed, 62 subtests at PR submission); `black`/`isort`/`flake8 --max-line-length=88` clean on every line this PR touches; `python -m build`: succeeds
- **`ContextGraph.add_edge` had no dedupe — identical edges were stored repeatedly under one shared edge ID, and re-ingest doubled the edge set** (#926, closes #922) by @pravit-amp
- `_add_internal_edge` appended to `self.edges`, `edge_type_index`, and `_adjacency` unconditionally, with no check for an edge already present. Edge identity is content-derived (`_resolve_edge_identity` builds `edge_id` from `source_id`/`target_id`/`edge_type`/`weight`/`metadata`/`valid_from`/`valid_until`), so two identical `add_edge` calls produced two edge objects sharing one `edge_id` — the graph already considered them the same edge, it just kept both copies. `self.nodes` already deduped by ID; edges did not, so `stats()["edge_count"]` inflated, `density()` could exceed its mathematical maximum of `1.0`, and a refresh/restore job calling `build_from_entities_and_relationships()` (or reloading a saved graph) doubled the edge set on every cycle
- Added an `edge_id -> ContextEdge` index (`_edge_index`), mirroring how `self.nodes` dedupes by node ID. `_add_internal_edge` now returns `False` when the `edge_id` already exists, checked before touching `edges`/`edge_type_index`/`_adjacency` and before firing the mutation callback, so a repeat `add_edge` is a silent no-op with no phantom `ADD_EDGE` audit event
- Genuinely parallel edges are unaffected: differing type/weight/metadata/validity still produce distinct content-derived `edge_id`s, so multigraph semantics are preserved
- Both state-reset paths (`load_from_file()` and `clear()`) also clear `_edge_index`
- New tests: repeat `add_edge` is a no-op, parallel edges with distinct attributes are preserved, re-ingest via `build_from_entities_and_relationships()` stays at one edge, and `clear()` resets the dedupe index
- `pytest tests/context/test_context.py`: 31 passed
- **`POST /api/enrich/extract` returned 503 on every request; the whole `/api/decisions*` family returned 500 as soon as a decision existed** (#886, closes #883, closes #884, closes #889) by @joseedson18jc, reviewed by @Sameer6305
- `semantica/explorer/routes/enrich.py` imported `extract_entities`/`extract_relations` from `semantica.semantic_extract.methods`, names that module never defined (only per-strategy variants like `extract_entities_ml` exist) — the `except ImportError` handler reported this as `"semantic_extract module not available"`, masking a wiring bug as a missing dependency. The route now calls `NamedEntityRecognizer`/`RelationExtractor` directly and forwards extracted entities into relation extraction instead of re-deriving them
- `ContextGraph.record_decision()` stores `timestamp` as `datetime.now().timestamp()` (a float), while `DecisionResponse.timestamp` was typed `Optional[str]`; passing the value through unconverted failed pydantic validation on every decision route (`/api/decisions`, `/{id}`, `/{id}/chain`, `/{id}/precedents`, `/{id}/compliance`). Added a `field_validator(mode="before")` on `DecisionResponse` normalizing float/int/datetime inputs to ISO-8601
- Folds in the fix for #889: `extract_entities_ml`/`extract_relations_similarity`/`extract_relations_dependency` called `spacy.load()` on every invocation (~120ms of a ~132ms call, ~60x the actual extraction work). Added a process-level, lock-guarded `load_spacy_model()` cache in `semantic_extract/methods.py`, keyed by model name; failed loads are not cached, and the separate `get_nlp_model()` cache (different `disable=` pipeline config for similarity work) is kept independent to avoid handing one caller's spaCy pipeline to another
- **Fixed during review** (@Sameer6305): capped previously-unbounded input text on `/api/enrich/extract`; tightened the route's exception handling
- **Fixed during review** (@KaifAhmad1): the timestamp validator's `math.isfinite()` guard only rejected NaN/inf — a finite-but-out-of-range epoch (e.g. milliseconds mistakenly stored instead of seconds, such as `1723600000000`) still raised an uncaught `OverflowError`/`OSError` from `datetime.fromtimestamp()`, reintroducing an unhandled 500 on `/api/decisions*` for exactly the class of bug this PR closes. Now caught and re-raised as a `ValueError`. Also excluded `bool` from the numeric branch (`isinstance(True, int)` is `True` in Python, so `timestamp=True` was silently coerced to epoch 1 instead of being rejected)
- New/updated tests: `tests/explorer/test_explorer_api.py` (`TestRecordedDecisions`, extraction coverage, 4 new `TestDecisionResponseTimestampValidator` cases for the range/bool fixes), `tests/semantic_extract/test_spacy_model_cache.py` (6 tests)
- `pytest tests/explorer tests/semantic_extract/test_spacy_model_cache.py`: 266 passed
- **Explorer UI hid backend failures: graph load hung forever, landing page always showed "System Online"** (#980, closes #977) by @ZohaibHassan16, reviewed by @Sameer6305
- `GraphWorkspace.tsx` only destructured `{ data, isLoading, isFetching }` from `useLoadGraph()`, ignoring the `isError`/`error`/`refetch` that `useQuery` (`retry: 0`) already returned. Combined with `GraphLoadingOverlay` having no error prop and `showLoadingOverlay` staying true whenever `loadingProgress` held a stale frame, a backend-down or failed fetch left the graph workspace stuck on the last progress frame indefinitely, with no error message and no way to recover short of a full page reload
- `GraphLoadingOverlay` now accepts `error`/`onRetry` and renders an error card with the real fetch error message and a Retry button (`refetch()`) instead of the stuck progress UI
+17 -3
View File
@@ -25,9 +25,9 @@ If you want to work on an open GitHub issue, please follow these steps to keep t
1. **Check the issue.** Look at the issue's assignees and recent comments. If someone is already actively working on it, consider a different issue or ask in the comments whether help is welcome.
2. **Comment before you start.** Leave a comment on the issue saying you'd like to work on it — something like *"I'd like to take this on"* is enough. This gives maintainers the context they need to assign the issue appropriately.
2. **Comment if you'd like the issue reserved.** Leaving a comment like *"I'd like to take this on"* is the fastest way to get assigned, but it isn't required — maintainers can also assign an issue directly to a contributor (e.g., based on recent activity in the repo) without waiting for a comment first.
3. **Wait for assignment.** A maintainer will review the request and assign the issue when appropriate. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
3. **Wait for assignment.** A maintainer will assign the issue when appropriate, whether or not a comment was left. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
4. **Create a branch and implement.** Once assigned, fork the repository (if you haven't already), create a dedicated branch, and begin your work.
@@ -37,12 +37,26 @@ If you want to work on an open GitHub issue, please follow these steps to keep t
5. **Open a focused PR and link the issue.** When you're ready, open a pull request and reference the issue in the description (e.g., `Closes #123`). Keep the PR scoped to the work described in the issue.
> **Why this matters:** Commenting before opening a PR helps maintainers track who is working on what, assign issues correctly, and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
> **Why this matters:** Assignment (with or without a comment) helps maintainers track who is working on what and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
Not sure where to start? Try a [`good first issue`](https://github.com/semantica-agi/semantica/labels/good%20first%20issue) or ask in [Discord](https://discord.gg/sV34vps5hH).
---
## 🔀 Duplicate PRs & Issue Priority
When more than one pull request targets the same issue, maintainers triage using this order of priority. These rules decide between PRs that are otherwise following the [assignment workflow above](#-working-on-an-existing-issue) — opening a PR before being assigned doesn't grant priority on its own, and an unassigned PR can still be closed as a duplicate once someone else is assigned to the issue.
1. **Contributor-raised issue with an existing PR.** If the person who opened the issue has also opened a PR for it, that PR is prioritized (they still need to be assigned before it's merged).
2. **Maintainer-raised issue with a claim comment.** If we opened the issue and someone has commented asking to work on it, we assign it to them and check their PR before picking up any other PR for the same issue.
3. **No prior assignment or comment.** If multiple PRs exist and no one was assigned or claimed the issue first, priority goes to whichever contributor has the most consistent activity in the repo over the last 60 days (e.g., merged PRs, substantive reviews, or issue triage participation) — not just PR volume.
4. **Late duplicate PRs.** If a PR is opened after another contributor has already been assigned to the issue, we close the duplicate early rather than let it sit open, and point the author to another open issue (or ask them to check `main` for newly opened ones). This avoids contributors spending time updating a PR that won't be merged.
5. **Overlapping scope.** If a PR covers multiple issues, or there's genuine overlap between competing PRs, maintainers discuss it on [Discord](https://discord.gg/sV34vps5hH) before deciding rather than resolving it unilaterally.
**Why this matters:** it keeps triage predictable, avoids wasted contributor effort on PRs that won't merge, and helps retain active contributors.
---
## 🎯 Ways to Contribute
### 💻 Code
+1 -1
View File
@@ -101,7 +101,7 @@ When using the all-contributors bot, use these codes:
- `infra` - Infrastructure
- `maintenance` - Maintenance
See [all-contributors specification](https://allcontributors.org/docs/en/emoji-key) for complete list.
See [all-contributors specification](https://github.com/all-contributors/all-contributors#emoji-key) for complete list.
---
+12 -12
View File
@@ -52,6 +52,8 @@ Most AI agents act without a trail. They store embeddings, not meaning: context
Semantica sits underneath your LLM, vector store, and agent framework as a deterministic infrastructure layer: no LLM required for graph construction, reasoning, or provenance.
> ⚠️ **System-level explainability, not foundation-model explainability.** Semantica does not expose or reconstruct what happens *inside* the LLM — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. Semantica explains what's *outside* the model: the context and data fed in, the decision produced, its provenance, relevant relationships, applied policies, and the full execution trail.
**Who it's for:**
- **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index
@@ -77,7 +79,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
- **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
- **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
- **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
- **Drop-in Integrations:** Native Agno support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
- **Drop-in Integrations:** Native Agno and CrewAI support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
---
@@ -1189,7 +1191,7 @@ Start with `semantica`, verify with `doctor`, build a graph, and explore the com
## Integrations
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno support for multi-agent shared context. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
Native plugin bundles for Claude Code, Cursor, Codex, Windsurf, Cline, Continue, VS Code, and OpenClaw; a full-featured MCP server for any MCP-compatible client; a comprehensive REST API; and first-class Agno and CrewAI support for agentic frameworks. Every major LLM provider is already supported via `semantica.llms` and LiteLLM: OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Azure, Bedrock, Ollama, DeepSeek, HuggingFace, and more.
MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
@@ -1303,6 +1305,11 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<strong>Agno</strong><br/>
<sub>First-class · <code>pip install semantica[agno]</code></sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/crewAIInc/crewAI"><img src="https://github.com/crewAIInc.png?size=120" alt="CrewAI" width="48" height="48" /></a><br/>
<strong>CrewAI</strong><br/>
<sub>First-class · <code>pip install semantica[crewai]</code></sub>
</td>
</tr>
<tr>
<th colspan="8" align="left">Already Supported via REST API &amp; MCP</th>
@@ -1319,11 +1326,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<sub>REST API · MCP</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/crewAIInc/crewAI"><img src="https://github.com/crewAIInc.png?size=120" alt="CrewAI" width="48" height="48" /></a><br/>
<strong>CrewAI</strong><br/>
<sub>REST API · MCP</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/run-llama/llama_index"><img src="https://github.com/run-llama.png?size=120" alt="LlamaIndex" width="48" height="48" /></a><br/>
<strong>LlamaIndex</strong><br/>
<sub>REST API · MCP</sub>
@@ -1354,11 +1356,6 @@ MCP setup takes 30 seconds — see [MCP Server](#mcp-server) below.
<sub>Dedicated toolkit</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/crewAIInc/crewAI"><img src="https://github.com/crewAIInc.png?size=120" alt="CrewAI" width="48" height="48" /></a><br/>
<strong>CrewAI</strong><br/>
<sub>Dedicated toolkit</sub>
</td>
<td align="center" width="12.5%">
<a href="https://github.com/run-llama/llama_index"><img src="https://github.com/run-llama.png?size=120" alt="LlamaIndex" width="48" height="48" /></a><br/>
<strong>LlamaIndex</strong><br/>
<sub>Dedicated toolkit</sub>
@@ -1503,6 +1500,8 @@ Semantica is designed for environments where AI outputs must be explainable, aud
- **Cybersecurity:** Threat attribution, incident response timelines, and IOC provenance tracking
- **Autonomous Systems:** Decision logs, safety validation, and explainable AI for certification
> ⚠️ **This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits what the AI system did, not the LLM's private internal reasoning.
---
## Installation
@@ -1514,6 +1513,7 @@ pip install semantica[all] # everything
```bash
pip install semantica[agno] # Agno multi-agent integration
pip install semantica[crewai] # CrewAI integration
pip install semantica[llm-litellm] # OpenAI, Anthropic, Gemini, Mistral, Llama, Groq, Cohere, Bedrock, Ollama, DeepSeek, and more
pip install semantica[graph-neo4j] # Neo4j graph store (LPG)
pip install semantica[graph-falkordb] # FalkorDB graph store (LPG)
+3
View File
@@ -16,6 +16,9 @@ At its core, Semantica adds a **context and accountability layer** on top of you
- **Accountability Layer** — Provenance tracking, decision intelligence, conflict detection, and W3C PROV-O compliance make every claim in your AI stack auditable and explainable.
- **Extension Layer**`PluginRegistry` and `MethodRegistry` let you replace or augment any component: ingestors, extractors, reasoning engines, backends: without changing framework code.
<Warning>
**This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. In short, Semantica explains and audits *what the AI system did*, not the foundation model's private internal reasoning.
</Warning>
## Knowledge Graphs
+1
View File
@@ -102,6 +102,7 @@
"group": "Integrations",
"pages": [
"integrations/agno",
"integrations/crewai",
"integrations/docling",
"integrations/snowflake",
"integrations/databricks"
+10
View File
@@ -52,6 +52,16 @@ Semantica works alongside these frameworks, not against them.
</Accordion>
<Accordion title="Does Semantica explain an LLM's internal reasoning or chain-of-thought?" icon="triangle-exclamation">
No. This is **system-level explainability, not foundation-model explainability**. Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system.
What Semantica explains is *outside* the model: what context and data were used, what decision was produced, the provenance behind it, the relevant relationships, the policies applied, and the resulting decision trail.
In short: Semantica explains and audits *what the AI system did* — not the foundation model's private internal reasoning.
</Accordion>
<Accordion title="Is Semantica free?" icon="tag">
Yes: MIT licensed, no vendor lock-in, no paywalled features. Some capabilities require third-party API keys (e.g., OpenAI embeddings, Groq inference), but Semantica itself is always free and open source.
+5 -1
View File
@@ -192,7 +192,11 @@ decision_id = context.record_decision(
## Built for Where Mistakes Have Consequences
Semantica was designed for domains where every decision must be explainable and every fact must be traceable:
Semantica was designed for domains where every decision must be explainable and every fact must be traceable.
<Warning>
**This is system-level explainability, not foundation-model explainability.** Semantica does not expose, reconstruct, or explain what happens *inside* the LLM/foundation model — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. What Semantica explains is *outside* the model: the context and data fed in, the decision produced, its provenance, the relevant relationships, the policies applied, and the full execution trail. See [Core Concepts](concepts) for the full scope note.
</Warning>
**Healthcare & Life Sciences**
- Clinical decision support with full audit trails
+147
View File
@@ -0,0 +1,147 @@
---
title: "CrewAI Integration"
description: "Give CrewAI crews a shared semantic knowledge graph, decision intelligence, and graph-based retrieval via three drop-in components."
icon: "users"
---
> Three drop-in components that bring Semantica's knowledge graph and decision intelligence into any CrewAI crew.
## Installation
```bash
pip install "semantica[crewai]"
```
Requires `crewai >= 0.80.0`. If `crewai` is not installed, the integration still imports — every class carries the full Semantica API and degrades gracefully, but cannot be passed to a `Crew`.
## Components at a Glance
- **SemanticaKGTool**`Agent(tools=[…])`: 5 KG construction/query actions: extract entities, extract relations, add to graph, query graph, find related.
- **SemanticaDecisionTool**`Agent(tools=[…])`: 5 decision intelligence actions: record decisions, find precedents, trace causal chains, analyze impact, check policies.
- **SemanticaKnowledgeSource**`Crew(knowledge_sources=[…])`: Serializes a `ContextGraph` into CrewAI knowledge storage so every agent gets retrieval access to the graph.
## Component Details
<Tabs>
<Tab title="SemanticaKGTool">
Lets agents actively **build and query** a shared `ContextGraph` mid-reasoning.
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKGTool
graph = ContextGraph()
analyst = Agent(
role="Knowledge Analyst",
goal="Build and explore a knowledge graph from documents",
backstory="You map entities and relationships into a shared graph.",
tools=[SemanticaKGTool(graph=graph)],
)
crew = Crew(
agents=[analyst],
tasks=[Task(
description="Extract and link key entities from the brief",
expected_output="JSON",
agent=analyst,
)],
)
crew.kickoff()
```
| Tool | Description |
| :------ | :------------- |
| `extract_entities` | Extract named entities from `text` |
| `extract_relations` | Extract relationships between entities in `text` |
| `add_to_graph` | Extract entities/relations from `text` and add them to the shared graph |
| `query_graph` | Keyword-search the graph by node id, type, and content using `query` |
| `find_related` | Find concepts related to `entity` within `hops` hops |
All actions return JSON so agents get parseable results.
**Sharing a graph:** the tool reads/writes whatever `graph` you pass in. When no `graph` is given, a fresh in-memory `ContextGraph()` is created (and a warning is logged) — two tool instances that each auto-create their own graph do **not** share knowledge. Pass the same `ContextGraph` to every agent that must share state.
</Tab>
<Tab title="SemanticaDecisionTool">
Exposes Semantica's decision intelligence as a native CrewAI tool, backed by `AgentContext`.
```python
from crewai import Agent, Crew, Task
from integrations.crewai import SemanticaDecisionTool
planner = Agent(
role="Decision Planner",
goal="Make grounded, precedented decisions",
backstory="You record decisions and validate them against policy.",
tools=[SemanticaDecisionTool()],
)
crew = Crew(agents=[planner], tasks=[...])
```
When no `AgentContext` is passed, one is created in-memory with `decision_tracking=True` and its own `ContextGraph`, so decision actions work out of the box (a warning is logged — pass the same `AgentContext` to every agent that must share decision state). Missing optional fields in `record_decision` fall back to `category="general"`, `reasoning="agent decision"`, and `outcome="recorded"`. `find_precedents` returns up to `max_precedents` results. If a knowledge graph cannot trace causality, `trace_causal_chain` returns an explicit error rather than substituting similarity-based results.
| Tool | Description |
| :------ | :------------- |
| `record_decision` | Record a decision with reasoning, outcome, and confidence |
| `find_precedents` | Search for similar past decisions |
| `trace_causal_chain` | Trace the causal chain from a decision |
| `analyze_impact` | Assess downstream influence of a decision |
| `check_policy` | Validate a proposed decision against policy rules |
</Tab>
<Tab title="SemanticaKnowledgeSource">
Gives **every agent in the crew** retrieval access to a `ContextGraph`.
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKnowledgeSource
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="...")
researcher = Agent(
role="Policy Researcher",
goal="Answer questions from the knowledge base",
backstory="You retrieve from graph knowledge to answer accurately.",
)
crew = Crew(
agents=[researcher],
tasks=[...],
knowledge_sources=[SemanticaKnowledgeSource(graph=graph)],
)
```
On kickoff the graph's nodes and edges are serialized, chunked, and stored through CrewAI's knowledge pipeline.
> **Embedder required:** storing chunks goes through CrewAI's knowledge pipeline, which needs an embedder to be configured. Set `Crew(embedder=...)` (or provide the default credentials CrewAI falls back to, e.g. `OPENAI_API_KEY`). If no working embedder is configured, storage fails, an ERROR is logged, and agents will retrieve **nothing** — the crew still runs, but its knowledge queries return empty.
**Compatibility:** CrewAI's `BaseKnowledgeSource` contract changed between `0.80.x` and current releases (`load_content()``validate_content()`/`aadd()`). `SemanticaKnowledgeSource` implements both legacy and current methods, so it works across `crewai>=0.80.0`.
</Tab>
</Tabs>
## Checkpoints & Serialization
CrewAI serializes tools and knowledge sources to JSON for checkpointing/resume. Live Semantica state (`ContextGraph`, `AgentContext`, extractors) is **excluded from that serialization** — a restored tool/source comes back with a fresh in-memory `ContextGraph` and logs a warning. Until you re-attach the live graph/context, the restored objects answer queries against an **empty** graph, so re-wire them after resuming (e.g. `restored_tool.graph = live_graph`) before agents continue.
## API Reference
```python
from integrations.crewai import (
SemanticaKGTool, # BaseTool: KG construction/query actions
SemanticaDecisionTool, # BaseTool: decision intelligence actions
SemanticaKnowledgeSource, # BaseKnowledgeSource: graph → crew knowledge
CREWAI_AVAILABLE, # bool: True if crewai is installed
)
```
All three classes are usable without `crewai` installed: they carry the full Semantica API and degrade gracefully.
## See Also
- [Context Module](../reference/context) — AgentContext and ContextGraph backing the integration.
- [Semantic Extraction](../reference/semantic_extract) — NERExtractor / RelationExtractor used by SemanticaKGTool.
- [LLMs](../reference/llms) — Configure LLM providers for your crew's agents.
- [Vector Store](../reference/vector_store) — Vector backend used by SemanticaDecisionTool.
+13
View File
@@ -203,6 +203,13 @@ export_lpg(graph, "import.cypher", method="cypher")
exporter = SemanticNetworkYAMLExporter()
exporter.export(graph, "graph.yaml")
```
The YAML exporters read `entities`/`relationships`/`triplets` (with
`nodes`/`edges` accepted as aliases, so `ContextGraph.to_dict()` exports
directly). A non-empty mapping supplying none of them raises
`ValidationError` rather than writing a file with every collection empty,
as does one whose collection value is not a list of records
(`{"entities": "abc"}`).
</Tab>
<Tab title="Graph DB Import">
**LPGExporter** writes Cypher `CREATE` statements for Neo4j and Memgraph:
@@ -236,6 +243,12 @@ export_lpg(graph, "import.cypher", method="cypher")
Both exporters write to a file and return `None`.
`LPGExporter`, `ArangoAQLExporter`, and `Neo4jCSVExporter` resolve mapping
payloads on the same terms as the YAML exporters above, so an unrecognized
or malformed mapping is rejected instead of exported as an empty graph.
`Neo4jCSVExporter` still reads graph *objects* off their
`nodes`/`entities` and `edges`/`relationships` attributes.
<Warning>
**`ArangoAQLExporter.export()` and `LPGExporter.export()` write to a file and return `None`.** They do not return the AQL/Cypher string. Write to a file and read it back if you need the string.
</Warning>
+1 -1
View File
@@ -9,7 +9,7 @@
"lint": "eslint .",
"preview": "vite preview",
"test:graph-store": "node --test tests/graphStore.multi-edge.test.mjs",
"test:graph-workspace": "node --import tsx --test tests/graphSceneState.display.test.ts",
"test:graph-workspace": "node --import tsx --test tests/graphSceneState.display.test.ts tests/temporalLifecycle.test.ts",
"test:plugin-registry": "node --import tsx --test tests/pluginRegistry.temporal.test.mjs"
},
"dependencies": {
@@ -40,6 +40,7 @@ import {
type GraphPluginToolbarItem,
} from "./plugins";
import { explorationEffectsShouldLoad, neighborhoodPanelShouldLoad, temporalOverlayShouldLoad } from "./pluginRegistryPredicates";
import { shouldFetchTemporalBounds, shouldFetchTemporalSnapshot } from "./temporalLifecyclePredicates";
import type { LinkPrediction, PathResponse } from "./GraphInspectorPanel";
import type { GraphSceneHandle, GraphSceneRuntime } from "./scene";
import type {
@@ -1440,7 +1441,18 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
applyGraphReadySummary(summary);
}, [applyGraphReadySummary, graphReady, summary]);
const canFetchTemporalBounds = shouldFetchTemporalBounds(summary);
const canFetchTemporalSnapshot = shouldFetchTemporalSnapshot({
debouncedTime,
isLoading,
summary,
});
useEffect(() => {
if (!canFetchTemporalBounds) {
return;
}
let cancelled = false;
const loadBounds = async () => {
try {
@@ -1460,10 +1472,21 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
return () => {
cancelled = true;
};
}, [summary?.nodeCount, summary?.edgeCount]);
}, [
canFetchTemporalBounds,
summary?.nodeCount,
summary?.edgeCount,
]);
useEffect(() => {
if (!debouncedTime || isLoading) return;
if (!canFetchTemporalSnapshot) {
return;
}
if (!debouncedTime) {
return;
}
let cancelled = false;
const applySnapshot = async () => {
@@ -1505,7 +1528,10 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
return () => {
cancelled = true;
};
}, [debouncedTime, isLoading]);
}, [
canFetchTemporalSnapshot,
debouncedTime,
]);
const resolveNodeIdForFocusedMode = useCallback((
nodeId: string,
@@ -0,0 +1,31 @@
import type { GraphLoadSummary } from "./types";
/**
* Predicates for gating GraphWorkspace temporal API requests.
*
* Temporal bounds and snapshot requests must strictly not execute until the
* initial graph load has succeeded (summary !== undefined). An empty graph
* (nodeCount: 0) is still a successful load and must not be rejected.
*/
export function shouldFetchTemporalBounds(
summary: GraphLoadSummary | undefined,
): boolean {
return summary !== undefined;
}
export function shouldFetchTemporalSnapshot({
debouncedTime,
isLoading,
summary,
}: {
debouncedTime: Date | null;
isLoading: boolean;
summary: GraphLoadSummary | undefined;
}): boolean {
return (
summary !== undefined &&
debouncedTime !== null &&
!isLoading
);
}
+114
View File
@@ -0,0 +1,114 @@
import test from "node:test";
import assert from "node:assert/strict";
import {
shouldFetchTemporalBounds,
shouldFetchTemporalSnapshot,
} from "../src/workspaces/GraphWorkspace/temporalLifecyclePredicates.ts";
import type { GraphLoadSummary } from "../src/workspaces/GraphWorkspace/types.ts";
const sampleSummary: GraphLoadSummary = {
nodeCount: 42,
edgeCount: 78,
loadTimeMs: 120,
hasCoordinates: true,
layoutSource: "provided",
layoutReady: true,
};
const emptyGraphSummary: GraphLoadSummary = {
nodeCount: 0,
edgeCount: 0,
loadTimeMs: 15,
hasCoordinates: false,
layoutSource: "runtime",
layoutReady: false,
};
// ── shouldFetchTemporalBounds ────────────────────────────────────────────────
test("temporal bounds: false when summary is undefined (initial mount or failed load)", () => {
assert.equal(
shouldFetchTemporalBounds(undefined),
false,
"bounds request must not run before graph load succeeds",
);
});
test("temporal bounds: true when non-empty summary is present", () => {
assert.equal(
shouldFetchTemporalBounds(sampleSummary),
true,
"bounds request should run when successful graph summary exists",
);
});
test("temporal bounds: true when successful summary has nodeCount of 0", () => {
assert.equal(
shouldFetchTemporalBounds(emptyGraphSummary),
true,
"an empty graph is still a successful load and must allow bounds fetching",
);
});
// ── shouldFetchTemporalSnapshot ──────────────────────────────────────────────
test("temporal snapshot: false when summary is undefined even if scrubber time is set and isLoading is false", () => {
assert.equal(
shouldFetchTemporalSnapshot({
debouncedTime: new Date("2024-01-01T00:00:00Z"),
isLoading: false,
summary: undefined,
}),
false,
"snapshot request must not run when graph load failed",
);
});
test("temporal snapshot: false when graph is currently loading", () => {
assert.equal(
shouldFetchTemporalSnapshot({
debouncedTime: new Date("2024-01-01T00:00:00Z"),
isLoading: true,
summary: sampleSummary,
}),
false,
"snapshot request must not run while graph is loading",
);
});
test("temporal snapshot: false when debouncedTime is null", () => {
assert.equal(
shouldFetchTemporalSnapshot({
debouncedTime: null,
isLoading: false,
summary: sampleSummary,
}),
false,
"snapshot request must not run without a scrubber timestamp",
);
});
test("temporal snapshot: true when summary exists, isLoading is false, and time is set", () => {
assert.equal(
shouldFetchTemporalSnapshot({
debouncedTime: new Date("2024-01-01T00:00:00Z"),
isLoading: false,
summary: sampleSummary,
}),
true,
"snapshot request should run after graph load succeeds and time is set",
);
});
test("temporal snapshot: true when successful summary has 0 nodes, isLoading is false, and time is set", () => {
assert.equal(
shouldFetchTemporalSnapshot({
debouncedTime: new Date("2024-01-01T00:00:00Z"),
isLoading: false,
summary: emptyGraphSummary,
}),
true,
"empty successful graph must allow snapshot requests once ready",
);
});
+108
View File
@@ -0,0 +1,108 @@
# Semantica × CrewAI
First-class integration between Semantica and [CrewAI](https://github.com/crewAIInc/crewAI) — give your crews a shared semantic knowledge graph, decision intelligence, and graph-based retrieval.
## Installation
```bash
pip install semantica[crewai]
```
Requires `crewai >= 0.80.0`. If `crewai` is not installed, the integration still imports (classes degrade gracefully), but you can't pass the objects to a `Crew`.
> **⚠️ Security note:** crewai hard-requires `chromadb~=1.1.0`, which is currently affected by the unpatched pre-authentication code-injection advisory **CVE-2026-45829** (no fixed release — even the latest chromadb 1.5.9 is affected). Installing `semantica[crewai]` pulls that dependency into your environment. The `crewai` extra is intentionally **not** part of `semantica[all]` for this reason — only install it where you actually use CrewAI, and follow chromadb for a patched release.
## 1. SemanticaKGTool
A `BaseTool` that lets agents **build and query** a shared `ContextGraph` mid-reasoning:
- `extract_entities` — extract named entities from `text`
- `extract_relations` — extract relationships from `text`
- `add_to_graph` — extract entities/relations from `text` and add them to the shared graph
- `query_graph` — keyword-search the graph using `query`
- `find_related` — find concepts related to `entity` within `hops`
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKGTool
graph = ContextGraph()
analyst = Agent(
role="Knowledge Analyst",
goal="Build and explore a knowledge graph from documents",
backstory="You map entities and relationships into a shared graph.",
tools=[SemanticaKGTool(graph=graph)],
)
crew = Crew(
agents=[analyst],
tasks=[Task(description="Extract and link key entities from the brief", expected_output="JSON", agent=analyst)],
)
result = crew.kickoff()
```
All actions return JSON, so agents get parseable results.
## 2. SemanticaDecisionTool
A `BaseTool` that wraps `AgentContext` and exposes decision intelligence:
- `record_decision` — record a decision with reasoning and outcome
- `find_precedents` — retrieve past decisions similar to a scenario
- `trace_causal_chain` — trace the causal chain from a decision
- `analyze_impact` — assess downstream influence using graph centrality
- `check_policy` — validate a proposed decision against rule-based policies
```python
from crewai import Agent, Crew, Task
from integrations.crewai import SemanticaDecisionTool
planner = Agent(
role="Decision Planner",
goal="Make grounded, precedented decisions",
backstory="You record decisions and validate them against policy.",
tools=[SemanticaDecisionTool()],
)
crew = Crew(agents=[planner], tasks=[...])
```
When no `AgentContext` is passed, one is created in-memory with `decision_tracking=True`.
## 3. SemanticaKnowledgeSource
A `BaseKnowledgeSource` that serializes the current state of a `ContextGraph` (nodes, edges, metadata) into CrewAI's knowledge storage, giving **every agent in the crew** retrieval access to the graph:
```python
from crewai import Agent, Crew, Task
from semantica.context import ContextGraph
from integrations.crewai import SemanticaKnowledgeSource
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="...")
researcher = Agent(
role="Policy Researcher",
goal="Answer questions from the knowledge base",
backstory="You retrieve from graph knowledge to answer accurately.",
)
crew = Crew(
agents=[researcher],
tasks=[...],
knowledge_sources=[SemanticaKnowledgeSource(graph=graph)],
)
```
> **Embedder required:** storing chunks goes through CrewAI's knowledge pipeline, which needs an embedder. Set `Crew(embedder=...)` (or provide CrewAI's default credentials, e.g. `OPENAI_API_KEY`). Without a working embedder, storage fails, an ERROR is logged, and agents retrieve nothing — the crew still runs with empty knowledge queries.
### Compatibility note
CrewAI's `BaseKnowledgeSource` contract changed between `0.80.x` and current releases (`load_content()``validate_content()`/`aadd()`). `SemanticaKnowledgeSource` implements both the legacy and current methods, so it works across `crewai>=0.80.0`.
### Sharing state & checkpoints
- Each tool/source holds whatever `graph`/`context` you pass it. When omitted, a fresh in-memory object is created and a warning is logged — instances that auto-create their own state do **not** share knowledge, so pass the same object to every agent that must share.
- Live state (`ContextGraph`, `AgentContext`, extractors) is excluded from CrewAI's JSON serialization. After restoring from a checkpoint, re-attach the live graph/context to the restored objects.
+44
View File
@@ -0,0 +1,44 @@
"""
Semantica × CrewAI Integration
==============================
First-class integration between the Semantica semantic intelligence stack and
the `CrewAI <https://github.com/crewAIInc/crewAI>`_ agentic framework.
Public surface
--------------
SemanticaKGTool CrewAI ``BaseTool`` exposing KG construction/query actions
SemanticaDecisionTool CrewAI ``BaseTool`` exposing decision-intelligence actions
SemanticaKnowledgeSource CrewAI ``BaseKnowledgeSource`` giving crews graph knowledge
Quick start
-----------
pip install semantica[crewai]
>>> from integrations.crewai import (
... SemanticaKGTool,
... SemanticaDecisionTool,
... SemanticaKnowledgeSource,
... )
Compatibility
-------------
Requires ``crewai >= 0.80.0``. All three classes degrade gracefully when
``crewai`` is not installed they are still importable and carry the full
Semantica API, but cannot be passed to ``Crew`` / ``Agent`` constructors.
"""
from ._availability import CREWAI_AVAILABLE, CREWAI_IMPORT_ERROR
from .decision_tool import SemanticaDecisionTool
from .kg_tool import SemanticaKGTool
from .knowledge_source import SemanticaKnowledgeSource
__all__ = [
"SemanticaKGTool",
"SemanticaDecisionTool",
"SemanticaKnowledgeSource",
"CREWAI_AVAILABLE",
"CREWAI_IMPORT_ERROR",
]
__version__ = "0.1.0"
+24
View File
@@ -0,0 +1,24 @@
"""
Shared CrewAI availability probe.
Every integration module needs to know whether the real ``crewai`` package is
installed. Probing once here (instead of once per module) guarantees the
exported ``CREWAI_AVAILABLE`` flag means the *whole* integration is ready a
caller gating on it will never see tools using CrewAI while a knowledge source
silently degrades (or vice versa).
"""
from typing import Optional
CREWAI_AVAILABLE = False
CREWAI_IMPORT_ERROR: Optional[str] = None
try:
from crewai.knowledge.source.base_knowledge_source import ( # noqa: F401
BaseKnowledgeSource,
)
from crewai.tools import BaseTool # noqa: F401
CREWAI_AVAILABLE = True
except ImportError as exc:
CREWAI_IMPORT_ERROR = str(exc)
+555
View File
@@ -0,0 +1,555 @@
"""
SemanticaDecisionTool a CrewAI ``BaseTool`` exposing Semantica's decision
intelligence (``AgentContext``) to agents.
Lets agents record decisions with reasoning, retrieve past precedents, trace
causal chains, analyse downstream impact, and validate proposed decisions
against policy rules.
Install
-------
pip install semantica[crewai]
Example
-------
>>> from integrations.crewai import SemanticaDecisionTool
>>> from crewai import Agent, Crew, Task
>>> tool = SemanticaDecisionTool()
>>> crew = Crew(
... agents=[Agent(role="...", goal="...", backstory="...", tools=[tool])],
... tasks=[...],
... )
Tools exposed
-------------
record_decision Record a decision with reasoning and outcome
find_precedents Search past decisions similar to a scenario
trace_causal_chain Trace the causal chain from a decision node
analyze_impact Assess downstream influence of a decision
check_policy Validate a proposed decision against policy rules
"""
from __future__ import annotations
import json
import re
from typing import Any, Dict, List, Literal, Optional, Type
from pydantic import BaseModel, Field
from semantica.utils.logging import get_logger
from ._availability import CREWAI_AVAILABLE
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: CrewAI BaseTool base class
# ---------------------------------------------------------------------------
_BaseTool: Any = object
if CREWAI_AVAILABLE:
from crewai.tools import BaseTool as _BaseTool # type: ignore
# ---------------------------------------------------------------------------
# Input schema
# ---------------------------------------------------------------------------
class SemanticaDecisionToolInput(BaseModel):
"""
Input schema for ``SemanticaDecisionTool``.
Exactly one action is dispatched per call; the remaining fields are only
used by the actions that need them.
"""
action: Literal[
"record_decision",
"find_precedents",
"trace_causal_chain",
"analyze_impact",
"check_policy",
] = Field(
...,
description=(
"Which decision-intelligence operation to run. One of: "
"'record_decision', 'find_precedents', 'trace_causal_chain', "
"'analyze_impact', 'check_policy'."
),
)
category: Optional[str] = Field(
None,
description="Domain category, e.g. 'loan_approval'. Used by 'record_decision'.",
)
scenario: Optional[str] = Field(
None,
description=(
"Short description of the situation. Used by 'record_decision' and "
"'find_precedents'."
),
)
reasoning: Optional[str] = Field(
None, description="Why this outcome was chosen. Used by 'record_decision'."
)
outcome: Optional[str] = Field(
None, description="The decision result. Used by 'record_decision'."
)
confidence: float = Field(
0.8,
ge=0.0,
le=1.0,
description="Confidence score in [0, 1]. Used by 'record_decision'.",
)
entities: Optional[str] = Field(
None,
description="Comma-separated entity names. Used by 'record_decision'.",
)
decision_id: Optional[str] = Field(
None,
description=(
"Identifier of a decision. Used by 'trace_causal_chain' and "
"'analyze_impact'."
),
)
depth: int = Field(
3,
ge=1,
le=20,
description="Maximum chain depth. Used by 'trace_causal_chain'.",
)
decision_data: Optional[str] = Field(
None,
description=(
"JSON object describing a proposed decision. Used by 'check_policy'."
),
)
policy_rules: Optional[str] = Field(
None,
description=(
"JSON list of rule strings like 'confidence >= 0.7'. Used by "
"'check_policy'."
),
)
# ---------------------------------------------------------------------------
# SemanticaDecisionTool
# ---------------------------------------------------------------------------
class SemanticaDecisionTool(_BaseTool): # type: ignore[misc]
"""
CrewAI tool that surfaces Semantica's decision intelligence as agent actions.
Parameters
----------
context:
A ``semantica.context.AgentContext`` (or compatible object exposing
``record_decision``, ``find_precedents_advanced``,
``analyze_decision_influence``). A fresh in-memory context is created
when ``None``.
max_precedents:
Default number of precedents returned by ``find_precedents``.
causal_depth:
Default chain depth used by ``trace_causal_chain``.
"""
name: str = "semantica_decision"
description: str = (
"Decision intelligence toolkit. Actions: 'record_decision' (record a "
"decision with category, scenario, reasoning, outcome, confidence), "
"'find_precedents' (search past decisions similar to 'scenario'), "
"'trace_causal_chain' (trace the causal chain from 'decision_id'), "
"'analyze_impact' (assess downstream influence of 'decision_id'), "
"'check_policy' (validate 'decision_data' JSON against 'policy_rules' "
"rules like 'confidence >= 0.7'). Returns JSON."
)
args_schema: Type[BaseModel] = SemanticaDecisionToolInput
context: Any = Field(default=None, exclude=True)
max_precedents: int = 5
causal_depth: int = 3
had_live_state: bool = False
reconstructed_state: bool = Field(default=False, exclude=True)
def __init__(
self,
context: Any = None,
max_precedents: int = 5,
causal_depth: int = 3,
**kwargs: Any,
) -> None:
if CREWAI_AVAILABLE:
super().__init__(
context=context,
max_precedents=max_precedents,
causal_depth=causal_depth,
**kwargs,
)
else:
super().__init__()
self.context = context
self.max_precedents = max_precedents
self.causal_depth = causal_depth
# Degraded mode is a plain class — no model_post_init lifecycle.
self._ensure_defaults()
logger.info("SemanticaDecisionTool initialised (crewai=%s)", CREWAI_AVAILABLE)
def model_post_init(self, __context: Any) -> None:
"""Re-create default state after validation/deserialisation.
``context`` is excluded from JSON serialisation (CrewAI checkpoints
serialise every tool via ``model_dump(mode="json")``), so a tool
restored from a checkpoint has ``None`` state until this runs.
"""
self._ensure_defaults()
super().model_post_init(__context)
def _ensure_defaults(self) -> None:
"""Lazy-import and build a real AgentContext when none is wired."""
if self.context is None:
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
self.context = AgentContext(
vector_store=VectorStore(backend="faiss"),
decision_tracking=True,
knowledge_graph=ContextGraph(),
)
if self.had_live_state:
self.reconstructed_state = True
logger.warning(
"SemanticaDecisionTool: the live decision context was lost "
"during serialization/checkpoint restore — an EMPTY "
"context was reconstructed; re-attach the original context "
"before continuing"
)
else:
logger.warning(
"SemanticaDecisionTool created a fresh in-memory "
"AgentContext — agents sharing decision state must be "
"wired to the same context"
)
self.had_live_state = True
# ------------------------------------------------------------------
# CrewAI entry points
# ------------------------------------------------------------------
def _run(
self,
action: str,
category: Optional[str] = None,
scenario: Optional[str] = None,
reasoning: Optional[str] = None,
outcome: Optional[str] = None,
confidence: float = 0.8,
entities: Optional[str] = None,
decision_id: Optional[str] = None,
depth: int = 3,
decision_data: Optional[str] = None,
policy_rules: Optional[str] = None,
**kwargs: Any,
) -> str:
valid = {
"record_decision",
"find_precedents",
"trace_causal_chain",
"analyze_impact",
"check_policy",
}
if action not in valid:
return json.dumps(
{
"error": f"Unknown action '{action}'. Valid actions: "
+ ", ".join(sorted(valid))
}
)
if action == "record_decision":
return self._record_decision(
category=category or "general",
scenario=scenario or "decision recorded",
reasoning=reasoning or "agent decision",
outcome=outcome or "recorded",
confidence=confidence,
entities=entities,
)
if action == "find_precedents":
return self._find_precedents(scenario=scenario or "", category=category)
if action == "trace_causal_chain":
return self._trace_causal_chain(decision_id or "", depth=depth)
if action == "analyze_impact":
return self._analyze_impact(decision_id or "")
return self._check_policy(decision_data or "", policy_rules)
async def _arun(self, action: str, **kwargs: Any) -> str:
"""Async variant of ``_run`` for CrewAI's async tool path."""
return self._run(action=action, **kwargs)
# ------------------------------------------------------------------
# Actions
# ------------------------------------------------------------------
def _record_decision(
self,
category: str,
scenario: str,
reasoning: str,
outcome: str,
confidence: float = 0.8,
entities: Optional[str] = None,
) -> str:
entity_list: Optional[List[str]] = None
if entities:
entity_list = [e.strip() for e in entities.split(",") if e.strip()]
try:
decision_id = self.context.record_decision(
category=category,
scenario=scenario,
reasoning=reasoning,
outcome=outcome,
confidence=float(confidence),
entities=entity_list,
)
result = {"decision_id": str(decision_id), "status": "recorded"}
logger.info("record_decision → %s", decision_id)
except Exception as exc:
result = {"error": str(exc), "status": "failed"}
logger.warning("record_decision failed: %s", exc)
return json.dumps(result)
def _find_precedents(
self,
scenario: str,
category: Optional[str] = None,
limit: Optional[int] = None,
) -> str:
k = limit if limit is not None else self.max_precedents
try:
precedents = self.context.find_precedents_advanced(
scenario=scenario,
category=category,
limit=k,
)
out: List[Dict[str, Any]] = []
for p in (precedents or [])[:k]:
if isinstance(p, dict):
out.append(p)
else:
out.append(
{
"scenario": getattr(p, "scenario", str(p)),
"outcome": getattr(p, "outcome", ""),
"confidence": getattr(p, "confidence", 0.0),
"category": getattr(p, "category", ""),
}
)
logger.info("find_precedents('%s') → %d results", scenario, len(out))
return json.dumps({"precedents": out, "count": len(out)})
except Exception as exc:
logger.warning("find_precedents failed: %s", exc)
return json.dumps({"precedents": [], "count": 0, "error": str(exc)})
def _trace_causal_chain(self, decision_id: str, depth: Optional[int] = None) -> str:
if not decision_id:
return json.dumps(
{
"error": "decision_id is required for trace_causal_chain",
"causal_chain": [],
"decision_id": "",
}
)
max_depth = depth or self.causal_depth
try:
graph = getattr(self.context, "knowledge_graph", None)
if graph is None:
return json.dumps(
{
"error": (
"causal tracing is not available on this knowledge "
"graph (the decision context has no knowledge_graph)"
),
"causal_chain": [],
"decision_id": decision_id,
}
)
trace = getattr(graph, "trace_decision_causality", None)
if trace is None:
return json.dumps(
{
"error": (
"causal tracing is not available on this knowledge graph "
"(graph.trace_decision_causality is not implemented)"
),
"causal_chain": [],
"decision_id": decision_id,
}
)
chain = trace(decision_id, max_depth=max_depth)
return json.dumps({"causal_chain": chain, "decision_id": decision_id})
except Exception as exc:
logger.warning("trace_causal_chain failed: %s", exc)
return json.dumps(
{"error": str(exc), "causal_chain": [], "decision_id": decision_id}
)
def _analyze_impact(self, decision_id: str) -> str:
try:
influence = self.context.analyze_decision_influence(decision_id)
if not isinstance(influence, dict):
influence = {"influence": str(influence)}
influence["decision_id"] = decision_id
return json.dumps(influence)
except Exception as exc:
logger.warning("analyze_impact failed: %s", exc)
return json.dumps({"error": str(exc), "decision_id": decision_id})
def _check_policy(
self,
decision_data: str,
policy_rules: Optional[str] = None,
) -> str:
try:
data = (
json.loads(decision_data)
if isinstance(decision_data, str)
else decision_data
)
except json.JSONDecodeError as exc:
return json.dumps(
{
"compliant": False,
"violations": [f"Invalid decision_data JSON: {exc}"],
"warnings": [],
}
)
if not isinstance(data, dict):
return json.dumps(
{
"compliant": False,
"violations": [
f"decision_data must decode to a JSON object, "
f"got {type(data).__name__}: {data!r}"
],
"warnings": [],
}
)
violations: List[str] = []
warnings: List[str] = []
rules: List[str] = []
if policy_rules:
try:
parsed_rules = json.loads(policy_rules)
except json.JSONDecodeError:
rules = [r.strip() for r in policy_rules.split(",") if r.strip()]
else:
if isinstance(parsed_rules, str):
rules = [parsed_rules]
elif isinstance(parsed_rules, list):
for item in parsed_rules:
if isinstance(item, str):
rules.append(item)
else:
warnings.append(
f"Ignoring non-string policy rule entry: {item!r}"
)
else:
warnings.append(
f"policy_rules must decode to a JSON list of rule strings, "
f"got {type(parsed_rules).__name__}: {parsed_rules!r}"
)
for rule in rules:
try:
if not self._eval_rule(rule, data):
violations.append(f"Rule violated: {rule}")
except Exception as exc:
warnings.append(f"Could not evaluate rule '{rule}': {exc}")
compliant = len(violations) == 0
logger.debug(
"check_policy: compliant=%s, violations=%d", compliant, len(violations)
)
return json.dumps(
{
"compliant": compliant,
"violations": violations,
"warnings": warnings,
}
)
def _eval_rule(self, rule: str, data: Dict[str, Any]) -> bool:
"""Evaluate a simple comparison rule (``field op value``) against data.
This is a small standalone evaluator for the tool's ``check_policy``
action it is intentionally independent of Semantica's policy engine
so agents get a bounded, side-effect-free rule check. Rules are
``<field> <op> <value>`` comparisons only; there is no expression
evaluation (no ``eval``), so untrusted rule strings are safe to pass.
Values are coerced type-aware: ``true``/``false`` (and ``1``/``0``)
become booleans, numeric literals become numbers, and string values
that parse as numbers are compared numerically, so ``score == 0.9``
holds for ``score: "0.90"`` and ``enabled == false`` holds for
``enabled: false``. Field names may contain hyphens, dots and spaces
(e.g. ``risk-score >= 0.9``); they are matched against ``data`` keys
as-is.
"""
m = re.match(r"(.+?)\s*(>=|<=|!=|==|>|<)\s*(.+)$", rule.strip())
if not m:
raise ValueError(f"unrecognised rule format: {rule!r}")
field, op, val_str = m.group(1), m.group(2), m.group(3).strip().strip("\"'")
if field not in data:
raise ValueError(f"rule references undefined field {field!r}")
actual = data[field]
if actual is None:
raise ValueError(f"field {field!r} is null — cannot evaluate rule")
val = self._coerce_value(val_str)
if isinstance(actual, str):
actual = self._coerce_value(actual)
ops = {
">=": lambda a, b: a >= b,
"<=": lambda a, b: a <= b,
"!=": lambda a, b: a != b,
"==": lambda a, b: a == b,
">": lambda a, b: a > b,
"<": lambda a, b: a < b,
}
return ops[op](actual, val)
@staticmethod
def _coerce_value(value: str) -> Any:
"""Parse a rule literal into its most specific Python type."""
text = value.strip()
lowered = text.lower()
if lowered in ("true", "1"):
return True
if lowered in ("false", "0"):
return False
try:
return int(text)
except ValueError:
pass
try:
return float(text)
except ValueError:
pass
return text
# When crewai is absent there is no BaseTool to provide the public
# ``run``/``arun`` entry points, so expose them directly. With crewai
# installed these are left untouched so crewai's own implementations
# (usage tracking, ``result_as_answer``) win.
if not CREWAI_AVAILABLE:
def run(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool synchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
async def arun(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool asynchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
+573
View File
@@ -0,0 +1,573 @@
"""
SemanticaKGTool a CrewAI ``BaseTool`` exposing Semantica's knowledge-graph
pipeline (``NERExtractor``, ``RelationExtractor``, ``ContextGraph``) to agents.
Lets agents build and query a shared ``ContextGraph`` as part of their
reasoning loop.
Install
-------
pip install semantica[crewai]
Example
-------
>>> from integrations.crewai import SemanticaKGTool
>>> from semantica.context import ContextGraph
>>> from crewai import Agent, Crew, Task
>>> graph = ContextGraph()
>>> tool = SemanticaKGTool(graph=graph)
>>> crew = Crew(
... agents=[Agent(role="...", goal="...", backstory="...", tools=[tool])],
... tasks=[...],
... )
Tools exposed
-------------
extract_entities Extract named entities from text
extract_relations Extract relationships between entities
add_to_graph Extract entities/relations from text and add them to the graph
query_graph Query the graph by keyword
find_related Find concepts related to a given entity within ``hops``
"""
from __future__ import annotations
import json
import threading
import weakref
from typing import Any, Dict, List, Literal, Optional, Sequence, Type
from pydantic import BaseModel, Field
from semantica.utils.logging import get_logger
from ._availability import CREWAI_AVAILABLE, CREWAI_IMPORT_ERROR # noqa: F401
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: CrewAI BaseTool base class
# ---------------------------------------------------------------------------
_BaseTool: Any = object
if CREWAI_AVAILABLE:
from crewai.tools import BaseTool as _BaseTool # type: ignore
# One re-entrant lock per graph so concurrent tool invocations sharing a graph
# cannot double-count duplicate adds (check-then-act is not atomic), while
# independent graphs are never serialised against each other. An RLock also
# means an extractor callback that re-enters add_to_graph on the same graph
# cannot deadlock.
_graph_locks_guard = threading.Lock()
_graph_locks: "weakref.WeakKeyDictionary[Any, threading.RLock]" = (
weakref.WeakKeyDictionary()
)
# ---------------------------------------------------------------------------
# Input schema
# ---------------------------------------------------------------------------
class SemanticaKGToolInput(BaseModel):
"""
Input schema for ``SemanticaKGTool``.
Exactly one action is dispatched per call; the remaining fields are only
used by the actions that need them.
"""
action: Literal[
"extract_entities",
"extract_relations",
"add_to_graph",
"query_graph",
"find_related",
] = Field(
...,
description=(
"Which graph operation to run. One of: 'extract_entities', "
"'extract_relations', 'add_to_graph', 'query_graph', 'find_related'."
),
)
text: Optional[str] = Field(
None,
description=(
"Input text. Used by 'extract_entities', 'extract_relations' and "
"'add_to_graph'."
),
)
query: Optional[str] = Field(
None, description="Search query. Used by 'query_graph'."
)
entity: Optional[str] = Field(
None,
description="Root entity name. Used by 'find_related'.",
)
hops: int = Field(
1,
ge=1,
le=10,
description="Maximum relationship hops. Used by 'find_related'.",
)
# ---------------------------------------------------------------------------
# SemanticaKGTool
# ---------------------------------------------------------------------------
class SemanticaKGTool(_BaseTool): # type: ignore[misc]
"""
CrewAI tool that surfaces Semantica's KG pipeline as agent actions.
Parameters
----------
graph:
A ``semantica.context.ContextGraph`` to read/write. A fresh in-memory
graph is used when ``None``.
ner_extractor:
A ``semantica.semantic_extract.NERExtractor`` instance; auto-created
when ``None``.
relation_extractor:
A ``semantica.semantic_extract.RelationExtractor`` instance; auto-
created when ``None``.
"""
name: str = "semantica_knowledge_graph"
description: str = (
"Build and query a semantic knowledge graph. Actions: "
"'extract_entities' (extract named entities from 'text'), "
"'extract_relations' (extract relationships from 'text'), "
"'add_to_graph' (extract entities/relations from 'text' and add them "
"to the shared graph), 'query_graph' (keyword search using 'query'), "
"'find_related' (find concepts related to 'entity' within 'hops' "
"hops). Returns JSON."
)
args_schema: Type[BaseModel] = SemanticaKGToolInput
graph: Any = Field(default=None, exclude=True)
ner_extractor: Any = Field(default=None, exclude=True)
relation_extractor: Any = Field(default=None, exclude=True)
had_live_state: bool = False
reconstructed_state: bool = Field(default=False, exclude=True)
def __init__(
self,
graph: Any = None,
ner_extractor: Any = None,
relation_extractor: Any = None,
**kwargs: Any,
) -> None:
if CREWAI_AVAILABLE:
super().__init__(
graph=graph,
ner_extractor=ner_extractor,
relation_extractor=relation_extractor,
**kwargs,
)
else:
super().__init__()
self.graph = graph
self.ner_extractor = ner_extractor
self.relation_extractor = relation_extractor
# Degraded mode is a plain class — no model_post_init lifecycle.
self._ensure_defaults()
logger.info("SemanticaKGTool initialised (crewai=%s)", CREWAI_AVAILABLE)
def model_post_init(self, __context: Any) -> None:
"""Re-create default state after validation/deserialisation.
``graph``/extractors are excluded from JSON serialisation (CrewAI
checkpoints serialise every tool via ``model_dump(mode="json")``), so a
tool restored from a checkpoint has ``None`` state until this runs.
"""
self._ensure_defaults()
super().model_post_init(__context)
def _ensure_defaults(self) -> None:
"""Lazy-import and build defaults for any missing shared state."""
# Lazy imports keep the module importable without heavy deps
if self.graph is None:
from semantica.context import ContextGraph
self.graph = ContextGraph()
if self.had_live_state:
self.reconstructed_state = True
logger.warning(
"SemanticaKGTool: the live graph was lost during "
"serialization/checkpoint restore — an EMPTY graph was "
"reconstructed; re-attach the original graph before "
"continuing"
)
else:
logger.warning(
"SemanticaKGTool created a fresh in-memory ContextGraph — "
"agents sharing this tool's graph must be wired explicitly"
)
self.had_live_state = True
if self.ner_extractor is None:
from semantica.semantic_extract import NERExtractor
self.ner_extractor = NERExtractor()
if self.relation_extractor is None:
from semantica.semantic_extract import RelationExtractor
self.relation_extractor = RelationExtractor()
# ------------------------------------------------------------------
# CrewAI entry points
# ------------------------------------------------------------------
def _run(
self,
action: str,
text: Optional[str] = None,
query: Optional[str] = None,
entity: Optional[str] = None,
hops: int = 1,
**kwargs: Any,
) -> str:
"""
Dispatch a graph action. Always returns a JSON string so the agent
receives a structured, parseable result.
"""
valid = {
"extract_entities",
"extract_relations",
"add_to_graph",
"query_graph",
"find_related",
}
if action not in valid:
return json.dumps(
{
"error": f"Unknown action '{action}'. Valid actions: "
+ ", ".join(sorted(valid))
}
)
if action == "extract_entities":
return self._extract_entities(text or "")
if action == "extract_relations":
return self._extract_relations(text or "")
if action == "add_to_graph":
return self._add_from_text(text or "")
if action == "query_graph":
return self._query_graph(query or "")
return self._find_related(entity or "", hops=hops)
async def _arun(
self,
action: str,
text: Optional[str] = None,
query: Optional[str] = None,
entity: Optional[str] = None,
hops: int = 1,
**kwargs: Any,
) -> str:
"""
Async variant of ``_run`` for CrewAI's async tool path.
"""
return self._run(
action=action, text=text, query=query, entity=entity, hops=hops, **kwargs
)
# ------------------------------------------------------------------
# Entity/relation field access (handles both Semantica dataclasses and
# third-party shapes like MagicMock/plain dicts in stubs)
# ------------------------------------------------------------------
@staticmethod
def _first_str(obj: Any, attrs: Sequence[str]) -> str:
"""Return the first attribute value that is a non-empty string."""
for attr in attrs:
value = getattr(obj, attr, None)
if isinstance(value, str) and value:
return value
if isinstance(obj, dict):
for key in attrs:
value = obj.get(key)
if isinstance(value, str) and value:
return value
return ""
@classmethod
def _entity_name(cls, e: Any) -> str:
"""Best-effort name for an entity-like object."""
return cls._first_str(e, ("name", "text", "label", "node_id", "id"))
@classmethod
def _entity_type(cls, e: Any) -> str:
"""Best-effort type/label for an entity-like object."""
return cls._first_str(e, ("type", "label")) or "Entity"
@classmethod
def _relation_source(cls, r: Any) -> str:
"""Best-effort source of a relation-like object."""
src = cls._first_str(r, ("source",))
if not src:
src = cls._entity_name(getattr(r, "subject", None))
return src
@classmethod
def _relation_target(cls, r: Any) -> str:
"""Best-effort target of a relation-like object."""
tgt = cls._first_str(r, ("target",))
if not tgt:
tgt = cls._entity_name(getattr(r, "object", None))
return tgt
@classmethod
def _relation_type(cls, r: Any) -> str:
"""Best-effort relation type of a relation-like object."""
rtype = cls._first_str(r, ("type", "relation", "predicate"))
return rtype or "related_to"
@classmethod
def _confidence(cls, e: Any) -> float:
"""Normalise an entity/relation confidence value to a float."""
try:
val = getattr(e, "confidence", None)
if val is None:
return 1.0
return round(float(val), 4)
except (TypeError, ValueError):
return 1.0
@classmethod
def _graph_lock(cls, graph: Any) -> threading.RLock:
"""Return the re-entrant lock guarding a specific graph."""
with _graph_locks_guard:
lock = _graph_locks.get(graph)
if lock is None:
lock = threading.RLock()
_graph_locks[graph] = lock
return lock
# ------------------------------------------------------------------
# Actions
# ------------------------------------------------------------------
def _extract_entities(self, text: str) -> str:
"""Extract named entities from ``text``."""
try:
raw = self.ner_extractor.extract_entities(text) or []
entities = [
{
"name": self._entity_name(e),
"type": self._entity_type(e),
"confidence": self._confidence(e),
}
for e in raw
if self._entity_name(e)
]
logger.debug("extract_entities → %d entities", len(entities))
return json.dumps({"entities": entities, "count": len(entities)})
except Exception as exc:
logger.warning("extract_entities failed: %s", exc)
return json.dumps({"entities": [], "count": 0, "error": str(exc)})
def _extract_relations(self, text: str) -> str:
"""Extract relationships between entities in ``text``."""
try:
raw = self.relation_extractor.extract_relations(text) or []
relations = [
{
"source": self._relation_source(r),
"relation": self._relation_type(r),
"target": self._relation_target(r),
"confidence": self._confidence(r),
}
for r in raw
]
logger.debug("extract_relations → %d relations", len(relations))
return json.dumps({"relations": relations, "count": len(relations)})
except Exception as exc:
logger.warning("extract_relations failed: %s", exc)
return json.dumps({"relations": [], "count": 0, "error": str(exc)})
def _add_from_text(self, text: str) -> str:
"""
Extract entities and relations from ``text`` and add them to the graph.
Duplicate nodes/edges (same id, or same source/type/target) are
skipped so repeated calls are idempotent. Returns JSON with the
number of nodes/edges added.
"""
nodes_added = 0
edges_added = 0
try:
with self._graph_lock(self.graph):
existing_nodes = {
n.get("id") or n.get("node_id")
for n in (
self.graph.find_nodes() or [] # type: ignore[attr-defined]
)
if n.get("id") or n.get("node_id")
}
existing_edges = {
(e.get("source"), e.get("type") or "related_to", e.get("target"))
for e in (
self.graph.find_edges() or [] # type: ignore[attr-defined]
)
if e.get("source") and e.get("target")
}
raw_entities = self.ner_extractor.extract_entities(text) or []
entities: List[Any] = []
seen: set = set()
for e in raw_entities:
name = self._entity_name(e)
ntype = self._entity_type(e)
if not name or name in seen:
continue
seen.add(name)
entities.append(e)
if name in existing_nodes:
continue
try:
if self.graph.add_node(node_id=name, node_type=ntype):
nodes_added += 1
existing_nodes.add(name)
except Exception as exc:
logger.debug("add_node(%r) failed: %s", name, exc)
raw_relations = (
self.relation_extractor.extract_relations(text, entities=entities)
or []
)
for r in raw_relations:
src = self._relation_source(r)
tgt = self._relation_target(r)
rtype = self._relation_type(r)
if not src or not tgt:
continue
key = (src, rtype, tgt)
if key in existing_edges:
continue
try:
if self.graph.add_edge(
source_id=src, target_id=tgt, edge_type=rtype
):
edges_added += 1
existing_edges.add(key)
except Exception as exc:
logger.debug("add_edge(%r) failed: %s", key, exc)
logger.debug("add_to_graph: +%d nodes, +%d edges", nodes_added, edges_added)
return json.dumps({"nodes_added": nodes_added, "edges_added": edges_added})
except Exception as exc:
logger.warning("add_to_graph failed: %s", exc)
return json.dumps({"nodes_added": 0, "edges_added": 0, "error": str(exc)})
def _query_graph(self, query: str) -> str:
"""Keyword-search graph nodes by id, type and content."""
try:
q = (query or "").strip().lower()
out: List[dict] = []
seen: set = set()
query_method = getattr(self.graph, "query", None)
if query_method is not None:
for match in query_method(query) or []:
node = match.get("node") or {}
nid = node.get("id", "") or node.get("node_id", "")
if not nid or nid in seen:
continue
seen.add(nid)
content = match.get("content") or node.get("content", "")
out.append(
{
"id": nid,
"type": node.get("type", "") or node.get("node_type", ""),
"label": nid,
"content": str(content)[:500],
"score": round(float(match.get("score") or 0.0), 4),
}
)
if q:
for n in self.graph.find_nodes() or []: # type: ignore[attr-defined]
if isinstance(n, dict):
nid = n.get("id", "") or n.get("node_id", "")
ntype = n.get("type", "") or n.get("node_type", "")
content = str(
n.get("content")
or (n.get("properties") or {}).get("content", "")
or ""
)
else:
nid = getattr(n, "id", getattr(n, "label", ""))
ntype = getattr(n, "node_type", "")
content = str(getattr(n, "content", "") or "")
if not nid or nid in seen:
continue
if q in str(nid).lower() or q in str(ntype).lower():
seen.add(nid)
out.append(
{
"id": nid,
"type": ntype,
"label": nid,
"content": content[:500],
"score": 1.0,
}
)
return json.dumps({"results": out, "count": len(out)})
except Exception as exc:
logger.warning("query_graph failed: %s", exc)
return json.dumps({"results": [], "count": 0, "error": str(exc)})
def _find_related(self, entity: str, hops: int = 1) -> str:
"""Find concepts related to ``entity`` within ``hops`` graph hops.
Traversal is undirected an edge counts as related regardless of
direction, so both outgoing and incoming edges are honored.
"""
try:
adjacency: Dict[str, List[str]] = {}
for edge in self.graph.find_edges() or []: # type: ignore[attr-defined]
if isinstance(edge, dict):
src = edge.get("source")
tgt = edge.get("target")
else:
src = getattr(edge, "source", None)
tgt = getattr(edge, "target", None)
if not src or not tgt:
continue
adjacency.setdefault(src, []).append(tgt)
adjacency.setdefault(tgt, []).append(src)
related: List[str] = []
frontier = [entity]
visited = {entity}
for _ in range(max(1, hops)):
next_frontier: List[str] = []
for e in frontier:
for n in adjacency.get(e, []):
if n in visited:
continue
visited.add(n)
next_frontier.append(n)
related.append(n)
frontier = next_frontier
logger.debug("find_related('%s', hops=%d) → %d", entity, hops, len(related))
return json.dumps(
{"entity": entity, "related": related, "count": len(related)}
)
except Exception as exc:
logger.warning("find_related failed: %s", exc)
return json.dumps(
{"entity": entity, "related": [], "count": 0, "error": str(exc)}
)
# When crewai is absent there is no BaseTool to provide the public
# ``run``/``arun`` entry points, so expose them directly. With crewai
# installed these are left untouched so crewai's own implementations
# (usage tracking, ``result_as_answer``) win.
if not CREWAI_AVAILABLE:
def run(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool synchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
async def arun(self, *args: Any, **kwargs: Any) -> str:
"""Run the tool asynchronously (degraded mode, no crewai)."""
return self._run(*args, **kwargs)
+331
View File
@@ -0,0 +1,331 @@
"""
SemanticaKnowledgeSource expose a Semantica ``ContextGraph`` as a CrewAI
knowledge source.
Lets a ``Crew`` load the current state of a knowledge graph (nodes, edges,
metadata) into its knowledge storage, so every agent gets retrieval access to
graph knowledge during the kickoff.
Install
-------
pip install semantica[crewai]
Example
-------
>>> from integrations.crewai import SemanticaKnowledgeSource
>>> from semantica.context import ContextGraph
>>> from crewai import Agent, Crew, Task
>>> graph = ContextGraph()
>>> graph.add_node(node_id="privacy", node_type="policy")
>>> crew = Crew(
... agents=[...],
... tasks=[...],
... knowledge_sources=[SemanticaKnowledgeSource(graph=graph)],
... )
Compatibility
-------------
Works with ``crewai >= 0.80.0``. The ``BaseKnowledgeSource`` contract changed
between versions (``load_content`` ``validate_content``/``aadd``), so this
source implements both legacy and current methods. It degrades gracefully
when ``crewai`` is not installed: the class is still importable and carries the
full Semantica API, but cannot be passed to a ``Crew``.
"""
from __future__ import annotations
import asyncio
from typing import Any, Dict, List, Optional
from pydantic import Field
from semantica.utils.logging import get_logger
from ._availability import CREWAI_AVAILABLE
logger = get_logger(__name__)
# ---------------------------------------------------------------------------
# Optional: CrewAI BaseKnowledgeSource base class
# ---------------------------------------------------------------------------
_BaseKnowledgeSource: Any = object
if CREWAI_AVAILABLE:
from crewai.knowledge.source.base_knowledge_source import (
BaseKnowledgeSource as _BaseKnowledgeSource, # type: ignore
)
def _chunk_text_manual(text: str, chunk_size: int, chunk_overlap: int) -> List[str]:
"""Fallback plain-text chunker for when CrewAI helpers are unavailable."""
if not text:
return []
if int(chunk_size) <= 0:
return [text]
size = max(1, int(chunk_size))
overlap = max(0, int(chunk_overlap))
if len(text) <= size:
return [text]
step = max(1, size - overlap)
return [text[i : i + size] for i in range(0, len(text), step)]
class SemanticaKnowledgeSource(_BaseKnowledgeSource): # type: ignore[misc]
"""
CrewAI knowledge source backed by a Semantica ``ContextGraph``.
On ``add()`` the graph's nodes and edges are serialised into readable text
and pushed through the standard CrewAI chunking / storage pipeline, making
graph knowledge retrievable by every agent in the crew.
Parameters
----------
graph:
A ``semantica.context.ContextGraph`` to expose. A fresh in-memory
graph is created when ``None``.
name:
Source name. Defaults to ``"semantica_knowledge_graph"``.
chunk_size:
Max characters per chunk (default 4000).
chunk_overlap:
Character overlap between adjacent chunks (default 200).
"""
name: str = "semantica_knowledge_graph"
graph: Any = Field(default=None, exclude=True)
chunk_size: int = 4000
chunk_overlap: int = 200
had_live_state: bool = False
reconstructed_state: bool = Field(default=False, exclude=True)
def __init__(
self,
graph: Any = None,
name: Optional[str] = None,
chunk_size: int = 4000,
chunk_overlap: int = 200,
**kwargs: Any,
) -> None:
if CREWAI_AVAILABLE:
# Do NOT eagerly build a graph here: pydantic calls this ``__init__``
# during ``model_validate`` (checkpoint restore), and the eager
# build would hide that a live graph was lost. ``model_post_init``
# rebuilds defaults and flags ``reconstructed_state`` instead.
super().__init__(
graph=graph,
name=name or "semantica_knowledge_graph",
chunk_size=int(chunk_size),
chunk_overlap=int(chunk_overlap),
**kwargs,
)
else:
if graph is None:
from semantica.context import ContextGraph
graph = ContextGraph()
super().__init__()
self.graph = graph
self.name = name or "semantica_knowledge_graph"
self.chunk_size = int(chunk_size)
self.chunk_overlap = int(chunk_overlap)
logger.info(
"SemanticaKnowledgeSource initialised (crewai=%s, chunk_size=%d)",
CREWAI_AVAILABLE,
self.chunk_size,
)
self.had_live_state = True
def model_post_init(self, __context: Any) -> None:
"""Re-create default state after validation/deserialisation.
``graph`` is excluded from JSON serialisation (CrewAI checkpoints
serialise their models via ``model_dump(mode="json")``), so a source
restored from a checkpoint has ``None`` state until this runs.
"""
if self.graph is None:
from semantica.context import ContextGraph
self.graph = ContextGraph()
if self.had_live_state:
self.reconstructed_state = True
logger.warning(
"SemanticaKnowledgeSource: the live graph was lost during "
"serialization/checkpoint restore — an EMPTY graph was "
"reconstructed; re-attach the original graph before "
"continuing"
)
else:
logger.warning(
"SemanticaKnowledgeSource created a fresh in-memory "
"ContextGraph — sources sharing knowledge must be wired to "
"the same graph explicitly"
)
self.had_live_state = True
super().model_post_init(__context)
# ------------------------------------------------------------------
# Content extraction
# ------------------------------------------------------------------
def load_content(self) -> Dict[str, str]:
"""
Serialise the graph into ``{id: readable_text}`` pairs.
Nodes are rendered with their type/content/metadata, edges with their
source, relation type and target. This satisfies the legacy CrewAI
``BaseKnowledgeSource.load_content`` contract.
"""
content: Dict[str, str] = {}
graph = self.graph
if graph is None:
return content
try:
for node in graph.find_nodes() or []: # type: ignore[attr-defined]
nid = node.get("id") or node.get("node_id") or ""
if not nid:
continue
parts = [
"Entity",
str(nid),
"type: " + str(node.get("type", "entity")),
]
if node.get("content"):
parts.append("content: " + str(node["content"]))
if node.get("metadata"):
try:
import json
parts.append("metadata: " + json.dumps(node["metadata"]))
except Exception:
parts.append("metadata: " + str(node["metadata"]))
content[str(nid)] = " | ".join(parts)
except Exception as exc:
logger.warning(
"SemanticaKnowledgeSource.load_content (nodes) failed: %s", exc
)
try:
for idx, edge in enumerate(
graph.find_edges() or [] # type: ignore[attr-defined]
):
src = edge.get("source")
tgt = edge.get("target")
if not src or not tgt:
continue
rel = edge.get("type") or edge.get("edge_type") or "related_to"
weight = edge.get("weight")
text = f"{src} -[{rel}]-> {tgt}"
if weight is not None:
text += f" (weight: {weight})"
content[f"edge-{idx}"] = text
except Exception as exc:
logger.warning(
"SemanticaKnowledgeSource.load_content (edges) failed: %s", exc
)
return content
def validate_content(self) -> Any:
"""
Validate that a readable graph is attached.
Satisfies the current CrewAI ``BaseKnowledgeSource.validate_content``
contract.
"""
if self.graph is None:
raise ValueError("SemanticaKnowledgeSource requires a ContextGraph.")
return True
# ------------------------------------------------------------------
# Chunking + storage (abstract in both CrewAI generations)
# ------------------------------------------------------------------
def _chunk(self, text: str) -> List[str]:
"""Chunk ``text`` using CrewAI's helper when available, else manual."""
helper = getattr(self, "_chunk_text", None)
if helper is not None:
try:
return list(helper(text) or [])
except Exception as exc:
logger.debug(
"SemanticaKnowledgeSource._chunk_text failed, falling back: %s", exc
)
return _chunk_text_manual(text, self.chunk_size, self.chunk_overlap)
def add(self) -> None:
"""
Process the graph into chunks and store them via CrewAI storage.
Sets both ``chunks`` (current CrewAI) and ``_chunks`` (legacy CrewAI)
so either ``_save_documents`` implementation picks them up. If no
storage has been wired (e.g. not yet attached to a ``Crew``), chunks
are kept in memory.
"""
content = self.load_content()
if not content:
logger.debug("SemanticaKnowledgeSource.add: empty graph — nothing to store")
return
chunks: List[str] = []
for _, text in content.items():
if text:
chunks.extend(self._chunk(text))
self.chunks = chunks
self._chunks = chunks
save = getattr(self, "_save_documents", None)
if save is not None:
if getattr(self, "storage", None) is None:
logger.debug(
"SemanticaKnowledgeSource.add: storage not wired — "
"keeping chunks in memory"
)
else:
try:
save()
logger.info(
"SemanticaKnowledgeSource.add: stored %d chunks", len(chunks)
)
return
except Exception as exc:
logger.error(
"SemanticaKnowledgeSource.add: storage save FAILED (%s) — "
"chunks are only kept in memory and agents will retrieve "
"nothing. Configure the Crew embedder (e.g. an OpenAI "
"embedder with OPENAI_API_KEY, or a local embedder) before "
"running the crew.",
exc,
)
logger.info(
"SemanticaKnowledgeSource.add: %d chunks ready in memory", len(chunks)
)
async def aadd(self) -> None:
"""
Asynchronous variant of ``add()`` (current CrewAI contract).
The graph serialisation is CPU-bound, so it runs in a thread pool to
avoid blocking the event loop.
"""
loop = asyncio.get_running_loop()
await loop.run_in_executor(None, self.add)
# ------------------------------------------------------------------
# Inspection helpers
# ------------------------------------------------------------------
def get_content_summary(self) -> Dict[str, Any]:
"""
Summarise what the source exposes (helpful for debugging / testing).
"""
content = self.load_content()
return {
"name": self.name,
"source_count": len(content),
"chunks": len(getattr(self, "chunks", []) or []),
"crewai_available": CREWAI_AVAILABLE,
}
+8
View File
@@ -201,6 +201,10 @@ gpu = [
# ---- Agentic Framework Integrations ----
agno = ["agno>=1.0.0"]
# crewai core provides BaseTool and BaseKnowledgeSource; crewai-tools is not
# needed (it pulls vulnerable transitive deps like chromadb) and would only
# duplicate the prebuilt tooling users can install separately.
crewai = ["crewai>=0.80.0"]
# ---- File Watching ----
watch = ["watchdog>=6.0.0"]
@@ -242,6 +246,10 @@ explorer-lite = [
]
# Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
# NOTE: the ``crewai`` extra is intentionally NOT in ``all``: crewai hard-requires
# ``chromadb~=1.1.0``, which carries a pre-authentication code-injection advisory
# (CVE-2026-45829) with no fixed release — including it here would fail the CI
# dependency-audit/security gates. Install it explicitly via ``semantica[crewai]``.
all = [
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
+4 -4
View File
@@ -1,5 +1,5 @@
# This file was autogenerated by uv via the following command:
# uv pip compile -p 3.11 --extra all --generate-hashes -o requirements-ci.txt pyproject.toml
# uv pip compile pyproject.toml --python-version 3.11 --extra all --generate-hashes -o requirements-ci.txt
accelerate==1.14.0 \
--hash=sha256:41b9c4377a54e0b460a959b0defa1b736e4ca0a2373252d9a539964c2afe3c8d \
--hash=sha256:e94390c2863b873be18f623f9df48a0d8fe5eff13ea7f1a00092b0a7904888c6
@@ -4123,9 +4123,9 @@ pooch==1.9.0 \
--hash=sha256:de46729579b9857ffd3e741987a2f6d5e0e03219892c167c6578c0091fb511ed \
--hash=sha256:f265597baa9f760d25ceb29d0beb8186c243d6607b0f60b83ecf14078dbc703b
# via librosa
portalocker==3.2.0 \
--hash=sha256:1f3002956a54a8c3730586c5c77bf18fae4149e07eaf1c29fc3faf4d5a3f89ac \
--hash=sha256:3cdc5f565312224bc570c49337bd21428bba0ef363bbcf58b9ef4a9f11779968
portalocker==2.7.0 \
--hash=sha256:032e81d534a88ec1736d03f780ba073f047a06c478b06e2937486f334e955c51 \
--hash=sha256:a07c5b4f3985c3cf4798369631fb7011adb498e2a46d8440efc75a8f29a0f983
# via qdrant-client
pre-commit==4.6.2 \
--hash=sha256:8f5d7bfb021ecdbcd9d49d89847082dd24172ccde534390081a679ad046e2441 \
+550
View File
@@ -188,6 +188,26 @@ def _normalize_temporal_input(value: Optional[Union[str, int, float, datetime]])
raise ValueError("Temporal values must be datetime, epoch seconds, ISO strings, or None")
def _closing_valid_until(current: Optional[str], at_iso: str) -> str:
"""Return the earlier of an existing end bound and a retraction time.
Retraction closes a validity window and must never widen one: an entity
added with ``valid_until`` already in the past would otherwise be reported
active by ``is_active``/``state_at`` for the span between its original end
and the retraction. An unparseable ``current`` imposes no end bound at all
(see :func:`_parse_iso_dt`), so ``at_iso`` still closes it.
"""
if current is None:
return at_iso
existing = _parse_iso_dt(current)
if existing is None:
return at_iso
requested = _parse_iso_dt(at_iso)
if requested is None or existing <= requested:
return current
return at_iso
def _pick_first(*values: Any) -> Any:
for value in values:
if value is None:
@@ -464,6 +484,7 @@ class ContextGraph:
self.nodes: Dict[str, ContextNode] = {}
self.edges: List[ContextEdge] = []
self._edge_index: Dict[str, ContextEdge] = {}
self._adjacency: Dict[str, List[ContextEdge]] = defaultdict(list)
@@ -475,6 +496,15 @@ class ContextGraph:
self._unresolved_links: Dict[str, Dict[str, str]] = {}
# Retraction closes an entity's validity window but keeps it in the
# graph; a tombstone records that an entity was purged outright,
# without retaining the purged content. Keyed by
# ``(entity_kind, entity_id)`` -- node ids are caller-supplied strings
# and edge ids are UUID strings, so a single id keyspace would let a
# node record mask an edge of the same id, and vice versa.
self._retractions: Dict[Tuple[str, str], Dict[str, Any]] = {}
self._tombstones: Dict[Tuple[str, str], Dict[str, Any]] = {}
self.progress_tracker = get_progress_tracker()
@@ -1120,11 +1150,16 @@ class ContextGraph:
# Clear existing
self.nodes.clear()
self.edges.clear()
self._edge_index.clear()
self._adjacency.clear()
self.node_type_index.clear()
self.edge_type_index.clear()
self._linked_graphs.clear()
self._unresolved_links.clear()
# Deletion metadata belongs to the graph being replaced; keeping it
# would make entities in the loaded graph read as already retracted.
self._retractions.clear()
self._tombstones.clear()
if "graph_id" in data:
self.graph_id = data["graph_id"]
@@ -1518,16 +1553,384 @@ class ContextGraph:
max_edges = n * (n - 1)
return len(self.edges) / max_edges
def retract_node(
self,
node_id: str,
reason: Optional[str] = None,
at: Optional[Union[str, datetime]] = None,
cascade: bool = True,
) -> bool:
"""Retract a node: no longer active, but still visible in history.
Closes the node's validity window rather than deleting it, so
:meth:`state_at` before ``at`` still returns the node and any decision
recorded against it remains explainable. Use :meth:`purge_node` when
the data itself has to be gone.
Args:
node_id: Node to retract.
reason: Why it was retracted, stored on the retraction record.
at: When the retraction takes effect (ISO string or datetime).
Defaults to now, UTC.
cascade: Also retract every edge touching the node. Leaving edges
active around an inactive node means :meth:`find_active_nodes`
drops the node while its relationships still read as current,
so the default keeps the active view self-consistent.
Retraction is expressed through the temporal window, so it is visible
to the activity-aware views -- :meth:`find_active_nodes`,
:meth:`state_at`, ``ContextNode.is_active`` -- and not to membership
checks like :meth:`has_node` or :meth:`stats`, which continue to count
the retained record. That matches how ``valid_until`` already behaved
before retraction existed.
A node whose ``valid_until`` is already earlier than ``at`` keeps that
earlier bound: retraction only ever closes a validity window, never
widens one.
Returns:
True if the node was retracted; False if it does not exist or was
already retracted.
Note:
Emits ``UPDATE_NODE`` to the audit-trail callback, since retraction
changes the validity window rather than removing the record.
"""
at_iso = _normalize_temporal_input(at) or datetime.now(timezone.utc).isoformat()
with self._lock:
node = self.nodes.get(node_id)
if node is None:
self.logger.warning("Cannot retract unknown node: %r", node_id)
return False
if ("node", node_id) in self._retractions:
return False
node.valid_until = _closing_valid_until(node.valid_until, at_iso)
record = {
"entity_id": node_id,
"entity_kind": "node",
"retracted_at": at_iso,
"reason": reason,
}
self._retractions[("node", node_id)] = record
node_payload = {**node.to_dict(), "retraction": dict(record)}
cascaded: List[Tuple[str, Dict[str, Any]]] = []
if cascade:
# Snapshotted once, before the loop: edge_id is content-derived
# and not guaranteed unique (#922), so two distinct edge objects
# can share one id. Checking the live _retractions dict inside
# the loop would let the first duplicate's record block the
# second from ever being closed, leaving it active indefinitely
# while its retraction record claimed otherwise.
already_retracted_edge_ids = {
key[1] for key in self._retractions if key[0] == "edge"
}
for edge in self._incident_edges(node_id):
if edge.edge_id in already_retracted_edge_ids:
continue
edge.valid_until = _closing_valid_until(edge.valid_until, at_iso)
edge_record = {
"entity_id": edge.edge_id,
"entity_kind": "edge",
"retracted_at": at_iso,
"reason": reason,
"cascaded_from": node_id,
}
self._retractions[("edge", edge.edge_id)] = edge_record
# Payloads are snapshotted here, not read back after the
# lock is released: a concurrent clear() would otherwise
# wipe the record out from under the emission below.
cascaded.append(
(
edge.edge_id,
{**edge.to_dict(), "retraction": dict(edge_record)},
)
)
self._emit_mutation("UPDATE_NODE", node_id, node_payload)
for edge_id, edge_payload in cascaded:
self._emit_mutation("UPDATE_EDGE", edge_id, edge_payload)
self.logger.info(
"Retracted node %r at %s (cascaded %d edge(s))",
node_id,
at_iso,
len(cascaded),
)
return True
def retract_edge(
self,
edge_id: str,
reason: Optional[str] = None,
at: Optional[Union[str, datetime]] = None,
) -> bool:
"""Retract a single edge, leaving its endpoints untouched.
An edge whose ``valid_until`` is already earlier than ``at`` keeps that
earlier bound; retraction never widens a validity window.
Args:
edge_id: Edge to retract.
reason: Why it was retracted.
at: When the retraction takes effect. Defaults to now, UTC.
Returns:
True if the edge was retracted; False if it does not exist or was
already retracted.
Note:
``edge_id`` is content-derived and not guaranteed unique (#922):
two distinct edge objects can share one id. Every edge matching
``edge_id`` is closed under a single retraction record, so a
duplicate can never be left silently active while the record
claims it was retracted.
"""
at_iso = _normalize_temporal_input(at) or datetime.now(timezone.utc).isoformat()
with self._lock:
edges = [e for e in self.edges if e.edge_id == edge_id]
if not edges:
self.logger.warning("Cannot retract unknown edge: %r", edge_id)
return False
if ("edge", edge_id) in self._retractions:
return False
record = {
"entity_id": edge_id,
"entity_kind": "edge",
"retracted_at": at_iso,
"reason": reason,
}
self._retractions[("edge", edge_id)] = record
for edge in edges:
edge.valid_until = _closing_valid_until(edge.valid_until, at_iso)
payload = {**edges[0].to_dict(), "retraction": dict(record)}
self._emit_mutation("UPDATE_EDGE", edge_id, payload)
self.logger.info(
"Retracted edge %r at %s (%d underlying record(s))",
edge_id,
at_iso,
len(edges),
)
return True
def purge_node(
self,
node_id: str,
reason: Optional[str] = None,
at: Optional[Union[str, datetime]] = None,
cascade: bool = True,
) -> bool:
"""Permanently remove a node; history no longer contains it.
Unlike :meth:`retract_node` this is destructive: the node disappears
from :meth:`state_at` as well as from the active view. Only a tombstone
remains, recording that a purge happened and why -- deliberately
without the purged content, since retaining it would defeat the point.
Scope is this graph only. Copies held elsewhere (``AgentMemory``, a
bound vector store, an exported file) are not reached, so this is one
step of an erasure workflow, not the whole of it.
Args:
node_id: Node to purge.
reason: Why it was purged, e.g. an erasure-request reference.
at: When the purge takes effect, recorded as the tombstone's
``purged_at`` (ISO string or datetime). Defaults to now, UTC.
cascade: Also purge every edge touching the node, and the marker
node of any cross-graph link it exits through. Defaults to True
because leaving edges pointing at a removed node produces
dangling endpoints.
Cross-graph links registered by :meth:`link_graph` out of this node are
deregistered either way -- a link whose source no longer exists would
still resolve through :meth:`navigate_to` and still be serialized by
:meth:`save_to_file`.
Returns:
True if the node was purged; False if it does not exist.
Note:
Emits ``REMOVE_NODE``/``REMOVE_EDGE`` to the audit-trail callback.
"""
purged_at = (
_normalize_temporal_input(at) or datetime.now(timezone.utc).isoformat()
)
with self._lock:
if node_id not in self.nodes:
self.logger.warning("Cannot purge unknown node: %r", node_id)
return False
# The link marker node is scaffolding reachable only from the node
# being purged, so it goes with the cascade rather than surviving as
# an orphan. Resolve the markers before deregistering the links they
# are derived from.
targets = [node_id]
if cascade:
targets.extend(self._cross_graph_marker_nodes(node_id))
for link_id in self._cross_graph_links_for(node_id):
self._linked_graphs.pop(link_id, None)
self._unresolved_links.pop(link_id, None)
# Tombstones are snapshotted into locals before the lock is
# released; reading them back afterwards would race a clear().
purged_edges: List[Tuple[str, Dict[str, Any]]] = []
purged_nodes: List[Tuple[str, Dict[str, Any]]] = []
for target in targets:
cascaded_from = None if target == node_id else node_id
if cascade:
for edge in self._incident_edges(target):
self._drop_edge_from_indexes(edge)
edge_record = {
"entity_id": edge.edge_id,
"entity_kind": "edge",
"purged_at": purged_at,
"reason": reason,
"cascaded_from": node_id,
}
self._tombstones[("edge", edge.edge_id)] = edge_record
self._retractions.pop(("edge", edge.edge_id), None)
purged_edges.append((edge.edge_id, dict(edge_record)))
self._drop_node_from_indexes(target)
node_record = {
"entity_id": target,
"entity_kind": "node",
"purged_at": purged_at,
"reason": reason,
}
if cascaded_from is not None:
node_record["cascaded_from"] = cascaded_from
self._tombstones[("node", target)] = node_record
self._retractions.pop(("node", target), None)
purged_nodes.append((target, dict(node_record)))
for edge_id, payload in purged_edges:
self._emit_mutation("REMOVE_EDGE", edge_id, payload)
for purged_id, payload in purged_nodes:
self._emit_mutation("REMOVE_NODE", purged_id, payload)
self.logger.info(
"Purged node %r (cascaded %d edge(s), %d node(s))",
node_id,
len(purged_edges),
len(purged_nodes) - 1,
)
return True
def purge_edge(
self,
edge_id: str,
reason: Optional[str] = None,
at: Optional[Union[str, datetime]] = None,
) -> bool:
"""Permanently remove a single edge, leaving its endpoints in place.
If the edge is the bridge of a cross-graph link, the link is also
deregistered -- :meth:`navigate_to` should not keep resolving a link
whose bridge is gone. The marker node itself is an endpoint and is left
in place; purge it directly, or purge the link's source node, to remove
it too.
Args:
edge_id: Edge to purge.
reason: Why it was purged.
at: When the purge takes effect, recorded as the tombstone's
``purged_at``. Defaults to now, UTC.
Returns:
True if the edge was purged; False if it does not exist.
Note:
``edge_id`` is content-derived and not guaranteed unique (#922):
two distinct edge objects can share one id. Every edge matching
``edge_id`` is dropped under a single tombstone, so a duplicate
can never be left live in the graph while the tombstone claims
the edge is gone.
"""
purged_at = (
_normalize_temporal_input(at) or datetime.now(timezone.utc).isoformat()
)
with self._lock:
edges = [e for e in self.edges if e.edge_id == edge_id]
if not edges:
self.logger.warning("Cannot purge unknown edge: %r", edge_id)
return False
for edge in edges:
self._drop_edge_from_indexes(edge)
link_id = (edge.metadata or {}).get("link_id")
if (edge.metadata or {}).get("cross_graph") and link_id:
self._linked_graphs.pop(link_id, None)
self._unresolved_links.pop(link_id, None)
self._retractions.pop(("edge", edge_id), None)
record = {
"entity_id": edge_id,
"entity_kind": "edge",
"purged_at": purged_at,
"reason": reason,
}
self._tombstones[("edge", edge_id)] = record
payload = dict(record)
self._emit_mutation("REMOVE_EDGE", edge_id, payload)
self.logger.info(
"Purged edge %r (%d underlying record(s))", edge_id, len(edges)
)
return True
def get_retraction(
self, entity_id: str, entity_kind: Optional[str] = None
) -> Optional[Dict[str, Any]]:
"""Return the retraction record for a node or edge, or None.
Args:
entity_id: Node id or edge id.
entity_kind: ``"node"`` or ``"edge"``. Records are keyed by kind as
well as id, so pass this when a node id and an edge id could
collide; without it a node record is preferred over an edge one.
"""
with self._lock:
return self._find_removal_record(self._retractions, entity_id, entity_kind)
def get_tombstone(
self, entity_id: str, entity_kind: Optional[str] = None
) -> Optional[Dict[str, Any]]:
"""Return the purge tombstone for a node or edge, or None.
The tombstone records that a purge happened, when, and why. It never
contains the purged content.
Args:
entity_id: Node id or edge id.
entity_kind: ``"node"`` or ``"edge"``; disambiguates a node id that
collides with an edge id, as for :meth:`get_retraction`.
"""
with self._lock:
return self._find_removal_record(self._tombstones, entity_id, entity_kind)
def list_retractions(self) -> List[Dict[str, Any]]:
"""Return every retraction record."""
with self._lock:
return [dict(record) for record in self._retractions.values()]
def list_tombstones(self) -> List[Dict[str, Any]]:
"""Return every purge tombstone."""
with self._lock:
return [dict(record) for record in self._tombstones.values()]
def clear(self) -> None:
"""Fully reset the graph state and indexes."""
with self._lock:
self.nodes.clear()
self.edges.clear()
self._edge_index.clear()
self._adjacency.clear()
self.node_type_index.clear()
self.edge_type_index.clear()
self._linked_graphs.clear()
self._unresolved_links.clear()
self._retractions.clear()
self._tombstones.clear()
self.logger.debug("Graph state fully cleared.")
# --- Internal Helpers ---
@@ -1595,6 +1998,11 @@ class ContextGraph:
self.logger.warning("Skipping internal edge with invalid endpoints: %r", edge)
return False
with self._lock:
# Edge identity is content-derived, so an existing edge_id means this
# exact edge is already stored; re-adding it is a no-op (issue #922).
if edge.edge_id in self._edge_index:
return False
# Ensure nodes exist
if edge.source_id not in self.nodes:
self._add_internal_node(
@@ -1605,6 +2013,7 @@ class ContextGraph:
ContextNode(edge.target_id, "entity", edge.target_id)
)
self._edge_index[edge.edge_id] = edge
self.edges.append(edge)
self.edge_type_index[edge.edge_type].append(edge)
self._adjacency[edge.source_id].append(edge)
@@ -1620,6 +2029,147 @@ class ContextGraph:
)
return True
def _emit_mutation(
self, operation: str, entity_id: str, payload: Dict[str, Any]
) -> None:
"""Fire the audit-trail callback, mirroring the add paths.
Kept in one place so retraction and purge record themselves the same
way ``_add_internal_node``/``_add_internal_edge`` already do, including
the ``_suspend_mutation_callback`` guard used during restores.
"""
if not getattr(self, "mutation_callback", None):
return
if getattr(self, "_suspend_mutation_callback", False):
return
try:
self.mutation_callback(operation, entity_id, payload)
except Exception as e:
self.logger.warning(
f"Audit trail callback failed for {operation} {entity_id}: {e}"
)
def _incident_edges(self, node_id: str) -> List[ContextEdge]:
"""Every edge touching ``node_id``, in either direction.
``_adjacency`` is keyed by source only, so incoming edges have to come
from a scan of ``self.edges``; relying on ``_adjacency`` alone would
silently leave inbound edges pointing at a removed node.
"""
return [
edge
for edge in self.edges
if edge.source_id == node_id or edge.target_id == node_id
]
@staticmethod
def _find_removal_record(
store: Dict[Tuple[str, str], Dict[str, Any]],
entity_id: str,
entity_kind: Optional[str],
) -> Optional[Dict[str, Any]]:
"""Look a retraction/tombstone up by id, optionally narrowed by kind.
The caller must hold ``self._lock``. Records are keyed by
``(entity_kind, entity_id)``; with no kind given, both keyspaces are
tried so callers that know an id is unambiguous can pass it alone.
"""
if entity_kind is not None:
if entity_kind not in ("node", "edge"):
raise ValueError(
f"entity_kind must be 'node', 'edge' or None, got {entity_kind!r}"
)
kinds: Tuple[str, ...] = (entity_kind,)
else:
kinds = ("node", "edge")
for kind in kinds:
record = store.get((kind, entity_id))
if record is not None:
return dict(record)
return None
def _cross_graph_links_for(self, node_id: str) -> List[str]:
"""Link ids that ``node_id`` participates in, as exit point or marker.
The caller must hold ``self._lock``. :meth:`link_graph` registers a link
in three places -- ``_linked_graphs``, a marker node and the bridge edge
-- so removing only the node would leave :meth:`navigate_to` resolving a
link whose source is gone.
"""
link_ids = [
link_id
for link_id, (_, source_node_id, _) in self._linked_graphs.items()
if source_node_id == node_id
]
link_ids.extend(
link_id
for link_id, meta in self._unresolved_links.items()
if meta.get("source_node_id") == node_id
)
node = self.nodes.get(node_id)
metadata = getattr(node, "metadata", None) or {}
if metadata.get("cross_graph") and metadata.get("link_id"):
link_ids.append(metadata["link_id"])
return list(dict.fromkeys(link_ids))
def _cross_graph_marker_nodes(self, node_id: str) -> List[str]:
"""Marker nodes of the cross-graph links ``node_id`` exits through.
The caller must hold ``self._lock``.
"""
return [
marker_id
for marker_id in (
f"__cross_graph_{link_id}"
for link_id in self._cross_graph_links_for(node_id)
)
if marker_id != node_id and marker_id in self.nodes
]
def _drop_node_from_indexes(self, node_id: str) -> None:
"""Remove one node from ``nodes``, ``node_type_index`` and ``_adjacency``.
The caller must hold ``self._lock``. Incident edges are not touched --
see :meth:`_drop_edge_from_indexes`.
"""
node = self.nodes.pop(node_id, None)
if node is None:
return
bucket = self.node_type_index.get(node.node_type)
if bucket is not None:
bucket.discard(node_id)
if not bucket:
del self.node_type_index[node.node_type]
self._adjacency.pop(node_id, None)
def _drop_edge_from_indexes(self, edge: ContextEdge) -> None:
"""Remove one edge from every structure that references it.
The caller must hold ``self._lock``. ``edges``, ``edge_type_index`` and
``_adjacency`` must be updated together or the indexes drift out of
step with the edge list.
"""
try:
self.edges.remove(edge)
except ValueError:
pass
bucket = self.edge_type_index.get(edge.edge_type)
if bucket is not None:
try:
bucket.remove(edge)
except ValueError:
pass
if not bucket:
del self.edge_type_index[edge.edge_type]
adjacent = self._adjacency.get(edge.source_id)
if adjacent is not None:
try:
adjacent.remove(edge)
except ValueError:
pass
if not adjacent:
del self._adjacency[edge.source_id]
# --- Builder Methods (Legacy/Utility) ---
def build_from_conversations(
+19 -15
View File
@@ -176,26 +176,30 @@ async def extract_entities(
session: GraphSession = Depends(get_session),
):
try:
from ...semantic_extract.methods import extract_entities as _extract_entities
from ...semantic_extract.methods import extract_relations as _extract_relations
entities = await asyncio.to_thread(_extract_entities, body.text)
relations = await asyncio.to_thread(_extract_relations, body.text)
ent_list = entities if isinstance(entities, list) else getattr(entities, "entities", [])
rel_list = relations if isinstance(relations, list) else getattr(relations, "relations", [])
return EnrichExtractResponse(
entities=[_safe_dict(entity) for entity in ent_list],
relations=[_safe_dict(relation) for relation in rel_list],
)
from ...semantic_extract import NamedEntityRecognizer, RelationExtractor
except ImportError:
raise HTTPException(
status_code=503,
detail="semantic_extract module not available. Ensure spacy and transformers are installed.",
)
except Exception as exc:
raise HTTPException(status_code=422, detail=f"Extraction failed: {exc}")
recognizer = NamedEntityRecognizer(confidence_threshold=0.7)
extractor = RelationExtractor(confidence_threshold=0.6)
entities = await asyncio.to_thread(recognizer.extract_entities, body.text)
ent_list = entities if isinstance(entities, list) else getattr(entities, "entities", [])
relations = await asyncio.to_thread(
extractor.extract_relations, body.text, ent_list
)
rel_list = relations if isinstance(relations, list) else getattr(relations, "relations", [])
return EnrichExtractResponse(
entities=[_safe_dict(entity) for entity in ent_list],
relations=[_safe_dict(relation) for relation in rel_list],
)
@router.post("/api/enrich/links", response_model=LinkPredictionResponse)
+36 -3
View File
@@ -2,10 +2,10 @@
Shared Pydantic schemas for the Semantica Knowledge Explorer API.
"""
from datetime import datetime
from datetime import datetime, timezone
from typing import Any, Dict, List, Literal, Optional, Tuple
from pydantic import BaseModel, Field
from pydantic import BaseModel, Field, field_validator
class ErrorResponse(BaseModel):
@@ -144,6 +144,37 @@ class DecisionResponse(BaseModel):
timestamp: Optional[str] = None
metadata: Dict[str, Any] = Field(default_factory=dict)
@field_validator("timestamp", mode="before")
@classmethod
def _normalize_timestamp(cls, value: Any) -> Optional[str]:
"""Accept the epoch floats ContextGraph.record_decision() writes.
Decision nodes store ``timestamp`` as ``datetime.now().timestamp()``, a
float, so passing the stored value through unconverted fails validation
and turns every decision route into a 500. Normalize to ISO-8601 here so
the wire format stays a single string type whatever the producer wrote.
"""
if value is None or isinstance(value, str):
return value
if isinstance(value, datetime):
return value.isoformat()
if isinstance(value, (int, float)) and not isinstance(value, bool):
import math
if not math.isfinite(value):
raise ValueError(
f"timestamp must be a finite number, got {value!r}"
)
try:
return datetime.fromtimestamp(value, tz=timezone.utc).isoformat()
except (OverflowError, OSError) as exc:
raise ValueError(
f"timestamp {value!r} is out of the representable epoch range"
) from exc
raise ValueError(
f"timestamp must be None, a string, a datetime, or a numeric epoch; "
f"got {type(value).__name__!r}"
)
class CausalChainResponse(BaseModel):
decision_id: str
@@ -174,7 +205,9 @@ class TemporalPatternResponse(BaseModel):
class EnrichExtractRequest(BaseModel):
text: str
# 10 000 characters is sufficient for a substantial document paragraph while
# preventing unbounded spaCy NLP processing on arbitrarily large payloads.
text: str = Field(..., max_length=10_000)
class EnrichExtractResponse(BaseModel):
+13 -11
View File
@@ -28,7 +28,7 @@ import json
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.helpers import ensure_directory
from ..utils.helpers import _require_mapping, ensure_directory, normalize_graph_payload
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -204,17 +204,19 @@ class ArangoAQLExporter:
self._generate_collection_creation(vertex_collection, edge_collection)
)
# Extract entities and relationships
entities = knowledge_graph.get("entities", [])
relationships = knowledge_graph.get("relationships", [])
nodes = knowledge_graph.get("nodes", entities)
edges = knowledge_graph.get("edges", relationships)
# A non-mapping payload cannot reach normalize_graph_payload(): it
# raises ValidationError for that case, which would leave this
# exporter alone in raising a different exception type than the YAML
# and Neo4j exporters raise for the identical mistake.
_require_mapping(
knowledge_graph, ("entities", "relationships", "nodes", "edges")
)
# Use nodes/edges if entities/relationships are empty
if not entities and nodes:
entities = nodes
if not relationships and edges:
relationships = edges
# Accept either vocabulary; resolution is centralized so every
# exporter agrees on what a given payload means.
normalized = normalize_graph_payload(knowledge_graph)
entities = normalized["entities"]
relationships = normalized["relationships"]
# Generate vertex INSERT statements
vertex_statements = self._generate_vertex_inserts(entities, vertex_collection)
+68
View File
@@ -297,6 +297,57 @@ export_yaml(semantic_network, "network.yaml", method="semantic_network")
export_yaml(schema, "schema.yaml", method="schema")
```
### Accepted Input
Both YAML exporters read their payload by key, so the input must be a mapping;
anything else raises `ProcessingError`. A bare list is rejected rather than
wrapped, since these formats distinguish entities from relationships from
triplets and guessing which one a list holds would mislabel the records.
Each exporter then reads a fixed set of keys, and raises `ValidationError` on a
non-empty mapping that supplies none of them — such a payload would otherwise
serialize to a valid file with every collection empty. Naming a recognized key
is not enough on its own: `{"entities": [], "data": [...]}` also raises, since
nothing resolves while the records sit under a key the exporter never reads.
| Method | Recognized keys |
| :--- | :--- |
| `"semantic_network"` | `entities` (alias `nodes`), `relationships` (alias `edges`), `triplets` |
| `"schema"` | `classes`, `properties`, `namespaces`, `uri`, `title`, `description`, `version` |
`metadata` is carried through on both, but does not by itself make a payload
recognized — an `export_json` envelope (`{"data": [...], "count": N,
"metadata": {...}}`) carries one and is rejected.
```python
# ContextGraph.to_dict() exports directly via the nodes/edges aliases
export_yaml(context_graph.to_dict(), "graph.yaml")
# A bare list has no unambiguous meaning here
export_yaml(records, "out.yaml") # ProcessingError
# An export_json payload is refused rather than written out empty
export_yaml({"data": records}, "out.yaml") # ValidationError
# ...and so is one that names a recognized key but leaves it empty
export_yaml({"entities": [], "data": records}, "out.yaml") # ValidationError
```
The value under a recognized key must be a collection of records — a list or
tuple of mappings or objects. A string, a bare mapping, or a scalar raises
`ValidationError` naming the key, rather than being iterated into
character-sized "records" or surfacing as a `TypeError` from inside the
exporter. `None` is read as an absent collection, the same as `[]`.
```python
export_yaml({"entities": "abc"}, "out.yaml") # ValidationError
export_yaml({"entities": 42}, "out.yaml") # ValidationError
export_yaml({"nodes": {"id": "n1"}}, "out.yaml") # ValidationError — wrap it in a list
```
An empty mapping is still accepted: an empty graph is a legitimate export and
has no records to lose.
## OWL Export
### OWL/XML Format
@@ -479,6 +530,23 @@ Pass `validate=True` to run a post-export integrity check before returning:
export_neo4j_csv(kg, "neo4j_import/", validate=True)
```
#### Accepted Input
Mapping payloads are read on the same terms as the YAML exporters (see [Accepted
Input](#accepted-input) above): `entities`/`relationships`, with `nodes`/`edges`
accepted as aliases. A non-empty mapping that supplies neither — or that supplies
a malformed collection value — raises `ValidationError` rather than writing
header-only CSVs indistinguishable from a genuinely exported empty graph. The
payload is normalized before any file is opened, so a rejected export writes
nothing.
Graph *objects* are unaffected: they are still read off `nodes`/`entities` and
`edges`/`relationships` attributes.
```python
export_neo4j_csv({"data": [{"id": "e1"}]}, "neo4j_import/") # ValidationError
```
#### Importing into Neo4j
Once the CSV files are generated, they can be imported into a new Neo4j database using the `neo4j-admin database import` command:
+28 -12
View File
@@ -24,7 +24,7 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory
from ..utils.helpers import _require_mapping, ensure_directory, normalize_graph_payload
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -154,15 +154,26 @@ class LPGExporter:
"""
queries = []
# Generate indexes if requested
if self.include_indexes:
queries.extend(self._generate_indexes(knowledge_graph))
# A non-mapping payload cannot reach normalize_graph_payload(): it
# raises ValidationError for that case, which would leave this
# exporter alone in raising a different exception type than the YAML
# and Neo4j exporters raise for the identical mistake.
_require_mapping(
knowledge_graph, ("entities", "relationships", "nodes", "edges")
)
# Extract entities and relationships
entities = knowledge_graph.get("entities", [])
relationships = knowledge_graph.get("relationships", [])
nodes = knowledge_graph.get("nodes", entities)
edges = knowledge_graph.get("edges", relationships)
# Accept either vocabulary. Reading 'nodes' with 'entities' as the
# default dropped every entity when 'nodes' was present but empty --
# the shape JSONExporter emits -- so resolution is centralized.
normalized = normalize_graph_payload(knowledge_graph)
nodes = normalized["entities"]
edges = normalized["relationships"]
# Generate indexes if requested. Fed the normalized entities so index
# generation sees the same records as node generation; reading
# 'entities' directly here skipped indexes for nodes/edges payloads.
if self.include_indexes:
queries.extend(self._generate_indexes(nodes))
# Generate node creation queries
node_queries = self._generate_node_queries(nodes)
@@ -174,13 +185,18 @@ class LPGExporter:
return queries
def _generate_indexes(self, knowledge_graph: Dict[str, Any]) -> List[str]:
"""Generate Cypher index and constraint creation queries."""
def _generate_indexes(self, entities: List[Dict[str, Any]]) -> List[str]:
"""Generate Cypher index and constraint creation queries.
Args:
entities: Entity records, already resolved from whichever
vocabulary the caller supplied.
"""
indexes = []
# Get unique entity types for labels
entity_types = set()
for entity in knowledge_graph.get("entities", []):
for entity in entities:
entity_type = entity.get("type") or entity.get("entity_type")
if entity_type:
entity_types.add(entity_type)
+20 -2
View File
@@ -494,7 +494,7 @@ def export_graph(
def export_yaml(
data: Union[Dict[str, Any], List[Dict[str, Any]]],
data: Dict[str, Any],
file_path: Union[str, Path],
method: str = "semantic_network",
**kwargs,
@@ -504,14 +504,32 @@ def export_yaml(
This is a user-friendly wrapper that exports data to YAML format.
Unlike :func:`export_json` and :func:`export_csv`, which treat a list as
opaque records, both YAML methods are keyed formats: they distinguish
entities from relationships from triplets (and classes from properties
for ``method="schema"``). A bare list is therefore rejected rather than
guessed at, since inferring which collection it represents would silently
mislabel the records.
Args:
data: Data to export (semantic network, entities, relationships)
data: Data to export, as a mapping. For ``method="semantic_network"``,
keyed by 'entities'/'relationships'/'triplets'; for
``method="schema"``, by 'classes'/'properties'.
file_path: Output YAML file path
method: Export method (default: "semantic_network")
- "semantic_network": Semantic network YAML export
- "schema": Schema YAML export
**kwargs: Additional options passed to YAML exporters
Raises:
ProcessingError: if ``data`` is not a mapping, or if ``method`` is not
a known YAML export method.
ValidationError: if ``data`` is a mapping whose keys the selected
exporter does not read -- an ``export_json`` envelope
(``{"data": [...], "count": N, "metadata": {...}}``) is the
common case. Such a payload used to be written out as a valid
YAML file with every collection empty.
Examples:
>>> from semantica.export.methods import export_yaml
>>> export_yaml(semantic_network, "network.yaml", method="semantic_network")
+28 -4
View File
@@ -32,12 +32,13 @@ from __future__ import annotations
import csv
import hashlib
import json
from collections.abc import Mapping
from dataclasses import asdict, is_dataclass
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Sequence, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory
from ..utils.helpers import ensure_directory, normalize_graph_payload
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@@ -173,6 +174,19 @@ class Neo4jCSVExporter:
Returns:
Mapping with ``"nodes"`` and ``"relationships"`` output paths.
Raises:
ValidationError: if a mapping payload carries no recognized graph
key, resolves to nothing while an unread key still holds
records, or holds something other than records under one --
see
:func:`~semantica.utils.helpers.normalize_graph_payload`.
Each would otherwise be written out as header-only CSVs
indistinguishable from a genuinely empty graph. The payload is
normalized before any file is opened, so a rejected export
writes nothing.
ProcessingError: if a non-mapping payload exposes none of the
graph attributes.
"""
output_dir = Path(output_dir)
ensure_directory(output_dir)
@@ -494,9 +508,19 @@ class Neo4jCSVExporter:
return prepared
def _normalize_graph(self, graph: Any) -> Dict[str, List[Dict[str, Any]]]:
if isinstance(graph, dict):
nodes = graph.get("nodes") or graph.get("entities") or []
relationships = graph.get("edges") or graph.get("relationships") or []
if isinstance(graph, Mapping):
# Mapping payloads go through the shared resolver on its default
# terms, so this backend cannot drift from the others: an
# unrecognized mapping raises here rather than writing header-only
# CSVs that read as a successful export of an empty graph. Checked
# against Mapping rather than dict, so a non-dict Mapping (a
# MappingProxyType, a ChainMap) takes this path too, instead of
# falling through to the attribute branch below and being rejected
# as an unrecognized object -- the LPG, Arango, and YAML exporters
# already accept such payloads via the same resolver.
resolved = normalize_graph_payload(graph)
nodes = resolved["entities"]
relationships = resolved["relationships"]
else:
nodes = getattr(graph, "nodes", None)
if nodes is None:
+178 -17
View File
@@ -21,15 +21,83 @@ Author: Semantica Contributors
License: MIT
"""
from collections.abc import Mapping
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.helpers import ensure_directory
from ..utils.exceptions import ValidationError
from ..utils.helpers import (
_require_mapping,
_require_nothing_dropped,
_require_recognized_keys,
ensure_directory,
normalize_graph_payload,
)
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
# Keys YAMLSchemaExporter.export_ontology_schema reads. Graph payloads use the
# recognized set owned by normalize_graph_payload() instead; schemas are a
# separate vocabulary with no aliasing, so the set lives here.
_SCHEMA_KEYS = (
"classes",
"properties",
"namespaces",
"uri",
"title",
"description",
"version",
)
def _require_usable_schema(ontology: Mapping) -> None:
"""Reject a schema mapping this exporter cannot read.
Two ways an ontology mapping produces an empty file: it shares no key with
the recognized set at all, or it names a recognized key that is empty
while the real records sit under a key this exporter does not read
(``{"classes": [], "nodes": [...]}``). Both are refused, using the same
checks the graph payloads go through, so the two vocabularies cannot drift
apart in what they consider a silent-empty export.
An empty mapping is allowed through: it carries nothing that could be
lost, and an empty export is a legitimate result.
Note the deliberate split in exception types, which the codebase already
makes: a wrong *type* cannot be exported at all and raises
ProcessingError, matching ``Neo4jCSVExporter._normalize_graph``; a mapping
whose *contents* are unusable raises ValidationError, matching
``normalize_graph_payload``.
Args:
ontology: Mapping already checked by :func:`_require_mapping`.
Raises:
ValidationError: if the mapping shares no key with ``_SCHEMA_KEYS``,
or resolves to nothing while an unread key still holds records.
"""
_require_recognized_keys(ontology, _SCHEMA_KEYS, what="Ontology schema")
# Only non-empty list/tuple values from recognized schema keys count as
# evidence that records survived export. Scalar metadata fields such as
# 'uri', 'title', 'description', and 'version' are truthy strings, but
# their presence does not mean the caller's record collections were
# exported -- passing them as ``resolved`` would let any scalar value
# short-circuit the dropped-records check and silently discard a list
# under an unread key alongside e.g. {"version": "1.0", "nodes": [...]}.
resolved = [
v
for key in _SCHEMA_KEYS
for v in (ontology.get(key),)
if isinstance(v, (list, tuple)) and v
]
_require_nothing_dropped(
ontology,
_SCHEMA_KEYS,
resolved,
what="Ontology schema",
)
class SemanticNetworkYAMLExporter:
"""
@@ -90,15 +158,39 @@ class SemanticNetworkYAMLExporter:
Args:
semantic_network: Semantic network dictionary containing:
- entities: List of entity dictionaries
- entities: List of entity dictionaries (alias: 'nodes')
- relationships: List of relationship dictionaries
(alias: 'edges')
- triplets: List of triplet dictionaries (optional)
- metadata: Metadata dictionary (optional)
Key resolution is delegated to
:func:`~semantica.utils.helpers.normalize_graph_payload`, so
``ContextGraph.to_dict()`` output ('nodes'/'edges') exports
directly.
**options: Additional export options (unused)
Returns:
String containing YAML representation of semantic network
Raises:
ProcessingError: if ``semantic_network`` is not a mapping. A bare
list of records cannot be exported here because this format
distinguishes entities, relationships, and triplets, and
guessing which one a list represents would silently mislabel
it.
ValidationError: if the mapping carries both spellings of a
collection with different contents; if it is non-empty and
shares no key with the recognized set; or if it resolves to
nothing while an unread key still holds records
(``{"entities": [], "data": [...]}``). Each previously
serialized to a file with every collection empty while the log
reported success. An empty mapping is still accepted -- it has
no records to lose. Note that 'metadata' alone is not a
recognized key: an ``export_json`` envelope carries one, and
accepting it would readmit the silent-empty export it is the
most likely source of.
Example:
>>> network = {
... "entities": [...],
@@ -107,6 +199,8 @@ class SemanticNetworkYAMLExporter:
... }
>>> yaml_str = exporter.export_semantic_network(network)
"""
_require_mapping(semantic_network, ("entities", "relationships", "triplets"))
# Track YAML export
tracking_id = self.progress_tracker.start_tracking(
file=None,
@@ -119,15 +213,14 @@ class SemanticNetworkYAMLExporter:
self.progress_tracker.update_tracking(
tracking_id, message="Preparing YAML data..."
)
records = normalize_graph_payload(semantic_network)
yaml_data = {
"metadata": {
"exported_at": datetime.now().isoformat(),
"version": "1.0",
**semantic_network.get("metadata", {}),
},
"entities": semantic_network.get("entities", []),
"relationships": semantic_network.get("relationships", []),
"triplets": semantic_network.get("triplets", []),
**records,
}
self.progress_tracker.update_tracking(
@@ -140,7 +233,7 @@ class SemanticNetworkYAMLExporter:
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message="Exported semantic network to YAML",
message="Serialized semantic network to YAML",
)
return result
@@ -160,16 +253,46 @@ class SemanticNetworkYAMLExporter:
data: Data to export
file_path: Output file path
**options: Additional options
Raises:
ProcessingError: if ``data`` is not a mapping.
ValidationError: on the mappings :meth:`export_semantic_network`
rejects. Serialization runs before the output directory is
created, so a rejected export leaves nothing behind.
OSError: if the file cannot be written. The write is tracked
separately from serialization, so no progress entry reports a
completed export until the bytes are on disk.
"""
file_path = Path(file_path)
ensure_directory(file_path.parent)
yaml_content = self.export_semantic_network(data, **options)
with open(file_path, "w", encoding="utf-8") as f:
f.write(yaml_content)
# Serialization reports its own completion, but it says nothing about
# the file: without this second span, a failing write would leave the
# tracker showing a completed export and no output.
tracking_id = self.progress_tracker.start_tracking(
file=str(file_path),
module="export",
submodule="SemanticNetworkYAMLExporter",
message=f"Writing YAML to {file_path}",
)
self.logger.info(f"Exported YAML to: {file_path}")
try:
ensure_directory(file_path.parent)
with open(file_path, "w", encoding="utf-8") as f:
f.write(yaml_content)
self.logger.info(f"Exported YAML to: {file_path}")
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Exported YAML to: {file_path}",
)
except Exception as e:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e)
)
raise
def export_entities(
self, entities: List[Dict[str, Any]], include_metadata: bool = True, **options
@@ -263,18 +386,34 @@ class SemanticNetworkYAMLExporter:
Structure for definition generation
Include extraction metadata
Return pipeline-ready YAML
Args:
extracted_data: Semantic network mapping, read through
:func:`~semantica.utils.helpers.normalize_graph_payload` on
the same terms as :meth:`export_semantic_network`.
pipeline_stage: Stage number recorded in the output.
**options: Additional export options (unused)
Returns:
Pipeline-ready YAML string.
Raises:
ProcessingError: if ``extracted_data`` is not a mapping.
ValidationError: on the same mappings as
:meth:`export_semantic_network` -- this method built its
nested semantic network from the same defaulted lookups and
so had the same silent-empty failure.
"""
_require_mapping(extracted_data, ("entities", "relationships", "triplets"))
semantic_network = normalize_graph_payload(extracted_data)
yaml_data = {
"pipeline_stage": pipeline_stage,
"metadata": {
"extracted_at": datetime.now().isoformat(),
**extracted_data.get("metadata", {}),
},
"semantic_network": {
"entities": extracted_data.get("entities", []),
"relationships": extracted_data.get("relationships", []),
"triplets": extracted_data.get("triplets", []),
},
"semantic_network": semantic_network,
}
return self.yaml.dump(yaml_data, default_flow_style=False, sort_keys=False)
@@ -308,7 +447,29 @@ class YAMLSchemaExporter:
Include hierarchies and constraints
Structure for easy editing
Return YAML schema
Args:
ontology: Ontology mapping keyed by any of 'classes',
'properties', 'namespaces', 'uri', 'title', 'description',
'version'.
**options: Additional export options (unused)
Returns:
YAML schema string.
Raises:
ProcessingError: if ``ontology`` is not a mapping.
ValidationError: if ``ontology`` is a non-empty mapping sharing
no key with the recognized set, or resolves to nothing while
an unread key still holds records
(``{"classes": [], "nodes": [...]}``) -- each previously
produced a file with empty 'classes', 'properties' and
'namespaces' and no indication anything was dropped. An empty
mapping is still accepted.
"""
_require_mapping(ontology, ("classes", "properties"))
_require_usable_schema(ontology)
yaml_data = {
"ontology": {
"uri": ontology.get("uri", ""),
+206
View File
@@ -0,0 +1,206 @@
"""Internal graph view helpers shared by KG analytics modules."""
from dataclasses import dataclass
from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
@dataclass
class GraphView:
"""Normalized node and edge view used by graph analytics."""
nodes: List[Any]
edges: List[Tuple[Any, Any]]
def build_graph_view(graph: Any) -> GraphView:
"""Build a graph view without dropping explicitly declared nodes.
Graph analytics accepts graph dictionaries, ContextGraph-like objects, and
NetworkX graphs. Nodes declared without an incident edge remain in the
returned view so callers can choose how to handle isolated nodes.
"""
nodes: List[Any] = []
edges: List[Tuple[Any, Any]] = []
seen_nodes: Set[Any] = set()
seen_edges: Set[Tuple[Any, Any]] = set()
def add_node(value: Any) -> Optional[Any]:
node_id = _node_id(value)
if node_id is None or node_id == "":
return None
if node_id not in seen_nodes:
seen_nodes.add(node_id)
nodes.append(node_id)
return node_id
for node in _extract_nodes(graph):
add_node(node)
for raw_edge in _extract_edges(graph):
edge = _edge_endpoints(raw_edge)
if edge is None:
continue
source, target = edge
source = add_node(source)
target = add_node(target)
if source is None or target is None:
continue
if (source, target) not in seen_edges:
seen_edges.add((source, target))
edges.append((source, target))
return GraphView(nodes=nodes, edges=edges)
def build_adjacency(graph: Any, directed: bool = False) -> Dict[Any, List[Any]]:
"""Build an adjacency list while preserving isolated graph nodes."""
view = build_graph_view(graph)
adjacency: Dict[Any, List[Any]] = {node: [] for node in view.nodes}
for source, target in view.edges:
if target not in adjacency[source]:
adjacency[source].append(target)
if not directed and source not in adjacency[target]:
adjacency[target].append(source)
return adjacency
def _extract_nodes(graph: Any) -> Iterable[Any]:
if isinstance(graph, dict):
raw_nodes: List[Any] = []
for key in ("entities", "nodes"):
values = graph.get(key, [])
if isinstance(values, dict):
raw_nodes.extend(values.keys())
elif values:
raw_nodes.extend(values)
return raw_nodes
raw_nodes = getattr(graph, "nodes", None)
if callable(raw_nodes):
return raw_nodes()
if isinstance(raw_nodes, dict):
return raw_nodes.keys()
if raw_nodes is not None:
return raw_nodes
get_nodes = getattr(graph, "get_nodes", None)
if callable(get_nodes):
return get_nodes()
return []
def _extract_edges(graph: Any) -> Iterable[Any]:
if isinstance(graph, dict):
raw_edges: List[Any] = []
for key in ("relationships", "edges"):
values = graph.get(key, [])
if values:
raw_edges.extend(values)
return raw_edges
raw_edges: List[Any] = []
relationships = getattr(graph, "relationships", None)
if relationships is not None:
raw_edges.extend(relationships)
edges = getattr(graph, "edges", None)
if callable(edges):
raw_edges.extend(edges())
elif edges is not None:
raw_edges.extend(edges)
if raw_edges:
return raw_edges
get_relationships = getattr(graph, "get_relationships", None)
if callable(get_relationships):
return get_relationships()
return []
def _edge_endpoints(edge: Any) -> Optional[Tuple[Any, Any]]:
if isinstance(edge, (tuple, list)) and len(edge) >= 2:
return edge[0], edge[1]
if isinstance(edge, dict):
source = _first_value(
edge,
"source",
"source_id",
"subject",
"start",
"start_id",
"from",
"src",
"START_ID",
":START_ID",
)
target = _first_value(
edge,
"target",
"target_id",
"object",
"end",
"end_id",
"to",
"dst",
"END_ID",
":END_ID",
)
else:
source = _first_attribute(
edge,
"source_id",
"source",
"subject",
"start",
"start_id",
"from_id",
)
target = _first_attribute(
edge,
"target_id",
"target",
"object",
"end",
"end_id",
"to_id",
)
if source is None or target is None:
return None
return source, target
def _node_id(value: Any) -> Any:
if isinstance(value, dict):
value = _first_value(
value, "id", "node_id", "entity_id", "key", "name", "text"
)
elif not isinstance(value, (str, int, float, bool, bytes, tuple)):
value = _first_attribute(
value, "node_id", "id", "entity_id", "key", "name", "text"
)
if value is None:
return None
try:
hash(value)
except TypeError:
return str(value)
return value
def _first_value(mapping: Dict[str, Any], *keys: str) -> Any:
for key in keys:
if key in mapping and mapping[key] not in (None, ""):
return mapping[key]
return None
def _first_attribute(value: Any, *names: str) -> Any:
for name in names:
attribute = getattr(value, name, None)
if attribute not in (None, ""):
return attribute
return None
+6 -66
View File
@@ -43,7 +43,7 @@ Author: Semantica Contributors
License: MIT
"""
from collections import defaultdict, deque
from collections import deque
from typing import Any, Dict, List, Optional
import numpy as np
@@ -51,6 +51,7 @@ from scipy import sparse
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ._graph_view import build_adjacency, build_graph_view
class CentralityCalculator:
@@ -518,76 +519,15 @@ class CentralityCalculator:
def _build_adjacency(self, graph) -> Dict[str, List[str]]:
"""Build adjacency list from graph."""
adjacency = defaultdict(list)
# Extract relationships
relationships = []
if hasattr(graph, "relationships"):
relationships = graph.relationships
elif hasattr(graph, "get_relationships"):
relationships = graph.get_relationships()
elif isinstance(graph, dict):
relationships = graph.get("relationships", graph.get("edges", []))
elif hasattr(graph, "edges") and not callable(graph.edges):
# ContextGraph-style: edges is a list of dataclass objects with source_id/target_id
for edge in (graph.edges or []):
if isinstance(edge, dict):
src = edge.get("source") or edge.get("source_id")
tgt = edge.get("target") or edge.get("target_id")
else:
src = getattr(edge, "source_id", None) or getattr(edge, "source", None)
tgt = getattr(edge, "target_id", None) or getattr(edge, "target", None)
if src and tgt:
src, tgt = str(src), str(tgt)
if tgt not in adjacency[src]:
adjacency[src].append(tgt)
if src not in adjacency[tgt]:
adjacency[tgt].append(src)
return dict(adjacency)
# Build adjacency
for rel in relationships:
# Handle tuple/list edges (e.g., from NetworkX)
if isinstance(rel, (tuple, list)) and len(rel) >= 2:
source, target = str(rel[0]), str(rel[1])
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
continue
source = rel.get("source") or rel.get("subject")
target = rel.get("target") or rel.get("object")
# Extract IDs if objects are passed
if source and not isinstance(source, (str, int, float)):
if isinstance(source, dict):
source = source.get("id") or source.get("entity_id") or source.get("text") or str(source)
else:
source = getattr(source, "id", getattr(source, "text", str(source)))
if target and not isinstance(target, (str, int, float)):
if isinstance(target, dict):
target = target.get("id") or target.get("entity_id") or target.get("text") or str(target)
else:
target = getattr(target, "id", getattr(target, "text", str(target)))
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
return dict(adjacency)
return build_adjacency(graph)
def _to_networkx(self, graph):
"""Convert graph to NetworkX format."""
adjacency = self._build_adjacency(graph)
view = build_graph_view(graph)
nx_graph = self.nx.Graph()
for source, targets in adjacency.items():
for target in targets:
nx_graph.add_edge(source, target)
nx_graph.add_nodes_from(view.nodes)
nx_graph.add_edges_from(view.edges)
return nx_graph
+47 -80
View File
@@ -49,6 +49,16 @@ from typing import Any, Dict, List, Optional
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ._graph_view import build_adjacency, build_graph_view
def _is_hashable(value: Any) -> bool:
"""Return whether a community identifier can be used in a set."""
try:
hash(value)
except TypeError:
return False
return True
class CommunityDetector:
@@ -157,17 +167,18 @@ class CommunityDetector:
nx_graph = self._to_networkx(graph)
# Check if graph is empty or has no edges
# An empty graph has no communities. A graph with nodes but
# no edges still has singleton communities.
num_nodes = nx_graph.number_of_nodes()
num_edges = nx_graph.number_of_edges()
self.logger.debug(f"Graph stats: nodes={num_nodes}, edges={num_edges}")
if num_nodes == 0 or num_edges == 0:
self.logger.warning("Graph is empty or has no edges, returning 0 communities")
if num_nodes == 0:
self.logger.warning("Graph is empty, returning 0 communities")
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message="Detected 0 communities (empty graph/no edges)",
message="Detected 0 communities (empty graph)",
)
return {
"communities": [],
@@ -350,17 +361,7 @@ class CommunityDetector:
adjacency = self._build_adjacency(graph)
# Extract community structure
if isinstance(communities, dict):
node_communities = communities
elif isinstance(communities, dict) and "node_assignments" in communities:
node_communities = communities["node_assignments"]
else:
# Convert list of communities to node assignments
node_communities = {}
for i, community in enumerate(communities):
for node in community:
node_communities[node] = i
node_communities = self._to_node_assignments(communities)
# Calculate metrics
num_communities = len(set(node_communities.values()))
@@ -408,16 +409,7 @@ class CommunityDetector:
metrics = self.calculate_community_metrics(graph, communities)
# Extract node assignments
if isinstance(communities, dict) and "node_assignments" in communities:
node_communities = communities["node_assignments"]
elif isinstance(communities, dict):
node_communities = communities
else:
node_communities = {}
for i, community in enumerate(communities):
for node in community:
node_communities[node] = i
node_communities = self._to_node_assignments(communities)
# Analyze connectivity between communities
adjacency = self._build_adjacency(graph)
@@ -440,6 +432,32 @@ class CommunityDetector:
"edge_ratio": intra_community_edges / (inter_community_edges + 1),
}
@staticmethod
def _to_node_assignments(communities: Any) -> Dict[Any, Any]:
"""Normalize community results to a node-to-community mapping."""
if isinstance(communities, dict):
assignments = communities.get("node_assignments")
if isinstance(assignments, dict):
return assignments
detected_communities = communities.get("communities")
if isinstance(detected_communities, (list, tuple)):
communities = detected_communities
elif "communities" in communities:
raise ValueError("Community results must contain a list of communities")
elif not all(_is_hashable(value) for value in communities.values()):
raise ValueError(
"Community assignments must map nodes to hashable community IDs"
)
else:
return communities
node_assignments: Dict[Any, Any] = {}
for community_id, community in enumerate(communities or []):
for node in community:
node_assignments[node] = community_id
return node_assignments
def detect_communities(
self, graph: Any, algorithm: str = "louvain", method: str = None, **options
) -> Dict[str, Any]:
@@ -478,57 +496,7 @@ class CommunityDetector:
def _build_adjacency(self, graph) -> Dict[str, List[str]]:
"""Build adjacency list from graph."""
from collections import defaultdict
adjacency = defaultdict(list)
# Extract relationships
relationships = []
raw_edges = [] # flat (u, v) tuples
if hasattr(graph, "relationships"):
relationships = graph.relationships
elif hasattr(graph, "get_relationships"):
relationships = graph.get_relationships()
elif isinstance(graph, dict):
relationships = graph.get("relationships", [])
# Also handle 'edges' key (list of tuples or dicts)
for edge in graph.get("edges", []):
if isinstance(edge, (list, tuple)) and len(edge) >= 2:
raw_edges.append((str(edge[0]), str(edge[1])))
elif isinstance(edge, dict):
relationships.append(edge)
# Add raw (u, v) edges
for u, v in raw_edges:
if u and v:
adjacency[u].append(v)
adjacency[v].append(u)
# Build adjacency
for rel in relationships:
source = rel.get("source") or rel.get("subject")
target = rel.get("target") or rel.get("object")
# Extract IDs if objects are passed
if source and not isinstance(source, (str, int, float)):
if isinstance(source, dict):
source = source.get("id") or source.get("entity_id") or source.get("text") or str(source)
else:
source = getattr(source, "id", getattr(source, "text", str(source)))
if target and not isinstance(target, (str, int, float)):
if isinstance(target, dict):
target = target.get("id") or target.get("entity_id") or target.get("text") or str(target)
else:
target = getattr(target, "id", getattr(target, "text", str(target)))
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
return dict(adjacency)
return build_adjacency(graph)
def _to_networkx(self, graph):
"""Convert graph to NetworkX format."""
@@ -536,12 +504,11 @@ class CommunityDetector:
if hasattr(graph, 'nodes') and hasattr(graph, 'edges') and hasattr(graph, 'number_of_nodes'):
return graph
adjacency = self._build_adjacency(graph)
view = build_graph_view(graph)
nx_graph = self.nx.Graph()
for source, targets in adjacency.items():
for target in targets:
nx_graph.add_edge(source, target)
nx_graph.add_nodes_from(view.nodes)
nx_graph.add_edges_from(view.edges)
return nx_graph
+3 -46
View File
@@ -48,11 +48,12 @@ Author: Semantica Contributors
License: MIT
"""
from collections import defaultdict, deque
from collections import deque
from typing import Any, Dict, List, Optional, Set, Tuple
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ._graph_view import build_adjacency
class ConnectivityAnalyzer:
@@ -385,51 +386,7 @@ class ConnectivityAnalyzer:
def _build_adjacency(self, graph) -> Dict[str, List[str]]:
"""Build adjacency list from graph."""
adjacency = defaultdict(list)
# Extract relationships
relationships = []
if hasattr(graph, "relationships"):
relationships = graph.relationships
elif hasattr(graph, "get_relationships"):
relationships = graph.get_relationships()
elif isinstance(graph, dict):
relationships = graph.get("relationships", graph.get("edges", []))
# Build adjacency
for rel in relationships:
# Handle tuple/list edges (e.g., from NetworkX)
if isinstance(rel, (tuple, list)) and len(rel) >= 2:
source, target = str(rel[0]), str(rel[1])
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
continue
source = rel.get("source") or rel.get("subject")
target = rel.get("target") or rel.get("object")
# Extract IDs if objects are passed
if source and not isinstance(source, (str, int, float)):
if isinstance(source, dict):
source = source.get("id") or source.get("entity_id") or source.get("text") or str(source)
else:
source = getattr(source, "id", getattr(source, "text", str(source)))
if target and not isinstance(target, (str, int, float)):
if isinstance(target, dict):
target = target.get("id") or target.get("entity_id") or target.get("text") or str(target)
else:
target = getattr(target, "id", getattr(target, "text", str(target)))
if source and target:
if target not in adjacency[source]:
adjacency[source].append(target)
if source not in adjacency[target]:
adjacency[target].append(source)
return dict(adjacency)
return build_adjacency(graph)
def _bfs_shortest_path(
self, adjacency: Dict[str, List[str]], source: str, target: str
+162 -129
View File
@@ -127,66 +127,100 @@ class PathFinder:
try:
self.logger.info(f"Finding Dijkstra shortest path from {source} to {target}")
# Validate nodes exist
if not self._node_exists(graph, source):
raise ValueError(f"Source node {source} not found")
if not self._node_exists(graph, target):
raise ValueError(f"Target node {target} not found")
traversal_graph = graph if directed else self._make_undirected_view(graph)
# Dijkstra's algorithm
distances = {source: 0.0}
previous = {}
priority_queue = [(0.0, source)]
visited = set()
while priority_queue:
current_distance, current_node = heapq.heappop(priority_queue)
if current_node in visited:
continue
visited.add(current_node)
if current_node == target:
break
# Explore neighbors
for neighbor, edge_data in self._get_neighbors(traversal_graph, current_node):
if neighbor in visited:
continue
# Get edge weight
weight = self._get_edge_weight(edge_data, weight_attribute, default_weight)
distance = current_distance + weight
if neighbor not in distances or distance < distances[neighbor]:
distances[neighbor] = distance
previous[neighbor] = current_node
heapq.heappush(priority_queue, (distance, neighbor))
# Reconstruct path
if target not in previous and source != target:
return [] # No path found
path = []
current = target
while current is not None:
path.append(current)
current = previous.get(current)
path.reverse()
path = self._dijkstra_shortest_path(
graph,
source,
target,
weight_attribute,
default_weight,
directed,
)
self.logger.info(f"Found path of length {len(path)}")
return path
except ValueError:
# Re-raise ValueError for invalid nodes
raise
except Exception as e:
self.logger.error(f"Dijkstra path finding failed: {str(e)}")
raise RuntimeError(f"Path finding failed: {str(e)}")
def _dijkstra_shortest_path(
self,
graph: Any,
source: str,
target: str,
weight_attribute: str = "weight",
default_weight: float = 1.0,
directed: bool = True,
excluded_nodes: Optional[Set[str]] = None,
excluded_edges: Optional[Set[Tuple[str, str]]] = None,
) -> List[str]:
"""Find a shortest path without mutating the graph.
``excluded_nodes`` and ``excluded_edges`` are used internally by
Yen's algorithm to model its temporary graph modifications.
"""
excluded_nodes = excluded_nodes or set()
excluded_edges = excluded_edges or set()
# Validate nodes exist before applying the temporary exclusions.
if not self._node_exists(graph, source):
raise ValueError(f"Source node {source} not found")
if not self._node_exists(graph, target):
raise ValueError(f"Target node {target} not found")
if source in excluded_nodes or target in excluded_nodes:
return []
traversal_graph = graph if directed else self._make_undirected_view(graph)
# Dijkstra's algorithm
distances = {source: 0.0}
previous = {}
priority_queue = [(0.0, source)]
visited = set()
while priority_queue:
current_distance, current_node = heapq.heappop(priority_queue)
if current_node in visited or current_node in excluded_nodes:
continue
visited.add(current_node)
if current_node == target:
break
# Explore neighbors
for neighbor, edge_data in self._get_neighbors(traversal_graph, current_node):
if neighbor in visited or neighbor in excluded_nodes:
continue
if self._edge_is_excluded(
traversal_graph, current_node, neighbor, excluded_edges
):
continue
# Get edge weight
weight = self._get_edge_weight(edge_data, weight_attribute, default_weight)
distance = current_distance + weight
if neighbor not in distances or distance < distances[neighbor]:
distances[neighbor] = distance
previous[neighbor] = current_node
heapq.heappush(priority_queue, (distance, neighbor))
# Reconstruct path
if target not in previous and source != target:
return [] # No path found
path = []
current = target
while current is not None:
path.append(current)
current = previous.get(current)
path.reverse()
return path
def a_star_search(
self,
@@ -493,69 +527,89 @@ class PathFinder:
raise ValueError("k must be positive")
# Find first shortest path
first_path = self.dijkstra_shortest_path(graph, source, target, weight_attribute, default_weight)
first_path = self.dijkstra_shortest_path(
graph, source, target, weight_attribute, default_weight
)
if not first_path:
return []
paths = [first_path]
candidates = []
for i in range(1, k):
# Generate candidate paths
for j in range(len(paths[-1]) - 1):
spur_node = paths[-1][j]
root_path = paths[-1][:j + 1]
# Temporarily remove edges
removed_edges = []
candidate_paths = {tuple(first_path)}
candidate_order = 0
while len(paths) < k:
previous_path = paths[-1]
# Generate candidate paths from every spur node in the last path.
for j in range(len(previous_path) - 1):
spur_node = previous_path[j]
root_path = previous_path[:j + 1]
# Block the next edge of every accepted path sharing this root.
excluded_edges = set()
for path in paths:
if len(path) > j and path[:j + 1] == root_path:
if j + 1 < len(path):
edge_data = self._get_edge_data(graph, path[j], path[j + 1])
if edge_data is not None:
removed_edges.append((path[j], path[j + 1], edge_data))
self._remove_edge(graph, path[j], path[j + 1])
# Temporarily remove nodes (except spur node and nodes that don't exist)
removed_nodes = []
for node in root_path[:-1]:
if node != spur_node and node != source and self._node_exists(graph, node):
removed_nodes.append(node)
self._remove_node(graph, node)
# Find spur path
spur_path = self.dijkstra_shortest_path(graph, spur_node, target, weight_attribute, default_weight)
# Restore graph
for node in removed_nodes:
self._restore_node(graph, node)
for u, v, data in removed_edges:
self._restore_edge(graph, u, v, data)
# Combine root and spur paths
if spur_path:
candidate_path = root_path[:-1] + spur_path
if candidate_path not in candidates and candidate_path not in paths:
candidates.append(candidate_path)
# Calculate path lengths and sort
candidates_with_lengths = []
for path in candidates:
try:
length = self.path_length(graph, path, weight_attribute, default_weight)
candidates_with_lengths.append((path, length))
except ValueError:
# Skip invalid paths
continue
candidates_with_lengths.sort(key=lambda x: x[1])
# Add shortest unique paths
for path, length in candidates_with_lengths:
if len(paths) < k and path not in paths:
paths.append(path)
if len(path) > j + 1 and path[:j + 1] == root_path:
excluded_edges.add((path[j], path[j + 1]))
# Block root nodes so the combined path remains loopless.
excluded_nodes = set(root_path[:-1])
spur_path = self._dijkstra_shortest_path(
graph,
spur_node,
target,
weight_attribute,
default_weight,
excluded_nodes=excluded_nodes,
excluded_edges=excluded_edges,
)
if not spur_path:
continue
candidate_path = root_path[:-1] + spur_path
if len(candidate_path) != len(set(candidate_path)):
continue
candidate_key = tuple(candidate_path)
if candidate_key in candidate_paths:
continue
try:
length = self.path_length(
graph, candidate_path, weight_attribute, default_weight
)
except ValueError:
continue
candidate_paths.add(candidate_key)
heapq.heappush(candidates, (length, candidate_order, candidate_path))
candidate_order += 1
if not candidates:
break
_, _, next_path = heapq.heappop(candidates)
paths.append(next_path)
return paths
def _edge_is_excluded(
self,
graph: Any,
source: str,
target: str,
excluded_edges: Set[Tuple[str, str]],
) -> bool:
"""Check whether an edge is excluded for the current traversal."""
if (source, target) in excluded_edges:
return True
is_directed = getattr(graph, "is_directed", None)
if callable(is_directed) and not is_directed():
return (target, source) in excluded_edges
return False
def _node_exists(self, graph: Any, node: str) -> bool:
"""Check if node exists in graph."""
@@ -614,27 +668,6 @@ class PathFinder:
return edge_data.get(weight_attribute, default_weight)
return default_weight
def _remove_edge(self, graph: Any, u: str, v: str) -> None:
"""Remove edge from graph."""
if hasattr(graph, 'remove_edge'):
graph.remove_edge(u, v)
def _restore_edge(self, graph: Any, u: str, v: str, data: Any) -> None:
"""Restore edge to graph."""
if hasattr(graph, 'add_edge'):
graph.add_edge(u, v, **data)
def _remove_node(self, graph: Any, node: str) -> None:
"""Remove node from graph."""
if hasattr(graph, 'remove_node'):
graph.remove_node(node)
def _restore_node(self, graph: Any, node: str) -> None:
"""Restore node to graph (implementation depends on graph type)."""
# This is a simplified implementation
# In practice, you'd need to restore the node and its connections
pass
def _reconstruct_all_paths(
self,
previous: Dict[str, List[str]],
+12 -5
View File
@@ -562,6 +562,11 @@ class CurrencyNormalizer:
"SEK",
"NOK",
"DKK",
"RUB",
"KRW",
"ILS",
"NGN",
"PKR",
]
self.logger.debug("Currency normalizer initialized")
@@ -606,13 +611,15 @@ class CurrencyNormalizer:
# Check for currency code
if not currency_code:
for code in self.currency_codes:
if code in currency_input.upper():
match = re.search(
rf"(?<![A-Z]){re.escape(code)}(?![A-Z])",
currency_input.upper(),
)
if match:
currency_code = code
amount_str = (
currency_input.replace(code, "")
.replace(code.lower(), "")
.strip()
)
currency_input[: match.start()] + currency_input[match.end() :]
).strip()
amount_str = amount_str.replace(",", "").replace(" ", "")
try:
amount = float(amount_str)
+1
View File
@@ -37,6 +37,7 @@ from openpyxl import load_workbook
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@dataclass
+14 -7
View File
@@ -401,18 +401,25 @@ class SeedDataManager:
"""
try:
from ..ingest.db_ingestor import DBIngestor
except ImportError as e:
raise ProcessingError(
"Database ingestion module not available. Install required dependencies."
) from e
try:
# Initialize DB ingestor
db_ingestor = DBIngestor(config={"connection_string": connection_string})
# Execute query or export table
# Execute query or export table. Both ingestor methods take the
# connection string as their first argument — the constructor's
# config is not a substitute for it (#973).
if query:
# Execute custom query
result = db_ingestor.execute_query(query)
result = db_ingestor.execute_query(connection_string, query)
records = result if isinstance(result, list) else [result]
elif table_name:
# Export table
table_data = db_ingestor.export_table(table_name)
table_data = db_ingestor.export_table(connection_string, table_name)
records = table_data.rows if hasattr(table_data, "rows") else []
else:
raise ProcessingError("Either 'query' or 'table_name' must be provided")
@@ -429,11 +436,11 @@ class SeedDataManager:
self.logger.info(f"Loaded {len(records)} records from database")
return records
except (ImportError, OSError):
raise ProcessingError(
"Database ingestion module not available. Install required dependencies."
)
except ProcessingError:
raise
except Exception as e:
# OSError here is a real connection/driver failure, not a missing
# module — report the actual cause and keep the chain (#973).
raise ProcessingError(f"Failed to load from database: {e}") from e
def load_from_api(
+39 -5
View File
@@ -108,6 +108,7 @@ License: MIT
import re
import difflib
import threading
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any, Dict, List, Optional, Tuple, Union
@@ -153,6 +154,39 @@ spacy, SPACY_AVAILABLE = safe_import("spacy")
_nlp_cache = None
_embedder_cache = None
# Cache for models loaded by name, so extraction functions do not pay
# spacy.load() on every call. Entries record the spacy module they were loaded
# from: tests patch `methods.spacy` with a mock, and an entry produced by a
# different module object must not be handed back to a later caller.
_spacy_model_cache: Dict[str, Tuple[Any, Any]] = {}
_spacy_model_cache_lock = threading.Lock()
def load_spacy_model(name: str):
"""Load a spaCy model once per process, keyed by model name.
Raises whatever ``spacy.load`` raises (``OSError`` for a missing model), so
callers keep their existing fallback behavior.
"""
cached = _spacy_model_cache.get(name)
if cached is not None and cached[0] is spacy:
return cached[1]
with _spacy_model_cache_lock:
cached = _spacy_model_cache.get(name)
if cached is not None and cached[0] is spacy:
return cached[1]
nlp = spacy.load(name)
_spacy_model_cache[name] = (spacy, nlp)
return nlp
def clear_spacy_model_cache() -> None:
"""Drop every cached spaCy model. Intended for tests."""
with _spacy_model_cache_lock:
_spacy_model_cache.clear()
def get_text_embedder():
"""
Get or load the TextEmbedder model for high-accuracy semantic similarity.
@@ -676,11 +710,11 @@ def extract_entities_ml(
return extract_entities_pattern(text, **kwargs)
try:
nlp = spacy.load(model)
nlp = load_spacy_model(model)
except OSError:
logger.warning(f"spaCy model {model} not found, using en_core_web_sm")
try:
nlp = spacy.load("en_core_web_sm")
nlp = load_spacy_model("en_core_web_sm")
except OSError:
logger.warning(
"spaCy model not available, falling back to pattern extraction"
@@ -1400,12 +1434,12 @@ def extract_relations_similarity(
# Prefer larger models for vectors
for model_name in ["en_core_web_lg", "en_core_web_md", "en_core_web_sm"]:
if spacy.util.is_package(model_name):
nlp = spacy.load(model_name)
nlp = load_spacy_model(model_name)
break
if not nlp:
# Try loading what we have
try:
nlp = spacy.load("en_core_web_sm")
nlp = load_spacy_model("en_core_web_sm")
except:
pass
except Exception:
@@ -1505,7 +1539,7 @@ def extract_relations_dependency(
return extract_relations_pattern(text, entities, **kwargs)
try:
nlp = spacy.load(model)
nlp = load_spacy_model(model)
except OSError:
logger.warning(f"spaCy model {model} not found")
return extract_relations_pattern(text, entities, **kwargs)
+12 -1
View File
@@ -50,12 +50,23 @@ _DISALLOWED_URI_CHARS_RE = re.compile(r"[\s<>\"{}|\\^`]")
#
# Shared by BlazegraphStore and RDF4JStore so the detection logic has one
# canonical implementation rather than being duplicated per-backend.
#
# The comment alternative must consume the whole comment up to a line
# terminator. Written as a bare `\#[^\n]*`, the trailing `*` backtracks: for
# "# CONSTRUCT ...\nSELECT ...", the engine gives back everything after the
# '#', letting the CONSTRUCT *inside the comment* satisfy the query-form
# keyword and misreporting a SELECT as a CONSTRUCT. Requiring a terminator
# ([\n\r], or end of input for a trailing comment) makes that backtracking
# impossible: if the character class gives a character back, the next
# character is by definition not a terminator, so the group cannot match.
# Both LF and CR are treated as terminators because the SPARQL grammar ends
# a comment at either.
CONSTRUCT_QUERY_RE = re.compile(
r"""
\A # anchor to start of string
(?: # skip zero or more of:
\s+ # whitespace
| \#[^\n]* # comments (until newline)
| \#[^\n\r]*(?:[\n\r]|\Z) # comment, to end of line or end of input
| PREFIX\s+[\w\-]*:\s*<[^>]*> # PREFIX declaration
| BASE\s+<[^>]*> # BASE declaration
)*
+2
View File
@@ -80,6 +80,7 @@ from .helpers import (
hash_data,
merge_dicts,
normalize_entities,
normalize_graph_payload,
parse_timestamp,
read_json_file,
retry_on_error,
@@ -183,6 +184,7 @@ __all__ = [
"format_data",
"clean_text",
"normalize_entities",
"normalize_graph_payload",
"hash_data",
"safe_filename",
"ensure_directory",
+372 -1
View File
@@ -63,9 +63,16 @@ import importlib
import json
import os
import re
import types
from collections import Counter
from collections.abc import Iterable as IterableABC
from collections.abc import Mapping
from dataclasses import asdict, is_dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Type, Union
from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple, Type, Union
from .exceptions import ProcessingError, ValidationError
def format_data(data: Any, format_type: str = "json") -> str:
@@ -584,3 +591,367 @@ def classify_path_distance(hop_count: int) -> str:
if hop_count <= 6:
return "mid-range"
return "distant"
# Graph payloads circulate under two vocabularies: 'entities'/'relationships'
# (kg builders, most exporters) and 'nodes'/'edges' (ContextGraph.to_dict,
# Neo4jCSVExporter, the Explorer routes). Consumers each reconciled them
# locally, with at least three competing idioms, so the same payload could be
# exported, silently dropped, or rejected depending on which consumer read it.
# This is the single place that decision is made.
_ENTITY_KEYS = ("entities", "nodes")
_RELATIONSHIP_KEYS = ("relationships", "edges")
_TRIPLET_KEYS = ("triplets",)
# Keys that legitimately travel alongside the collections without being
# records themselves, so their presence is never evidence that records were
# dropped: ContextGraph.to_dict() carries 'statistics', JSON envelopes carry
# 'metadata' and 'count'.
_CONTEXT_KEYS = ("metadata", "statistics", "count")
def _require_recognized_keys(
payload: Mapping, recognized_keys: Sequence[str], *, what: str
) -> None:
"""Reject a mapping that shares no key with the recognized set.
A consumer that reads a fixed set of keys turns an unrecognized mapping
into an empty result that looks like a legitimate one. An empty mapping is
allowed through -- it carries nothing that could be lost.
Args:
payload: Mapping to check.
recognized_keys: Keys the consumer reads.
what: Noun for the error message, e.g. ``"Graph payload"``.
Raises:
ValidationError: if ``payload`` is non-empty and shares no key with
``recognized_keys``.
"""
if not payload or any(key in payload for key in recognized_keys):
return
supplied = ", ".join(f"'{key}'" for key in sorted(map(str, payload)))
expected = ", ".join(f"'{key}'" for key in recognized_keys)
raise ValidationError(
f"{what} has no recognized key. Supplied: {supplied}. "
f"Expected at least one of: {expected}."
)
def _require_nothing_dropped(
payload: Mapping,
recognized_keys: Sequence[str],
resolved: Iterable[Any],
*,
what: str,
) -> None:
"""Reject a mapping that resolved to nothing while still holding records.
Checking that a recognized key is *present* is not enough:
``{"entities": [], "data": [...]}`` clears that bar and still resolves to
empty, dropping every record under 'data'. Presence answers "did the
caller use our vocabulary"; this answers the question that actually
matters, "did anything the caller supplied survive".
Only non-empty lists count as evidence of dropped records. A payload can
carry scalars and dicts that are not collections -- ContextGraph.to_dict()
always includes 'statistics' -- and an empty graph must stay exportable.
Args:
payload: Mapping to check.
recognized_keys: Keys the consumer reads.
resolved: The collections the consumer resolved from ``payload``.
what: Noun for the error message, e.g. ``"Graph payload"``.
Raises:
ValidationError: if nothing resolved and an unread key holds a
non-empty list.
"""
if any(resolved):
return
dropped = sorted(
str(key)
for key, value in payload.items()
if key not in recognized_keys
and key not in _CONTEXT_KEYS
and isinstance(value, (list, tuple))
and value
)
if not dropped:
return
named = ", ".join(f"'{key}'" for key in dropped)
expected = ", ".join(f"'{key}'" for key in recognized_keys)
raise ValidationError(
f"{what} resolved to nothing, but {named} still holds records. "
f"Exporting it would drop them silently. Supply the records under "
f"one of: {expected}."
)
def _is_record(value: Any) -> bool:
"""Report whether a value can stand in for a graph record.
Consumers read records either as mappings (``entity.get("type")`` in the
LPG and Arango exporters) or as objects with attributes
(``Neo4jCSVExporter._record_to_dict`` accepts dataclasses and anything
carrying a ``__dict__``). Both are legitimate, so both are accepted here;
strings, numbers, and nested sequences are not records under either
reading.
Modules and class/type objects are excluded even though they carry
``__dict__``: they are not graph records under any supported reading, and
passing them through the boundary would produce ``AttributeError`` inside
exporters rather than a ``ValidationError`` at the boundary where the
problem is visible.
"""
return isinstance(value, Mapping) or is_dataclass(value) or (
hasattr(value, "__dict__")
and not isinstance(value, (types.ModuleType, type))
)
def _record_to_dict(record: Any) -> Dict[str, Any]:
"""Convert an accepted record to a plain dict.
:func:`_is_record` accepts mappings, dataclasses, and objects carrying
``__dict__`` as legitimate record shapes, but consumers of
:func:`normalize_graph_payload` -- YAML serialization, ``entity.get(...)``
in the LPG and Arango exporters -- read records as dicts. Converting here,
at the boundary, means every exporter gets the same shape regardless of
which reading the caller used; previously only ``Neo4jCSVExporter``
converted object-shaped records locally, so a dataclass record passed
validation for the other exporters only to crash with a raw
``AttributeError`` once used.
"""
if isinstance(record, Mapping):
return dict(record)
if is_dataclass(record):
return asdict(record)
return {
key: value for key, value in vars(record).items() if not key.startswith("_")
}
def _coerce_records(key: str, value: Any) -> List[Any]:
"""Validate one collection value and materialize it as a list of records.
This runs before any truthiness or ``list()`` call, because both mislead
on malformed input: ``list("abc")`` quietly turns a string into three
single-character "records", and ``list(42)`` raises a bare ``TypeError``
from deep inside the exporter that named the exporter rather than the
offending payload key. Neither reaches the caller as an actionable
message, so the shapes that produce them are rejected by name instead.
``None`` is deliberately not rejected: JSON round-trips an absent
collection to null, and treating that as "no records under this key" is
the same answer an explicit ``[]`` gets. It is not silent data loss --
a null collection alongside records under an unread key is still caught
by :func:`_require_nothing_dropped`.
Args:
key: Payload key the value came from, for the error message.
value: The raw value stored under ``key``.
Returns:
The records as a new list, so the result never aliases the input.
Raises:
ValidationError: if ``value`` is a string, bytes, a mapping, or any
non-iterable scalar; or if any element is not a record.
"""
if value is None:
return []
if isinstance(value, (str, bytes, bytearray)):
raise ValidationError(
f"Graph payload key '{key}' holds a {type(value).__name__}, not a "
f"collection of records. Iterating it would yield characters, not "
f"records. Supply a list of records."
)
if isinstance(value, Mapping):
raise ValidationError(
f"Graph payload key '{key}' holds a mapping, not a collection of "
f"records. If it is a single record, wrap it in a list; if it is "
f"keyed by ID, supply its values as a list."
)
if not isinstance(value, IterableABC):
raise ValidationError(
f"Graph payload key '{key}' holds a "
f"{type(value).__name__}, not a collection of records. Supply a "
f"list of records."
)
records = list(value)
for index, record in enumerate(records):
if not _is_record(record):
raise ValidationError(
f"Graph payload key '{key}' holds a "
f"{type(record).__name__} at index {index}, not a record. "
f"Records must be mappings or objects with attributes."
)
return [_record_to_dict(record) for record in records]
def _canonical_record_multiset(records: List[Dict[str, Any]]) -> "Counter[str]":
"""Represent records as an order-independent multiset for equality checks.
Two spellings of the same collection (``entities`` and ``nodes``) can
legitimately list identical records in a different order -- a caller
round-tripping through a dict-keyed cache or a set has no reason to
preserve list order. Comparing with plain list equality would treat that
as a conflict and reject a payload that carries no real data loss, so
records are compared as a multiset of their canonical JSON form instead.
"""
return Counter(
json.dumps(record, sort_keys=True, default=str) for record in records
)
def _resolve_collection(
payload: Dict[str, Any], keys: Tuple[str, ...]
) -> List[Dict[str, Any]]:
"""Pick one collection from a payload that may use either vocabulary.
Both spellings may legitimately be present: ``JSONExporter`` writes
'entities' and 'nodes' side by side, so a round-trip of its output carries
both, one of them empty. Where only one holds records, that one wins.
Two non-empty, unequal spellings are a different matter -- there is no
basis for preferring either, and picking one would silently discard the
other -- so that is refused rather than guessed at.
Every spelling present is validated, not just the one that wins: a
malformed 'nodes' alongside a well-formed 'entities' is a payload the
caller should hear about, and validating only the winner would let it
through on the strength of the other key.
Args:
payload: Mapping to read from.
keys: Accepted spellings, most canonical first.
Returns:
The resolved collection, or an empty list if no spelling is present.
Raises:
ValidationError: if a spelling holds something other than a collection
of records; or if two spellings are both present, both non-empty,
and hold different records, order ignored.
"""
present = {
key: _coerce_records(key, payload[key]) for key in keys if key in payload
}
populated = {key: value for key, value in present.items() if value}
if len(populated) > 1:
values = list(populated.values())
canonical = [_canonical_record_multiset(value) for value in values]
if any(entry != canonical[0] for entry in canonical[1:]):
named = " and ".join(f"'{key}'" for key in populated)
raise ValidationError(
f"Graph payload carries {named} with different contents; "
f"cannot determine which to export. Supply one, or make them "
f"identical."
)
for key in keys:
value = present.get(key)
if value:
# Already a fresh list from _coerce_records, so the result cannot
# alias the caller's collection.
return value
# Every spelling present is empty (or none is): an explicit empty
# collection is a legitimate answer, distinct from "unrecognized".
return []
def _require_mapping(data: Any, expected_keys: Sequence[str]) -> None:
"""Reject non-mapping export input with an actionable error.
Shared by every consumer of :func:`normalize_graph_payload` so that a
wrong *type* fails the same way everywhere. Handed a sequence (or any
other non-mapping), every downstream key lookup would fail with a bare
``AttributeError: 'list' object has no attribute 'get'``, which tells the
caller nothing about the shape expected -- and ``normalize_graph_payload``
itself raises ``ValidationError`` for this case, which would leave
exporters that skip this guard raising a different exception type than
the ones that call it, for the identical mistake.
A list is rejected rather than wrapped: these formats distinguish
entities from relationships from triplets (or nodes/edges), so inferring
which one a bare list represents would silently mislabel the records.
Args:
data: Candidate export payload.
expected_keys: Key names the caller reads, named in the error so the
caller learns the expected shape.
Raises:
ProcessingError: if ``data`` is not a mapping.
"""
if not isinstance(data, Mapping):
keys = "/".join(f"'{key}'" for key in expected_keys)
raise ProcessingError(
f"Cannot export object of type '{type(data).__name__}': "
f"expected a dict with {keys}."
)
def normalize_graph_payload(
payload: Dict[str, Any],
) -> Dict[str, List[Dict[str, Any]]]:
"""Reduce a graph payload to one canonical vocabulary.
Accepts either 'entities'/'relationships' or 'nodes'/'edges' (or a mix)
and returns the canonical spelling, so consumers read one shape instead of
reimplementing the reconciliation.
This is the validation boundary for graph payloads: it either returns
collections of records or raises. Nothing that reaches an exporter through
it needs re-checking, and nothing malformed passes through it as a
valid-looking empty graph.
Args:
payload: Graph payload mapping.
Returns:
``{"entities": [...], "relationships": [...], "triplets": [...]}``.
Raises:
ValidationError: if ``payload`` is not a mapping; if a recognized key
holds something other than a collection of records; if two
spellings of the same collection are both non-empty and differ; if
a non-empty mapping contains no recognized key; or if it resolves
to nothing while an unread key still holds records. The last two
would otherwise hand the caller a valid-looking result with their
records silently dropped.
Example:
>>> normalize_graph_payload({"nodes": [{"id": "n1"}], "edges": []})
{'entities': [{'id': 'n1'}], 'relationships': [], 'triplets': []}
"""
if not isinstance(payload, Mapping):
raise ValidationError(
f"Cannot normalize graph payload of type "
f"'{type(payload).__name__}': expected a mapping."
)
recognized = _ENTITY_KEYS + _RELATIONSHIP_KEYS + _TRIPLET_KEYS
_require_recognized_keys(payload, recognized, what="Graph payload")
resolved = {
"entities": _resolve_collection(payload, _ENTITY_KEYS),
"relationships": _resolve_collection(payload, _RELATIONSHIP_KEYS),
"triplets": _resolve_collection(payload, _TRIPLET_KEYS),
}
_require_nothing_dropped(
payload, recognized, resolved.values(), what="Graph payload"
)
return resolved
+55
View File
@@ -109,6 +109,61 @@ class TestContextModule(unittest.TestCase):
self.assertEqual(neighbors[0]["id"], "n2")
self.assertEqual(neighbors[0]["relationship"], "knows")
def test_add_edge_is_idempotent(self):
graph = ContextGraph()
graph.add_node("a", "t")
graph.add_node("b", "t")
self.assertTrue(graph.add_edge("a", "b", "rel"))
self.assertFalse(graph.add_edge("a", "b", "rel"))
self.assertFalse(graph.add_edge("a", "b", "rel"))
self.assertEqual(len(graph.edges), 1)
self.assertEqual(len(graph.edge_type_index["rel"]), 1)
self.assertEqual(len(graph._adjacency["a"]), 1)
self.assertEqual(graph.stats()["edge_count"], 1)
self.assertLessEqual(graph.density(), 1.0)
def test_parallel_edges_with_distinct_attributes_are_kept(self):
graph = ContextGraph()
graph.add_node("a", "t")
graph.add_node("b", "t")
graph.add_edge("a", "b", "rel", confidence=0.9)
graph.add_edge("a", "b", "rel", confidence=0.5)
graph.add_edge("a", "b", "other")
self.assertEqual(len(graph.edges), 3)
self.assertEqual(len({e.edge_id for e in graph.edges}), 3)
def test_reingest_does_not_duplicate_edges(self):
graph = ContextGraph()
entities = [
{"id": "alice", "type": "person"},
{"id": "acme", "type": "org"},
]
relationships = [
{"source_id": "alice", "target_id": "acme", "type": "works_at"}
]
for _ in range(3):
graph.build_from_entities_and_relationships(entities, relationships)
self.assertEqual(len(graph.edges), 1)
def test_clear_resets_edge_dedupe_index(self):
graph = ContextGraph()
graph.add_node("a", "t")
graph.add_node("b", "t")
graph.add_edge("a", "b", "rel")
graph.clear()
graph.add_node("a", "t")
graph.add_node("b", "t")
self.assertTrue(graph.add_edge("a", "b", "rel"))
self.assertEqual(len(graph.edges), 1)
def test_get_nodes_by_label_returns_metadata_copy(self):
graph = ContextGraph()
graph.add_node("n1", "person", "Alice", role="engineer")
@@ -0,0 +1,595 @@
"""Tests for ContextGraph retraction and purge (issue #955).
``ContextGraph`` had 56 public methods and none that removed anything: the only
option was ``clear()``, which discards the whole graph. Two operations are
added, with deliberately different contracts.
Retraction closes an entity's validity window. The entity stops being active
going forward, but ``state_at()`` before the retraction still returns it, so
decisions recorded against it remain explainable. Purge is destructive: the
entity is gone from history too, leaving only a tombstone recording that a
purge happened and why -- never the purged content.
The audit-trail assertions run against a real ``TemporalVersionManager`` rather
than a mock callback, since the behaviour under test is precisely that these
operations reach the existing mutation-recording path.
"""
import json
import os
import tempfile
import threading
import unittest
from datetime import datetime
from semantica.change_management import TemporalVersionManager
from semantica.context import ContextEdge, ContextGraph
BEFORE = "2025-06-01T00:00:00Z"
BETWEEN = "2025-09-01T00:00:00Z"
CUTOFF = "2026-01-01T00:00:00Z"
AFTER = "2026-06-01T00:00:00Z"
def _graph():
"""alice --works_at--> acme, plus an unrelated bob."""
graph = ContextGraph(advanced_analytics=False)
graph.add_node("alice", "person")
graph.add_node("acme", "org")
graph.add_node("bob", "person")
graph.add_edge("alice", "acme", "works_at")
return graph
def _ids_at(graph, when):
return {node.get("id") for node in graph.state_at(when).get("nodes", [])}
def _index_totals(graph):
return {
"nodes": len(graph.nodes),
"node_index": sum(len(v) for v in graph.node_type_index.values()),
"edges": len(graph.edges),
"edge_index": sum(len(v) for v in graph.edge_type_index.values()),
"adjacency": sum(len(v) for v in graph._adjacency.values()),
}
class TestRetractNode(unittest.TestCase):
def test_retracted_node_leaves_the_active_view(self):
graph = _graph()
self.assertTrue(graph.retract_node("alice", at=CUTOFF))
active = {node["id"] for node in graph.find_active_nodes()}
self.assertNotIn("alice", active)
self.assertIn("bob", active)
def test_history_before_the_retraction_is_preserved(self):
graph = _graph()
graph.retract_node("alice", at=CUTOFF)
self.assertIn("alice", _ids_at(graph, BEFORE))
self.assertNotIn("alice", _ids_at(graph, AFTER))
def test_retraction_record_captures_reason_and_time(self):
graph = _graph()
graph.retract_node("alice", reason="employment ended", at=CUTOFF)
record = graph.get_retraction("alice")
self.assertEqual(record["entity_id"], "alice")
self.assertEqual(record["entity_kind"], "node")
self.assertEqual(record["reason"], "employment ended")
self.assertIn("2026-01-01", record["retracted_at"])
def test_retracting_twice_is_a_no_op(self):
graph = _graph()
self.assertTrue(graph.retract_node("alice", reason="first", at=CUTOFF))
self.assertFalse(graph.retract_node("alice", reason="second"))
self.assertEqual(graph.get_retraction("alice")["reason"], "first")
def test_retracting_an_unknown_node_returns_false(self):
graph = _graph()
self.assertFalse(graph.retract_node("nobody"))
self.assertIsNone(graph.get_retraction("nobody"))
def test_cascade_retracts_incident_edges_in_both_directions(self):
graph = _graph()
graph.add_edge("bob", "alice", "knows") # inbound, not in _adjacency['alice']
graph.retract_node("alice", at=CUTOFF)
for edge in graph.edges:
self.assertIsNotNone(
graph.get_retraction(edge.edge_id),
f"edge {edge.edge_type} touching alice was not retracted",
)
def test_cascade_can_be_disabled(self):
graph = _graph()
graph.retract_node("alice", at=CUTOFF, cascade=False)
edge = graph.edges[0]
self.assertIsNone(graph.get_retraction(edge.edge_id))
def test_retraction_does_not_remove_the_record(self):
"""Retraction is a temporal change, not a deletion."""
graph = _graph()
graph.retract_node("alice", at=CUTOFF)
self.assertTrue(graph.has_node("alice"))
self.assertIsNotNone(graph.find_node("alice"))
class TestRetractEdge(unittest.TestCase):
def test_edge_is_retracted_without_touching_endpoints(self):
graph = _graph()
edge_id = graph.edges[0].edge_id
self.assertTrue(graph.retract_edge(edge_id, reason="wrong extraction"))
self.assertIsNotNone(graph.get_retraction(edge_id))
active = {node["id"] for node in graph.find_active_nodes()}
self.assertIn("alice", active)
self.assertIn("acme", active)
def test_retracting_an_unknown_edge_returns_false(self):
self.assertFalse(_graph().retract_edge("no-such-edge"))
def test_retracting_an_edge_twice_is_a_no_op(self):
graph = _graph()
edge_id = graph.edges[0].edge_id
self.assertTrue(graph.retract_edge(edge_id))
self.assertFalse(graph.retract_edge(edge_id))
class TestPurge(unittest.TestCase):
def test_purged_node_is_absent_from_history(self):
graph = _graph()
self.assertTrue(graph.purge_node("alice", reason="erasure request #1"))
self.assertNotIn("alice", _ids_at(graph, BEFORE))
self.assertFalse(graph.has_node("alice"))
def test_tombstone_records_the_purge_without_the_content(self):
graph = ContextGraph(advanced_analytics=False)
graph.add_node("alice", "person", email="alice@example.com")
graph.purge_node("alice", reason="erasure request #1")
tombstone = graph.get_tombstone("alice")
self.assertEqual(tombstone["entity_id"], "alice")
self.assertEqual(tombstone["reason"], "erasure request #1")
self.assertIn("purged_at", tombstone)
self.assertNotIn(
"alice@example.com",
str(tombstone),
"tombstone retained purged content, defeating the purpose of a purge",
)
def test_purge_cascades_to_incident_edges(self):
graph = _graph()
graph.add_edge("bob", "alice", "knows")
graph.purge_node("alice")
remaining = {(e.source_id, e.target_id) for e in graph.edges}
self.assertEqual(remaining, set())
def test_purge_keeps_every_index_consistent(self):
"""The invariant clear() already upholds must hold here too."""
graph = ContextGraph(advanced_analytics=False)
for i in range(5):
graph.add_node(f"n{i}", f"t{i % 2}")
graph.add_edge("n0", "n1", "a")
graph.add_edge("n1", "n2", "b")
graph.add_edge("n2", "n0", "a")
graph.add_edge("n3", "n0", "b")
graph.purge_node("n0")
totals = _index_totals(graph)
self.assertEqual(totals["node_index"], totals["nodes"])
self.assertEqual(totals["edge_index"], totals["edges"])
self.assertEqual(totals["adjacency"], totals["edges"])
self.assertEqual(totals["edges"], 1) # only n1->n2 survives
def test_purge_edge_leaves_endpoints_in_place(self):
graph = _graph()
edge_id = graph.edges[0].edge_id
self.assertTrue(graph.purge_edge(edge_id))
self.assertEqual(len(graph.edges), 0)
self.assertTrue(graph.has_node("alice"))
self.assertTrue(graph.has_node("acme"))
totals = _index_totals(graph)
self.assertEqual(totals["edge_index"], 0)
self.assertEqual(totals["adjacency"], 0)
def test_purging_unknown_entities_returns_false(self):
graph = _graph()
self.assertFalse(graph.purge_node("nobody"))
self.assertFalse(graph.purge_edge("no-such-edge"))
def test_purge_supersedes_an_earlier_retraction(self):
graph = _graph()
graph.retract_node("alice", reason="left", at=CUTOFF)
graph.purge_node("alice", reason="erasure request #2")
self.assertIsNone(graph.get_retraction("alice"))
self.assertIsNotNone(graph.get_tombstone("alice"))
class TestRetractionNeverWidensTheWindow(unittest.TestCase):
"""Retraction closes a validity window; it must never extend one.
An entity added with ``valid_until`` already in the past was inactive from
that point on. Overwriting the bound with a later retraction time would
make ``state_at`` report it active over a span it previously was not.
"""
def test_a_node_keeps_an_earlier_valid_until(self):
graph = ContextGraph(advanced_analytics=False)
graph.add_node("alice", "person", valid_until=BEFORE)
self.assertTrue(graph.retract_node("alice", at=AFTER))
self.assertEqual(graph.nodes["alice"].valid_until, BEFORE)
self.assertNotIn("alice", _ids_at(graph, BETWEEN))
def test_an_edge_keeps_an_earlier_valid_until(self):
graph = ContextGraph(advanced_analytics=False)
graph.add_node("alice", "person")
graph.add_node("acme", "org")
graph.add_edge("alice", "acme", "works_at", valid_until=BEFORE)
edge = graph.edges[0]
self.assertTrue(graph.retract_edge(edge.edge_id, at=AFTER))
self.assertEqual(edge.valid_until, BEFORE)
self.assertFalse(edge.is_active(datetime(2025, 9, 1)))
def test_cascade_keeps_an_earlier_edge_bound(self):
graph = ContextGraph(advanced_analytics=False)
graph.add_node("alice", "person")
graph.add_node("acme", "org")
graph.add_edge("alice", "acme", "works_at", valid_until=BEFORE)
graph.retract_node("alice", at=AFTER)
self.assertEqual(graph.edges[0].valid_until, BEFORE)
def test_an_open_window_is_still_closed_at_the_retraction_time(self):
graph = _graph()
graph.retract_node("alice", at=CUTOFF)
self.assertEqual(graph.nodes["alice"].valid_until, "2026-01-01T00:00:00")
class TestPurgeTimestamp(unittest.TestCase):
"""Purge accepts an explicit effective time, as retraction does."""
def test_node_tombstone_records_the_supplied_time(self):
graph = _graph()
graph.purge_node("alice", reason="erasure request #4", at=CUTOFF)
self.assertEqual(
graph.get_tombstone("alice")["purged_at"], "2026-01-01T00:00:00"
)
def test_edge_tombstone_records_the_supplied_time(self):
graph = _graph()
edge_id = graph.edges[0].edge_id
graph.purge_edge(edge_id, at=CUTOFF)
self.assertEqual(
graph.get_tombstone(edge_id)["purged_at"], "2026-01-01T00:00:00"
)
def test_cascaded_edge_tombstones_share_the_supplied_time(self):
graph = _graph()
edge_id = graph.edges[0].edge_id
graph.purge_node("alice", at=CUTOFF)
self.assertEqual(
graph.get_tombstone(edge_id)["purged_at"], "2026-01-01T00:00:00"
)
def test_purge_time_defaults_to_now(self):
graph = _graph()
graph.purge_node("alice")
self.assertIn("purged_at", graph.get_tombstone("alice"))
class TestIdKeyspaces(unittest.TestCase):
"""Node ids are caller-supplied and edge ids are UUIDs, so they can collide."""
def _colliding(self):
graph = _graph()
edge_id = graph.edges[0].edge_id
graph.add_node(edge_id, "person")
return graph, edge_id
def test_an_edge_retraction_does_not_block_a_colliding_node(self):
graph, edge_id = self._colliding()
self.assertTrue(graph.retract_edge(edge_id, reason="edge"))
self.assertTrue(graph.retract_node(edge_id, reason="node"))
self.assertEqual(graph.get_retraction(edge_id, "edge")["reason"], "edge")
self.assertEqual(graph.get_retraction(edge_id, "node")["reason"], "node")
def test_purging_a_node_leaves_a_colliding_edge_alone(self):
graph, edge_id = self._colliding()
self.assertTrue(graph.purge_node(edge_id))
self.assertEqual(len(graph.edges), 1)
self.assertIsNone(graph.get_tombstone(edge_id, "edge"))
self.assertIsNotNone(graph.get_tombstone(edge_id, "node"))
def test_an_unknown_entity_kind_is_rejected(self):
with self.assertRaises(ValueError):
_graph().get_retraction("alice", "vertex")
class TestDuplicateEdgeId(unittest.TestCase):
"""``edge_id`` is content-derived; before #926, two identical ``add_edge``
calls produced two edge objects sharing one id. #926 stops *new*
duplicates through ``add_edge``/``add_edges``, but a graph can still carry
one from a save made before that fix, or from any other path that builds
a ``ContextEdge`` directly -- so retraction/purge must still handle it.
Every duplicate must be reached, or a retraction/tombstone record can
claim an edge is gone/inactive while a live copy remains in the graph.
"""
def _duplicated(self):
"""A graph with two distinct ``ContextEdge`` objects sharing one
edge_id, reproducing pre-#926 (or any hand-built) duplicate state
without going through the now-deduping ``add_edge``.
"""
graph = _graph()
original = graph.edges[0]
duplicate = ContextEdge(
source_id=original.source_id,
target_id=original.target_id,
edge_type=original.edge_type,
weight=original.weight,
)
self.assertEqual(duplicate.edge_id, original.edge_id)
graph.edges.append(duplicate)
graph.edge_type_index[duplicate.edge_type].append(duplicate)
graph._adjacency[duplicate.source_id].append(duplicate)
edge_id = original.edge_id
self.assertEqual({e.edge_id for e in graph.edges}, {edge_id})
self.assertEqual(len(graph.edges), 2)
return graph, edge_id
def test_retract_edge_closes_every_duplicate(self):
graph, edge_id = self._duplicated()
self.assertTrue(graph.retract_edge(edge_id, reason="dup", at=CUTOFF))
for edge in graph.edges:
self.assertEqual(edge.valid_until, "2026-01-01T00:00:00")
self.assertFalse(edge.is_active(datetime(2026, 6, 1)))
def test_retract_node_cascade_closes_every_duplicate(self):
graph, edge_id = self._duplicated()
self.assertTrue(graph.retract_node("alice", at=CUTOFF))
for edge in graph.edges:
self.assertEqual(edge.valid_until, "2026-01-01T00:00:00")
def test_purge_edge_removes_every_duplicate(self):
graph, edge_id = self._duplicated()
self.assertTrue(graph.purge_edge(edge_id, reason="dup"))
self.assertFalse(any(e.edge_id == edge_id for e in graph.edges))
def test_purge_node_cascade_removes_every_duplicate(self):
graph, edge_id = self._duplicated()
self.assertTrue(graph.purge_node("alice"))
self.assertFalse(any(e.edge_id == edge_id for e in graph.edges))
def test_repeat_purge_edge_does_not_overwrite_the_tombstone(self):
"""Once every duplicate is gone, a second call must no-op, not
silently 'complete' the purge again and clobber the original record."""
graph, edge_id = self._duplicated()
self.assertTrue(graph.purge_edge(edge_id, reason="first"))
self.assertFalse(graph.purge_edge(edge_id, reason="second"))
self.assertEqual(graph.get_tombstone(edge_id)["reason"], "first")
class TestPurgeCrossGraphLinks(unittest.TestCase):
"""link_graph() registers a link, a marker node and a bridge edge."""
def _linked(self):
graph = _graph()
other = ContextGraph(advanced_analytics=False)
other.add_node("target", "topic")
return graph, other, graph.link_graph(other, "alice", "target")
def test_purging_the_source_removes_link_marker_and_registration(self):
graph, _, link_id = self._linked()
graph.purge_node("alice", reason="erasure request #5")
self.assertFalse(graph.has_node(f"__cross_graph_{link_id}"))
with self.assertRaises(KeyError):
graph.navigate_to(link_id)
totals = _index_totals(graph)
self.assertEqual(totals["node_index"], totals["nodes"])
self.assertEqual(totals["edge_index"], totals["edges"])
self.assertEqual(totals["adjacency"], totals["edges"])
def test_the_marker_purge_is_recorded_as_cascaded(self):
graph, _, link_id = self._linked()
graph.purge_node("alice")
tombstone = graph.get_tombstone(f"__cross_graph_{link_id}")
self.assertEqual(tombstone["cascaded_from"], "alice")
def test_a_purged_link_is_not_serialized(self):
graph, _, _ = self._linked()
graph.purge_node("alice")
with tempfile.TemporaryDirectory() as directory:
path = os.path.join(directory, "graph.json")
graph.save_to_file(path)
with open(path, encoding="utf-8") as handle:
data = json.load(handle)
self.assertEqual(data["links"], [])
def test_cascade_disabled_still_deregisters_the_link(self):
"""The source node is gone either way, so the link cannot resolve."""
graph, _, link_id = self._linked()
graph.purge_node("alice", cascade=False)
with self.assertRaises(KeyError):
graph.navigate_to(link_id)
self.assertTrue(graph.has_node(f"__cross_graph_{link_id}"))
def test_purging_the_bridge_edge_deregisters_the_link(self):
graph, _, link_id = self._linked()
bridge = next(
edge for edge in graph.edges if edge.metadata.get("link_id") == link_id
)
self.assertTrue(graph.purge_edge(bridge.edge_id))
with self.assertRaises(KeyError):
graph.navigate_to(link_id)
def test_purging_the_marker_node_deregisters_the_link(self):
graph, _, link_id = self._linked()
self.assertTrue(graph.purge_node(f"__cross_graph_{link_id}"))
with self.assertRaises(KeyError):
graph.navigate_to(link_id)
self.assertTrue(graph.has_node("alice"))
def test_an_unrelated_link_survives(self):
graph, other, link_id = self._linked()
graph.purge_node("bob")
self.assertEqual(graph.navigate_to(link_id), (other, "target"))
class TestClearResetsRecords(unittest.TestCase):
def test_clear_drops_retractions_and_tombstones(self):
graph = _graph()
graph.retract_node("alice", at=CUTOFF)
graph.purge_node("bob")
graph.clear()
self.assertEqual(graph.list_retractions(), [])
self.assertEqual(graph.list_tombstones(), [])
def test_load_from_file_drops_records_from_the_previous_graph(self):
source = _graph()
graph = ContextGraph(advanced_analytics=False)
graph.add_node("alice", "person")
graph.add_node("carol", "person")
graph.retract_node("alice", at=CUTOFF)
graph.purge_node("carol")
with tempfile.TemporaryDirectory() as directory:
path = os.path.join(directory, "graph.json")
source.save_to_file(path)
graph.load_from_file(path)
self.assertEqual(graph.list_retractions(), [])
self.assertEqual(graph.list_tombstones(), [])
# The reloaded alice is a fresh record, not one already retracted.
self.assertTrue(graph.retract_node("alice", at=CUTOFF))
class TestAuditTrailIntegration(unittest.TestCase):
"""Against the real TemporalVersionManager, not a mock callback."""
def _attached(self):
manager = TemporalVersionManager()
graph = _graph()
manager.attach_to_graph(graph)
return manager, graph
def _ops(self, manager, entity_id):
history = manager.storage.get_entity_history(entity_id) or []
return [entry.get("operation") for entry in history]
def test_retraction_is_recorded_as_an_update(self):
manager, graph = self._attached()
graph.retract_node("alice", reason="left", at=CUTOFF)
self.assertIn("UPDATE_NODE", self._ops(manager, "alice"))
def test_purge_is_recorded_as_a_removal(self):
manager, graph = self._attached()
graph.purge_node("acme", reason="erasure request #3")
self.assertIn("REMOVE_NODE", self._ops(manager, "acme"))
def test_operations_use_the_documented_mutation_vocabulary(self):
"""MutationRecord documents ADD/UPDATE/REMOVE for nodes and edges."""
manager, graph = self._attached()
graph.retract_node("alice", at=CUTOFF)
graph.purge_node("bob")
allowed = {
"ADD_NODE",
"UPDATE_NODE",
"REMOVE_NODE",
"ADD_EDGE",
"UPDATE_EDGE",
"REMOVE_EDGE",
}
seen = set()
for entity_id in ("alice", "acme", "bob"):
seen.update(self._ops(manager, entity_id))
self.assertTrue(seen)
self.assertTrue(
seen <= allowed, f"undocumented mutation operation(s): {seen - allowed}"
)
class TestMutationEmissionIsSelfContained(unittest.TestCase):
"""Audit payloads must be snapshotted before the lock is released.
The callback fires outside the lock, so anything read from
``_retractions``/``_tombstones`` at emission time can already have been
wiped by a concurrent ``clear()``. A callback that clears the graph on its
first call stands in for that interleaving deterministically.
"""
def _clearing_callback(self, graph, seen):
def callback(operation, entity_id, payload):
seen.append((operation, entity_id, payload))
if len(seen) == 1:
graph.clear()
return callback
def test_purge_emits_every_mutation_after_a_concurrent_clear(self):
graph = _graph()
graph.add_edge("bob", "alice", "knows")
seen = []
graph.mutation_callback = self._clearing_callback(graph, seen)
self.assertTrue(graph.purge_node("alice", reason="erasure request #6"))
self.assertEqual(
[operation for operation, _, _ in seen],
["REMOVE_EDGE", "REMOVE_EDGE", "REMOVE_NODE"],
)
for _, entity_id, payload in seen:
self.assertEqual(payload["entity_id"], entity_id)
self.assertEqual(payload["reason"], "erasure request #6")
def test_retraction_emits_every_mutation_after_a_concurrent_clear(self):
graph = _graph()
graph.add_edge("bob", "alice", "knows")
seen = []
graph.mutation_callback = self._clearing_callback(graph, seen)
self.assertTrue(graph.retract_node("alice", reason="left", at=CUTOFF))
self.assertEqual(
[operation for operation, _, _ in seen],
["UPDATE_NODE", "UPDATE_EDGE", "UPDATE_EDGE"],
)
for _, _, payload in seen:
self.assertEqual(payload["retraction"]["reason"], "left")
class TestConcurrency(unittest.TestCase):
def test_concurrent_purges_keep_indexes_consistent(self):
"""Post-condition, not timing: threads must finish and indexes agree."""
graph = ContextGraph(advanced_analytics=False)
for i in range(60):
graph.add_node(f"n{i}", "t")
for i in range(59):
graph.add_edge(f"n{i}", f"n{i + 1}", "rel")
errors = []
def purge(start):
try:
for i in range(start, 60, 4):
graph.purge_node(f"n{i}")
except Exception as exc: # surfaced below, never swallowed
errors.append(f"{type(exc).__name__}: {exc}")
threads = [
threading.Thread(target=purge, args=(offset,)) for offset in range(4)
]
for thread in threads:
thread.start()
for thread in threads:
thread.join(timeout=30)
self.assertEqual([t.name for t in threads if t.is_alive()], [])
self.assertEqual(errors, [])
self.assertEqual(len(graph.nodes), 0)
totals = _index_totals(graph)
self.assertEqual(totals["node_index"], 0)
self.assertEqual(totals["edge_index"], 0)
self.assertEqual(totals["adjacency"], 0)
self.assertEqual(totals["edges"], 0)
if __name__ == "__main__":
unittest.main()
+224 -1
View File
@@ -1,5 +1,6 @@
"""Integration tests for the explorer API."""
from datetime import datetime
import json
from pathlib import Path
import uuid
@@ -444,6 +445,63 @@ class TestDecisions:
assert violation_response.json()["compliant"] is False
@pytest.fixture(scope="module")
def recorded_client():
"""Client over a graph whose decisions were written by record_decision()."""
graph = ContextGraph(advanced_analytics=False)
entities = ["applicant_A7291"]
graph.record_decision(
category="credit_application",
scenario="Personal loan, $85k income, 31% DTI",
reasoning="Income meets threshold; employment stable",
outcome="proceed_to_underwriting",
confidence=0.88,
entities=entities,
)
graph.record_decision(
category="loan_underwriting",
scenario="Underwriting review for A-7291",
reasoning="DTI within policy; clean 36-month credit history",
outcome="approved",
confidence=0.94,
entities=entities,
)
with TestClient(create_app(session=GraphSession(graph))) as test_client:
yield test_client
class TestRecordedDecisions:
"""Decisions written by record_decision(), not hand-built decision nodes.
record_decision() stores ``timestamp`` as a float epoch. The fixtures above
set no timestamp at all, so these routes were only ever exercised against
decision nodes that could not trigger the float/str mismatch.
"""
def test_list_decisions_serializes_float_timestamp(self, recorded_client):
response = recorded_client.get("/api/decisions")
assert response.status_code == 200
payload = response.json()
assert len(payload) == 2
for item in payload:
assert isinstance(item["timestamp"], str)
datetime.fromisoformat(item["timestamp"])
def test_get_decision(self, recorded_client):
listed = recorded_client.get("/api/decisions").json()
decision_id = listed[0]["decision_id"]
response = recorded_client.get(f"/api/decisions/{decision_id}")
assert response.status_code == 200
assert response.json()["decision_id"] == decision_id
def test_filter_by_category(self, recorded_client):
response = recorded_client.get("/api/decisions?category=loan_underwriting")
assert response.status_code == 200
payload = response.json()
assert len(payload) == 1
assert payload[0]["outcome"] == "approved"
class TestTemporal:
def test_snapshot_now(self, client):
response = client.get("/api/temporal/snapshot")
@@ -602,7 +660,20 @@ class TestEnrichment:
def test_extract(self, client):
response = client.post("/api/enrich/extract", json={"text": "Alice works at Acme Corp."})
assert response.status_code in (200, 422, 503)
# 503 is reserved for a genuinely absent semantic_extract module; it must
# not be reachable on an install where the module imports cleanly.
# Runtime errors from the extraction stack surface as 500, not 422.
assert response.status_code in (200, 422, 500)
def test_extract_returns_entities(self, client):
response = client.post(
"/api/enrich/extract",
json={"text": "Apple CEO Tim Cook announced record earnings in Cupertino."},
)
assert response.status_code == 200
payload = response.json()
assert payload["entities"], "extraction returned no entities"
assert any("Tim Cook" in str(entity) for entity in payload["entities"])
def test_link_prediction(self, client):
response = client.post("/api/enrich/links", json={"node_id": "python", "top_n": 5})
@@ -1157,3 +1228,155 @@ class TestClassifyDistance:
def test_large_hop_count_is_distant(self):
assert classify_path_distance(20) == "distant"
# ---------------------------------------------------------------------------
# Timestamp validator unit tests (no HTTP server needed)
# ---------------------------------------------------------------------------
class TestDecisionResponseTimestampValidator:
"""Unit tests for DecisionResponse._normalize_timestamp.
These run directly against the Pydantic model, not through the HTTP stack,
so they are fast and isolated from the rest of the Explorer infrastructure.
"""
def _make(self, ts):
from semantica.explorer.schemas import DecisionResponse
import pytest as _pytest
return DecisionResponse(decision_id="x", timestamp=ts)
def test_none_passes_through(self):
from semantica.explorer.schemas import DecisionResponse
dr = DecisionResponse(decision_id="x", timestamp=None)
assert dr.timestamp is None
def test_string_passes_through_unchanged(self):
from semantica.explorer.schemas import DecisionResponse
iso = "2024-08-14T10:23:45+00:00"
dr = DecisionResponse(decision_id="x", timestamp=iso)
assert dr.timestamp == iso
def test_float_epoch_becomes_iso_string(self):
from datetime import datetime, timezone
from semantica.explorer.schemas import DecisionResponse
epoch = 1723600000.5
dr = DecisionResponse(decision_id="x", timestamp=epoch)
assert isinstance(dr.timestamp, str)
parsed = datetime.fromisoformat(dr.timestamp)
assert abs(parsed.timestamp() - epoch) < 1.0
def test_int_epoch_becomes_iso_string(self):
from datetime import datetime
from semantica.explorer.schemas import DecisionResponse
epoch = 1723600000
dr = DecisionResponse(decision_id="x", timestamp=epoch)
assert isinstance(dr.timestamp, str)
datetime.fromisoformat(dr.timestamp)
def test_nan_raises_validation_error(self):
import math
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp=math.nan)
def test_positive_inf_raises_validation_error(self):
import math
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp=math.inf)
def test_negative_inf_raises_validation_error(self):
import math
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp=-math.inf)
def test_dict_raises_validation_error(self):
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp={"$date": 1723600000})
def test_list_raises_validation_error(self):
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp=[1723600000])
def test_bool_raises_validation_error(self):
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp=True)
def test_oserror_range_epoch_raises_validation_error(self):
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
# Milliseconds mistakenly stored where seconds were expected.
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp=1723600000000)
def test_overflow_range_epoch_raises_validation_error(self):
import pytest
from pydantic import ValidationError
from semantica.explorer.schemas import DecisionResponse
with pytest.raises(ValidationError):
DecisionResponse(decision_id="x", timestamp=1e20)
# ---------------------------------------------------------------------------
# /api/enrich/extract input-size and import-boundary tests
# ---------------------------------------------------------------------------
class TestEnrichExtractValidation:
"""Tests for the input constraints and exception handling added to
POST /api/enrich/extract."""
def test_oversized_input_rejected_before_nlp(self, client):
"""A payload exceeding the 10 000-character limit must be rejected with
422 before any NLP work is attempted."""
oversized = "a " * 5_001 # 10 002 characters
response = client.post("/api/enrich/extract", json={"text": oversized})
assert response.status_code == 422
def test_input_at_limit_is_accepted(self, client):
"""A payload at exactly the maximum length must not be rejected by the
schema validator (NLP may still fail, but the schema must accept it)."""
at_limit = "a" * 10_000
response = client.post("/api/enrich/extract", json={"text": at_limit})
# 503 = module missing, 500 = runtime error from the extraction stack,
# 200 = success. What must NOT happen is a schema rejection (422 from
# Pydantic due to max_length), since this input is exactly at the limit.
assert response.status_code in (200, 500, 503)
def test_import_failure_returns_503_not_422(self, client, monkeypatch):
"""A genuine ImportError on the semantic_extract import must produce 503
(dependency unavailable), NOT 422 (extraction failed)."""
import semantica.explorer.routes.enrich as enrich_module
def _failing_import(name, *args, **kwargs):
if "semantic_extract" in name:
raise ImportError("semantic_extract not installed")
return original_import(name, *args, **kwargs)
import builtins
original_import = builtins.__import__
monkeypatch.setattr(builtins, "__import__", _failing_import)
response = client.post(
"/api/enrich/extract",
json={"text": "Apple was founded by Steve Jobs."},
)
assert response.status_code == 503
assert "semantic_extract" in response.json()["detail"].lower()
+57
View File
@@ -301,3 +301,60 @@ def test_nested_properties_are_json_serialized(tmp_path):
by_id = {row[0]: row for row in rows[1:]}
assert by_id["node1"][2] == '{"k":"v"}'
assert by_id["node1"][3] == "[1,2,3]"
def test_unrecognized_mapping_is_refused_rather_than_exported_empty(tmp_path):
"""The Neo4j path reads mappings on the shared normalizer's default terms.
An ``export_json`` envelope names no graph key, so it resolves to nothing.
Written out, that is a pair of header-only CSVs indistinguishable from a
genuinely empty graph -- the silent-empty export the shared contract
exists to prevent.
"""
exporter = Neo4jCSVExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export({"data": [{"id": "e1"}]}, tmp_path)
message = str(excinfo.value)
assert "data" in message
assert "entities" in message
assert not (tmp_path / "nodes.csv").exists()
assert not (tmp_path / "relationships.csv").exists()
def test_records_under_an_unread_key_are_not_dropped_silently(tmp_path):
"""Naming a recognized key is not enough if nothing resolves from it."""
exporter = Neo4jCSVExporter()
with pytest.raises(ValidationError):
exporter.export({"nodes": [], "data": [{"id": "e1"}]}, tmp_path)
assert not (tmp_path / "nodes.csv").exists()
def test_malformed_collection_value_is_refused(tmp_path):
"""``list("abc")`` would otherwise export one node per character."""
exporter = Neo4jCSVExporter()
for value in ("abc", 42, {"id": "n1"}):
with pytest.raises(ValidationError) as excinfo:
exporter.export({"nodes": value}, tmp_path)
assert "nodes" in str(excinfo.value)
assert not (tmp_path / "nodes.csv").exists()
def test_graph_objects_still_use_the_attribute_path(tmp_path):
"""Only mappings changed; objects are not mappings and are unaffected."""
class Graph:
def __init__(self):
self.nodes = [{"id": "e1", "type": "Person", "name": "Acme"}]
self.edges = []
exporter = Neo4jCSVExporter()
exporter.export(Graph(), tmp_path)
assert "Acme" in (tmp_path / "nodes.csv").read_text(encoding="utf-8")
@@ -0,0 +1,181 @@
"""Regression tests for YAML export input validation (issue #952).
``export_yaml`` declared ``Union[Dict[str, Any], List[Dict[str, Any]]]`` but
both YAML exporters read their payload by key, so a list reached
``semantic_network.get(...)`` and surfaced as a bare
``AttributeError: 'list' object has no attribute 'get'`` from inside the
exporter an error that names neither the offending argument nor the shape
expected.
A list is rejected rather than wrapped. These formats distinguish entities
from relationships from triplets, so inferring which collection a bare list
represents would silently mislabel the records; and wrapping it under an
unrecognised key would write a structurally valid file with every collection
empty, trading a loud failure for silent data loss.
Both directions are pinned: non-mappings raise ``ProcessingError`` with an
actionable message, and every mapping that worked before still exports.
"""
import os
import shutil
import tempfile
import unittest
from collections import OrderedDict, defaultdict
import yaml
from semantica.export.methods import export_yaml
from semantica.export.yaml_exporter import (
SemanticNetworkYAMLExporter,
YAMLSchemaExporter,
)
from semantica.utils.exceptions import ProcessingError
# Non-mapping payloads that must be rejected. A list of dicts is the shape
# from #952; the rest guard the same path against other sequence/scalar types.
NON_MAPPINGS = {
"list_of_dicts": [{"id": "1", "name": "Acme"}],
"empty_list": [],
"tuple_of_dicts": ({"id": "1"},),
"list_of_scalars": ["a", "b"],
"string": "entities",
"bytes": b"entities",
"int": 42,
"none": None,
"set": {"a"},
}
# Both YAML methods, with a minimal valid payload and the key names the
# corresponding error message must mention.
METHODS = {
"semantic_network": {
"valid": {
"entities": [{"id": "1", "name": "Acme"}],
"relationships": [],
"triplets": [],
},
"expected_key": "entities",
"top_level_key": "entities",
},
"schema": {
"valid": {"classes": [{"name": "Thing"}], "properties": []},
"expected_key": "classes",
"top_level_key": "classes",
},
}
class TestExportYamlRejectsNonMappings(unittest.TestCase):
"""Non-mapping input fails loudly, through the public wrapper."""
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True)
def _path(self, name="out.yaml"):
return os.path.join(self.tmpdir, name)
def test_fixture_tables_are_populated(self):
"""Guard against a vacuous suite.
Every test below iterates a table; emptying or renaming one would let
those loops pass without asserting anything.
"""
self.assertGreaterEqual(len(NON_MAPPINGS), 9)
self.assertEqual(set(METHODS), {"semantic_network", "schema"})
def test_non_mapping_raises_processing_error(self):
for method in METHODS:
for label, payload in NON_MAPPINGS.items():
with self.subTest(method=method, case=label):
with self.assertRaises(ProcessingError):
export_yaml(payload, self._path(), method=method)
def test_error_names_the_offending_type_and_expected_keys(self):
"""The message must be actionable, not just the right exception type."""
for method, spec in METHODS.items():
with self.subTest(method=method):
with self.assertRaises(ProcessingError) as ctx:
export_yaml([{"id": "1"}], self._path(), method=method)
message = str(ctx.exception)
self.assertIn("list", message)
self.assertIn(spec["expected_key"], message)
def test_no_file_is_written_when_input_is_rejected(self):
"""A rejected export must not leave a partial or empty artefact."""
for method in METHODS:
with self.subTest(method=method):
path = self._path(f"{method}_rejected.yaml")
with self.assertRaises(ProcessingError):
export_yaml([{"id": "1"}], path, method=method)
self.assertFalse(os.path.exists(path))
def test_exporter_classes_reject_non_mappings_directly(self):
"""Validation lives in the exporters, not only the convenience wrapper.
Callers using the classes directly get the same contract.
"""
for label, payload in NON_MAPPINGS.items():
with self.subTest(exporter="SemanticNetworkYAMLExporter", case=label):
with self.assertRaises(ProcessingError):
SemanticNetworkYAMLExporter().export_semantic_network(payload)
with self.subTest(exporter="YAMLSchemaExporter", case=label):
with self.assertRaises(ProcessingError):
YAMLSchemaExporter().export_ontology_schema(payload)
class TestExportYamlStillAcceptsMappings(unittest.TestCase):
"""Everything that exported before must still export."""
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True)
def _path(self, name="out.yaml"):
return os.path.join(self.tmpdir, name)
def _load(self, path):
with open(path, encoding="utf-8") as handle:
return yaml.safe_load(handle)
def test_valid_mapping_exports_for_each_method(self):
for method, spec in METHODS.items():
with self.subTest(method=method):
path = self._path(f"{method}.yaml")
export_yaml(spec["valid"], path, method=method)
self.assertTrue(os.path.exists(path))
loaded = self._load(path)
self.assertIn(spec["top_level_key"], loaded)
def test_semantic_network_records_survive_the_round_trip(self):
path = self._path("network.yaml")
export_yaml(METHODS["semantic_network"]["valid"], path)
loaded = self._load(path)
self.assertEqual(loaded["entities"], [{"id": "1", "name": "Acme"}])
def test_empty_mapping_is_still_accepted(self):
"""An empty dict is a mapping; rejecting it would be a behaviour change."""
for method in METHODS:
with self.subTest(method=method):
path = self._path(f"{method}_empty.yaml")
export_yaml({}, path, method=method)
self.assertTrue(os.path.exists(path))
def test_mapping_subclasses_are_accepted(self):
"""Validation is by Mapping, not dict, so these must keep working."""
valid = METHODS["semantic_network"]["valid"]
subclasses = {
"OrderedDict": OrderedDict(valid),
"defaultdict": defaultdict(list, valid),
}
for label, payload in subclasses.items():
with self.subTest(case=label):
path = self._path(f"{label}.yaml")
export_yaml(payload, path)
loaded = self._load(path)
self.assertEqual(loaded["entities"], valid["entities"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,434 @@
"""Tests for YAML export key recognition (issue #953).
``SemanticNetworkYAMLExporter`` built its output from ``.get(key, [])``
lookups, so a mapping keyed by anything it did not read -- an ``export_json``
envelope, a typo'd 'entitys', ``ContextGraph.to_dict()``'s 'nodes'/'edges' --
serialized to a structurally valid file with every collection empty. Nothing
signalled the loss: no exception, no warning, and the progress log reported a
completed export. ``YAMLSchemaExporter`` had the same defect over a different
key set.
The exporters are run for real rather than mocked, and the written files are
parsed back, since the behaviour under test is what actually lands on disk.
"""
from pathlib import Path
import pytest
import yaml
from semantica.context.context_graph import ContextGraph
from semantica.export.methods import export_json, export_yaml
from semantica.export.yaml_exporter import (
SemanticNetworkYAMLExporter,
YAMLSchemaExporter,
)
from semantica.utils.exceptions import ProcessingError, ValidationError
ENTITIES = [{"id": "e1", "name": "Acme"}, {"id": "e2", "name": "Beta"}]
RELATIONSHIPS = [{"id": "r1", "source": "e1", "target": "e2", "type": "PARTNER"}]
TRIPLETS = [{"subject": "e1", "predicate": "partner_of", "object": "e2"}]
def _load(path):
with open(path, "r", encoding="utf-8") as handle:
return yaml.safe_load(handle)
class TestSemanticNetworkKeyRecognition:
"""An unrecognized mapping is refused instead of silently emptied."""
def test_export_json_envelope_is_rejected(self, tmp_path):
"""The realistic trigger: re-exporting an export_json payload.
``export_json`` wraps records as ``{"data": [...], "count": N,
"metadata": {...}}``. Feeding that straight to ``export_yaml`` used to
write a file with every record gone. Note the envelope's 'metadata'
key is deliberately not enough to make the payload recognized --
treating it as sufficient would readmit exactly this case.
"""
json_path = tmp_path / "records.json"
export_json(ENTITIES, json_path)
envelope = yaml.safe_load(json_path.read_text(encoding="utf-8"))
assert "data" in envelope and "metadata" in envelope
yaml_path = tmp_path / "records.yaml"
with pytest.raises(ValidationError) as excinfo:
export_yaml(envelope, yaml_path)
message = str(excinfo.value)
assert "'data'" in message, "error should name the supplied keys"
assert "'entities'" in message, "error should name the expected keys"
assert not yaml_path.exists(), "a rejected export must write nothing"
@pytest.mark.parametrize(
"payload",
[
{"records": ENTITIES},
{"entitys": ENTITIES},
{"data": ENTITIES},
{"metadata": {"source": "test"}},
],
ids=["records", "typo", "data", "metadata-only"],
)
def test_unrecognized_mappings_are_rejected(self, payload):
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export_semantic_network(payload)
@pytest.mark.parametrize(
"payload",
[
{"entities": [], "data": ENTITIES},
{"nodes": [], "edges": [], "records": ENTITIES},
{"triplets": [], "data": ENTITIES, "metadata": {"source": "test"}},
],
ids=["entities-empty", "nodes-edges-empty", "triplets-empty"],
)
def test_recognized_but_empty_with_records_elsewhere_is_rejected(self, payload):
"""Presence of a recognized key is not proof the records survived.
``{"entities": [], "data": [...]}`` clears a presence-only check and
still resolves to empty, dropping everything under 'data' -- the same
silent-empty export by a narrower route.
"""
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export_semantic_network(payload)
message = str(excinfo.value)
assert "holds records" in message
assert "'entities'" in message, "error should name where records belong"
def test_empty_graph_with_non_record_keys_still_exports(self, tmp_path):
"""The rejection must key on dropped *records*, not on unread keys.
``ContextGraph.to_dict()`` always carries a populated 'statistics'
dict, so an empty graph would be refused if any unread key counted.
"""
graph = ContextGraph()
path = tmp_path / "empty_graph.yaml"
export_yaml(graph.to_dict(), path)
written = _load(path)
assert written["entities"] == []
assert written["relationships"] == []
def test_empty_mapping_still_exports(self, tmp_path):
"""An empty graph is legitimate and carries nothing that could be lost."""
path = tmp_path / "empty.yaml"
export_yaml({}, path)
written = _load(path)
assert written["entities"] == []
assert written["relationships"] == []
assert written["triplets"] == []
def test_recognized_keys_still_export(self, tmp_path):
path = tmp_path / "network.yaml"
export_yaml(
{
"entities": ENTITIES,
"relationships": RELATIONSHIPS,
"triplets": TRIPLETS,
"metadata": {"source": "test"},
},
path,
)
written = _load(path)
assert written["entities"] == ENTITIES
assert written["relationships"] == RELATIONSHIPS
assert written["triplets"] == TRIPLETS
assert written["metadata"]["source"] == "test"
def test_nodes_edges_alias_exports_records(self, tmp_path):
path = tmp_path / "aliased.yaml"
export_yaml({"nodes": ENTITIES, "edges": RELATIONSHIPS}, path)
written = _load(path)
assert written["entities"] == ENTITIES
assert written["relationships"] == RELATIONSHIPS
def test_context_graph_to_dict_round_trips(self, tmp_path):
"""The most direct path from this library's own graph type to YAML.
Built from a real ``ContextGraph`` rather than a hand-written
'nodes'/'edges' dict, so the test breaks if ``to_dict()`` changes
vocabulary.
"""
graph = ContextGraph()
graph.add_node("n1", node_type="Person", content="Alice")
graph.add_node("n2", node_type="Org", content="Acme")
graph.add_edge("n1", "n2", "WORKS_FOR")
path = tmp_path / "context.yaml"
export_yaml(graph.to_dict(), path)
written = _load(path)
assert len(written["entities"]) == 2
assert len(written["relationships"]) == 1
def test_conflicting_spellings_are_refused(self):
"""Two populated spellings of one collection: no basis to pick either."""
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export_semantic_network(
{"entities": ENTITIES, "nodes": [{"id": "other"}]}
)
def test_non_mapping_raises_processing_error(self):
"""A wrong type is a different failure from a wrong-keyed mapping.
ProcessingError says the object cannot be exported at all;
ValidationError says the mapping's contents are unusable. Pinned here
so the two do not quietly converge.
"""
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ProcessingError):
exporter.export_semantic_network(ENTITIES)
def test_rejected_export_creates_no_output_directory(self, tmp_path):
"""Validation runs before the output directory is created."""
target = tmp_path / "nested" / "out.yaml"
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export({"data": ENTITIES}, target)
assert not target.parent.exists()
class TestPipelineExportKeyRecognition:
"""export_for_pipeline read the same defaulted lookups, so it had the bug too."""
def test_unrecognized_mapping_is_rejected(self):
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ValidationError):
exporter.export_for_pipeline({"data": ENTITIES})
def test_non_mapping_raises_processing_error(self):
exporter = SemanticNetworkYAMLExporter()
with pytest.raises(ProcessingError):
exporter.export_for_pipeline(ENTITIES)
def test_aliases_resolve_into_the_semantic_network(self):
exporter = SemanticNetworkYAMLExporter()
written = yaml.safe_load(
exporter.export_for_pipeline({"nodes": ENTITIES, "edges": RELATIONSHIPS})
)
assert written["semantic_network"]["entities"] == ENTITIES
assert written["semantic_network"]["relationships"] == RELATIONSHIPS
def test_metadata_is_preserved(self):
exporter = SemanticNetworkYAMLExporter()
written = yaml.safe_load(
exporter.export_for_pipeline(
{"entities": ENTITIES, "metadata": {"source": "test"}}
)
)
assert written["metadata"]["source"] == "test"
assert written["semantic_network"]["entities"] == ENTITIES
class TestSchemaKeyRecognition:
"""method="schema" emitted empty classes/properties/namespaces the same way."""
def test_unrecognized_mapping_is_rejected(self, tmp_path):
path = tmp_path / "schema.yaml"
with pytest.raises(ValidationError) as excinfo:
export_yaml({"nodes": [{"id": "1"}]}, path, method="schema")
message = str(excinfo.value)
assert "'nodes'" in message
assert "'classes'" in message
assert not path.exists()
def test_non_mapping_raises_processing_error(self):
exporter = YAMLSchemaExporter()
with pytest.raises(ProcessingError):
exporter.export_ontology_schema([{"id": "1"}])
def test_recognized_but_empty_with_records_elsewhere_is_rejected(self):
"""The schema path had the same presence-only hole."""
exporter = YAMLSchemaExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export_ontology_schema({"classes": [], "nodes": [{"id": "1"}]})
assert "holds records" in str(excinfo.value)
def test_ontology_metadata_without_records_still_exports(self):
"""A schema described only by its identity is not a dropped export."""
exporter = YAMLSchemaExporter()
written = yaml.safe_load(
exporter.export_ontology_schema(
{"uri": "http://example.org/o", "classes": []}
)
)
assert written["ontology"]["uri"] == "http://example.org/o"
assert written["classes"] == []
def test_empty_mapping_still_exports(self, tmp_path):
path = tmp_path / "schema.yaml"
export_yaml({}, path, method="schema")
written = _load(path)
assert written["classes"] == []
assert written["properties"] == []
assert written["namespaces"] == {}
@pytest.mark.parametrize(
"payload",
[
{"classes": ["Person"], "properties": ["WORKS_FOR"]},
{"namespaces": {"ex": "http://example.org/"}},
{"uri": "http://example.org/ontology"},
],
ids=["classes-properties", "namespaces-only", "uri-only"],
)
def test_recognized_keys_still_export(self, payload, tmp_path):
path = tmp_path / "schema.yaml"
export_yaml(payload, path, method="schema")
written = _load(path)
assert written["classes"] == payload.get("classes", [])
assert written["properties"] == payload.get("properties", [])
assert written["ontology"]["uri"] == payload.get("uri", "")
# ── Fix regression: scalar recognized keys must not short-circuit the ──
# ── dropped-records check (version, uri, title, description). ──────────
@pytest.mark.parametrize(
"scalar_key, scalar_value",
[
("version", "1.0"),
("uri", "http://example.org/ontology"),
("title", "My Ontology"),
("description", "A test ontology"),
],
ids=["version", "uri", "title", "description"],
)
def test_scalar_recognized_key_does_not_excuse_records_under_unread_key(
self, scalar_key, scalar_value
):
"""A truthy scalar such as version='1.0' must not silence the dropped-
records check. Before the fix, any truthy value from _SCHEMA_KEYS
would make _require_nothing_dropped believe something resolved and
return early, silently discarding a list under an unread key.
"""
exporter = YAMLSchemaExporter()
with pytest.raises(ValidationError) as excinfo:
exporter.export_ontology_schema(
{scalar_key: scalar_value, "nodes": [{"id": "c1"}]}
)
assert "holds records" in str(excinfo.value), str(excinfo.value)
def test_valid_classes_with_scalar_metadata_is_accepted(self):
"""classes/properties populated alongside version/uri must still work."""
exporter = YAMLSchemaExporter()
written = yaml.safe_load(
exporter.export_ontology_schema(
{
"classes": [{"id": "Person"}],
"properties": [{"id": "name"}],
"version": "2.0",
"uri": "http://example.org/o",
}
)
)
assert written["classes"] == [{"id": "Person"}]
assert written["properties"] == [{"id": "name"}]
assert written["ontology"]["version"] == "2.0"
assert written["ontology"]["uri"] == "http://example.org/o"
class TestFailureIsObservable:
"""The complaint in #953 was that the logs affirmatively reported success."""
def test_no_success_is_logged_for_a_rejected_export(self, tmp_path, caplog):
path = tmp_path / "out.yaml"
with caplog.at_level("DEBUG"):
with pytest.raises(ValidationError):
export_yaml({"data": ENTITIES}, path)
assert "Exported YAML to" not in caplog.text
assert any(
record.levelname in ("WARNING", "ERROR", "CRITICAL")
for record in caplog.records
), "a rejected export should leave something at warning or above"
class _RecordingTracker:
"""Records the exporter's own progress calls, which are what is under test."""
def __init__(self):
self.stopped = []
self._next_id = 0
def start_tracking(self, **kwargs):
self._next_id += 1
return str(self._next_id)
def update_tracking(self, tracking_id, **kwargs):
pass
def stop_tracking(self, tracking_id, status=None, message=None):
self.stopped.append((status, message))
class TestProgressReflectsTheWrite:
"""Serialization completing is not the same as the file landing on disk."""
def test_failed_write_is_not_reported_as_completed(self, tmp_path):
"""A write failure after serialization must not leave a clean tracker.
The path's parent is an existing *file*, so directory creation fails
after `export_semantic_network` has already reported its own
completion.
"""
blocker = tmp_path / "blocker"
blocker.write_text("not a directory", encoding="utf-8")
target = blocker / "nested" / "out.yaml"
exporter = SemanticNetworkYAMLExporter()
tracker = _RecordingTracker()
exporter.progress_tracker = tracker
with pytest.raises(OSError):
exporter.export({"entities": ENTITIES}, target)
assert not target.exists()
statuses = [status for status, _ in tracker.stopped]
assert "failed" in statuses, f"write failure went unreported: {tracker.stopped}"
assert not any(
status == "completed" and "Exported YAML" in (message or "")
for status, message in tracker.stopped
), "no span may claim a completed export when nothing was written"
def test_successful_write_is_reported_as_completed(self, tmp_path):
target = tmp_path / "out.yaml"
exporter = SemanticNetworkYAMLExporter()
tracker = _RecordingTracker()
exporter.progress_tracker = tracker
exporter.export({"entities": ENTITIES}, target)
assert target.exists()
assert all(status == "completed" for status, _ in tracker.stopped)
assert any(
"Exported YAML" in (message or "") for _, message in tracker.stopped
), "the write should report its own completion, not just serialization"
class TestUnaffectedExporters:
"""export_json's own behaviour is untouched -- only the YAML path changed."""
def test_export_json_still_accepts_a_bare_list(self, tmp_path):
path = tmp_path / "records.json"
export_json(ENTITIES, path)
assert Path(path).exists()
+151
View File
@@ -0,0 +1,151 @@
"""
Shared pytest configuration for CrewAI integration tests.
Installs comprehensive crewai stubs into sys.modules before any test in this
directory runs, so every test file can import the integration modules with
``CREWAI_AVAILABLE == True`` and exercise the real subclassing code paths
without a real crewai installation.
The stubs mirror the current CrewAI contracts:
- ``crewai.tools.BaseTool`` Pydantic ``BaseModel`` (arbitrary types allowed)
- ``crewai.knowledge.source.base_knowledge_source.BaseKnowledgeSource``
Pydantic model with ``validate_content``/``add``/``aadd`` abstract methods
and ``_chunk_text``/``_save_documents`` helpers.
The graceful-degradation path (crewai genuinely absent) is covered separately
in ``test_degradation.py`` via a subprocess, so this stub never has to be torn
down mid-session.
"""
from __future__ import annotations
import sys
import types
from typing import Any, Optional
from pydantic import BaseModel, ConfigDict, Field, field_serializer, field_validator
def _install_crewai_stubs() -> None:
"""Install a full set of crewai stubs into sys.modules."""
# -----------------------------------------------------------------------
# crewai.tools — BaseTool
# -----------------------------------------------------------------------
class BaseTool(BaseModel): # noqa: D101
"""Stub mirroring crewai.tools.base_tool.BaseTool."""
model_config = ConfigDict(arbitrary_types_allowed=True)
name: str = "base_tool"
description: str = ""
args_schema: Any = None
result_as_answer: bool = False
@field_serializer("args_schema", when_used="json")
def _ser_args_schema(self, schema): # noqa: D102
if schema is None:
return None
return {"__schema__": f"{schema.__module__}.{schema.__qualname__}"}
@field_validator("args_schema", mode="before")
@classmethod
def _restore_args_schema(cls, v): # noqa: D102
if isinstance(v, dict) and "__schema__" in v:
import importlib
mod_name, cls_name = v["__schema__"].rsplit(".", 1)
return getattr(importlib.import_module(mod_name), cls_name)
return v
def run(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
return self._run(*args, **kwargs)
async def arun(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
return await self._arun(*args, **kwargs)
def _run(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
raise NotImplementedError
async def _arun(self, *args: Any, **kwargs: Any) -> str: # noqa: D102
raise NotImplementedError
tools_mod = types.ModuleType("crewai.tools")
tools_mod.BaseTool = BaseTool # type: ignore[attr-defined]
tools_base_mod = types.ModuleType("crewai.tools.base_tool")
tools_base_mod.BaseTool = BaseTool # type: ignore[attr-defined]
# -----------------------------------------------------------------------
# crewai.knowledge.source.base_knowledge_source — BaseKnowledgeSource
# -----------------------------------------------------------------------
class BaseKnowledgeSource(BaseModel): # noqa: D101
"""Stub mirroring crewai.knowledge.source.base_knowledge_source."""
model_config = ConfigDict(arbitrary_types_allowed=True)
chunk_size: int = 4000
chunk_overlap: int = 200
chunks: list = Field(default_factory=list)
chunk_embeddings: list = Field(default_factory=list, exclude=True)
storage: Any = None
metadata: dict = Field(default_factory=dict)
collection_name: Optional[str] = None
def _chunk_text(self, text: str) -> list: # noqa: D102
return [
text[i : i + self.chunk_size]
for i in range(0, len(text), self.chunk_size - self.chunk_overlap)
]
def _save_documents(self) -> None: # noqa: D102
if self.storage is not None:
self.storage.save(self.chunks)
else:
raise ValueError("No storage found to save documents.")
async def _asave_documents(self) -> None: # noqa: D102
if self.storage is not None:
await self.storage.asave(self.chunks)
else:
raise ValueError("No storage found to save documents.")
def validate_content(self) -> Any: # noqa: D102
raise NotImplementedError
def add(self) -> None: # noqa: D102
raise NotImplementedError
async def aadd(self) -> None: # noqa: D102
raise NotImplementedError
knowledge_pkg = types.ModuleType("crewai.knowledge")
source_pkg = types.ModuleType("crewai.knowledge.source")
source_base_mod = types.ModuleType("crewai.knowledge.source.base_knowledge_source")
source_base_mod.BaseKnowledgeSource = ( # type: ignore[attr-defined]
BaseKnowledgeSource
)
source_pkg.BaseKnowledgeSource = BaseKnowledgeSource # type: ignore[attr-defined]
knowledge_pkg.source = source_pkg
# -----------------------------------------------------------------------
# Register everything
# -----------------------------------------------------------------------
crewai = types.ModuleType("crewai")
crewai.tools = tools_mod # type: ignore[attr-defined]
crewai.knowledge = knowledge_pkg # type: ignore[attr-defined]
_mods = {
"crewai": crewai,
"crewai.tools": tools_mod,
"crewai.tools.base_tool": tools_base_mod,
"crewai.knowledge": knowledge_pkg,
"crewai.knowledge.source": source_pkg,
"crewai.knowledge.source.base_knowledge_source": source_base_mod,
}
for name, mod in _mods.items():
sys.modules[name] = mod
# Install once at import time (conftest is imported before any test file)
_install_crewai_stubs()
@@ -0,0 +1,562 @@
"""
Tests for SemanticaDecisionTool decision intelligence CrewAI tool.
Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is
``True`` and the real Pydantic/BaseTool subclassing path is exercised. A
MagicMock ``AgentContext`` is used so no vector store / faiss is required.
"""
from __future__ import annotations
import json
import unittest
from unittest.mock import MagicMock
from integrations.crewai import SemanticaDecisionTool
from integrations.crewai.decision_tool import (
CREWAI_AVAILABLE,
SemanticaDecisionToolInput,
)
def _make_context() -> MagicMock:
ctx = MagicMock()
ctx.record_decision.return_value = "dec-test-001"
ctx.find_precedents_advanced.return_value = [
{
"scenario": "past loan",
"outcome": "approved",
"confidence": 0.9,
"category": "loan",
}
]
ctx.analyze_decision_influence.return_value = {"centrality": 0.75, "influenced": 3}
ctx.knowledge_graph = MagicMock()
ctx.knowledge_graph.trace_decision_causality = MagicMock(
return_value=["step1", "step2"]
)
return ctx
class TestSemanticaDecisionToolInit(unittest.TestCase):
def test_crewai_available_via_stub(self):
self.assertTrue(CREWAI_AVAILABLE)
def test_is_base_tool_subclass(self):
from crewai.tools import BaseTool
self.assertTrue(issubclass(SemanticaDecisionTool, BaseTool))
def test_creates_with_explicit_context(self):
ctx = _make_context()
tool = SemanticaDecisionTool(context=ctx)
self.assertIs(tool.context, ctx)
def test_creates_context_when_none(self):
tool = SemanticaDecisionTool()
self.assertIsNotNone(tool.context)
def test_default_metadata(self):
tool = SemanticaDecisionTool(context=_make_context())
self.assertEqual(tool.name, "semantica_decision")
self.assertTrue(tool.description)
self.assertEqual(tool.args_schema, SemanticaDecisionToolInput)
def test_input_schema_validates(self):
inp = SemanticaDecisionToolInput(action="record_decision", confidence=0.5)
self.assertEqual(inp.confidence, 0.5)
with self.assertRaises(Exception):
SemanticaDecisionToolInput(action="bogus")
def test_max_precedents_and_causal_depth_defaults(self):
tool = SemanticaDecisionTool(context=_make_context())
self.assertEqual(tool.max_precedents, 5)
self.assertEqual(tool.causal_depth, 3)
class TestSemanticaDecisionToolSerialization(unittest.TestCase):
"""CrewAI checkpoints serialise tools via ``model_dump(mode="json")`` — the
live context must not break that (regression for PydanticSerializationError
on arbitrary state objects)."""
def test_model_dump_json_excludes_context(self):
tool = SemanticaDecisionTool(context=_make_context())
dumped = tool.model_dump(mode="json")
self.assertNotIn("context", dumped)
self.assertEqual(dumped["max_precedents"], 5)
self.assertEqual(dumped["causal_depth"], 3)
def test_model_validate_restores_defaults(self):
tool = SemanticaDecisionTool(context=_make_context())
restored = SemanticaDecisionTool.model_validate(tool.model_dump(mode="json"))
self.assertIsNotNone(restored.context)
self.assertEqual(restored.max_precedents, 5)
self.assertEqual(restored.causal_depth, 3)
def test_restore_flags_lost_live_state(self):
"""A tool restored from a checkpoint must signal that its live context
was excluded and an empty one reconstructed (``reconstructed_state``)."""
tool = SemanticaDecisionTool(context=_make_context())
dumped = tool.model_dump(mode="json")
self.assertTrue(dumped["had_live_state"])
self.assertNotIn("reconstructed_state", dumped)
restored = SemanticaDecisionTool.model_validate(dumped)
self.assertTrue(restored.reconstructed_state)
self.assertFalse(SemanticaDecisionTool().reconstructed_state)
class TestRecordDecision(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_decision_id(self):
result = json.loads(
self.tool._run(
action="record_decision",
category="loan",
scenario="Customer A loan application",
reasoning="Good credit score 740",
outcome="approved",
confidence=0.95,
)
)
self.assertEqual(result["decision_id"], "dec-test-001")
self.assertEqual(result["status"], "recorded")
def test_delegates_to_context(self):
self.tool._run(
action="record_decision",
category="content",
scenario="Moderation check",
reasoning="No violations",
outcome="allowed",
confidence=0.88,
)
self.ctx.record_decision.assert_called_once()
def test_parses_entities_string(self):
self.tool._run(
action="record_decision",
category="hr",
scenario="Hire decision",
reasoning="Qualified",
outcome="hired",
confidence=0.9,
entities="Alice, ACME Corp, Senior Engineer",
)
call_kwargs = self.ctx.record_decision.call_args[1]
self.assertIsInstance(call_kwargs["entities"], list)
self.assertEqual(len(call_kwargs["entities"]), 3)
def test_returns_error_json_on_failure(self):
self.ctx.record_decision.side_effect = RuntimeError("DB unavailable")
result = json.loads(
self.tool._run(
action="record_decision",
category="x",
scenario="y",
reasoning="z",
outcome="failed",
)
)
self.assertEqual(result["status"], "failed")
self.assertIn("error", result)
def test_default_confidence_used(self):
self.tool._run(
action="record_decision",
category="test",
scenario="Default confidence test",
reasoning="N/A",
outcome="pass",
)
call_kwargs = self.ctx.record_decision.call_args[1]
self.assertEqual(call_kwargs["confidence"], 0.8)
def test_malformed_confidence_returns_error_json(self):
"""A non-numeric confidence must not crash the tool — it is coerced
inside ``_record_decision``'s error handling and reported as JSON."""
for bad in ("high", None, "0.9"):
result = json.loads(
self.tool._run(
action="record_decision",
category="x",
scenario="y",
reasoning="z",
outcome="failed",
confidence=bad,
)
)
if bad == "0.9":
self.assertEqual(result["status"], "recorded")
else:
self.assertEqual(result["status"], "failed")
self.assertIn("error", result)
def test_missing_fields_get_sane_defaults(self):
"""record_decision must not hard-fail when the agent omits optional
fields category/reasoning/outcome get defaults."""
result = json.loads(self.tool._run(action="record_decision"))
self.assertEqual(result["status"], "recorded")
call_kwargs = self.ctx.record_decision.call_args[1]
self.assertEqual(call_kwargs["category"], "general")
self.assertEqual(call_kwargs["scenario"], "decision recorded")
self.assertEqual(call_kwargs["reasoning"], "agent decision")
self.assertEqual(call_kwargs["outcome"], "recorded")
class TestRealAutoCreatedContext(unittest.TestCase):
"""The no-context path builds a real AgentContext with a knowledge graph so
decision tracking is actually enabled (regression for the live
'Decision tracking is not enabled' failure)."""
def setUp(self):
self.tool = SemanticaDecisionTool()
def test_context_is_real_agent_context(self):
from semantica.context import AgentContext
self.assertIsInstance(self.tool.context, AgentContext)
self.assertIsNotNone(self.tool.context.knowledge_graph)
def test_record_decision_actually_records(self):
result = json.loads(
self.tool.run(
action="record_decision",
scenario="ship v2",
reasoning="user demand",
confidence=0.9,
)
)
self.assertEqual(result["status"], "recorded")
self.assertTrue(result["decision_id"])
def test_find_precedents_runs_against_real_context(self):
result = json.loads(self.tool.run(action="find_precedents", scenario="ship v2"))
self.assertIn("precedents", result)
def test_trace_causal_chain_runs_against_real_context(self):
"""Regression: trace_decision_causality takes ``max_depth``, not
``depth`` must not raise against a real ContextGraph."""
rec = json.loads(
self.tool.run(
action="record_decision",
scenario="ship v2",
reasoning="user demand",
confidence=0.9,
)
)
trace = json.loads(
self.tool.run(action="trace_causal_chain", decision_id=rec["decision_id"])
)
self.assertIn("causal_chain", trace)
self.assertEqual(trace["decision_id"], rec["decision_id"])
class TestFindPrecedents(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_precedents(self):
result = json.loads(
self.tool._run(action="find_precedents", scenario="new loan application")
)
self.assertIn("precedents", result)
self.assertIsInstance(result["precedents"], list)
def test_count_in_result(self):
result = json.loads(
self.tool._run(action="find_precedents", scenario="test scenario")
)
self.assertEqual(result["count"], len(result["precedents"]))
def test_category_filter_passed(self):
self.tool._run(
action="find_precedents", scenario="scenario", category="finance"
)
call_kwargs = self.ctx.find_precedents_advanced.call_args[1]
self.assertEqual(call_kwargs.get("category"), "finance")
def test_limit_propagated_to_backend(self):
self.tool.max_precedents = 20
self.tool._run(action="find_precedents", scenario="scenario")
call_kwargs = self.ctx.find_precedents_advanced.call_args[1]
self.assertEqual(call_kwargs.get("limit"), 20)
def test_handles_exception_gracefully(self):
self.ctx.find_precedents_advanced.side_effect = RuntimeError("fail")
result = json.loads(self.tool._run(action="find_precedents", scenario="broken"))
self.assertEqual(result["precedents"], [])
self.assertIn("error", result)
class TestTraceCausalChain(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_causal_chain(self):
result = json.loads(
self.tool._run(action="trace_causal_chain", decision_id="dec-001")
)
self.assertIn("causal_chain", result)
self.assertEqual(result["decision_id"], "dec-001")
def test_honest_error_when_causal_trace_unavailable(self):
"""When the graph cannot trace causality, the tool must say so — it
must NOT substitute similarity-based precedents as a causal chain."""
del self.ctx.knowledge_graph.trace_decision_causality
result = json.loads(
self.tool._run(action="trace_causal_chain", decision_id="dec-002")
)
self.assertEqual(result["causal_chain"], [])
self.assertIn("error", result)
self.ctx.knowledge_graph.find_precedents.assert_not_called()
def test_missing_decision_id_reports_error(self):
result = json.loads(self.tool._run(action="trace_causal_chain"))
self.assertIn("error", result)
self.assertEqual(result["causal_chain"], [])
def test_depth_used(self):
self.tool._run(action="trace_causal_chain", decision_id="dec-001", depth=5)
self.ctx.knowledge_graph.trace_decision_causality.assert_called_once_with(
"dec-001", max_depth=5
)
def test_graceful_error_when_context_has_no_knowledge_graph(self):
"""Regression: an unguarded ``self.context.knowledge_graph`` read raised
AttributeError out of ``_run`` and could hard-fail a crew task. It must
return honest error JSON instead."""
del self.ctx.knowledge_graph
result = json.loads(
self.tool._run(action="trace_causal_chain", decision_id="dec-003")
)
self.assertEqual(result["causal_chain"], [])
self.assertIn("error", result)
class TestAnalyzeImpact(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_decision_id(self):
result = json.loads(
self.tool._run(action="analyze_impact", decision_id="dec-001")
)
self.assertEqual(result["decision_id"], "dec-001")
def test_includes_influence_metrics(self):
result = json.loads(
self.tool._run(action="analyze_impact", decision_id="dec-001")
)
self.assertIn("centrality", result)
class TestCheckPolicy(unittest.TestCase):
def setUp(self):
self.ctx = _make_context()
self.tool = SemanticaDecisionTool(context=self.ctx)
def test_returns_json_with_compliant_key(self):
decision = json.dumps(
{"category": "loan", "outcome": "approved", "confidence": 0.9}
)
result = json.loads(
self.tool._run(action="check_policy", decision_data=decision)
)
self.assertIn("compliant", result)
def test_invalid_json_returns_error(self):
result = json.loads(
self.tool._run(action="check_policy", decision_data="{not valid json}")
)
self.assertFalse(result["compliant"])
self.assertGreater(len(result["violations"]), 0)
def test_rule_violation_detected(self):
decision = json.dumps({"confidence": 0.5})
rules = json.dumps(["confidence >= 0.9"])
result = json.loads(
self.tool._run(
action="check_policy", decision_data=decision, policy_rules=rules
)
)
self.assertFalse(result["compliant"])
self.assertEqual(len(result["violations"]), 1)
def test_bool_false_rule_is_compliant(self):
"""Regression: ``enabled == false`` with ``enabled: false`` must be
compliant bool("false") is truthy, so the old coercion inverted it."""
decision = json.dumps({"enabled": False, "confidence": 0.95})
rules = json.dumps(["enabled == false"])
result = json.loads(
self.tool._run(
action="check_policy", decision_data=decision, policy_rules=rules
)
)
self.assertTrue(result["compliant"])
self.assertEqual(result["violations"], [])
def test_bool_true_rule_is_compliant(self):
decision = json.dumps({"enabled": True})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["enabled == true"]),
)
)
self.assertTrue(result["compliant"])
def test_whitespace_padded_strings_are_trimmed(self):
"""Regression: ``_coerce_value`` must return the *stripped* string for
non-numeric literals, or padded decision_data fields never match."""
decision = json.dumps({"status": " approved "})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["status == approved"]),
)
)
self.assertTrue(result["compliant"])
self.assertEqual(result["violations"], [])
def test_bool_false_rule_violated_when_true(self):
decision = json.dumps({"enabled": True})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["enabled == false"]),
)
)
self.assertFalse(result["compliant"])
self.assertEqual(len(result["violations"]), 1)
def test_zero_one_flag_parsed_as_bool(self):
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"flag": 1}),
policy_rules=json.dumps(["flag != 0"]),
)
)
self.assertTrue(result["compliant"])
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"flag": 0}),
policy_rules=json.dumps(["flag != 0"]),
)
)
self.assertFalse(result["compliant"])
def test_numeric_string_value_compared_numerically(self):
"""Regression: a string datum like "0.90" must compare numerically to
rule literal 0.9, not lexicographically."""
decision = json.dumps({"score": "0.90"})
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["score == 0.9"]),
)
)
self.assertTrue(result["compliant"])
def test_numeric_string_ordering(self):
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"pct": "0.95"}),
policy_rules=json.dumps(["pct >= 0.9"]),
)
)
self.assertTrue(result["compliant"])
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps({"pct": "0.85"}),
policy_rules=json.dumps(["pct >= 0.9"]),
)
)
self.assertFalse(result["compliant"])
def test_field_names_with_hyphens_dots_spaces(self):
"""Rule field names are not limited to ``\\w+`` — hyphenated/dotted
(and space-containing) JSON keys must be addressable."""
decision = json.dumps({"risk-score": 0.95, "max.risk": 0.2, "min score": 0.4})
compliant = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(
["risk-score >= 0.9", "max.risk <= 0.5", "min score >= 0.3"]
),
)
)
self.assertTrue(compliant["compliant"])
self.assertEqual(compliant["violations"], [])
violated = json.loads(
self.tool._run(
action="check_policy",
decision_data=decision,
policy_rules=json.dumps(["max.risk >= 0.5"]),
)
)
self.assertFalse(violated["compliant"])
self.assertEqual(len(violated["violations"]), 1)
def test_rule_missing_field_warns_not_silently_compliant(self):
decision = json.dumps({"confidence": 0.95})
rules = json.dumps(["minimum_score >= 0.9"])
result = json.loads(
self.tool._run(
action="check_policy", decision_data=decision, policy_rules=rules
)
)
self.assertTrue(result["compliant"])
self.assertEqual(result["violations"], [])
self.assertEqual(len(result["warnings"]), 1)
self.assertIn("minimum_score", result["warnings"][0])
def test_decision_data_non_object_rejected(self):
result = json.loads(
self.tool._run(
action="check_policy",
decision_data=json.dumps(["confidence", 0.95]),
policy_rules=json.dumps(["confidence >= 0.9"]),
)
)
self.assertFalse(result["compliant"])
self.assertEqual(len(result["violations"]), 1)
self.assertIn("JSON object", result["violations"][0])
def test_unknown_action_returns_error(self):
result = json.loads(self.tool._run(action="nope"))
self.assertIn("error", result)
def test_run_entrypoint(self):
result = json.loads(
self.tool.run(
action="check_policy",
decision_data=json.dumps({"confidence": 0.95}),
policy_rules=json.dumps(["confidence >= 0.9"]),
)
)
self.assertTrue(result["compliant"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,103 @@
"""
Graceful-degradation tests for the CrewAI integration.
These run the integration modules in a fresh subprocess (no conftest crewai
stubs, no real crewai) to prove that every public class remains importable and
functional when ``crewai`` is absent. A subprocess is used because the other
test files in this directory install crewai stubs into ``sys.modules`` for the
whole pytest session; a subprocess keeps the two scenarios isolated.
"""
from __future__ import annotations
import os
import subprocess
import sys
import unittest
REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
_SCRIPT = r"""
import json
import sys
try:
import crewai # noqa: F401
real_crewai = True
except ImportError:
real_crewai = False
from integrations.crewai import (
CREWAI_AVAILABLE,
SemanticaKGTool,
SemanticaDecisionTool,
SemanticaKnowledgeSource,
)
from semantica.context import ContextGraph
assert CREWAI_AVAILABLE == real_crewai, (
f"CREWAI_AVAILABLE={CREWAI_AVAILABLE} but real crewai={real_crewai}"
)
# --- SemanticaKGTool: importable + functional without crewai -----------------
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="privacy policy doc")
tool = SemanticaKGTool(graph=graph)
assert tool.name == "semantica_knowledge_graph"
assert tool.args_schema is not None
res = json.loads(tool._run(action="query_graph", query="privacy"))
assert res["count"] == 1, res
res = json.loads(tool._run(action="find_related", entity="ghost", hops=1))
assert res["count"] == 0, res
# The public run()/arun() entry points must exist without crewai too.
res = json.loads(tool.run(action="query_graph", query="privacy"))
assert res["count"] == 1, res
import asyncio
res = json.loads(asyncio.run(tool.arun(action="query_graph", query="privacy")))
assert res["count"] == 1, res
# --- SemanticaKnowledgeSource: importable + functional without crewai --------
src = SemanticaKnowledgeSource(graph=graph, chunk_size=40, chunk_overlap=5)
assert src.load_content() != {}
assert src.validate_content() is True
src.add() # must not raise; chunks kept in memory
assert len(src.chunks) > 0
# --- SemanticaDecisionTool: importable, builds its own context --------------
dt = SemanticaDecisionTool()
assert dt.name == "semantica_decision"
res = json.loads(dt.run(action="find_precedents", scenario="x"))
assert "precedents" in res, res
res = json.loads(asyncio.run(dt.arun(action="find_precedents", scenario="x")))
assert "precedents" in res, res
print("DEGRADATION_OK")
"""
class TestDegradation(unittest.TestCase):
def test_importable_and_functional_without_crewai(self):
result = subprocess.run(
[sys.executable, "-c", _SCRIPT],
cwd=REPO_ROOT,
capture_output=True,
text=True,
timeout=180,
)
self.assertEqual(
result.returncode,
0,
msg=(
f"subprocess failed:\nSTDOUT:\n{result.stdout}\n"
f"STDERR:\n{result.stderr}"
),
)
self.assertIn("DEGRADATION_OK", result.stdout)
if __name__ == "__main__":
unittest.main()
+453
View File
@@ -0,0 +1,453 @@
"""
Tests for SemanticaKGTool knowledge graph CrewAI tool.
Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is
``True`` and the real Pydantic/BaseTool subclassing path is exercised.
"""
from __future__ import annotations
import asyncio
import json
import unittest
from unittest.mock import MagicMock
from integrations.crewai import SemanticaKGTool as ImportedSemanticaKGTool
from integrations.crewai.kg_tool import (
CREWAI_AVAILABLE,
CREWAI_IMPORT_ERROR,
SemanticaKGTool,
SemanticaKGToolInput,
)
from semantica.context import ContextGraph
# ---------------------------------------------------------------------------
# Fakes
# ---------------------------------------------------------------------------
def _fake_entity(name="Tesla", etype="ORG", conf=0.9):
e = MagicMock()
e.name = name
e.type = etype
e.confidence = conf
return e
def _fake_relation(src="Tesla", rel="FOUNDED_BY", tgt="Elon Musk", conf=0.85):
r = MagicMock()
r.source = src
r.type = rel
r.target = tgt
r.confidence = conf
return r
class _FakeNER:
def extract_entities(self, text):
return [_fake_entity("Tesla"), _fake_entity("Elon Musk", "PERSON")]
class _FakeRelExtractor:
def extract_relations(self, text, entities=None):
return [_fake_relation()]
class _DataclassNER:
"""Returns Semantica's real ``Entity`` dataclass shape (text/label, no name)."""
def extract_entities(self, text):
from semantica.semantic_extract.types import Entity
return [
Entity(text="Tesla", label="ORG", start_char=0, end_char=5),
Entity(text="Elon Musk", label="PERSON", start_char=17, end_char=26),
]
class _DataclassRelExtractor:
"""Returns Semantica's real ``Relation`` dataclass shape (subject/object)."""
def __init__(self):
self.received_entities = None
def extract_relations(self, text, entities=None):
from semantica.semantic_extract.types import Entity, Relation
self.received_entities = entities
return [
Relation(
subject=Entity(text="Tesla", label="ORG", start_char=0, end_char=5),
predicate="FOUNDED_BY",
object=Entity(
text="Elon Musk", label="PERSON", start_char=17, end_char=26
),
)
]
class TestSemanticaKGToolInit(unittest.TestCase):
def test_crewai_available_via_stub(self):
self.assertTrue(CREWAI_AVAILABLE)
self.assertIsNone(CREWAI_IMPORT_ERROR)
def test_is_base_tool_subclass(self):
from crewai.tools import BaseTool
self.assertTrue(issubclass(SemanticaKGTool, BaseTool))
def test_exposed_from_package_init(self):
self.assertIs(ImportedSemanticaKGTool, SemanticaKGTool)
def test_creates_with_explicit_graph(self):
graph = ContextGraph()
tool = SemanticaKGTool(graph=graph)
self.assertIs(tool.graph, graph)
def test_creates_fresh_graph_when_none(self):
tool = SemanticaKGTool(
ner_extractor=_FakeNER(), relation_extractor=_FakeRelExtractor()
)
self.assertIsNotNone(tool.graph)
self.assertIsInstance(tool.graph, ContextGraph)
def test_default_metadata(self):
tool = SemanticaKGTool(
ner_extractor=_FakeNER(), relation_extractor=_FakeRelExtractor()
)
self.assertEqual(tool.name, "semantica_knowledge_graph")
self.assertTrue(tool.description)
self.assertEqual(tool.args_schema, SemanticaKGToolInput)
def test_input_schema_validates(self):
inp = SemanticaKGToolInput(action="query_graph", query="privacy", hops=2)
self.assertEqual(inp.hops, 2)
with self.assertRaises(Exception):
SemanticaKGToolInput(action="bogus")
def test_custom_kwargs_forwarded(self):
tool = SemanticaKGTool(
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
result_as_answer=True,
)
self.assertTrue(tool.result_as_answer)
class TestSemanticaKGToolSerialization(unittest.TestCase):
"""CrewAI checkpoints serialise tools via ``model_dump(mode="json")`` — the
live graph/extractors must not break that (regression for
PydanticSerializationError on arbitrary state objects)."""
def setUp(self):
self.tool = SemanticaKGTool(
graph=ContextGraph(),
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
)
def test_model_dump_json_excludes_shared_state(self):
dumped = self.tool.model_dump(mode="json")
self.assertNotIn("graph", dumped)
self.assertNotIn("ner_extractor", dumped)
self.assertNotIn("relation_extractor", dumped)
self.assertEqual(dumped["name"], "semantica_knowledge_graph")
def test_model_validate_restores_defaults(self):
restored = SemanticaKGTool.model_validate(self.tool.model_dump(mode="json"))
self.assertIsInstance(restored.graph, ContextGraph)
self.assertIs(restored.args_schema, SemanticaKGToolInput)
self.assertEqual(restored.name, "semantica_knowledge_graph")
def test_model_validate_restored_tool_still_runs(self):
restored = SemanticaKGTool.model_validate(self.tool.model_dump(mode="json"))
restored.graph.add_node(node_id="privacy", node_type="policy")
result = json.loads(restored._run(action="query_graph", query="privacy"))
self.assertEqual(result["count"], 1)
def test_restore_flags_lost_live_state(self):
"""A tool restored from a checkpoint must signal that its live graph
was excluded and an empty one reconstructed (``reconstructed_state``)."""
dumped = self.tool.model_dump(mode="json")
self.assertTrue(dumped["had_live_state"])
self.assertNotIn("reconstructed_state", dumped)
restored = SemanticaKGTool.model_validate(dumped)
self.assertTrue(restored.reconstructed_state)
self.assertFalse(SemanticaKGTool().reconstructed_state)
class TestSemanticaKGToolActions(unittest.TestCase):
def setUp(self):
self.graph = ContextGraph()
self.tool = SemanticaKGTool(
graph=self.graph,
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
)
def test_extract_entities(self):
result = json.loads(
self.tool._run(
action="extract_entities", text="Tesla was founded by Elon Musk"
)
)
self.assertEqual(result["count"], 2)
self.assertEqual(result["entities"][0]["name"], "Tesla")
self.assertEqual(result["entities"][0]["type"], "ORG")
def test_extract_relations(self):
result = json.loads(
self.tool._run(
action="extract_relations", text="Tesla was founded by Elon Musk"
)
)
self.assertEqual(result["count"], 1)
self.assertEqual(result["relations"][0]["source"], "Tesla")
self.assertEqual(result["relations"][0]["target"], "Elon Musk")
def test_add_to_graph_populates_graph(self):
result = json.loads(
self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk")
)
self.assertGreaterEqual(result["nodes_added"], 2)
self.assertGreaterEqual(result["edges_added"], 1)
nodes = self.graph.find_nodes()
node_ids = {n["id"] for n in nodes}
self.assertIn("Tesla", node_ids)
self.assertIn("Elon Musk", node_ids)
def test_add_to_graph_is_idempotent(self):
self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk")
second = json.loads(
self.tool._run(action="add_to_graph", text="Tesla was founded by Elon Musk")
)
self.assertEqual(second["nodes_added"], 0)
self.assertEqual(second["edges_added"], 0)
def test_query_graph_finds_matching_node(self):
self.graph.add_node(
node_id="privacy", node_type="policy", content="privacy policy doc"
)
result = json.loads(self.tool._run(action="query_graph", query="privacy"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["results"][0]["id"], "privacy")
def test_query_graph_no_match(self):
result = json.loads(
self.tool._run(action="query_graph", query="nothing-matches")
)
self.assertEqual(result["count"], 0)
self.assertEqual(result["results"], [])
def test_query_graph_searches_node_content(self):
"""query_graph must match node content, not just ids/types."""
self.graph.add_node(
node_id="n1",
node_type="policy",
content="all refunds must be processed within 30 days",
)
result = json.loads(self.tool._run(action="query_graph", query="refunds"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["results"][0]["id"], "n1")
def test_query_graph_matches_type(self):
self.graph.add_node(node_id="n2", node_type="risk")
result = json.loads(self.tool._run(action="query_graph", query="risk"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["results"][0]["id"], "n2")
def test_query_graph_result_shape_is_consistent(self):
"""Every result — content match or id/type match — must carry the same
keys (id, type, label, content, score) so agents get one schema."""
self.graph.add_node(
node_id="n1",
node_type="policy",
content="all refunds within 30 days",
)
by_content = json.loads(self.tool._run(action="query_graph", query="refunds"))[
"results"
][0]
expected_keys = {"id", "type", "label", "content", "score"}
self.assertEqual(set(by_content.keys()), expected_keys)
by_id = json.loads(self.tool._run(action="query_graph", query="n1"))["results"][
0
]
self.assertEqual(set(by_id.keys()), expected_keys)
self.assertEqual(by_id["content"], "all refunds within 30 days")
self.assertEqual(by_id["score"], 1.0)
def test_extract_entities_skips_nameless_entities(self):
class _NamelessNER:
def extract_entities(self, text):
e = MagicMock()
e.name = None
e.type = "MISC"
e.confidence = 0.5
return [e]
tool = SemanticaKGTool(
graph=self.graph,
ner_extractor=_NamelessNER(),
relation_extractor=_FakeRelExtractor(),
)
result = json.loads(tool._run(action="extract_entities", text="text"))
self.assertEqual(result["count"], 0)
self.assertEqual(result["entities"], [])
def test_find_related_multi_hop(self):
self.graph.add_node(node_id="A", node_type="concept")
self.graph.add_node(node_id="B", node_type="concept")
self.graph.add_node(node_id="C", node_type="concept")
self.graph.add_edge(source_id="A", target_id="B", edge_type="related_to")
self.graph.add_edge(source_id="B", target_id="C", edge_type="related_to")
result = json.loads(self.tool._run(action="find_related", entity="A", hops=2))
self.assertEqual(result["count"], 2)
self.assertIn("B", result["related"])
self.assertIn("C", result["related"])
def test_find_related_unknown_entity(self):
result = json.loads(
self.tool._run(action="find_related", entity="Ghost", hops=1)
)
self.assertEqual(result["count"], 0)
self.assertEqual(result["related"], [])
def test_find_related_honors_incoming_edges(self):
"""find_related must be undirected: a node whose only edge is
incoming (A -> B) is still related to A."""
self.graph.add_node(node_id="OpenAI", node_type="ORG")
self.graph.add_node(node_id="Google", node_type="ORG")
self.graph.add_edge(
source_id="OpenAI", target_id="Google", edge_type="related_to"
)
result = json.loads(self.tool._run(action="find_related", entity="Google"))
self.assertEqual(result["related"], ["OpenAI"])
result_out = json.loads(self.tool._run(action="find_related", entity="OpenAI"))
self.assertEqual(result_out["related"], ["Google"])
def test_unknown_action_returns_error(self):
result = json.loads(self.tool._run(action="do_something_else"))
self.assertIn("error", result)
self.assertIn("do_something_else", result["error"])
def test_extract_entities_empty_text_is_graceful(self):
result = json.loads(self.tool._run(action="extract_entities", text=""))
self.assertIn("entities", result)
def test_extract_entities_confidence_none_defaults_to_one(self):
"""A single entity with ``confidence=None`` must not nuke the whole
extract result it normalises to 1.0 instead of raising float(None)."""
class _NoneConfNER:
def extract_entities(self, text):
e = MagicMock()
e.name = "X"
e.type = "MISC"
e.confidence = None
return [e]
tool = SemanticaKGTool(
graph=self.graph,
ner_extractor=_NoneConfNER(),
relation_extractor=_FakeRelExtractor(),
)
result = json.loads(tool._run(action="extract_entities", text="text"))
self.assertEqual(result["count"], 1)
self.assertEqual(result["entities"][0]["name"], "X")
self.assertEqual(result["entities"][0]["confidence"], 1.0)
self.assertNotIn("error", result)
def test_graph_lock_is_per_graph(self):
"""Independent graphs must not share a batch lock."""
g2 = ContextGraph()
lock_a = self.tool._graph_lock(self.graph)
lock_a_again = self.tool._graph_lock(self.graph)
lock_b = self.tool._graph_lock(g2)
self.assertIs(lock_a, lock_a_again)
self.assertIsNot(lock_a, lock_b)
class TestSemanticaKGToolDataclassShapes(unittest.TestCase):
"""Real Semantica ``Entity``/``Relation`` dataclasses (text/label,
subject/object) instead of MagicMock-shaped fakes."""
def setUp(self):
self.ner = _DataclassNER()
self.rel = _DataclassRelExtractor()
self.graph = ContextGraph()
self.tool = SemanticaKGTool(
graph=self.graph, ner_extractor=self.ner, relation_extractor=self.rel
)
def test_extract_entities_reads_text_label(self):
result = json.loads(
self.tool._run(action="extract_entities", text="Tesla founded by Elon Musk")
)
self.assertEqual(result["count"], 2)
self.assertEqual(result["entities"][0]["name"], "Tesla")
self.assertEqual(result["entities"][0]["type"], "ORG")
self.assertEqual(result["entities"][1]["name"], "Elon Musk")
self.assertEqual(result["entities"][1]["type"], "PERSON")
def test_extract_relations_reads_subject_object(self):
result = json.loads(
self.tool._run(
action="extract_relations", text="Tesla founded by Elon Musk"
)
)
self.assertEqual(result["count"], 1)
self.assertEqual(result["relations"][0]["source"], "Tesla")
self.assertEqual(result["relations"][0]["relation"], "FOUNDED_BY")
self.assertEqual(result["relations"][0]["target"], "Elon Musk")
def test_add_to_graph_passes_entity_objects_to_relation_extractor(self):
result = json.loads(
self.tool._run(action="add_to_graph", text="Tesla founded by Elon Musk")
)
self.assertEqual(result["nodes_added"], 2)
self.assertEqual(result["edges_added"], 1)
from semantica.semantic_extract.types import Entity
self.assertIsNotNone(self.rel.received_entities)
for e in self.rel.received_entities:
self.assertIsInstance(e, Entity)
node_ids = {n["id"] for n in self.graph.find_nodes()}
self.assertIn("Tesla", node_ids)
self.assertIn("Elon Musk", node_ids)
edge_keys = {
(e["source"], e["type"], e["target"]) for e in self.graph.find_edges()
}
self.assertIn(("Tesla", "FOUNDED_BY", "Elon Musk"), edge_keys)
class TestSemanticaKGToolCrewAIEntrypoints(unittest.TestCase):
def setUp(self):
self.tool = SemanticaKGTool(
graph=ContextGraph(),
ner_extractor=_FakeNER(),
relation_extractor=_FakeRelExtractor(),
)
def test_run_delegates_to_run(self):
result = json.loads(
self.tool.run(action="extract_entities", text="Tesla led by Elon Musk")
)
self.assertEqual(result["count"], 2)
def test_arun_async(self):
async def _call():
return await self.tool.arun(action="query_graph", query="x")
result = json.loads(asyncio.run(_call()))
self.assertIn("results", result)
def test_run_returns_string(self):
out = self.tool.run(action="extract_entities", text="hello world")
self.assertIsInstance(out, str)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,228 @@
"""
Tests for SemanticaKnowledgeSource CrewAI knowledge source backed by a
Semantica ContextGraph.
Runs with the crewai stubs installed by conftest, so ``CREWAI_AVAILABLE`` is
``True`` and the real Pydantic/BaseKnowledgeSource subclassing path (including
the current ``validate_content`` / ``add`` / ``aadd`` contract) is exercised.
"""
from __future__ import annotations
import asyncio
import unittest
from integrations.crewai import SemanticaKnowledgeSource
from integrations.crewai.knowledge_source import CREWAI_AVAILABLE, _chunk_text_manual
from semantica.context import ContextGraph
class _FakeStorage:
def __init__(self):
self.saved_chunks: list = []
def save(self, chunks: list) -> None:
self.saved_chunks.extend(chunks)
async def asave(self, chunks: list) -> None:
self.saved_chunks.extend(chunks)
class _RaisingStorage(_FakeStorage):
"""Mirrors real crewai: storage is wired but ``save`` raises ``ValueError``
(e.g. the embedder has no credentials configured)."""
def save(self, chunks: list) -> None:
raise ValueError("The OPENAI_API_KEY environment variable is not set.")
async def asave(self, chunks: list) -> None:
raise ValueError("The OPENAI_API_KEY environment variable is not set.")
def _build_graph() -> ContextGraph:
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy", content="privacy policy doc")
graph.add_node(node_id="fraud", node_type="risk", content="fraud detection rules")
graph.add_edge(source_id="privacy", target_id="fraud", edge_type="constrains")
return graph
class TestSemanticaKnowledgeSourceInit(unittest.TestCase):
def test_crewai_available_via_stub(self):
self.assertTrue(CREWAI_AVAILABLE)
def test_is_base_knowledge_source_subclass(self):
from crewai.knowledge.source import BaseKnowledgeSource
self.assertTrue(issubclass(SemanticaKnowledgeSource, BaseKnowledgeSource))
def test_creates_with_explicit_graph(self):
graph = _build_graph()
src = SemanticaKnowledgeSource(graph=graph)
self.assertIs(src.graph, graph)
def test_creates_fresh_graph_when_none(self):
src = SemanticaKnowledgeSource()
self.assertIsNotNone(src.graph)
self.assertIsInstance(src.graph, ContextGraph)
def test_default_metadata(self):
src = SemanticaKnowledgeSource(graph=_build_graph())
self.assertEqual(src.name, "semantica_knowledge_graph")
self.assertEqual(src.chunk_size, 4000)
self.assertEqual(src.chunk_overlap, 200)
def test_custom_chunking_params(self):
src = SemanticaKnowledgeSource(
graph=_build_graph(), chunk_size=50, chunk_overlap=10
)
self.assertEqual(src.chunk_size, 50)
self.assertEqual(src.chunk_overlap, 10)
class TestLoadContent(unittest.TestCase):
def setUp(self):
self.graph = _build_graph()
self.src = SemanticaKnowledgeSource(graph=self.graph)
def test_nodes_serialized(self):
content = self.src.load_content()
text = "\n".join(content.values())
self.assertIn("privacy", text)
self.assertIn("fraud", text)
self.assertIn("policy", text)
def test_edges_serialized(self):
content = self.src.load_content()
text = "\n".join(content.values())
self.assertIn("-[" + "constrains" + "]->", text)
def test_empty_graph_returns_empty(self):
src = SemanticaKnowledgeSource(graph=ContextGraph())
self.assertEqual(src.load_content(), {})
def test_validate_content_passes(self):
self.assertTrue(self.src.validate_content())
def test_validate_content_raises_without_graph(self):
self.src.graph = None
with self.assertRaises(ValueError):
self.src.validate_content()
class TestAdd(unittest.TestCase):
def setUp(self):
self.graph = _build_graph()
self.src = SemanticaKnowledgeSource(
graph=self.graph, chunk_size=40, chunk_overlap=5
)
def test_add_saves_chunks_to_storage(self):
storage = _FakeStorage()
self.src.storage = storage
self.src.add()
self.assertGreater(len(storage.saved_chunks), 0)
self.assertTrue(all(isinstance(c, str) and c for c in storage.saved_chunks))
def test_add_without_storage_keeps_chunks_in_memory(self):
self.src.add()
self.assertGreater(len(self.src.chunks), 0)
self.assertGreater(len(self.src._chunks), 0)
def test_add_wired_storage_failure_logs_error_not_debug(self):
"""Regression: real crewai raises ``ValueError`` for a missing embedder
even though storage IS wired. That used to fall into the "storage not
wired" DEBUG branch, silently hiding the failure — it must log an
actionable ERROR instead."""
self.src.storage = _RaisingStorage()
with self.assertLogs(
f"semantica.{SemanticaKnowledgeSource.__module__}", level="ERROR"
) as caught:
self.src.add()
joined = "\n".join(caught.output)
self.assertIn("storage save FAILED", joined)
self.assertIn("OPENAI_API_KEY", joined)
self.assertGreater(len(self.src.chunks), 0)
def test_add_empty_graph_no_chunks(self):
src = SemanticaKnowledgeSource(
graph=ContextGraph(), chunk_size=40, chunk_overlap=5
)
src.add()
self.assertEqual(src.chunks, [])
def test_aadd_async(self):
storage = _FakeStorage()
self.src.storage = storage
asyncio.run(self.src.aadd())
self.assertGreater(len(storage.saved_chunks), 0)
def test_content_summary(self):
summary = self.src.get_content_summary()
self.assertEqual(summary["name"], "semantica_knowledge_graph")
self.assertGreater(summary["source_count"], 0)
self.assertTrue(summary["crewai_available"])
class TestSemanticaKnowledgeSourceSerialization(unittest.TestCase):
"""CrewAI checkpoints serialise their models via ``model_dump(mode="json")``
the live graph must not break that (regression for
PydanticSerializationError on arbitrary state objects)."""
def test_model_dump_json_excludes_graph(self):
src = SemanticaKnowledgeSource(graph=_build_graph())
dumped = src.model_dump(mode="json")
self.assertNotIn("graph", dumped)
self.assertEqual(dumped["name"], "semantica_knowledge_graph")
def test_model_validate_restores_graph(self):
src = SemanticaKnowledgeSource(graph=_build_graph())
restored = SemanticaKnowledgeSource.model_validate(src.model_dump(mode="json"))
self.assertIsInstance(restored.graph, ContextGraph)
def test_restored_source_still_loads_content(self):
"""A checkpoint-restored source gets a fresh graph (the live graph is
excluded from serialisation); once a graph is attached it works."""
src = SemanticaKnowledgeSource(graph=_build_graph())
restored = SemanticaKnowledgeSource.model_validate(src.model_dump(mode="json"))
restored.graph = _build_graph()
self.assertNotEqual(restored.load_content(), {})
def test_restore_flags_lost_live_state(self):
"""A source restored from a checkpoint must signal that its live graph
was excluded and an empty one reconstructed (``reconstructed_state``).
Regression: an eager graph build in ``__init__`` used to hide this."""
src = SemanticaKnowledgeSource(graph=_build_graph())
dumped = src.model_dump(mode="json")
self.assertTrue(dumped["had_live_state"])
self.assertNotIn("reconstructed_state", dumped)
restored = SemanticaKnowledgeSource.model_validate(dumped)
self.assertTrue(restored.reconstructed_state)
self.assertFalse(SemanticaKnowledgeSource().reconstructed_state)
self.assertIsInstance(SemanticaKnowledgeSource().graph, ContextGraph)
class TestManualChunker(unittest.TestCase):
def test_short_text_single_chunk(self):
self.assertEqual(_chunk_text_manual("hello", 40, 5), ["hello"])
def test_empty_text(self):
self.assertEqual(_chunk_text_manual("", 40, 5), [])
def test_long_text_overlaps(self):
chunks = _chunk_text_manual("a" * 100, 40, 10)
self.assertGreater(len(chunks), 1)
self.assertTrue(all(len(c) <= 40 for c in chunks))
# Overlap means consecutive chunks share tail/head content
self.assertIn("a" * 10, chunks[0][-10:] + chunks[1][:10])
def test_zero_chunk_size_guarded(self):
self.assertEqual(_chunk_text_manual("hello world", 0, 5), ["hello world"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,123 @@
"""
End-to-end integration tests against the REAL crewai package.
These run in a subprocess because the stubs in ``conftest.py`` install a fake
``crewai`` module into ``sys.modules`` for the whole pytest session the same
interpreter can never see both. Each test launches a fresh interpreter; if
crewai is genuinely not installed there, the test is skipped.
This covers the failure class the stubs cannot: ``Crew``-level serialization
(list[BaseTool] inside Agent.tools), checkpoint restore via ``model_validate``,
and knowledge-source behaviour with a real ``Crew``.
"""
from __future__ import annotations
import subprocess
import sys
import textwrap
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[3]
_SCRIPT = textwrap.dedent(
"""
import os
import json
import sys
sys.path.insert(0, os.getcwd())
try:
import crewai
except ImportError:
print("CREWAI_IMPORT_FAILED")
sys.exit(2)
import crewai as crewai_mod
from crewai import Agent, Task, Crew
from semantica.context import ContextGraph
from integrations.crewai import (
SemanticaKGTool,
SemanticaDecisionTool,
SemanticaKnowledgeSource,
)
os.environ["CREWAI_DESERIALIZE_CALLBACKS"] = "1"
# --- 1. Crew-level serialization round-trip ------------------------------
graph = ContextGraph()
graph.add_node(node_id="privacy", node_type="policy",
content="privacy policy: no data sharing")
tool = SemanticaKGTool(graph=graph)
decision_ctx = SemanticaDecisionTool()
decision_tool = SemanticaDecisionTool(context=decision_ctx.context)
agent = Agent(role="researcher", goal="answer questions",
backstory="retrieves from a knowledge graph",
tools=[tool, decision_tool])
task = Task(description="answer", expected_output="an answer", agent=agent)
crew = Crew(agents=[agent], tasks=[task])
dump = crew.model_dump(mode="json")
agents = dump["agents"]
assert len(agents) == 1, f"expected 1 agent, got {len(agents)}"
dumped_tools = agents[0]["tools"]
assert len(dumped_tools) == 2, f"expected 2 tools, got {len(dumped_tools)}"
for t in dumped_tools:
assert isinstance(t, dict), f"tool not serialized to dict: {type(t)}"
assert "graph" not in t, "live graph leaked into serialized tool"
assert "context" not in t, "live context leaked into serialized tool"
assert "ner_extractor" not in t, "extractor leaked into serialized tool"
# --- 2. Restore a tool from the crew dump --------------------------------
kg_dump = dumped_tools[0]
assert kg_dump["name"] == "semantica_knowledge_graph", kg_dump["name"]
restored = SemanticaKGTool.model_validate(kg_dump)
assert restored.graph is not None, "restored tool did not self-heal a graph"
q = json.loads(restored._run(action="query_graph", query="privacy"))
assert "results" in q, f"restored tool query_graph failed: {q}"
# --- 3. Knowledge source with no embedder must not crash a Crew ----------
ks = SemanticaKnowledgeSource(graph=graph)
agent2 = Agent(role="researcher2", goal="answer",
backstory="retrieves from knowledge")
task2 = Task(description="q", expected_output="a", agent=agent2)
crew2 = Crew(agents=[agent2], tasks=[task2],
knowledge_sources=[ks])
assert ks.chunks, "knowledge source retained no chunks in memory"
assert crew2.knowledge is not None, "crew.knowledge not created"
print("REAL_CREWAI_OK")
"""
)
class TestRealCrewAIIntegration(unittest.TestCase):
def _run(self) -> subprocess.CompletedProcess:
return subprocess.run(
[sys.executable, "-c", _SCRIPT],
cwd=str(REPO_ROOT),
capture_output=True,
text=True,
timeout=240,
)
def test_crew_level_round_trip_with_real_crewai(self):
proc = self._run()
if proc.returncode == 2:
self.skipTest("real crewai is not installed in this environment")
self.assertEqual(
proc.returncode,
0,
msg=f"subprocess failed:\n{proc.stdout}\n{proc.stderr}",
)
self.assertIn("REAL_CREWAI_OK", proc.stdout)
if __name__ == "__main__":
unittest.main()
+112
View File
@@ -0,0 +1,112 @@
"""Regression tests for KG analytics node scope handling."""
import networkx as nx
from semantica.kg.centrality_calculator import CentralityCalculator
from semantica.kg.community_detector import CommunityDetector
from semantica.kg.connectivity_analyzer import ConnectivityAnalyzer
def _graph_with_isolated_node():
return {
"entities": [{"id": "A"}, {"id": "B"}, {"id": "C"}],
"relationships": [{"source": "A", "target": "B"}],
}
def test_centrality_keeps_declared_isolated_nodes():
result = CentralityCalculator().calculate_degree_centrality(
_graph_with_isolated_node()
)
assert result["total_nodes"] == 3
assert result["centrality"]["C"] == 0.0
def test_connectivity_reports_declared_isolated_nodes():
result = ConnectivityAnalyzer().analyze_connectivity(
_graph_with_isolated_node()
)
assert result["num_nodes"] == 3
assert result["num_components"] == 2
assert ["C"] in result["components"]
assert result["is_connected"] is False
def test_community_detection_keeps_declared_isolated_nodes():
detector = CommunityDetector()
result = detector.detect_communities(_graph_with_isolated_node())
assert set(result["node_assignments"]) == {"A", "B", "C"}
metrics = detector.calculate_community_metrics(
_graph_with_isolated_node(), result
)
assert metrics["num_communities"] == 2
structure = detector.analyze_community_structure(
_graph_with_isolated_node(), result
)
assert structure["num_communities"] == 2
def test_community_detection_returns_singletons_for_edgeless_graph():
graph = {"entities": [{"id": "A"}, {"id": "B"}], "relationships": []}
result = CommunityDetector().detect_communities(graph)
assert {frozenset(community) for community in result["communities"]} == {
frozenset({"A"}),
frozenset({"B"}),
}
def test_networkx_graph_keeps_isolated_nodes_for_analytics():
graph = nx.Graph()
graph.add_nodes_from(["A", "B", "C"])
graph.add_edge("A", "B")
centrality = CentralityCalculator().calculate_degree_centrality(graph)
connectivity = ConnectivityAnalyzer().analyze_connectivity(graph)
assert centrality["total_nodes"] == 3
assert centrality["centrality"]["C"] == 0.0
assert connectivity["num_nodes"] == 3
assert connectivity["num_components"] == 2
def test_nodes_edges_payload_keeps_declared_isolated_nodes():
graph = {
"nodes": [{"id": "A"}, {"id": "B"}, {"id": "C"}],
"edges": [("A", "B")],
}
result = CentralityCalculator().calculate_degree_centrality(graph)
assert result["total_nodes"] == 3
assert result["centrality"]["C"] == 0.0
def test_name_and_text_nodes_are_kept_when_ids_are_missing():
graph = {
"entities": [{"name": "Alice"}, {"text": "Bob"}],
"relationships": [],
}
result = CentralityCalculator().calculate_degree_centrality(graph)
assert result["total_nodes"] == 2
assert set(result["centrality"]) == {"Alice", "Bob"}
def test_community_metrics_accepts_communities_payload():
detector = CommunityDetector()
graph = {
"entities": [{"id": "A"}, {"id": "B"}, {"id": "C"}],
"relationships": [{"source": "A", "target": "B"}],
}
result = {"communities": [["A", "B"], ["C"]]}
metrics = detector.calculate_community_metrics(graph, result)
assert metrics["num_communities"] == 2
assert metrics["community_sizes"] == {0: 2, 1: 1}
+45 -1
View File
@@ -241,7 +241,51 @@ class TestPathFinder:
if len(paths) > 1:
lengths = [self.finder.path_length(multi_path_graph, path) for path in paths]
assert all(lengths[i] <= lengths[i+1] for i in range(len(lengths)-1))
def test_find_k_shortest_paths_preserves_graph(self):
"""Test k-shortest path search does not mutate the input graph."""
graph = nx.Graph()
graph.add_edges_from([
("A", "X"), ("X", "Y"), ("Y", "E"),
("A", "B"), ("B", "C"), ("C", "E"),
])
original_nodes = set(graph.nodes)
original_edges = set(graph.edges)
paths = self.finder.find_k_shortest_paths(graph, "A", "E", k=5)
assert len(paths) == 2
assert set(graph.nodes) == original_nodes
assert set(graph.edges) == original_edges
def test_find_k_shortest_paths_returns_loopless_paths(self):
"""Test k-shortest paths do not repeat nodes."""
graph = nx.Graph()
graph.add_edges_from([
("A", "D"), ("A", "E"), ("A", "C"),
("B", "D"), ("B", "C"),
])
paths = self.finder.find_k_shortest_paths(graph, "A", "B", k=5)
assert paths == [["A", "C", "B"], ["A", "D", "B"]]
assert all(len(path) == len(set(path)) for path in paths)
def test_dijkstra_exclusion_respects_undirected_traversal(self):
"""Test exclusions apply in both directions for undirected traversal."""
graph = nx.DiGraph()
graph.add_edge("A", "B")
path = self.finder._dijkstra_shortest_path(
graph,
"B",
"A",
directed=False,
excluded_edges={("A", "B")},
)
assert path == []
def test_find_k_shortest_paths_no_path(self):
"""Test finding k shortest paths with no path available."""
paths = self.finder.find_k_shortest_paths(self.disconnected_graph, "A", "D", k=3)
+18
View File
@@ -49,6 +49,24 @@ class TestCurrencyNormalizer(unittest.TestCase):
self.assertEqual(result["amount"], 100.0)
self.assertEqual(result["currency"], "EUR")
def test_symbol_currencies_are_validated_as_supported_codes(self):
for symbol, expected_code in self.normalizer.currency_symbols.items():
result = self.normalizer.normalize_currency(f"{symbol}100")
self.assertEqual(result["currency"], expected_code)
self.assertTrue(self.normalizer.validate_currency_code(result["currency"]))
def test_currency_codes_match_boundaries_without_matching_words(self):
for value in ("RUB100", "100 RUB", "rub 100"):
result = self.normalizer.normalize_currency(value)
self.assertEqual(result["amount"], 100.0)
self.assertEqual(result["currency"], "RUB")
for value in ("ruby 100", "wilson 100"):
result = self.normalizer.normalize_currency(value)
self.assertEqual(result["amount"], 100.0)
self.assertEqual(result["currency"], "USD")
class TestScientificNotationHandler(unittest.TestCase):
def setUp(self):
self.handler = ScientificNotationHandler()
+73
View File
@@ -0,0 +1,73 @@
"""Construction coverage for the parse module's public parser classes.
Regression tests for #1014: ``ExcelParser.__init__`` called ``get_progress_tracker()``
without importing it, so every instantiation raised ``NameError``. The class was
covered by an import-only test, which passes regardless of whether ``__init__``
works, so nothing caught it. #530 was the same bug in ``SimilarityCalculator``.
These tests deliberately do **not** patch ``get_logger``/``get_progress_tracker``.
``tests/parse/test_parse_comprehensive.py`` patches both into every parse module
that exposes them, which would mock away the exact interaction under test here and
let the regression back in silently.
"""
import unittest
import semantica.parse as parse_module
from semantica.parse.excel_parser import ExcelParser
def _exported_parser_classes():
"""Public parser classes, taken from the package's own ``__all__``.
Driven off ``__all__`` rather than a hand-written list so a parser added later
is covered without anyone remembering to update this file.
"""
return [
(name, getattr(parse_module, name))
for name in parse_module.__all__
if name.endswith("Parser")
]
class TestExcelParserConstruction(unittest.TestCase):
"""ExcelParser must be constructible -- see #1014."""
def test_excel_parser_constructs(self):
parser = ExcelParser()
self.assertIsNotNone(parser)
def test_excel_parser_wires_progress_tracker(self):
"""The missing import was for the tracker, so assert it is actually set.
A bare construction check would pass against a version that dropped the
tracker call entirely; this pins the attribute the import exists to provide.
"""
parser = ExcelParser()
self.assertIsNotNone(parser.progress_tracker)
class TestExportedParsersConstruct(unittest.TestCase):
"""Every parser the package exports must survive ``__init__``."""
def test_all_exported_parsers_construct(self):
classes = _exported_parser_classes()
self.assertGreater(len(classes), 0, "no exported parser classes found")
for name, cls in classes:
with self.subTest(parser=name):
try:
self.assertIsNotNone(cls())
except ImportError as exc:
# Parsers backed by an optional dependency raise a deliberate,
# actionable ImportError when it is absent (e.g. DoclingParser
# without `docling`). That is correct behavior, not a defect.
self.assertIn(
"install",
str(exc).lower(),
f"{name} raised ImportError without install guidance: {exc}",
)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,114 @@
"""Tests for the process-level spaCy model cache in semantic_extract.methods.
Before this cache existed, extract_entities_ml(), extract_relations_similarity()
and extract_relations_dependency() called spacy.load() on every invocation, so a
short sentence cost ~120 ms of model loading on top of ~2 ms of actual work.
"""
from types import SimpleNamespace
from unittest.mock import MagicMock
import pytest
from semantica.semantic_extract import methods
@pytest.fixture(autouse=True)
def clear_cache():
methods.clear_spacy_model_cache()
yield
methods.clear_spacy_model_cache()
def _fake_spacy(load):
return SimpleNamespace(load=load, util=SimpleNamespace(is_package=lambda name: True))
def test_model_loaded_once_across_calls(monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append(name)
return MagicMock()
monkeypatch.setattr(methods, "spacy", _fake_spacy(fake_load))
methods.load_spacy_model("en_core_web_sm")
methods.load_spacy_model("en_core_web_sm")
methods.load_spacy_model("en_core_web_sm")
assert calls == ["en_core_web_sm"], "spacy.load should run once per model name"
def test_same_object_returned(monkeypatch):
sentinel = MagicMock()
monkeypatch.setattr(methods, "spacy", _fake_spacy(lambda name, **kw: sentinel))
assert methods.load_spacy_model("en_core_web_sm") is sentinel
assert methods.load_spacy_model("en_core_web_sm") is sentinel
def test_distinct_models_cached_separately(monkeypatch):
calls = []
monkeypatch.setattr(
methods,
"spacy",
_fake_spacy(lambda name, **kw: (calls.append(name), MagicMock())[1]),
)
methods.load_spacy_model("en_core_web_sm")
methods.load_spacy_model("en_core_web_lg")
methods.load_spacy_model("en_core_web_sm")
assert calls == ["en_core_web_sm", "en_core_web_lg"]
def test_load_errors_propagate_and_are_not_cached(monkeypatch):
"""Callers rely on OSError to trigger their fallback path."""
attempts = []
def failing_load(name, **kwargs):
attempts.append(name)
raise OSError(f"Can't find model '{name}'")
monkeypatch.setattr(methods, "spacy", _fake_spacy(failing_load))
with pytest.raises(OSError):
methods.load_spacy_model("en_core_web_missing")
with pytest.raises(OSError):
methods.load_spacy_model("en_core_web_missing")
assert len(attempts) == 2, "a failed load must not populate the cache"
def test_cache_ignores_entries_from_a_replaced_spacy_module(monkeypatch):
"""Patching methods.spacy must not hand back a model from the old module.
Existing tests patch this attribute with a mock and assert on load calls, so
a cache keyed on model name alone would leak objects across those tests.
"""
first = MagicMock()
monkeypatch.setattr(methods, "spacy", _fake_spacy(lambda name, **kw: first))
assert methods.load_spacy_model("en_core_web_sm") is first
second = MagicMock()
monkeypatch.setattr(methods, "spacy", _fake_spacy(lambda name, **kw: second))
assert methods.load_spacy_model("en_core_web_sm") is second
def test_extract_entities_ml_reuses_the_cached_model(monkeypatch):
calls = []
def fake_load(name, **kwargs):
calls.append(name)
nlp = MagicMock()
nlp.return_value = SimpleNamespace(ents=[])
return nlp
monkeypatch.setattr(methods, "spacy", _fake_spacy(fake_load))
monkeypatch.setattr(methods, "SPACY_AVAILABLE", True)
methods.extract_entities_ml("Alice works at Acme Corp.")
methods.extract_entities_ml("Bob works at Globex.")
assert len(calls) == 1, "the model should be loaded once, not once per call"
+22 -1
View File
@@ -126,7 +126,11 @@ def test_load_from_database(mock_db_ingestor_cls, seed_manager):
assert len(records) == 1
assert records[0]["id"] == 1
assert records[0]["entity_type"] == "User"
mock_db_ingestor.execute_query.assert_called_once_with("SELECT * FROM users")
# Regression for #973: the ingestor methods receive the connection
# string as their first argument — the constructor config is not enough.
mock_db_ingestor.execute_query.assert_called_once_with(
"sqlite:///:memory:", "SELECT * FROM users"
)
# Mock export_table result
mock_table_data = MagicMock()
@@ -139,6 +143,23 @@ def test_load_from_database(mock_db_ingestor_cls, seed_manager):
)
assert len(records) == 1
assert records[0]["id"] == 2
mock_db_ingestor.export_table.assert_called_once_with("sqlite:///:memory:", "users")
def test_load_from_database_os_error_not_misreported(seed_manager):
# Regression for #973: a real OSError from the ingestor must surface as a
# database failure with the cause chained, not as a missing module.
import semantica.ingest.db_ingestor as dbi
with patch.object(
dbi.DBIngestor, "execute_query", side_effect=OSError(111, "Connection refused")
):
with pytest.raises(ProcessingError) as excinfo:
seed_manager.load_from_database(
"postgresql://u:p@10.0.0.9/db", query="SELECT 1"
)
assert "Failed to load from database" in str(excinfo.value)
assert "module not available" not in str(excinfo.value)
assert isinstance(excinfo.value.__cause__, OSError)
def test_load_from_database_import_error(seed_manager):
with patch.dict("sys.modules", {"semantica.ingest.db_ingestor": None}):
@@ -0,0 +1,210 @@
"""Regression tests for CONSTRUCT query-form detection (issue #931).
``CONSTRUCT_QUERY_RE``'s comment alternative used to be written ``\\#[^\\n]*``.
The trailing ``*`` backtracks, so for a query like::
# CONSTRUCT in a comment
SELECT * WHERE { }
the engine consumed the ``#``, gave back everything after it, and let the
CONSTRUCT *inside the comment* satisfy the query-form keyword. Every SPARQL
backend delegates to this one regex, so a SELECT/ASK carrying such a leading
comment was routed down the CONSTRUCT path of ``execute_sparql`` which sends
``Accept: text/turtle`` and parses the body as Turtle, failing with a
misleading "Failed to parse CONSTRUCT response as Turtle".
Both directions are pinned here: the false positives that motivated the fix,
and the queries that were already detected correctly, so a future tightening
cannot silently start dropping real CONSTRUCT queries instead.
"""
import unittest
from unittest.mock import patch
from semantica.triplet_store import sparql_escaping
from semantica.triplet_store.anzo_store import AnzoStore
from semantica.triplet_store.blazegraph_store import BlazegraphStore
from semantica.triplet_store.jena_store import JenaStore
from semantica.triplet_store.rdf4j_store import RDF4JStore
# Queries whose *form* is CONSTRUCT. Each must be detected.
CONSTRUCT_CASES = {
"bare": "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"lowercase": "construct { ?s ?p ?o } where { ?s ?p ?o }",
"mixed_case": "Construct { ?s ?p ?o } Where { ?s ?p ?o }",
"leading_whitespace": " \n\t CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"prefix_preamble": (
"PREFIX e: <http://e/>\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
),
"base_preamble": "BASE <http://e/> CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"comment_then_construct": "# a comment\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"two_comments_then_construct": "#\n#\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"comment_crlf": "# a comment\r\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"comment_cr_only": "# a comment\rCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"empty_comment": "#\nCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }",
"mixed_preamble": (
" \n # header \n PREFIX e: <http://e/>\n # note \n "
"CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
),
}
# Queries whose form is NOT CONSTRUCT. None may be detected.
NON_CONSTRUCT_CASES = {
"plain_select": "SELECT ?s WHERE { ?s ?p ?o }",
"plain_ask": "ASK { ?s ?p ?o }",
"describe": "DESCRIBE <urn:x>",
"keyword_in_literal": 'SELECT * WHERE { ?s ?p "please CONSTRUCT this" }',
"keyword_in_trailing_comment": "SELECT * WHERE { } # CONSTRUCT",
"keyword_as_substring": 'SELECT ?s WHERE { ?s <urn:p> "CONSTRUCTOR" }',
# The issue #931 payloads: CONSTRUCT inside a *leading* comment.
"leading_comment_lf": "# CONSTRUCT in a comment\nSELECT * WHERE { }",
"leading_comment_crlf": "# CONSTRUCT in a comment\r\nSELECT * WHERE { }",
"leading_comment_cr_only": "# CONSTRUCT in a comment\rSELECT * WHERE { }",
"leading_comment_no_space": "#CONSTRUCT\nSELECT * WHERE { }",
"leading_comment_mid_sentence": "# we will CONSTRUCT later\nSELECT * WHERE { }",
"leading_comment_second_line": "#\n# CONSTRUCT\nSELECT * WHERE { }",
"leading_comment_before_ask": "# TODO: CONSTRUCT\nASK { ?s ?p ?o }",
"leading_comment_word_construction": "# CONSTRUCTION notes\nSELECT * WHERE { }",
"comment_only_no_newline": "# CONSTRUCT",
}
def _blazegraph_store() -> BlazegraphStore:
with patch.object(BlazegraphStore, "_connect", autospec=True):
store = BlazegraphStore(endpoint="http://localhost:9999/blazegraph")
store.connected = True
return store
def _rdf4j_store() -> RDF4JStore:
with patch.object(RDF4JStore, "_connect", autospec=True):
store = RDF4JStore(
endpoint="http://localhost:8080/rdf4j-server", repository_id="repo1"
)
store.connected = True
return store
def _anzo_store() -> AnzoStore:
with patch.object(AnzoStore, "_connect", autospec=True):
store = AnzoStore(
endpoint="http://localhost:8080",
dataset_uri="http://cambridgesemantics.com/Graphmart/abc123",
)
store.connected = True
return store
def _jena_store() -> JenaStore:
return JenaStore()
# Every backend that delegates to CONSTRUCT_QUERY_RE. Detection is shared, so
# a per-backend regression would otherwise only surface in whichever backend
# happened to be covered.
BACKENDS = {
"blazegraph": _blazegraph_store,
"rdf4j": _rdf4j_store,
"anzo": _anzo_store,
"jena": _jena_store,
}
class TestConstructQueryRegex(unittest.TestCase):
"""Direct tests of the shared regex."""
def test_case_tables_are_populated(self):
"""Guard against a vacuous suite.
Every test below iterates a table; if a table were emptied or renamed
away, those loops would pass without asserting anything.
"""
self.assertGreaterEqual(len(CONSTRUCT_CASES), 12)
self.assertGreaterEqual(len(NON_CONSTRUCT_CASES), 15)
self.assertEqual(len(BACKENDS), 4)
def test_detects_construct_query_forms(self):
for label, query in CONSTRUCT_CASES.items():
with self.subTest(case=label):
self.assertIsNotNone(
sparql_escaping.CONSTRUCT_QUERY_RE.search(query),
f"{label}: real CONSTRUCT query was not detected",
)
def test_rejects_non_construct_query_forms(self):
for label, query in NON_CONSTRUCT_CASES.items():
with self.subTest(case=label):
self.assertIsNone(
sparql_escaping.CONSTRUCT_QUERY_RE.search(query),
f"{label}: non-CONSTRUCT query was misdetected as CONSTRUCT",
)
def test_comment_alternative_does_not_backtrack(self):
"""The specific mechanism behind #931.
A comment must be consumed up to its terminator. If the character
class backtracks, the match ends *inside* the comment instead of
failing, which is what let CONSTRUCT-in-a-comment win.
"""
query = "# CONSTRUCT in a comment\nSELECT * WHERE { }"
self.assertIsNone(sparql_escaping.CONSTRUCT_QUERY_RE.search(query))
def test_carriage_return_terminates_a_comment(self):
"""CR alone ends a comment, so CONSTRUCT after it is a real CONSTRUCT.
Pins the difference between `[^\\n]*(?:\\n|\\Z)` and the shipped
`[^\\n\\r]*(?:[\\n\\r]|\\Z)`: the former treats a CR-terminated comment
as running to end of input, swallowing the query form after it.
"""
self.assertIsNotNone(
sparql_escaping.CONSTRUCT_QUERY_RE.search(
"# a comment\rCONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
)
)
self.assertIsNone(
sparql_escaping.CONSTRUCT_QUERY_RE.search(
"# CONSTRUCT in a comment\rSELECT * WHERE { }"
)
)
class TestConstructDetectionAcrossBackends(unittest.TestCase):
"""The regex is shared, so assert every backend's public detector agrees."""
def test_all_backends_detect_construct_query_forms(self):
for backend, factory in BACKENDS.items():
store = factory()
for label, query in CONSTRUCT_CASES.items():
with self.subTest(backend=backend, case=label):
self.assertTrue(
store._is_construct_query(query),
f"{backend}/{label}: real CONSTRUCT query was not detected",
)
def test_all_backends_reject_non_construct_query_forms(self):
for backend, factory in BACKENDS.items():
store = factory()
for label, query in NON_CONSTRUCT_CASES.items():
with self.subTest(backend=backend, case=label):
self.assertFalse(
store._is_construct_query(query),
f"{backend}/{label}: non-CONSTRUCT query was misdetected",
)
def test_every_backend_exposes_the_detector(self):
"""Fail loudly if a backend stops delegating to the shared regex.
Without this, a backend that dropped `_is_construct_query` would make
the loops above error rather than report a meaningful failure.
"""
for backend, factory in BACKENDS.items():
with self.subTest(backend=backend):
store = factory()
self.assertTrue(
callable(getattr(store, "_is_construct_query", None)),
f"{backend}: no callable _is_construct_query",
)
if __name__ == "__main__":
unittest.main()
+487
View File
@@ -0,0 +1,487 @@
"""Tests for the shared graph-payload normalizer (issue #956).
Graph payloads circulate under two vocabularies -- 'entities'/'relationships'
and 'nodes'/'edges' -- and consumers each reconciled them locally with at
least three competing idioms. The same payload could therefore be exported,
silently dropped, or rejected depending on which consumer read it:
``export_lpg`` dropped every entity when 'nodes' was present but empty, which
is precisely the shape ``JSONExporter`` emits.
The end-to-end assertions run the real exporters rather than mocking them,
since the behaviour under test is that the exporters now agree.
"""
import os
import shutil
import tempfile
import unittest
from dataclasses import dataclass
from semantica.export import methods as export_methods
from semantica.utils import normalize_graph_payload
from semantica.utils.exceptions import ValidationError
ENTITY = {"id": "e1", "name": "Acme"}
RELATIONSHIP = {"id": "r1", "source": "e1", "target": "e2"}
class TestVocabularyResolution(unittest.TestCase):
def test_canonical_keys_pass_through(self):
result = normalize_graph_payload(
{"entities": [ENTITY], "relationships": [RELATIONSHIP]}
)
self.assertEqual(result["entities"], [ENTITY])
self.assertEqual(result["relationships"], [RELATIONSHIP])
self.assertEqual(result["triplets"], [])
def test_aliases_are_mapped_to_canonical_keys(self):
result = normalize_graph_payload({"nodes": [ENTITY], "edges": [RELATIONSHIP]})
self.assertEqual(result["entities"], [ENTITY])
self.assertEqual(result["relationships"], [RELATIONSHIP])
def test_empty_alias_does_not_mask_a_populated_canonical_key(self):
"""The JSONExporter round-trip shape, and the #956 data-loss case."""
result = normalize_graph_payload(
{"entities": [ENTITY], "nodes": [], "relationships": [], "edges": []}
)
self.assertEqual(result["entities"], [ENTITY])
def test_empty_canonical_key_does_not_mask_a_populated_alias(self):
result = normalize_graph_payload({"entities": [], "nodes": [ENTITY]})
self.assertEqual(result["entities"], [ENTITY])
def test_identical_spellings_are_accepted(self):
result = normalize_graph_payload({"entities": [ENTITY], "nodes": [ENTITY]})
self.assertEqual(result["entities"], [ENTITY])
def test_conflicting_spellings_are_refused(self):
"""No basis to prefer either, and picking one would lose the other."""
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload(
{"entities": [ENTITY], "nodes": [{"id": "different"}]}
)
message = str(ctx.exception)
self.assertIn("entities", message)
self.assertIn("nodes", message)
def test_reordered_identical_spellings_are_accepted(self):
"""Same records, different order, is not a conflict.
A caller round-tripping through a dict-keyed cache or a set has no
reason to preserve list order; comparing spellings with plain list
equality rejected this as if the records differed.
"""
other = {"id": "e2", "name": "Beta"}
result = normalize_graph_payload(
{"entities": [ENTITY, other], "nodes": [other, ENTITY]}
)
self.assertCountEqual(result["entities"], [ENTITY, other])
def test_reordered_spellings_with_duplicate_records_still_conflict(self):
"""Multiset comparison must still catch a real count mismatch."""
with self.assertRaises(ValidationError):
normalize_graph_payload({"entities": [ENTITY, ENTITY], "nodes": [ENTITY]})
def test_triplets_are_carried_through(self):
result = normalize_graph_payload({"triplets": [{"s": "a", "p": "b", "o": "c"}]})
self.assertEqual(result["triplets"], [{"s": "a", "p": "b", "o": "c"}])
def test_missing_collections_default_to_empty_lists(self):
result = normalize_graph_payload({"entities": [ENTITY]})
self.assertEqual(result["relationships"], [])
self.assertEqual(result["triplets"], [])
def test_result_does_not_alias_the_input_collections(self):
payload = {"entities": [ENTITY]}
result = normalize_graph_payload(payload)
result["entities"].append({"id": "e2"})
self.assertEqual(len(payload["entities"]), 1)
class TestUnrecognizedInput(unittest.TestCase):
def test_unrecognized_keys_raise_by_default(self):
for payload in ({"data": [ENTITY]}, {"records": [ENTITY]}, {"foo": "bar"}):
with self.subTest(payload=payload):
with self.assertRaises(ValidationError):
normalize_graph_payload(payload)
def test_error_names_supplied_and_expected_keys(self):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"data": [ENTITY]})
message = str(ctx.exception)
self.assertIn("data", message)
self.assertIn("entities", message)
self.assertIn("nodes", message)
def test_empty_mapping_is_accepted(self):
"""An empty graph is legitimate and carries nothing that could be lost."""
result = normalize_graph_payload({})
self.assertEqual(result, {"entities": [], "relationships": [], "triplets": []})
def test_non_mapping_input_raises(self):
for payload in ([ENTITY], (ENTITY,), "entities", 42, None):
with self.subTest(payload=repr(payload)):
with self.assertRaises(ValidationError):
normalize_graph_payload(payload)
class TestExportersAgree(unittest.TestCase):
"""The divergence from #956, run against the real exporters."""
# export_csv is excluded: it writes entities/relationships/nodes/edges to
# four separate files by design, so it is not resolving two spellings of
# one collection and is out of scope for this change.
EXPORTERS = ("export_json", "export_arango", "export_neo4j_csv", "export_lpg")
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.tmpdir, ignore_errors=True)
def _export_and_read(self, name, payload):
outdir = os.path.join(self.tmpdir, name)
os.makedirs(outdir, exist_ok=True)
getattr(export_methods, name)(payload, os.path.join(outdir, "out"))
blob = ""
for root, _, files in os.walk(outdir):
for filename in files:
with open(os.path.join(root, filename), errors="ignore") as handle:
blob += handle.read()
return blob
def test_exporter_list_is_populated(self):
"""Guard against a vacuous suite if the list is emptied."""
self.assertGreaterEqual(len(self.EXPORTERS), 4)
def test_every_exporter_keeps_records_when_an_alias_is_empty(self):
payload = {
"entities": [ENTITY],
"nodes": [],
"relationships": [],
"edges": [],
}
for name in self.EXPORTERS:
with self.subTest(exporter=name):
self.assertIn(
"Acme",
self._export_and_read(name, payload),
f"{name} dropped the entity when 'nodes' was present but empty",
)
def test_every_exporter_accepts_the_alias_vocabulary(self):
payload = {"nodes": [ENTITY], "edges": []}
for name in self.EXPORTERS:
with self.subTest(exporter=name):
self.assertIn(
"Acme",
self._export_and_read(name, payload),
f"{name} dropped the entity supplied as 'nodes'",
)
def test_every_exporter_raises_processing_error_for_non_mapping_input(self):
"""A wrong-type payload is rejected the same way everywhere.
export_yaml and export_neo4j_csv raised ProcessingError for a bare
list; export_lpg and export_arango called normalize_graph_payload()
directly with no type guard, so they alone raised ValidationError
(from inside the resolver) for the identical mistake.
"""
from semantica.utils.exceptions import ProcessingError
for name in ("export_arango", "export_neo4j_csv", "export_lpg"):
with self.subTest(exporter=name):
outdir = os.path.join(self.tmpdir, name + "_bad_type")
os.makedirs(outdir, exist_ok=True)
with self.assertRaises(ProcessingError):
getattr(export_methods, name)([ENTITY], os.path.join(outdir, "out"))
def test_every_exporter_converts_object_shaped_records(self):
"""A dataclass record must not merely pass validation.
normalize_graph_payload() accepts dataclass/attribute-bearing
records (Neo4jCSVExporter reads them off attributes), but
export_lpg and export_arango read records with ``.get(...)``. A
record that passed validation unconverted crashed with a raw
AttributeError once used -- the exact failure the boundary exists
to prevent.
"""
@dataclass
class Node:
id: str
name: str
payload = {"entities": [Node(id="e1", name="Acme")], "relationships": []}
for name in ("export_arango", "export_neo4j_csv", "export_lpg"):
with self.subTest(exporter=name):
self.assertIn("Acme", self._export_and_read(name, payload))
def test_neo4j_accepts_non_dict_mappings(self):
"""Neo4jCSVExporter's mapping path must not be narrower than the rest.
_normalize_graph checked isinstance(graph, dict), so a non-dict
Mapping (a MappingProxyType, a ChainMap) fell into the
object-attribute branch and was rejected as an unrecognized object,
even though the identical payload exports fine via LPG/Arango/YAML.
"""
import types
payload = types.MappingProxyType({"entities": [ENTITY], "relationships": []})
self.assertIn("Acme", self._export_and_read("export_neo4j_csv", payload))
class TestRecordsCannotBeDroppedSilently(unittest.TestCase):
"""Presence of a recognized key is not proof the records survived.
``{"entities": [], "data": [...]}`` clears a presence-only check and still
resolves to empty, so the records under 'data' would be dropped with no
signal -- the same failure the recognition check exists to prevent.
"""
def test_empty_recognized_key_does_not_excuse_records_elsewhere(self):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [], "data": [ENTITY]})
message = str(ctx.exception)
self.assertIn("'data'", message)
self.assertIn("holds records", message)
def test_check_applies_to_every_recognized_spelling(self):
for key in ("entities", "nodes", "relationships", "edges", "triplets"):
with self.subTest(key=key):
with self.assertRaises(ValidationError):
normalize_graph_payload({key: [], "records": [ENTITY]})
def test_non_record_keys_are_not_mistaken_for_dropped_records(self):
"""ContextGraph.to_dict() always carries 'statistics'.
An empty graph must stay exportable, so only a non-empty list counts
as evidence that records were dropped.
"""
result = normalize_graph_payload(
{"nodes": [], "edges": [], "statistics": {"node_count": 0}}
)
self.assertEqual(result["entities"], [])
self.assertEqual(result["relationships"], [])
def test_records_alongside_a_populated_collection_are_not_refused(self):
"""Something resolved, so the export is not silently empty."""
result = normalize_graph_payload({"entities": [ENTITY], "statistics": {"n": 1}})
self.assertEqual(result["entities"], [ENTITY])
class TestCollectionValuesAreValidated(unittest.TestCase):
"""A recognized key is not proof its value is a collection of records.
Resolving on truthiness alone let ``{"entities": "abc"}`` through as three
single-character "records" and let ``{"entities": 42}`` surface as a raw
``TypeError`` from ``list()`` inside an exporter, naming the exporter
rather than the payload key at fault. Both are rejected here, at the
boundary that owns the question.
"""
COLLECTION_KEYS = ("entities", "nodes", "relationships", "edges", "triplets")
# Every public export path that reads its payload through the normalizer.
# export_json is excluded: it treats the payload as opaque records rather
# than resolving graph collections, so it never calls the normalizer.
NORMALIZING_EXPORTERS = (
"export_arango",
"export_neo4j_csv",
"export_lpg",
"export_yaml",
)
def test_string_value_is_not_treated_as_a_collection(self):
for key in self.COLLECTION_KEYS:
with self.subTest(key=key):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({key: "abc"})
message = str(ctx.exception)
self.assertIn(f"'{key}'", message)
self.assertIn("str", message)
def test_bytes_value_is_not_treated_as_a_collection(self):
for value in (b"abc", bytearray(b"abc")):
with self.subTest(value=repr(value)):
with self.assertRaises(ValidationError):
normalize_graph_payload({"entities": value})
def test_scalar_value_raises_validation_error_not_type_error(self):
for key in self.COLLECTION_KEYS:
for value in (42, 3.5, True, object()):
with self.subTest(key=key, value=repr(value)):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({key: value})
self.assertIn(f"'{key}'", str(ctx.exception))
def test_mapping_value_is_not_treated_as_a_collection(self):
"""``{"nodes": {"id": "n1"}}`` -- a single record, or an ID index."""
for payload in (
{"nodes": {"id": "n1"}},
{"entities": {"e1": ENTITY}},
{"edges": {"id": "r1"}},
):
with self.subTest(payload=payload):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload(payload)
self.assertIn("mapping", str(ctx.exception))
def test_non_record_elements_are_rejected(self):
for value in (["Acme"], [ENTITY, "Acme"], [42], [None], [[ENTITY]]):
with self.subTest(value=repr(value)):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": value})
self.assertIn("'entities'", str(ctx.exception))
def test_error_names_the_offending_index(self):
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [ENTITY, ENTITY, "Acme"]})
self.assertIn("index 2", str(ctx.exception))
def test_object_records_are_accepted(self):
"""Attribute-bearing objects are accepted and converted to dicts.
LPGExporter and ArangoAQLExporter read records with ``.get(...)``, so
an object record that merely passed validation unconverted would
still crash with AttributeError once used; the boundary converts it.
"""
class Node:
def __init__(self):
self.id = "e1"
self.name = "Acme"
node = Node()
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "e1", "name": "Acme"}])
def test_dataclass_records_are_accepted(self):
@dataclass
class Node:
id: str
node = Node(id="e1")
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "e1"}])
def test_tuple_collections_are_accepted_and_materialized(self):
result = normalize_graph_payload({"entities": (ENTITY,)})
self.assertEqual(result["entities"], [ENTITY])
def test_none_is_read_as_an_absent_collection(self):
"""JSON round-trips an absent collection to null."""
result = normalize_graph_payload(
{"entities": None, "relationships": [RELATIONSHIP]}
)
self.assertEqual(result["entities"], [])
self.assertEqual(result["relationships"], [RELATIONSHIP])
def test_null_collection_still_cannot_hide_dropped_records(self):
with self.assertRaises(ValidationError):
normalize_graph_payload({"entities": None, "data": [ENTITY]})
def test_every_spelling_is_validated_not_just_the_winner(self):
"""A malformed alias is a defect even when the canonical key resolves."""
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [ENTITY], "nodes": "abc"})
self.assertIn("'nodes'", str(ctx.exception))
def test_malformed_value_reaches_no_exporter(self):
"""The end-to-end half: no exporter sees a TypeError from list()."""
tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, tmpdir, ignore_errors=True)
for name in self.NORMALIZING_EXPORTERS:
for value in ("abc", 42, {"id": "n1"}):
with self.subTest(exporter=name, value=repr(value)):
outdir = os.path.join(tmpdir, f"{name}_{type(value).__name__}")
os.makedirs(outdir, exist_ok=True)
with self.assertRaises(ValidationError):
getattr(export_methods, name)(
{"entities": value}, os.path.join(outdir, "out")
)
class TestIsRecordBoundary(unittest.TestCase):
"""_is_record gates the validation boundary introduced by this PR.
Modules and class/type objects carry ``__dict__`` but are not graph
records. Passing them through previously produced ``AttributeError``
inside exporters rather than a ``ValidationError`` at the boundary.
"""
def test_python_module_in_entities_raises_validation_error(self):
"""import math; {"entities": [math]} must be rejected at the boundary."""
import math
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [math]})
self.assertIn("'entities'", str(ctx.exception))
def test_class_object_in_entities_raises_validation_error(self):
"""A class (type object) is not a graph record."""
class MyNode:
pass
with self.assertRaises(ValidationError) as ctx:
normalize_graph_payload({"entities": [MyNode]})
self.assertIn("'entities'", str(ctx.exception))
def test_user_defined_instance_with_attributes_is_accepted(self):
"""Attribute-bearing instances are the legitimate use-case, converted
to a dict so every exporter -- not just Neo4jCSVExporter -- can read
it with ``.get(...)``."""
class Node:
def __init__(self):
self.id = "n1"
self.name = "Alice"
node = Node()
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "n1", "name": "Alice"}])
def test_dataclass_instance_is_accepted(self):
"""Dataclasses are a common record type used by Neo4jCSVExporter,
converted to a dict at the boundary so LPGExporter and
ArangoAQLExporter can read it too."""
node = dataclass_node()
result = normalize_graph_payload({"entities": [node]})
self.assertEqual(result["entities"], [{"id": "dc1"}])
def test_mapping_record_is_accepted(self):
"""Plain dicts are the canonical record shape."""
result = normalize_graph_payload({"entities": [ENTITY]})
self.assertEqual(result["entities"], [ENTITY])
def test_module_rejected_through_normalizing_exporter(self):
"""End-to-end: a module element must not reach an exporter's internals."""
import math
tmpdir = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, tmpdir, ignore_errors=True)
for name in ("export_arango", "export_neo4j_csv", "export_lpg"):
with self.subTest(exporter=name):
outdir = os.path.join(tmpdir, name)
os.makedirs(outdir, exist_ok=True)
with self.assertRaises(ValidationError):
getattr(export_methods, name)(
{"entities": [math]}, os.path.join(outdir, "out")
)
@dataclass
class _DataclassNode:
id: str
def dataclass_node():
return _DataclassNode(id="dc1")
if __name__ == "__main__":
unittest.main()