Compare commits

...
6 Commits
Author SHA1 Message Date
Kevin Zhang da642f12fa fix(export): @vocab mints into the shipped ns# namespace (#1236)
fix(export): keep caller data out of the shipped ns# namespace

Every JSON-LD context set @vocab to https://semantica.dev/vocab/,
which 404s, so every bare term in caller data (extracted entity/
relationship types, arbitrary metadata keys) minted under a namespace
the package never ships. The obvious fix, pointing @vocab at
SEMANTICA_NS instead, turned out to be worse than the dead link: since
that namespace is real and populated, every bare term a caller happens
to use now expands into something that looks like official Semantica
vocabulary. An extracted type "ORG" became ns#ORG, a class the
vocabulary never defines. A metadata key "source" attached a plain
string value to sem:source, an owl:ObjectProperty that already exists
in semantica-ns.ttl with a resource-valued range, silently corrupting
its semantics.

@vocab is now removed from all five contexts (four in
json_exporter.py, one in rdf_exporter.py) rather than repointed.
Every document already used explicit semantica: prefixes for its own
terms, so nothing else in the output changes; an unscoped bare term
now simply fails to expand, which is standard JSON-LD behavior for a
context that doesn't know it, instead of being silently claimed by
our namespace.

Two call sites needed to stop handing caller data to @type/bare terms
in the first place:

- Entity nodes are always typed semantica:Entity now, with the
  caller's label carried as a semantica:type string instead of
  minted into @type. This matches how relationship nodes already
  carried their type. sem:type's domain in semantica-ns.ttl opens up
  to cover entities as well as relationships, following the
  sem:confidence precedent, since the property is now legitimately
  emitted for both.
- semantica:metadata gets an explicit @json term definition, so a
  caller's metadata dict travels as one rdf:JSON literal instead of
  having its keys expand as separate predicates. A metadata key can
  no longer collide with a real ontology term no matter what the
  caller names it.

Both JSONExporter and RDFExporter.serialize_to_jsonld got the same
treatment, since they build separate JSON-LD structures for the same
underlying data.

The regression tests assert the negative space this bug lived in: no
context declares @vocab, no caller type label appears as an rdf:type
under ns#, and no caller metadata key appears as a predicate under
ns# at all, only as content inside the single JSON literal.

Closes #1146
2026-08-28 19:53:35 +05:00
Aldrin Joseph 5376f046ca fix(explorer): dedupe temporal snapshot requests and apply latest-wins (#1241)
fix(explorer): dedupe temporal snapshot requests and apply latest-wins

The temporal snapshot effect fetched /api/temporal/snapshot with no
idempotency or ordering guards. Upstream churn (timeline recreation
while bounds settle, play ticks resetting the playhead, drag events)
could re-request the same `at` repeatedly, and with variable network
latency an older position's response could land after a newer one's,
overwriting the active-node count, so the chip visibly lagged the
scrubber.

Add a small stateful guard module (temporalSnapshotGuards.ts) built
around a per-position cache, keyed by the debounced timestamp's
primitive millisecond value rather than the Date object, so upstream
object-identity churn cannot defeat the dedup on its own:

- at most one in-flight request per scrubber position, so identical
  `at` values arriving while a request is pending are dropped instead
  of firing a fresh fetch, breaking the idle/play polling loop;
- successful snapshots are cached per position and re-applied when the
  scrubber returns to it (play wrap-around, back-scrubbing) without a
  network round trip;
- a response is applied only while the scrubber is still on the
  position it was requested for, so an out-of-order response can never
  clobber a newer position's count;
- failed, cancelled, or superseded requests release their position so
  it can be fetched again the next time it's visited, rather than
  stalling it permanently;
- reset() drops all cached and in-flight state when the underlying
  graph summary changes (reload/retry), since snapshots cached against
  the previous graph no longer describe anything real. Keyed on the
  summary query's data identity, which react-query keeps stable
  (staleTime: Infinity plus structural sharing) unless the graph data
  itself was replaced, so reset fires exactly on a real reload and not
  on cosmetic re-renders.

The snapshot effect is wired through the guards end to end: begin()
returns either a fresh sequence number to fetch under or a cached
snapshot to reapply directly; the same shouldApply()/apply() gate
handles both the network and cached-reapply paths so they can't drift
apart; finish() runs from both the fetch's failure branch and its
cleanup function, so a cancelled or failed request is always retryable
on the next visit instead of leaving its position stuck in-flight.

16 unit tests cover dedup, independent positions, revisit re-apply,
play wrap-around, failure retry, stale-sequence protection (a late
response or a late release from a superseded request cannot act on a
newer request's position), reset-on-reload, and cache-bound eviction.

Closes #1128
2026-08-28 19:45:13 +05:00
Guofang.Tang 56d9e9a857 fix(ontology): coalesce normalized property collisions (#1231)
fix(ontology): coalesce normalized property collisions

Different raw property spellings can normalize to the same ontology
name and IRI. works_for and worksFor, for example, both normalize to
worksFor, but property inference emitted a separate definition for
each spelling, so the generated ontology declared two distinct
properties under what would become the same IRI once minted. The same
collapse could also happen across kinds: a relationship type and an
entity attribute that normalize to the same name would previously
produce a data property and an object property sharing one name, with
no signal that anything was wrong.

infer_properties() now runs a coalescing pass after object and data
properties are both inferred. Properties are grouped by (kind, name).
Object properties that collide are merged in occurrence order:
domains and ranges are unioned rather than overwritten, so a property
seen across several source classes keeps every domain instead of
losing all but the first, and occurrence_count is summed across the
merged spellings so downstream confidence/frequency signals stay
correct. Data properties merge domains the same way and reconcile
differing ranges through the existing _get_more_general_type()
widening logic already used elsewhere in this file, rather than a new
implementation.

A name that resolves to both an object property and a data property
is not silently coalesced into either one, since the two kinds mean
different things in the emitted ontology. That case raises a
ValidationError up front, naming every colliding name and which kinds
collided, so the conflict surfaces before an ambiguous ontology is
written rather than after.

Verified beyond the two cases in the new test file: a data property
colliding across two different domain classes correctly unions the
domain instead of keeping only the first class, and three distinct
spellings of the same relationship type collapse into one property
with the occurrence count correctly summed across all three.

Follow-up to #1170 (relationship endpoint types) and #1171 (retained
data properties for normalized class names).
2026-08-28 16:23:46 +05:00
KaifAhmad1 ecb33a5b7d chore(release): prepare v0.6.7
Bump version, cut CHANGELOG's Unreleased section into 0.6.7, backfill
changelog entries for merged PRs missing from it, and refresh
version-dependent references in README/docs.
2026-08-28 15:51:17 +05:30
Kevin Zhang 100e95a098 feat(ingest): add SAP OData ingestor (#1228) (#1234)
Adds `SAPODataConnector`, `SAPODataEntity`, and `SAPIngestor` for ingesting master and transactional data from SAP OData services, mainly things like Business Partners and Sales Orders.

Tested around S/4HANA Cloud, SuccessFactors, and on-prem NetWeaver Gateway style OData endpoints.

Main pieces included:

* OAuth2 client credentials and Basic auth support. Both go through the existing `ssrf.py` checks, including the OAuth token request.
* Small EDMX parser used by `discover_service()` so we don't need to pull in `pyodata`.
* Server-side pagination support for both OData versions:

  * v2: `__next`, including plain string and `__deferred` formats
  * v4: `@odata.nextLink`
* Keeps the service path in the base URL correctly whether the URL has a trailing slash or not. This is normalized in `SAPIngestor.__init__`.
* Adds an `ingest-sap` extra with just `requests`, so there is no SAP/proprietary SDK dependency.

This is meant to be a fairly small first version of the connector without adding a lot of SAP-specific dependencies.

Closes #1228
2026-08-28 14:54:02 +05:00
cxzg007and江俊杰 cce5ea177c fix(pipeline): set_parallelism now enables dependency-layer parallel execution (closes #1223) (#1226)
PipelineBuilder.set_parallelism() validated and stored a level in
pipeline config, but ExecutionEngine._execute_steps() had no parallel
code path and no code ever read it back, so steps always ran strictly
sequentially regardless of the configured value. parallelism was also
lost across a serialize/deserialize round trip, since the nested
config key was never promoted to the top-level dict build_pipeline()
reads.

Steps are now grouped into dependency layers (declaration order
preserved within each layer). A layer runs concurrently, bounded by
ThreadPoolExecutor(max_workers=min(configured parallelism, engine
max_workers)), only when every one of the following holds: more than
one step in the layer, the shared input is a dict, every step is
opted in via the new PipelineStep.parallel_safe flag, and no step is
in delta_mode. Any layer that doesn't meet all four falls back to the
existing sequential path unchanged.

Each step's input is deep-copied before any handler in the layer
starts, so concurrent steps never share mutable state. Layer results
are merged back in declaration order, not completion order; keys
whose value is unchanged from the shared input are treated as an
echo rather than a write, so two handlers both returning {**data, ...}
don't spuriously conflict on keys neither of them actually touched.
Genuinely conflicting values for the same key raise ProcessingError
naming both the key and the two steps involved. Retry policy, step
status, result/error tracking, and progress reporting are shared
between the sequential and parallel paths so behavior stays identical
either way. On step failure, not yet started futures in the same
layer are cancelled and the error propagates, so no downstream layer
ever runs.

parallel_safe is opt-in per step because handlers that share mutable
state or depend on strict ordering are not safe to run concurrently.
ParallelismManager.execute_pipeline_steps_parallel() is deliberately
not reused here; the engine implements its own bounded layer
scheduler so retry/status semantics stay identical between the
sequential and parallel code paths instead of diverging.

fix(pipeline): address qodo review findings on PR #1226

- detect circular/unknown dependencies in parallel grouping
  (ValidationError instead of RecursionError/KeyError)
- skip unchanged echoed keys in parallel result merging to
  avoid false conflicts
- fail before COMPLETED status when a parallel step returns a
  non-dict; never retry such contract violations
- require strict bool parallel_safe in builder and engine gate
- make per-step progress tracking IDs unique across same-type
  parallel steps

---------

Co-authored-by: 江俊杰 <jiangjunjie.37@jd.com>
2026-08-28 12:33:39 +05:00
26 changed files with 3311 additions and 117 deletions
+94
View File
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.6.7] - 2026-08-28
### Added
- **First-class LangChain integration** (closes #963; recreates #969)
@@ -20,6 +22,98 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- All adapters remain importable without langchain-core (`LANGCHAIN_AVAILABLE` flag)
- Docs: `docs/integrations/langchain.md`, README native-integration matrix, and `docs.json` nav entry
- **SAP OData ingestor** (#1234, closes #1228) by @pkupt
- New `SAPODataEntity` / `SAPODataConnector` / `SAPIngestor` (`semantica.ingest`, lazy exports), following the three-layer connector pattern already used for Snowflake/Databricks, to pull master/transactional data (Business Partners, Sales Orders) from SAP OData v2/v4 services into the Context Graph
- Dual auth (OAuth2 client-credentials for BTP/S4HANA Cloud, Basic for on-prem NetWeaver); every outbound request, including the token exchange, routes through `request_with_ssrf_guard`
- `$metadata` (CSDL XML) is parsed with a hand-rolled `xml.etree` reader rather than pulling in `pyodata`; pagination follows OData v2 `__next`/`__deferred` and v4 `@odata.nextLink`
- New `pip install semantica[ingest-sap]` extra (`requests>=2.28.0`)
- **Known phase-1 limits** (documented in docstrings): the OAuth2 token is cached but never refreshed, and pagination has no `max_pages` fuse (`top` bounds it when supplied)
- New `tests/ingest/test_sap_ingestor.py`: 22 tests (auth, EDMX parsing, v2/v4 pagination, SSRF routing, error paths, service-root normalization)
- **`ContextGraph` gains deterministic, human-editable Markdown round-trip persistence** (#852) by @SaurabhScripts
- `save_to_markdown()`/`load_from_markdown()` write one file per node plus a graph manifest, so a graph can be reviewed and hand-edited outside the application without giving up the existing JSON API or its default behavior
- An existing destination is validated as a complete, canonical managed export before atomic replacement, so the loader can't silently clobber an unrelated or manually-extended directory
- Import/export paths and their ancestors reject symlinks, Windows junctions, and other reparse points, with pre-open and post-open validation — the same hardening applied to `AgentMemory`'s existing Markdown import in the companion fix below
- Dangling edge endpoints import as JSON-compatible entity stubs rather than being rejected outright (matching what the JSON loader already accepts); node/edge indexes, adjacency, and analytics/retraction/tombstone state are rebuilt after a Markdown load, and granular node/edge events are still emitted so temporal audit history stays useful
- New `tests/context/test_context_graph_markdown.py`: 29 passed, 1 skipped (the skipped case creates a real Windows junction and runs on Windows CI); full `tests/context/` suite: 614 passed, 1 skipped
- **Explorer graph inspector gains a read-only Markdown content viewer** (#1078, closes #900) by @sakshi04-ui — Preview (rendered GFM) and Source (exact, whitespace-preserving) tabs for node content, with a copy-to-clipboard action. A URL allowlist restricts links to `http:`/`https:`/`mailto:`/in-document anchors, raw HTML execution is disabled, and external links carry `rel="noopener noreferrer"`. A first, focused step toward human-editable memory (#765); no write path yet. New `explorer/tests/markdownContentViewer.test.ts`: 8 tests
- **Follow-up (perf)** (#1195, addresses #1118) by @pravit-amp: `remarkPlugins` and the ~20-entry renderer `components` map were inline literals, so every unrelated re-render (e.g. clicking Copy) re-ran the full remark parse and remounted the whole subtree — up to 1.1s of main-thread block on a 2000-row GFM table. Both are now hoisted to module scope and the rendered element is memoized on content, cutting re-render cost from as much as 1121ms to ~0.1ms across all measured fixtures with no change to rendered output. A separate, upstream `remark-gfm` table-parse cost (~O(n^1.9), not fixed here) is left open on the issue as a product decision
- **Follow-up (cleanup)** (#1194, closes #1119) by @pravit-amp: the pure `isSafeUrl` URL-safety helper is extracted out of `MarkdownContentViewer.tsx` into its own `markdownUrlSafety.ts` module (behavior-preserving — moved verbatim), so the component module exports only components and stops tripping `react-refresh/only-export-components`
- **`reasoning` gains a structured Action layer — rule-driven side effects with optional provenance** (#1096, closes #1095) by @cxzg007`AssertAction`/`RetractAction`/`CallAction`/`EmitEventAction` let a matched rule write facts back to a `KnowledgeGraph`, retract facts, call a structured handler (replacing the previously-unused `Rule.handler`), or emit to a sink registered via `Reasoner.on_event`, turning the reasoner from a pure inference engine into a production-rule system. With `provenance=True`, fired actions are recorded to `Reasoner.action_log`. Fully additive — rules without `actions` are unaffected, and the legacy `handler` field still fires (now wrapped internally as a `CallAction`). Also fixes a latent dangling import in `reasoning_provenance.py` (`ReasoningEngine`/`infer``Reasoner`/`infer_facts`). New `tests/reasoning/test_rule_actions.py`: 9 tests; full `tests/reasoning/` suite: 54 passed
- **`run_shacl_validation` is now a public, documented entry point** (#1189, closes #1186) by @mikemikimike — the SHACL guide had documented the private `_run_pyshacl` helper as the canonical API; it's now exposed through `semantica.ontology`, with `_run_pyshacl` kept as a compatibility alias over the same implementation. `tests/ontology/test_ontology_advanced.py`: 33 passed (also fixes a flaky comparison against pySHACL's non-deterministic blank-node shape identifiers by comparing stable report fields instead)
- **`docs/storage-backends.md`: adapter inventory and RDF/LPG feature matrix** (#899, addresses #888) by @yulinlina — which graph storage backends are built-in vs. bring-your-own, and where provenance/context support is partial
- **`docs/guides/shacl-validation.md`: documented that `rdfs:range` + RDFS entailment makes `sh:class` unfalsifiable** (#1182, fixes #1130) by @ALDRIN121 — with entailment on, pyshacl infers the declared range class onto every object, so a `sh:class` constraint can never fail and reports `conforms: True` on non-conforming data; added to Common Pitfalls with the `inference="none"` vs `inference="rdfs"` contrast and guidance to re-run `sh:class` shape sets with entailment off before trusting a pass
- **Cookbook: four new module notebooks**`22_Provenance_Tracking.ipynb` (#989, lineage walks, revision history, invalidation, checksums), `23_Reasoning.ipynb` (#990, `Reasoner`/`DatalogReasoner`/`ExplanationGenerator`), `24_Change_Management.ipynb` (#991, versioned snapshots, named tags, checksum tamper-detection), and `25_Seed_Data.ipynb` (#992, bootstrapping a foundation graph from a trusted CSV source) — all by @LeonSGP43, filling gaps where the corresponding module shipped a usage doc but no runnable tutorial; every cell verified against current module source. `docs/cookbook.md` index entries for all four added in #1225
- **README "Cite Us" section and `docs/citation.md` cross-link** (#1210) by @KaifAhmad1 — BibTeX/APA/MLA/Chicago/IEEE citation forms; also corrects the copyright holder in `LICENSE`/`docs/project-license.md` from the stale "Hawksight AI" to "Semantica" and replaces the retired `Hawksight-AI` GitHub org slug with `semantica-agi` across ~40 files (READMEs, issue templates, plugin manifests, cookbook notebooks, docs)
### Changed
- **A registered custom method can now refuse, instead of being silently overridden by the default implementation** (#1127, closes #1108) by @fabio-rovai — every module supporting custom methods wrapped the registered callable in a `try`/`except` that logged a warning and ran the built-in default on *any* exception, including one a validator or policy gate raised on purpose to say "do not produce this output." That made every registered gate advisory rather than authoritative. `semantica/utils/custom_methods.py` now centralizes the policy: an exception from a registered method propagates to the caller by default; `fallback_on_custom_error=True` restores the previous warn-and-continue behavior per call. Applied mechanically across all 58 call sites in `export/`, `ingest/`, `normalize/`, `parse/`, `embeddings/`, and `kg/` methods modules. New `tests/utils/test_custom_method_can_refuse.py`: 13 tests, including the reported gate-deletes-and-raises scenario and a guard that no call site still swallows
- **Removed 13 confirmed-dead symbols across 9 files** (#1176, closes #1174) by @Vinv-AI — private helpers and Explorer app-layer code with zero callers in code, tests, or docs, none part of the public API or a FastAPI `response_model`; 289 deletions, no behavior change
- **Consolidated the two duplicate Turtle/N-Triples literal escapers in `rdf_exporter.py`** (#1221, closes #1218) by @pkupt`_escape_turtle_literal` (added in #1148) escaped the same five characters in the same order as the older module-level `_escape_literal`; the redundant one is dropped and all four call sites route through the original. Behavior no-op, verified against the full export suite (301 passed, 1 skipped)
- **Removed the unreachable `_extract_with_spacy()` method and the unused `self.nlp` attribute from `NERExtractor`** (#1220, fixes #1058) by @yunaremaia — the ML dispatch path has always gone through `methods.py`'s process-level model cache instead; `__init__` still validates the spaCy runtime up front but no longer eagerly loads a model nothing on the instance reads
- **Cleaned up an unused `sys` import and import ordering in `semantica/worker.py`** (#1061) by @aoright
- **Test-only contributions**: isolated `sys.modules` mock leakage between `tests/visualization/` files so the suite passes in any collection order (#897, closes #859, by @luantaraschi); added coverage for 4 previously-untested `ConflictResolver` strategies and 3 `ConflictDetector` conflict types (#902, fixes #865, by @Devansh070); added a regression test tracking relationship provenance through `ProvenanceManager` (#1071, closes #1055, by @dex0shubham); added `max_tokens`-propagation regression coverage for LLM extraction methods, later folded into the cache-key fix below (#925, by @saiganesh47)
### Fixed
- **`SPARQLReasoner.execute_query()` claimed to run a query but always returned an empty result** (#1087, fixes #1083) by @ALDRIN121 — both the store-configured and unconfigured branches returned an empty `SPARQLQueryResult` with no real execution behind it, so a caller trusting "no matches" (e.g. a compliance check) could draw a false-negative conclusion from a method that never actually queried anything. Until a real triplet-store execution path lands, it now raises `NotImplementedError` explaining why, and the dead cache/inference scaffolding after the unreachable execution point is removed. 3 new regression tests
- **`DuplicateDetector` merged entities that share no identifier, type, or name** (#1149, fixes #1137) by @pkupt`_create_duplicate_candidate()` only ever boosted confidence for matching types and never penalized a mismatch, so two sparse, differently-typed entities (e.g. a `Person` and an `Organization`) could land above the merge threshold and collapse into one node, silently dropping the second. Two non-empty, differing types are now never a duplicate candidate. `tests/deduplication/`: 92 passed
- **`TemporalGraphQuery.analyze_evolution()`'s `stability` metric was a hardcoded placeholder** (#1143, closes #1142) by @cxzg007 — every bounded relationship contributed a constant `1`, so `stability` was always `1.0` or `0` regardless of how long relationships actually stayed valid. Now computes the mean valid-time duration in seconds across relationships with both `valid_from`/`valid_until` set; unbounded/half-open intervals are skipped and negative intervals clamp to zero. 3 new tests in `tests/kg/test_kg.py`
- **CodeQL false-positive on a JSON-LD test's URL check** (#1183) by @KaifAhmad1`"https://schema.org/" in flattened` pattern-matched CodeQL's substring-sanitization heuristic even though `flattened` is always a `list` (exact membership, no sanitization or SSRF path involved); rewritten as an explicit `any(entry == ... for entry in flattened)` with identical behavior
- **HuggingFace NER extraction crashed on `huggingface_model` being forwarded as an unexpected pipeline loader kwarg** (#1188, fixes #1063) by @shahzaib-ahmadcs — while preserving genuinely supported pipeline kwargs like `aggregation_strategy`. 5 tests pass
- **JSON-LD document/graph `@id` was minted from the wall clock, so re-exporting an unchanged graph produced a new subject every time** (#1181, closes #1147) by @reddynitish — merging repeated exports duplicated graph identity instead of recognizing them as the same graph. The `@id` is now content-derived, with optional `graph_uri`/`document_uri` overrides for callers with a stable graph name; `semantica:exportedAt` still records export time separately. Applies to both JSON-LD export paths
- **`ContextGraph.get_causal_chain()` only matched the canonical uppercase causal-edge spellings, silently missing edges recorded in `CausalChainAnalyzer`'s present-tense vocabulary** (#1187, fixes #1184) by @ALDRIN121`causes`/`influences`/`precedes` differ from `CAUSED`/`INFLUENCED`/`PRECEDENT_FOR` in word form, not just case, so an edge recorded with the analyzer's spelling produced an empty audit chain — silent, and in the dangerous direction for a compliance trace. `add_causal_relationship()` now normalizes through an alias map before storing the canonical form; traversal accepts the union vocabulary. 2 new regression tests, full `tests/context/` suite: 587 passed
- **`semantica embed generate` corrupted its own output and could recurse into a stack overflow** (#996/#1004/#1005, closes #994) by @varunsahni18, @yzxcj797 — three compounding defects in one pipeline. (1) `generate_embeddings`/`embed_text`/`calculate_similarity`/`pool_embeddings` all registered themselves as their own custom-method-registry default, so an unqualified call (exactly what the CLI does) re-entered the same wrapper until Python's recursion limit; each of the four dispatch sites now guards on registry identity before recursing (#996, #1005). A second self-recursion in `EmbeddingGeneratorWithProvenance.__getattr__` (re-entering itself when `_generator` is unset, e.g. during a `deepcopy` probe) now raises a normal `AttributeError` for private names instead (#1005). (2) `--output embeddings.parquet` wrote `json.dumps(result, default=str)` regardless of extension, turning a numpy array into its plain-text `repr()` — a file `embed index` then failed to open as Parquet; the writer now detects `.parquet`/`.json`/`.jsonl` and produces real Parquet/JSON, rejecting any other extension with a clear message (#996, #1004). (3) `pyarrow` was only in optional extras despite being required by the documented quick-start flow; promoted to a core dependency (#996)
- **`AgentMemory`'s existing Markdown import accepted symbolic links, NTFS junctions, and other Windows reparse points** (#851) by @SaurabhScripts — a direct linked import path is now rejected with an actionable error, and a linked entry found inside an otherwise-valid directory is skipped rather than aborting the whole import; hardened with pre-open/post-open checks, `O_NOFOLLOW` where available, and `fstat`-based regular-file validation. `tests/context/`: 595 passed, 1 skipped (Windows-junction test, runs on Windows CI)
- **A caught vector-similarity scoring exception left stale partial state behind, risking a misleading match on the next call** (#885, fixes #875) by @ArmanGrewal007 — the exception is now logged at debug level and `vector_score`/`vector_idx` reset to neutral values before the remaining matching stages continue
- **`RDFExporter` could write invalid or unintended relative IRIs for `GraphBuilder`-default entity/relationship identifiers** (#1112, closes #1099) by @mikemikimike — normalization is now applied at the RDF export boundary across Turtle (including temporal Turtle), RDF/XML, and N-Triples: bare/relative identifiers are minted under the Semantica namespace with safe percent-encoding, absolute IRIs pass through unchanged, and configured/input-context prefixes expand through the effective namespace mapping. 38 focused regression tests; 175 export tests plus 46 subtests pass
- **`RDF4JStore`'s `repository_id` constructor argument had no effect** (#1192, closes #1191) by @Freakz2z — the explicit id is now honored when selecting the repository; stale documentation caveats claiming otherwise are removed. 65 tests pass across the affected triplet-store suites
- **Non-interactive stdout (piped/redirected output, CI logs) was flooded with progress-bar escape sequences** (#1193, fixes #1185) by @ALDRIN121 — a plain `python demo.py > out.txt` captured 173 bytes of progress noise around 10 bytes of real output. `ProgressTracker` now attaches its console display only for an interactive terminal, Jupyter, or the new `SEMANTICA_FORCE_PROGRESS` opt-in (following the `NO_COLOR`/`FORCE_COLOR` convention); file-based progress logging is untouched. Both switches are now documented in the README and `docs/reference/utils.md`. 11 tests pass (6 new)
- **Entity `metadata` was dropped by every RDF serializer except the JSON-LD path**, so an entity kept its confidence but lost its source document, page, extractor, and reviewer on Turtle/N-Triples/RDF/XML/`RDFExporter`'s own JSON-LD (#1165, closes #1154) by @fabio-rovai — Semantica's own metadata keys (`num_entities`, `snapshot_time`, Neo4j loader fields, etc.) are now mapped to declared vocabulary terms and carried through on every path; a caller-supplied key with no mapped term is skipped with an explicit warning (rather than silently vanishing) naming the override needed, pending the caller-key namespace decision tracked in #1146. 21 new tests in `tests/export/test_metadata_passthrough.py`; `tests/export`+`tests/ontology`: 274 pass
- **`extract_relations_llm` silently dropped caller-supplied generation parameters** (`max_tokens`, `top_p`, `seed`, etc.), and the extraction cache didn't distinguish calls made with different generation settings (#1213, with test coverage from #925) by @Sameer6305 — a small hardcoded allowlist forwarded only `temperature`/`verbose` to `generate_typed`, discarding the rest; fixed by forwarding all caller kwargs. Once forwarded, those parameters also needed to enter the cache key, since two calls differing only in `max_tokens` previously shared one cache entry and the second could silently reuse a result generated under the first's settings — now applied consistently across entity, relation, and triplet LLM extraction. New regression tests for cache bypass/reuse under differing `max_tokens`/`temperature`
- **`OxigraphStore` silently ignored the `storage_path` constructor argument and never flushed writes before a reopen**, both causing silent on-disk data loss (#970) by @logan-jl-cc — `__init__`'s parameter is named `path`, so the project-conventional `storage_path` landed in `**config` and was ignored, degrading a supposedly-persistent store to in-memory with no error; `storage_path` is now accepted as an alias. Separately, pyoxigraph's background flush can lag behind a write, so a reopen immediately after `add_triplets` could observe fewer triples than were written; writes to an on-disk store now call `flush()` explicitly. 2 new regression tests, full suite: 9 passed
- **MCP server's `export_graph` tool was broken on every output format** (#1151) by @Arasz — the `json` branch called `JSONExporter().export()` without the `file_path` it requires, and every RDF branch passed a `ContextGraph` object where the exporters expect the canonical kg dict, both surfacing as a raw exception string. A third bug compounded both: the RDF export path's progress bar wrote to stdout, which over stdio MCP *is* the JSON-RPC framing, corrupting the protocol and hanging the client (a 300s timeout on an empty graph). Fixed by converting through `ContextGraph.to_kg_dict()`, serializing the JSON branch to match the RDF branches' string contract, and forcing `SEMANTICA_DISABLE_PROGRESS=1` for the server process. 5 new tests, verified failing against 0.6.6 beforehand
- **`OntologyIngestor` dropped every class and property from a JSON-LD document using a named graph** (#1156, fixes #1129) by @13g4d0 — a top-level `@id` beside `@graph` names the graph, and `rdflib.Graph.parse()` silently loads only the default graph, discarding the rest; `POST /api/ontology/load` returned `status: "success"` with `class_count: 0`. Now parses into a `Dataset` and flattens all quads into the working graph (the same `Graph``Dataset` migration #757 made for `JenaStore`, extended to the ingest path). On the PR's real-world reproduction: 25 triples/1 subject before, 719 triples/45 classes/40 object properties after. 4 new tests including a default-graph canary so the fix can't trade one blind spot for another
- **Turtle and N-Triples RDF export interpolated entity `text` into string literals with no escaping**, so a `"`, backslash, newline, CR, or tab in the source text emitted invalid RDF other parsers rejected (#1148, closes #1098) by @pkupt — a shared `_escape_turtle_literal()` (later consolidated in #1221) now escapes per the RDF 1.1 Turtle grammar and is reused for the N-Triples path, which previously escaped only quotes and newlines. `tests/export/`: 161 passed, 1 skipped
- **`PipelineSerializer` round trips dropped step dependencies and delta-processing metadata, and could rehydrate a legacy stringified handler as a non-callable string** (#1217, fixes #1216) by @cxzg007 — step dependencies, delta mode, and base/target version IDs are now restored from the serialized schema; runtime handler callables are treated as process-local state and excluded from serialized business configuration rather than (mis)serialized. 52 tests pass
- **`PipelineBuilder` never actually dispatched to a handler registered by `step_type`**, and a serialize/deserialize round trip could leak `handler`/`dependencies` into a step's business config (#1215, fixes #1214) by @cxzg007 — a registered handler is now resolved by `step_type` when no explicit `handler=` is supplied (explicit handlers still take precedence), and the two builder-control fields are kept out of `PipelineStep.config` so a strict handler signature can't receive them as unexpected kwargs. `tests/core`+`tests/pipeline`: 50 passed
- **`PipelineBuilder.set_parallelism()` was accepted and stored but never read — pipeline steps always ran strictly sequentially**, and the setting didn't survive a serialize/deserialize round trip (#1226, fixes #1223) by @cxzg007 — wired through builder → serializer → execution engine, plus a new opt-in `PipelineStep.parallel_safe` flag. A dependency layer now runs in parallel only when every step in it is marked `parallel_safe`, the layer has more than one step, the input is dict-typed, and no step is in delta mode; otherwise it falls back to sequential execution. Each parallel step's input is deep-copied for isolation, execution is bounded by `ThreadPoolExecutor(max_workers=min(configured parallelism, max_workers))`, a failure cancels pending futures in the layer, and layer results merge back in declaration order (a same-key conflict raises `ProcessingError`). 22 new tests in `tests/pipeline/test_pipeline_parallel.py`
- **`Config.get()` silently dropped boolean environment-variable overrides** (#1038, fixes #1035) by @Kyou12138 — the type dispatch checked `isinstance(default, int)` before `isinstance(default, bool)`, and since `bool` subclasses `int` in Python, the bool branch was unreachable: `CONFLICT_ZZTESTFLAG=true` with a `False` default returned `False`, and `=1` returned the int `1` rather than `True`. Bool is now checked first (with whitespace stripped before parsing truthy/falsy spellings), fixed across all ten affected config modules (`conflicts`, `deduplication`, `split`, `embeddings`, `export`, `ingest`, `kg`, `parse`, `ontology`, `normalize`). 12 new tests plus 6 existing conflicts tests and 131 related module tests pass
- **Scanned (image-only) PDFs parsed with no error and no warning, returning empty text with a "completed" status** (#1021, closes #1020) by @shanyu910`PDFParser._parse_page` swallowed a missing text layer via `page.extract_text() or ""`, so the failure only surfaced far downstream as zero extracted entities. A warning now fires when every parsed page yields no text with `extract_text` enabled, pointing at `parse_pdf(..., method="docling", enable_ocr=True)`. Also fixes a separate `import semantica.parse` failure on a fresh interpreter (`email_parser.py` used `email.message.Message` without importing `email.message`) that was blocking the parse test suite from even collecting. 25 tests pass in `tests/parse/`
- **`GET /api/decisions` returned HTTP 422 for any graph containing real decisions**, breaking the Explorer Decisions workspace entirely (#937) by @logan-jl-cc — `record_decision()` stores the timestamp as a POSIX float, but `DecisionResponse.timestamp` is typed `Optional[str]` and Pydantic's strict mode rejected the coercion. Fixed by coercing to `str` (preserving `None`) at the response-adapter boundary
- **Decision persistence/query bugs, CJK text handling, and three missing MCP graph tools** (#967) by @toratto`mcp_server`'s `_get_graph` called a non-existent `graph.load` instead of `load_from_file`, so `SEMANTICA_KG_PATH` was silently ignored and the server always started with an empty graph; `query_decisions` read `category` from the wrong field, always returning nothing for a category filter; `find_precedents`/`query_decisions(query=)`'s similarity threshold was too high for short CJK queries, which also failed outright because `_calculate_decision_content_similarity`'s whitespace-Jaccard fallback is always zero for languages with no whitespace tokenization (now falls back further to a character-bigram overlap coefficient); `load_from_file` didn't rebuild the in-memory decision/entity/temporal indexes after loading, breaking `find_precedents_by_scenario` and decision counts post-reload; `extract_entities`/`extract_relations` returned the spaCy type label as `text` and dropped the actual entity text, and had no way to select a non-English NER model. Also adds three new MCP tools (`query_graph`, `update_node`, `delete_node`, the latter two persisting back to `SEMANTICA_KG_PATH`)
- **`sqlalchemy.text` was used but never imported in two `DBIngestor`/`DataExporter` methods**, raising `NameError` on every call before any query reached the database (#1017, closes #1015) by @pravit-amp — `connect()`/`test_connection()` imported `text` function-locally, so the binding never reached `export_table_data()` or `execute_query()`, which called it anyway; both raised immediately, re-wrapped by an `except Exception` into a `ProcessingError` that read like a database fault rather than a missing import. `docs/guides/ontology.md` documents `DBIngestor().execute_query()` as a supported entry point, so documented usage walked straight into it. 5 new tests against a temporary SQLite database, also repairing a previously-failing `tests/ingest/test_notebook_02.py` case
- **Ontology generation resolved relationship endpoint types incorrectly, producing wrong object-property domains/ranges** (#1170, closes #1168) by @T1mn — endpoint types are now resolved from the canonical `source_id`/`target_id` fields and supported aliases instead of defaulting to the first entity when a field was missing, preventing e.g. a `Person -> Organization` relationship from generating a `Person -> Person` property. 80 tests pass, 1 skipped
- **Ontology property generation dropped data properties when a raw entity type was normalized into a class name** (#1171, closes #1169) by @T1mn — e.g. `software engineer``SoftwareEngineer` lost its `email` property; attributes are now grouped by matching raw, normalized, and recorded class names, so the normalized class stays each property's domain. 79 tests pass, 1 skipped
- **`flatten_dict()` silently dropped data when a top-level key already containing the separator collided with a key produced by flattening a nested dict** (#1012, fixes #1010) by @yzxcj797`{"a.b": 1, "a": {"b": 2}}` flattened to `{"a.b": 2}` with no error, the `1` simply gone; collisions are now detected (unique-key count vs. item count) and raise `ValueError` naming the colliding key before data is lost. 6 new tests
- **Creating relationships after `GraphStore.add_edges`/`build_from_entities_and_relationships` silently produced zero edges against ID-minting backends** (#1173, fixes #1136) by @yzxcj797 — an id-space mismatch across three layers: `add_edges` reads application-level string ids and passes them to `create_relationship`, which is a pure passthrough into `Neo4jStore.create_relationship`'s `MATCH ... WHERE id(a) = $start_id` — a Neo4j-internal integer id. Every node was created and every relationship silently failed with one easily-missed warning per edge. `GraphStore` now keeps an application-id→internal-id map, populated by `add_nodes`/`create_node` from the backend's own creation results and consulted by `create_relationship`; unknown ids and identity-mapped backends are unaffected. `tests/graph_store/`: 100 passed
- **RDF export left `semantica:text`/`rdfs:label` empty for entities that only carry a `name` field**, across all four RDF formats (#1113, fixes #1097) by @cxzg007`RDFSerializer.convert_kg_to_rdf()` already implemented the `name``label`/`text` normalization, but `export_to_rdf()` never called it. Now called once at the export boundary (idempotent, non-destructive, falls back to a label derived from the id suffix). 7 new tests, `tests/export/test_rdf_exporter.py`: 17 passed
- **Docker Explorer image failed to build on Python 3.14**`gensim` has no prebuilt wheel for it and the slim base has no `gcc` to build from source (#1172, closes #1025) by @DwitiThaker — runtime pinned to `python:3.13-slim`, where `gensim` installs from a prebuilt wheel
- **Unit normalization rejected common aliases before conversion**`kg`, `g`, and other abbreviated/plural unit spellings failed category validation and the conversion-factor lookup ahead of it (#939) by @Mr-Neutr0n — aliases now normalize first; canonical aliases added for feet, yards, miles, and gallons. 7 tests pass
- **An oversized, caller-controlled mapping key could blow up a `ValidationError` message to megabyte scale**, and equally inflate application logs on repeated malformed input (#1088, fixes #1001) by @ALDRIN121 — follow-up to the graph-payload validation added in #958. The displayed key is now truncated at 64 characters with an ellipsis; the underlying input and validation decisions are unchanged. 4 new tests
- **`SeedDataManager.load_from_api()` mislabeled genuine connection failures as a missing `requests` dependency** (#972, closes #949) by @pravit-amp — `requests.exceptions.RequestException` (connection errors, timeouts, `raise_for_status()` failures) subclasses `OSError`, so an `except (ImportError, OSError)` block written to guard a lazy import that no longer existed (`requests` is a core dependency) caught real failures too and told users to reinstall an already-installed library while dropping the original exception chain. The block is removed; genuine failures now surface through the existing `Failed to load from API: {e}` path with `from e` intact. 5 new regression tests
- **`SHACLGenerator` produced shapes that matched nothing, and pySHACL reported `conforms: True` on data that plainly violated them** (#1124, closes #1104, closes #1105) by @fabio-rovai — `base_uri` was used both as where shape resources live and to expand every `sh:targetClass`/`sh:path`, so with the default shapes namespace, generated shapes targeted classes no data graph in the package actually uses; a shape with zero matching focus nodes is vacuously satisfied, so validation silently passed regardless of real violations. The target namespace now resolves independently (explicit argument → ontology's declared namespace → an existing absolute class/property IRI → ontology `uri` → the vocabulary namespace), never the shapes namespace. Separately, `_attach_property_shapes` attached a domain-less property's constraint to *every* shape ("no domain declared, attach to all"), asserting a constraint the ontology never stated; a domain-less property is now left unattached by default, with `attach_domainless_properties=True` to restore the old behavior. 17 new tests validate real data through pySHACL rather than reading shape text; `tests/ontology`+`tests/export`: 239 passed
- **OWL export dropped every generated property and collapsed distinct classes onto one node** (#1123, closes #1103) by @fabio-rovai — `OWLExporter` reads `object_properties`/`data_properties`, but `OntologyGenerator` emits one combined `properties` list, so every property was silently discarded; separately, a class built without a namespace manager gets `"uri": None`, which a `"uri" not in cls"` guard never catches (the key is present), so the exporter wrote a relative `<>` IRI for it — resolved by rdflib against the current working directory, meaning two classes could collapse onto one subject and that subject's identity changed with the export's working directory. Both dict shapes are now merged and classified correctly, and a class/property IRI resolves through `uri``iri``id`→a name joined onto the ontology base, skipping (with a warning) a term with none of those instead of minting `<>`. 10 new regression tests parse the real output with rdflib and Oxigraph; `tests/export`+`tests/ontology`: 231 passed
- **Confidence scores serialized as four different, mutually-disagreeing RDF terms depending on export format, and one non-numeric confidence value could break an entire Turtle export** (#1125, closes #1100, closes #1102) by @fabio-rovai — Turtle wrote a bare `xsd:decimal`, N-Triples an explicit `xsd:float`, RDF/XML an untyped plain literal, and JSON-LD's native number expanded to `xsd:double`; loading a Turtle and an N-Triples export of the same graph into one store gave the same entity two different confidence values. Separately, an unparseable confidence (e.g. the string `"high"`) was interpolated into Turtle with no validation, producing a syntax error that dropped every entity from the export. All four paths now write one canonical `xsd:decimal` lexical form (matching the pre-existing Turtle behavior and the only exact representation of the four); an unusable value is omitted with a warning instead of corrupting the document. The vocabulary's `sem:confidence` now declares `xsd:decimal` (previously left undeclared to avoid contradicting the disagreeing exporters). 20 new tests compare parsed graphs across all four formats; `tests/export`+`tests/ontology`: 240 passed
- **An OWL-Time validity interval was reified onto a relationship IRI the graph never actually referenced**, making it unreachable from the edge it described (#1126, closes #1106) by @fabio-rovai — a relationship serializes as a single triple with no node of its own, so `include_temporal=True` minted a well-formed `time:Interval` with zero inbound arcs to its subject. Turtle now also emits the `sem:Relationship`/`sem:source`/`sem:target`/`sem:type` reification the JSON-LD path already produced, but only when there's temporal data to attach — default and `include_temporal=False` output are byte-for-byte unchanged. 7 new tests include a SPARQL walk from the edge to its interval, the path the dangling node made impossible; `tests/export`+`tests/ontology`: 228 passed
- **JSON-LD exports were unreadable by Semantica's own default parser** (#1145, fixes #1144) by @fabio-rovai — every export was written as a named graph (a top-level `@id` beside `@graph`), which a plain `rdflib.Graph.parse()` silently discards in favor of the (empty) default graph; a two-entity graph parsed as 2 triples instead of 20. Compounded by `export_knowledge_graph` converting its payload to JSON-LD and then handing the *already-converted* document to `export()`, which converted it again, producing two `@context` blocks and two document nodes. Metadata now attaches beside `@graph` rather than naming it, and a payload that already declares `@context` is merged rather than re-wrapped. 9 new tests parse with both `Graph()` and `Dataset()` and assert identical counts; full-suite failure set unchanged before/after (539/539)
- **`GraphBuilder` didn't propagate entity-resolution's merged ids into the `source_id`/`target_id` relationship aliases**, only `source`/`target` (#1115, closes #1110) by @T1mn — a relationship's alias fields could still point at a pre-merge id after resolution. Both alias pairs are now kept in sync. 9 tests pass
- **`GraphValidator` indexed entities only by `id`, rejecting graphs that use the `entity_id` alias as invalid even when their relationships were fine** (#1116, closes #1111) by @T1mn — validation and endpoint checks now go through the shared `get_entity_id()` helper, accepting both fields consistently. 5 tests pass
- **Broken star history chart in README** (#1057) by @OctoBored — the embedded chart used the GitHub stargazer API, now access-restricted; switched to a token-free alternative data source
### Security
- **Agno's `AgnoKnowledgeGraph.load_urls()` made outbound requests with no SSRF protection beyond a scheme check** (#1212) by @Sameer6305 — caller-supplied URLs went straight to `urllib.request.urlopen()`, unguarded against loopback/private addresses, cloud metadata endpoints (`169.254.169.254`), IPv6-internal addresses, hostnames resolving to private space, or redirects into any of the above. Found during a project-wide SSRF audit following #936/#959. Now routed through the shared `request_with_ssrf_guard()`; an unsafe URL is skipped rather than aborting the rest of the ingestion batch. `OpenClawKGTool` (operator-configured, intentionally allowed to target `localhost` for local deployments) gains scheme/malformed-URL validation as defense in depth, without restricting its legitimate private-network use case. 29 new Agno tests, 26 new OpenClaw tests, all passing alongside the 15 pre-existing Agno integration tests
### Dependencies
- Routine version bumps with no application-facing behavior change: `anthropic` 0.121.0→0.122.0 (#1045), `botocore` 1.43.69→1.43.73 (#1047), `agno` 2.8.7→2.9.0 (#1050), `google-genai` 2.17.0→2.18.1→2.19.0 (#1163, #1205), `lxml` 6.1.1→6.1.2 (#1197), `charset-normalizer` 3.5.0→3.5.1 (#1201), `pypickle` 2.0.1→2.0.2 (#1203)
## [0.6.6] - 2026-08-20
### Added
+10 -10
View File
@@ -142,7 +142,7 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
```bash
semantica doctor
# Python 3.11.9 pass
# semantica 0.6.6 pass
# semantica 0.6.7 pass
# faiss vector store pass
# Config file pass ~/.semantica/config.yaml
```
@@ -1463,18 +1463,18 @@ For contributor / dev-server setup: **[explorer/README.md: Local Setup Guide](ex
---
## What's New in v0.6.6
## What's New in v0.6.7
**Security release — upgrading is strongly recommended.** Fixes for a privately disclosed batch of vulnerabilities spanning backup/restore, database export, outbound requests, and triplet-store backends, plus SSRF hardening across ingestion:
**Feature release**, plus one SSRF hardening fix and a large batch of correctness fixes across the RDF/ontology export pipeline:
- **Tarball restore path traversal**: `semantica backup restore` now validates every archive member for path containment and rejects symlink/hardlink escapes before extraction
- **Latent SQL injection in `DataExporter.export_table_data()`**: table/schema names are now identifier-allowlisted and `where`/`order_by` fragments are blocklist-checked
- **DNS-rebinding TOCTOU in the shared SSRF guard**: the resolved IP that passes validation is now the one the connection is pinned to, closing the check-then-use race (also closes the `100.64.0.0/10` CGNAT gap)
- **Stored XSS in HTML report generation** and **unvalidated SPARQL object IRIs in AnzoStore** (SPARQL injection): both now escape/validate before interpolation
- **`Authorization`/`Proxy-Authorization` credential leakage across redirects**, plus **SSRF gaps in `FeedIngestor`/`FeedMonitor`, `RepoIngestor`, and the MCP/public-API ingest paths**: all now route through the shared, redirect-safe SSRF guard
- **HTTP response header injection and an unbounded-memory DoS** in the Explorer API, and a **`fastapi`/`python-multipart` ReDoS** (PYSEC-2024-38): floors raised, inputs sanitized, candidate pools capped
- **First-class LangChain integration** (`semantica[langchain]`): a `BaseRetriever` and `VectorStore` over `HybridSearch`, plus graph/decision-query tools
- **SAP OData ingestor** (`semantica[ingest-sap]`): OAuth2/Basic-auth, SSRF-guarded ingestion for Business Partners and Sales Orders, following the existing Snowflake/Databricks connector pattern
- **`ContextGraph` gains deterministic, human-editable Markdown round-trip persistence** alongside the existing JSON API, and the Explorer graph inspector gains a read-only Markdown content viewer
- **`reasoning` gains a structured Action layer**: rule-driven `Assert`/`Retract`/`Call`/`EmitEvent` actions with optional provenance, turning the reasoner into a production-rule system
- **`run_shacl_validation` is now a public, documented API**, and a dozen ontology/RDF export correctness fixes land: OWL property/class export, SHACL target-namespace resolution, one canonical confidence datatype across all four RDF formats, reachable OWL-Time reification, JSON-LD default-graph and content-derived document identity, and full metadata passthrough on every RDF serializer
- **Security**: Agno's `AgnoKnowledgeGraph.load_urls()` and OpenClaw's MCP tool now route outbound requests through the shared SSRF guard
Also ships: **first-class CrewAI integration** (`semantica[crewai]`, extraction/decision tools + a knowledge source), **`ContextGraph` retraction and purge** (GDPR-style erasure without a full `clear()`), a declared **Semantica RDF vocabulary with deterministic entity/relationship IRIs** (stable, diffable exports), and **timezone-aware timestamps** across `export/` and `provenance/`.
Also fixes: `PipelineBuilder.set_parallelism()` now actually parallelizes independent pipeline steps, `flatten_dict()` no longer silently drops data on a key collision, `Config.get()` honors boolean environment overrides, and the MCP server's `export_graph` tool works again on every format.
→ [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md)
+1 -1
View File
@@ -17,7 +17,7 @@ icon: "circle-question"
| API key required? | Optional: pattern extraction works with no keys |
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
| Latest version? | **v0.6.6** (August 2026) |
| Latest version? | **v0.6.7** (August 2026) |
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
+1 -1
View File
@@ -42,7 +42,7 @@ icon: "rocket"
Verify installation:
```python
import semantica
print(semantica.__version__) # 0.6.6
print(semantica.__version__) # 0.6.7
```
</Check>
</Step>
+39
View File
@@ -382,6 +382,45 @@ For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-p
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
## Source 7 — SAP OData
`SAPIngestor` ingests an Entity Set from a SAP OData service (S/4HANA Cloud, SuccessFactors, or an on-prem NetWeaver Gateway over its REST surface). It speaks OData v2 and v4, follows server-driven pagination automatically, and flattens each record into a document dict via `export_as_documents()` — the same structured "transform to text, then store" pattern as the other sources.
```python
from semantica.ingest import SAPIngestor
ing = SAPIngestor(
base_url="https://my-sap.example.com/sap/opu/odata/sap/API_BUSINESS_PARTNER",
client_id="...", client_secret="...",
token_url="https://my-sap.example.com/oauth/token", # OAuth2 client-credentials (BTP/S/4HANA Cloud)
# On-prem NetWeaver often uses Basic auth instead — swap the block above for:
# username="erp_user", password="...",
)
# 1. Discover an unfamiliar service: entity sets + field types from $metadata
sets = ing.discover_service() # [{"name": "A_BusinessPartnerSet", "fields": [...]}, ...]
# 2. Pull a page-walked Entity Set (v2/v4 next links handled for you)
partners = ing.ingest_entity_set(
entity_set="A_BusinessPartnerSet",
select="BusinessPartner,BusinessPartnerFullName", # $select
top=1000, # cap on total rows
)
# 3. Flatten to document dicts, then build text for the graph
docs = ing.export_as_documents(partners)
partner_texts = [
f"Business Partner {d['BusinessPartner']}: {d['BusinessPartnerFullName']}"
for d in docs
]
```
- Use `expand="to_Item"` (e.g. on a sales-order header set) to pull nested line items in one request — handy for modeling order → line-item → material relationships.
- Every outbound request, including the OAuth2 token exchange, is routed through the SSRF guard, so a user-supplied SAP URL can never reach private/loopback/link-local address space.
- Install with `pip install 'semantica[ingest-sap]'`.
> **Security Note:** Never hardcode credentials (`client_secret`, `password`) in code; pass them via environment variables (e.g., `SAP_CLIENT_SECRET`, `SAP_PASSWORD`) or a secrets manager.
## Combining All Five Sources
Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly.
+12 -1
View File
@@ -241,7 +241,18 @@ engine = ExecutionEngine(max_workers=2, retry_on_failure=True)
result = engine.execute_pipeline(pipeline)
```
`set_parallelism(n)` tells the engine how many steps it may run simultaneously. The topological sort guarantees that only steps whose dependencies are all completed are eligible for concurrent execution — you cannot accidentally run a step before its inputs are ready.
`set_parallelism(n)` tells the engine how many steps it may run simultaneously; `n` must be a positive integer. The topological sort guarantees that only steps whose dependencies are all completed are eligible for concurrent execution — you cannot accidentally run a step before its inputs are ready. The effective concurrency is capped at `min(n, max_workers)`, so the engine's `max_workers` setting remains a hard resource ceiling.
Concurrency is opt-in per step. A dependency layer only runs in parallel when every step in that layer is marked `parallel_safe`, the layer has more than one step, and the data flowing into the layer is a dict:
```python
builder.add_step("ner", "ner_extract", parallel_safe=True, confidence_threshold=0.75)
builder.add_step("triplets", "triplet_extract", parallel_safe=True, include_temporal=True)
```
If any step in a layer is not marked `parallel_safe`, or if a step runs in delta mode, the entire layer falls back to sequential execution — parallelism never silently bypasses a step that was not declared safe. `parallel_safe` is a control field: like `dependencies`, it is consumed by the builder and never reaches your handler's config.
Parallel-safe handlers must return a dict. Each step in a parallel layer receives an isolated deep copy of the layer's input, so steps cannot see each other's mutations. The per-step results are merged key by key in step declaration order: a key written by one step is added to the merged output, a key written by several steps with equal values is kept, and two steps writing different values for the same key fail the pipeline with a `ProcessingError` naming the conflicting key and both steps. Handlers that touch shared mutable resources — database connections, in-memory stores, global caches — should not be marked `parallel_safe`.
## Common Pitfalls
+1
View File
@@ -28,6 +28,7 @@ icon: "database"
| `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries |
| `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
| `DatabricksIngestor` | Databricks Unity Catalog metadata, Delta table queries, and lineage |
| `SAPIngestor` | SAP OData services (S/4HANA Cloud, SuccessFactors, NetWeaver Gateway): entity-set discovery and ingestion with v2/v4 pagination |
| `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
| `ArrowIngestor` | Apache Arrow IPC and Feather file processing |
| `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |
@@ -41,6 +41,7 @@ import {
} from "./plugins";
import { explorationEffectsShouldLoad, neighborhoodPanelShouldLoad, temporalOverlayShouldLoad } from "./pluginRegistryPredicates";
import { shouldFetchTemporalBounds, shouldFetchTemporalSnapshot } from "./temporalLifecyclePredicates";
import { createTemporalSnapshotGuards, type TemporalSnapshotResponse } from "./temporalSnapshotGuards";
import type { LinkPrediction, PathResponse } from "./GraphInspectorPanel";
import type { GraphSceneHandle, GraphSceneRuntime } from "./scene";
import type {
@@ -1479,6 +1480,23 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
summary?.edgeCount,
]);
// Guards the snapshot lifecycle: at most one in-flight request per scrubber
// position (identical-`at` polls are deduplicated, breaking the idle/play
// polling loop), applied snapshots are cached and re-applied on revisit, and
// a response applies only while the scrubber is still on its position
// (out-of-order responses cannot clobber the active-node count).
const temporalSnapshotGuardsRef = useRef<ReturnType<typeof createTemporalSnapshotGuards> | null>(null);
if (temporalSnapshotGuardsRef.current === null) {
temporalSnapshotGuardsRef.current = createTemporalSnapshotGuards();
}
const temporalSnapshotGuards = temporalSnapshotGuardsRef.current;
// A new graph summary means the graph data was replaced (reload/retry);
// snapshots cached against the previous graph are stale, so reset all state.
useEffect(() => {
temporalSnapshotGuards.reset();
}, [summary]);
useEffect(() => {
if (!canFetchTemporalSnapshot) {
return;
@@ -1488,37 +1506,67 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
return;
}
const atMs = debouncedTime.getTime();
const { seq, cached } = temporalSnapshotGuards.begin(atMs);
if (seq === null) {
// An identical request is already in flight: one request per position.
return;
}
let cancelled = false;
const applyData = (data: TemporalSnapshotResponse) => {
const nextActiveIds = new Set(data.active_node_ids);
requestAnimationFrame(() => {
if (cancelled) return;
if (!temporalSnapshotGuards.shouldApply(atMs, seq)) {
// The scrubber moved on (or this request was superseded): release the
// position so a return to it refetches instead of stalling.
temporalSnapshotGuards.finish(atMs, seq);
return;
}
const previous = prevActiveIdsRef.current;
previous.forEach((id) => {
if (!nextActiveIds.has(id) && graph.hasNode(id)) {
graph.setNodeAttribute(id, "hidden", true);
}
});
nextActiveIds.forEach((id) => {
if (graph.hasNode(id)) {
graph.setNodeAttribute(id, "hidden", false);
}
});
prevActiveIdsRef.current = nextActiveIds;
setActiveNodeCount(data.active_node_count);
setGraphVersion((current) => current + 1);
sceneRef.current?.getRuntime()?.requestRender();
temporalSnapshotGuards.apply(atMs, seq, data);
});
};
if (cached) {
// Returning to a position whose snapshot was already applied: re-apply
// the cached result without a network request.
applyData(cached);
return;
}
const applySnapshot = async () => {
try {
const at = debouncedTime.toISOString();
const response = await fetch(`/api/temporal/snapshot?at=${encodeURIComponent(at)}`);
if (!response.ok || cancelled) return;
const data: { active_node_ids: string[]; active_node_count: number } = await response.json();
if (!response.ok) {
// A failed request must be retryable if the scrubber returns.
if (!cancelled) temporalSnapshotGuards.finish(atMs, seq);
return;
}
if (cancelled) return;
const nextActiveIds = new Set(data.active_node_ids);
requestAnimationFrame(() => {
if (cancelled) return;
const previous = prevActiveIdsRef.current;
previous.forEach((id) => {
if (!nextActiveIds.has(id) && graph.hasNode(id)) {
graph.setNodeAttribute(id, "hidden", true);
}
});
nextActiveIds.forEach((id) => {
if (graph.hasNode(id)) {
graph.setNodeAttribute(id, "hidden", false);
}
});
prevActiveIdsRef.current = nextActiveIds;
setActiveNodeCount(data.active_node_count);
setGraphVersion((current) => current + 1);
sceneRef.current?.getRuntime()?.requestRender();
});
const data: TemporalSnapshotResponse = await response.json();
if (cancelled) return;
applyData(data);
} catch (fetchError) {
temporalSnapshotGuards.finish(atMs, seq);
if (!cancelled) {
console.error("[Temporal] Snapshot fetch failed", fetchError);
}
@@ -1528,6 +1576,8 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
applySnapshot();
return () => {
cancelled = true;
// A cancelled request must be retryable when its position is revisited.
temporalSnapshotGuards.finish(atMs, seq);
};
}, [
canFetchTemporalSnapshot,
@@ -0,0 +1,113 @@
/**
* Guards for the temporal snapshot fetch/apply lifecycle.
*
* The snapshot effect previously fetched /api/temporal/snapshot with no
* idempotency or ordering protection. Upstream churn (timeline recreation
* while bounds settle, play ticks resetting the playhead, drag events) could
* re-request the same `at` repeatedly, and responses could arrive after the
* scrubber had moved on.
*
* The guards enforce:
* - at most one in-flight request per scrubber position (identical `at`
* values are deduplicated while a request is pending, breaking the
* idle/play polling loop);
* - successful snapshots are cached per position and re-applied when the
* scrubber returns (play wrap-around, back-scrubbing) without a refetch;
* - a response is applied only while the scrubber is still on its position,
* so out-of-order responses cannot clobber a newer position's count;
* - failed, cancelled, or superseded requests release their position so it
* can be fetched again on the next visit;
* - `reset()` drops all state when the underlying graph data is replaced
* (reload/retry), because cached snapshots describe the previous graph.
*
* `createTemporalSnapshotGuards()` is stateful by design.
*/
export interface TemporalSnapshotResponse {
active_node_ids: string[];
active_node_count: number;
}
export interface TemporalSnapshotRequest {
/** null when the request was deduplicated because one is already in flight. */
seq: number | null;
/** The snapshot previously applied for this position, when revisiting it. */
cached: TemporalSnapshotResponse | null;
}
export interface TemporalSnapshotGuards {
/** Begin (or dedupe) a request for `atMs`; marks it as the current position. */
begin(atMs: number): TemporalSnapshotRequest;
/** True when the response for `atMs`/`seq` may be applied (scrubber still on `atMs`). */
shouldApply(atMs: number, seq: number): boolean;
/** Record a successful application and cache its snapshot for revisits. */
apply(atMs: number, seq: number, data: TemporalSnapshotResponse): void;
/** Release a position whose request failed, was cancelled, or was superseded. */
finish(atMs: number, seq: number): void;
/** Drop all state; call when the underlying graph data is replaced (reload). */
reset(): void;
}
interface SnapshotEntry {
seq: number;
/** null while the request is in flight (or before the first success). */
data: TemporalSnapshotResponse | null;
}
/** Upper bound on cached positions so long scrubbing sessions stay bounded. */
const MAX_CACHED_POSITIONS = 256;
export function createTemporalSnapshotGuards(): TemporalSnapshotGuards {
const entries = new Map<number, SnapshotEntry>();
let latestRequestSeq = 0;
let currentAtMs: number | null = null;
const evictOldest = () => {
while (entries.size > MAX_CACHED_POSITIONS) {
const oldestAtMs = entries.keys().next().value;
if (oldestAtMs === undefined) return;
entries.delete(oldestAtMs);
}
};
return {
begin(atMs) {
const existing = entries.get(atMs);
if (existing && existing.data === null) {
// Identical request already in flight: dedupe, but the scrubber is here now.
currentAtMs = atMs;
return { seq: null, cached: null };
}
latestRequestSeq += 1;
const seq = latestRequestSeq;
entries.set(atMs, { seq, data: existing?.data ?? null });
currentAtMs = atMs;
evictOldest();
return { seq, cached: existing?.data ?? null };
},
shouldApply(atMs, seq) {
return atMs === currentAtMs && entries.get(atMs)?.seq === seq;
},
apply(atMs, seq, data) {
const entry = entries.get(atMs);
if (entry && entry.seq === seq) {
entry.data = data;
}
},
finish(atMs, seq) {
const entry = entries.get(atMs);
if (entry && entry.seq === seq && entry.data === null) {
entries.delete(atMs);
}
},
reset() {
entries.clear();
latestRequestSeq = 0;
currentAtMs = null;
},
};
}
@@ -0,0 +1,150 @@
import test from "node:test";
import assert from "node:assert/strict";
import { createTemporalSnapshotGuards } from "../src/workspaces/GraphWorkspace/temporalSnapshotGuards.ts";
const POSITION_1 = new Date("2023-07-02T00:00:00Z").getTime();
const POSITION_2 = new Date("2024-01-02T00:00:00Z").getTime();
const POSITION_3 = new Date("2024-07-02T00:00:00Z").getTime();
const SNAPSHOT = { active_node_ids: ["n1", "n2"], active_node_count: 2 };
// ── begin: one request per scrubber position ─────────────────────────────────
test("begin: a new position returns a fresh request sequence", () => {
const guards = createTemporalSnapshotGuards();
assert.deepEqual(guards.begin(POSITION_1), { seq: 1, cached: null });
});
test("begin: an identical in-flight request is deduplicated (no duplicate fetch)", () => {
const guards = createTemporalSnapshotGuards();
guards.begin(POSITION_1);
assert.deepEqual(guards.begin(POSITION_1), { seq: null, cached: null });
});
test("begin: distinct positions request independently", () => {
const guards = createTemporalSnapshotGuards();
assert.equal(guards.begin(POSITION_1).seq, 1);
assert.equal(guards.begin(POSITION_2).seq, 2);
});
test("begin: revisiting an applied position returns its cached snapshot", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
guards.apply(POSITION_1, seq, SNAPSHOT);
const revisit = guards.begin(POSITION_1);
assert.equal(revisit.seq, 2);
assert.deepEqual(revisit.cached, SNAPSHOT);
});
test("begin: a failed position (finished) can be requested again", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
guards.finish(POSITION_1, seq);
const retry = guards.begin(POSITION_1);
assert.equal(retry.seq, 2);
assert.equal(retry.cached, null);
});
test("finish: does not clear a position whose snapshot was already applied", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
guards.apply(POSITION_1, seq, SNAPSHOT);
guards.finish(POSITION_1, seq);
assert.deepEqual(guards.begin(POSITION_1).cached, SNAPSHOT);
});
test("finish: a stale sequence cannot release a newer request's position", () => {
const guards = createTemporalSnapshotGuards();
const first = guards.begin(POSITION_1);
guards.finish(POSITION_1, first.seq);
guards.begin(POSITION_1); // seq 2, in flight again
guards.finish(POSITION_1, first.seq); // stale seq: must not release seq 2
assert.deepEqual(guards.begin(POSITION_1), { seq: null, cached: null });
});
// ── shouldApply: applied only while the scrubber is on that position ─────────
test("shouldApply: the current position's response is applied", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
assert.equal(guards.shouldApply(POSITION_1, seq), true);
});
test("shouldApply: a response for a position the scrubber left is discarded", () => {
const guards = createTemporalSnapshotGuards();
const { seq: seq1 } = guards.begin(POSITION_1);
guards.begin(POSITION_2);
assert.equal(guards.shouldApply(POSITION_1, seq1), false);
assert.equal(guards.shouldApply(POSITION_2, 2), true);
});
test("shouldApply: a late response for the position the scrubber returned to is applied", () => {
const guards = createTemporalSnapshotGuards();
const { seq: seq1 } = guards.begin(POSITION_1);
const { seq: seq2 } = guards.begin(POSITION_2);
guards.begin(POSITION_1); // back to 1: deduplicated, no new request
assert.equal(guards.shouldApply(POSITION_1, seq1), true);
assert.equal(guards.shouldApply(POSITION_2, seq2), false);
});
test("shouldApply: an unknown sequence is discarded", () => {
const guards = createTemporalSnapshotGuards();
guards.begin(POSITION_1);
assert.equal(guards.shouldApply(POSITION_1, 99), false);
});
test("shouldApply: after a reset no pre-reset response applies", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
guards.reset();
assert.equal(guards.shouldApply(POSITION_1, seq), false);
});
// ── apply: caching for revisits ─────────────────────────────────────────────
test("apply: stores the snapshot so a revisit re-applies it without a request", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
guards.apply(POSITION_1, seq, SNAPSHOT);
guards.begin(POSITION_2);
assert.deepEqual(guards.begin(POSITION_1).cached, SNAPSHOT);
});
test("apply: play wrap-around re-applies the wrapped-to position's snapshot", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
guards.apply(POSITION_1, seq, SNAPSHOT);
guards.begin(POSITION_2);
guards.begin(POSITION_3);
const wrap = guards.begin(POSITION_1);
assert.deepEqual(wrap.cached, SNAPSHOT);
assert.equal(guards.shouldApply(POSITION_1, wrap.seq), true);
});
// ── reset: graph reload ─────────────────────────────────────────────────────
test("reset: clears requested and cached state so positions refetch", () => {
const guards = createTemporalSnapshotGuards();
const { seq } = guards.begin(POSITION_1);
guards.apply(POSITION_1, seq, SNAPSHOT);
guards.reset();
const fresh = guards.begin(POSITION_1);
assert.equal(fresh.seq, 1);
assert.equal(fresh.cached, null);
});
// ── cache bound ─────────────────────────────────────────────────────────────
test("cache: oldest positions are evicted when the cache is full", () => {
const guards = createTemporalSnapshotGuards();
const count = 300;
for (let i = 0; i < count; i++) {
const { seq } = guards.begin(POSITION_1 + i * 1000);
guards.apply(POSITION_1 + i * 1000, seq, SNAPSHOT);
}
const oldest = guards.begin(POSITION_1);
assert.equal(oldest.cached, null); // evicted: must refetch on revisit
const newest = guards.begin(POSITION_1 + (count - 1) * 1000);
assert.deepEqual(newest.cached, SNAPSHOT); // still cached
});
+2 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "semantica"
version = "0.6.6"
version = "0.6.7"
description = "Graph-Native Infrastructure for Context and Accountable AI Systems: context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
readme = "README.md"
license = { text = "MIT" }
@@ -126,6 +126,7 @@ db-databricks = ["databricks-sdk>=0.60.0", "databricks-sql-connector>=4.0.0"]
db-arrow = ["pyarrow>=24.0.0"]
ingest-parquet = ["pyarrow>=24.0.0"]
ingest-arrow = ["pyarrow>=24.0.0"]
ingest-sap = ["requests>=2.28.0"]
db-all = [
"semantica[db-snowflake,db-databricks,db-arrow]"
+1 -1
View File
@@ -10,7 +10,7 @@ Main exports:
- Config: Configuration management
"""
__version__ = "0.6.6"
__version__ = "0.6.7"
__author__ = "Semantica Contributors"
__license__ = "MIT"
+30 -11
View File
@@ -306,10 +306,10 @@ class JSONExporter:
self.logger.debug(f"Exporting {len(entities)} entity(ies) to JSON")
# Build JSON data with JSON-LD context
# Build JSON data with JSON-LD context. No @vocab: it would expand
# every bare key in the caller's entity dicts into ns# (#1146).
json_data = {
"@context": {
"@vocab": "https://semantica.dev/vocab/",
"semantica": SEMANTICA_NS,
"entities": {"@id": "semantica:entities", "@container": "@list"},
},
@@ -339,7 +339,6 @@ class JSONExporter:
"""
json_data = {
"@context": {
"@vocab": "https://semantica.dev/vocab/",
"semantica": SEMANTICA_NS,
"relationships": {
"@id": "semantica:relationships",
@@ -434,11 +433,14 @@ class JSONExporter:
Returns:
Dictionary in JSON-LD format with @context, @graph/@value, and metadata
"""
# Initialize JSON-LD structure with context
# Initialize JSON-LD structure with context. No @vocab: for a generic
# payload it turned whatever bare keys the caller happened to use into
# ns# terms (#1146). Undeclared terms now simply expand to nothing,
# which is standard JSON-LD behaviour for a context that does not
# know them; the raw payload is still in the document.
jsonld = {
"@context": {
"@vocab": "https://semantica.dev/vocab/",
"semantica": "https://semantica.dev/ns#",
"semantica": SEMANTICA_NS,
}
}
@@ -598,13 +600,21 @@ class JSONExporter:
Returns:
Dictionary in JSON-LD format with @context, @id, @type, and graph data
"""
# Initialize JSON-LD structure with RDF context
# Initialize JSON-LD structure with RDF context. No @vocab: it applied
# to every bare term in caller data, so an extracted type like "ORG"
# became ns#ORG and a metadata key like "source" collided with the
# real sem:source object property (#1146). Only explicit semantica:
# terms resolve now, and the caller's metadata dict is typed @json so
# it survives as one rdf:JSON literal instead of expanding its keys.
jsonld = {
"@context": {
"@vocab": "https://semantica.dev/vocab/",
"semantica": "https://semantica.dev/ns#",
"semantica": SEMANTICA_NS,
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"semantica:metadata": {
"@id": "semantica:metadata",
"@type": "@json",
},
},
# Minted from the graph's own content rather than the wall clock
# (#1147): re-exporting an unchanged graph must produce the same
@@ -664,14 +674,23 @@ class JSONExporter:
entity_text = entity.get("text") or entity.get("label", "unknown")
entity_id = entity.get("id") or mint_entity_iri(entity_text)
# The caller's type label is data, not a class we define: minting it
# into @type expanded it through @vocab into ns#ORG and friends, terms
# that look official but do not exist (#1146). The node is always a
# semantica:Entity and the label travels as semantica:type, exactly
# how _relationship_to_jsonld has always carried the relationship type.
jsonld = {
"@id": entity_id,
"@type": entity.get("type") or "semantica:Entity",
"@type": "semantica:Entity",
"semantica:text": entity.get("text") or entity.get("label", ""),
"semantica:confidence": entity.get("confidence", 1.0),
}
entity_type = entity.get("type")
if entity_type:
jsonld["semantica:type"] = entity_type
# Add metadata if present
# Add metadata if present. The @json term definition on
# semantica:metadata keeps the whole dict one rdf:JSON literal.
if "metadata" in entity:
jsonld["semantica:metadata"] = entity["metadata"]
+16 -4
View File
@@ -1226,11 +1226,14 @@ class RDFSerializer:
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
graph_uri: Optional[str] = options.pop("graph_uri", None)
# Initialize JSON-LD structure with context
# Initialize JSON-LD structure with context. No @vocab: it applied to
# every bare term in caller data, so an extracted type like "ORG"
# became ns#ORG and a metadata key like "source" collided with the
# real sem:source object property (#1146). Only explicit semantica:
# terms resolve now.
jsonld = {
"@context": {
"@vocab": "https://semantica.dev/vocab/",
"semantica": "https://semantica.dev/ns#",
"semantica": SEMANTICA_NS,
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
},
@@ -1252,11 +1255,20 @@ class RDFSerializer:
# and was dropped in full by a JSON-LD parser, silently.
entity_id = entity.get("id") or mint_entity_iri(entity.get("text", ""))
# The caller's type label is data, not a class we define: minting
# it into @type expanded it through @vocab into ns#ORG and
# friends, terms that look official but do not exist (#1146).
# The node is always a semantica:Entity and the label travels as
# semantica:type, matching the relationship node below and
# JSONExporter._entity_to_jsonld.
node = {
"@id": entity_id,
"@type": entity.get("type", "semantica:Entity"),
"@type": "semantica:Entity",
"semantica:text": entity.get("text") or entity.get("label", ""),
}
entity_type = entity.get("type")
if entity_type:
node["semantica:type"] = entity_type
confidence = normalize_confidence(entity.get("confidence", 1.0))
if confidence is None:
self.logger.warning(
+8
View File
@@ -218,6 +218,10 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
"SnowflakeIngestor": (".snowflake_ingestor", "SnowflakeIngestor"),
"SnowflakeData": (".snowflake_ingestor", "SnowflakeData"),
"SnowflakeConnector": (".snowflake_ingestor", "SnowflakeConnector"),
# SAP OData ingestion
"SAPIngestor": (".sap_ingestor", "SAPIngestor"),
"SAPODataEntity": (".sap_ingestor", "SAPODataEntity"),
"SAPODataConnector": (".sap_ingestor", "SAPODataConnector"),
# Databricks ingestion
"DatabricksIngestor": (".databricks_ingestor", "DatabricksIngestor"),
"DatabricksData": (".databricks_ingestor", "DatabricksData"),
@@ -345,6 +349,10 @@ __all__ = [
"SnowflakeIngestor",
"SnowflakeData",
"SnowflakeConnector",
# SAP OData ingestion
"SAPIngestor",
"SAPODataEntity",
"SAPODataConnector",
# Databricks ingestion
"DatabricksIngestor",
"DatabricksData",
+51
View File
@@ -875,6 +875,57 @@ schema = connector.get_schema(engine)
print(f" {table_name}: {[col['name'] for col in columns]}")
```
## SAP OData Ingestion
`SAPIngestor` reads an Entity Set from a SAP OData service — S/4HANA Cloud,
SuccessFactors, or an on-prem NetWeaver Gateway over its REST surface. It
follows OData v2/v4 server-driven pagination and flattens each record into a
document dict via `export_as_documents()`.
Install with `pip install 'semantica[ingest-sap]'`.
### Connector Construction & Authentication
```python
from semantica.ingest import SAPIngestor
# OAuth2 client-credentials (BTP / S/4HANA Cloud)
ing = SAPIngestor(
base_url="https://my-sap.example.com/sap/opu/odata/sap/API_BUSINESS_PARTNER",
client_id="...", client_secret="...",
token_url="https://my-sap.example.com/oauth/token",
)
# On-prem NetWeaver often uses Basic auth instead — swap the block above for:
# ing = SAPIngestor(base_url="...", username="erp_user", password="...")
```
### Entity-Set Ingestion & Document Export
```python
# 1. Discover entity sets + field types from $metadata
sets = ing.discover_service()
# 2. Page-walk an Entity Set (v2/v4 next links handled automatically)
partners = ing.ingest_entity_set(
entity_set="A_BusinessPartnerSet",
select="BusinessPartner,BusinessPartnerFullName",
top=1000,
)
# 3. Flatten to document dicts that GraphBuilder can consume directly
docs = ing.export_as_documents(partners)
```
- Use `expand="to_Item"` on a sales-order header set to pull nested line items
in one request — handy for modeling order → line-item → material relations.
- Every outbound request, including the OAuth2 token exchange, is routed through
the SSRF guard, and pagination never follows a next link that points to a
different host than the service root.
> **Security Note:** Never hardcode credentials (`client_secret`, `password`);
> pass them via environment variables (`SAP_CLIENT_SECRET`, `SAP_PASSWORD`) or a
> secrets manager.
## MCP Server Ingestion
**IMPORTANT**: This implementation supports **ONLY Python-based MCP servers and FastMCP servers**. Users can bring their own Python or FastMCP MCP servers via URL connections. JavaScript, TypeScript, C#, Java, and other language implementations are **NOT supported**.
+617
View File
@@ -0,0 +1,617 @@
"""SAP OData ingestion module.
Pulls an Entity Set from a SAP OData service (S/4HANA and on-prem NetWeaver
REST surfaces) and flattens it into document dicts that the pipeline can feed
to ``GraphBuilder``.
Why this exists
---------------
Semantica ingests from many sources; SAP is the ERP backbone of finance and
regulated industries, and its master/transactional data (customers, vendors,
sales orders) is exactly the "context" a Context Graph wants. SAP exposes that
data over OData (v2 on many on-prem NetWeaver systems, v4 on BTP / S/4HANA
Cloud). This connector speaks the REST surface of OData only.
Design notes
------------
Three classes, matching the Snowflake/Databricks ingestors:
- ``SAPODataEntity``: a collection fetch from one Entity Set (``records``,
``count``, ``service``, ``metadata``), flattened to document dicts for
``GraphBuilder`` via ``export_as_documents``.
- ``SAPODataConnector``: auth (OAuth2 client-credentials or Basic) + the
shared, SSRF-guarded :mod:`requests` session. *Every* outbound request,
including the OAuth2 token exchange, goes through
``request_with_ssrf_guard`` so user-supplied endpoints can not reach
private/loopback/link-local address space.
- ``SAPIngestor``: the three methods the issue requested
``discover_service``, ``ingest_entity_set`` and ``export_as_documents``
(plus ``close`` for symmetry with the SQL connectors).
EDMX
----
``$metadata`` is plain CSDL XML in *both* OData v2 and v4, so we hand-roll a
minimal parser with :mod:`xml.etree` instead of pulling in ``pyodata``. That
keeps phase 1 ``requests``-only, exactly as scoped in the issue.
Pagination
----------
OData uses a server-driven "next link": OData v2 surfaces it as the atom
``__next`` element, OData v4 as the ``@odata.nextLink`` field on the JSON
payload. ``ingest_entity_set`` follows whichever it sees until the set is
exhausted.
"""
from __future__ import annotations
import base64
import os
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any, Dict, List, Optional
from urllib.parse import urljoin, urlparse
import requests
from requests.adapters import HTTPAdapter
try:
from urllib3.util.retry import Retry
except (ImportError, OSError): # pragma: no cover - old urllib3 layout
from requests.packages.urllib3.util.retry import Retry # type: ignore
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from .ssrf import parse_bool, request_with_ssrf_guard
__all__ = [
"SAPODataEntity",
"SAPODataConnector",
"SAPIngestor",
]
_logger = get_logger("sap_ingestor")
def _prop_is_nullable(prop: Any) -> bool:
"""CSDL structural properties default to nullable=True when omitted."""
val = prop.get("Nullable", prop.get("nullable"))
return True if val is None else val.strip().lower() == "true"
def _match(elem: Any, localname: str) -> bool:
"""True if *elem* has the given local name in any namespace."""
return elem.tag.rsplit("}", 1)[-1] == localname
@dataclass
class SAPODataEntity:
"""A collection fetch from a SAP OData Entity Set.
Holds the rows pulled from one Entity Set (all paging fan-in'd), with the
shape the issue specifies: ``records`` (the row data), ``count``,
``service`` (the resolved service root), optional ``metadata`` (entity-set
schema from ``$metadata``) and ``ingested_at``.
"""
records: List[Dict[str, Any]]
entity_set: str
count: int
service: str
metadata: Optional[Dict[str, Any]] = None
ingested_at: datetime = field(default_factory=datetime.now)
def to_documents(self) -> List[Dict[str, Any]]:
"""Flatten each record to a document dict ``GraphBuilder`` can consume.
GraphBuilder only treats a dict as an entity when it carries
``id``/``entity_id``/``name`` (or ``text``+``type``); SAP records have
none of those, so they would be silently dropped. We inject an
identifier resolved from each record's primary-key-like field, falling
back to ``entity_set:index``, and expose it under both ``id`` and
``name``.
"""
docs: List[Dict[str, Any]] = []
for index, record in enumerate(self.records):
doc = dict(record)
key_value = self._id_value(record)
doc.setdefault("id", key_value or f"{self.entity_set}:{index}")
doc.setdefault("name", key_value or self.entity_set)
doc.setdefault("source", self.service)
docs.append(doc)
return docs
@staticmethod
def _id_value(record: Dict[str, Any]) -> str:
for key, value in record.items():
if "id" in key.lower() and value not in (None, ""):
return str(value)
return ""
class SAPODataConnector:
"""Connection + authentication management for a SAP OData REST service.
Supports the two auth landscapes called out in the issue:
- **OAuth2 client-credentials** (BTP / S/4HANA Cloud). The token URL is
user supplied; both the token exchange *and* every subsequent data
request are validated through the SSRF guard.
- **Basic** (on-prem NetWeaver). Username/password passed through as an
``Authorization: Basic`` header, also through the guard.
Example usage::
>>> connector = SAPODataConnector(
... base_url="https://myhost/sap/opu/odata/sap/",
... token_url="https://myhost/oauth/token",
... client_id="cid", client_secret="secret",
... )
>>> session = connector.get_session()
"""
def __init__(
self,
base_url: Optional[str] = None,
*,
auth: Optional[str] = None,
token_url: Optional[str] = None,
client_id: Optional[str] = None,
client_secret: Optional[str] = None,
username: Optional[str] = None,
password: Optional[str] = None,
allow_private_ips: bool = False,
**config: Any,
) -> None:
"""Initialize the SAP OData connector.
Args:
base_url: Base OData service URL, e.g. ``https://host/sap/opu
/odata/sap/``. The issue's ``service`` value.
auth: Explicit auth flow, ``"oauth2"`` or ``"basic"``. When
omitted, the flow is inferred from which credentials are set.
token_url: OAuth2 token endpoint. Required only for OAuth2 flow.
client_id: OAuth2 client id (OAuth2 flow).
client_secret: OAuth2 client secret (OAuth2 flow).
username: Basic-auth username (on-prem flow).
password: Basic-auth password (on-prem flow).
allow_private_ips: Opt into private/loopback/link-local endpoints.
Defaults to False (SSRF-safe).
**config: Extra options, notably ``timeout``, ``max_retries``,
``backoff_factor``, ``headers``.
"""
self.logger = _logger
self.base_url = base_url or os.getenv("SAP_BASE_URL")
self.auth = (auth or os.getenv("SAP_AUTH") or "").lower()
self.token_url = token_url or os.getenv("SAP_TOKEN_URL")
self.client_id = client_id or os.getenv("SAP_CLIENT_ID")
self.client_secret = client_secret or os.getenv("SAP_CLIENT_SECRET")
self.username = username or os.getenv("SAP_USERNAME")
self.password = password or os.getenv("SAP_PASSWORD")
self.allow_private_ips = parse_bool(
config.pop("allow_private_ips", allow_private_ips), default=False
)
self.config = config
if not self.base_url:
raise ValidationError(
"SAP base_url is required. Provide via 'base_url' or "
"SAP_BASE_URL environment variable."
)
oauth_configured = bool(self.client_id or self.client_secret or self.token_url)
if self.auth in ("oauth2", "oauth"):
if not (self.client_id and self.client_secret and self.token_url):
raise ValidationError(
"SAP OAuth2 flow requires client_id, client_secret and "
"token_url all set."
)
elif self.auth == "basic":
if not (self.username and self.password):
raise ValidationError("SAP Basic flow requires username and password.")
elif oauth_configured:
if not (self.client_id and self.client_secret and self.token_url):
raise ValidationError(
"SAP OAuth2 flow requires client_id, client_secret and "
"token_url all set."
)
elif not self.username:
raise ValidationError(
"SAP authentication requires either (username/password) or "
"(client_id/client_secret + token_url)."
)
self.session = requests.Session()
retry_strategy = Retry(
total=self.config.get("max_retries", 3),
backoff_factor=self.config.get("backoff_factor", 1),
status_forcelist=[429, 500, 502, 503, 504],
)
adapter = HTTPAdapter(max_retries=retry_strategy)
self.session.mount("http://", adapter)
self.session.mount("https://", adapter)
default_headers = self.config.get("headers", {})
if default_headers:
self.session.headers.update(default_headers)
self._token: Optional[str] = None
self.logger.debug(
"SAP OData connector initialized (base_url=%s, allow_private_ips=%s)",
self.base_url,
self.allow_private_ips,
)
def get_session(self) -> requests.Session:
"""Return an authenticated session for data requests.
For the Basic flow the credentials are attached eagerly; for the
OAuth2 flow a token is fetched (and cached) on first use. The token
is never refreshed, so a job that runs past the token TTL (typically
3600s on SAP) will fail with 401 -- re-create the connector instead.
"""
if self.username:
self.session.headers["Authorization"] = "Basic " + self._basic_header()
return self.session
if self._token is None:
self._token = self._fetch_token()
self.session.headers["Authorization"] = "Bearer " + self._token
return self.session
def _basic_header(self) -> str:
pair = f"{self.username}:{self.password or ''}".encode("utf-8")
return base64.b64encode(pair).decode("ascii")
def _fetch_token(self) -> str:
"""Perform the OAuth2 client-credentials token exchange (SSRF-guarded)."""
if not self.token_url or not self.client_id:
raise ProcessingError("OAuth2 flow requires token_url and client_id.")
body = {
"grant_type": "client_credentials",
"client_id": self.client_id,
"client_secret": self.client_secret or "",
}
resp = request_with_ssrf_guard(
"POST",
self.token_url,
session=self.session,
allow_private_ips=self.allow_private_ips,
data=body,
timeout=self.config.get("timeout", 30),
)
try:
resp.raise_for_status()
except requests.exceptions.RequestException as exc:
raise ProcessingError(f"SAP OAuth2 token exchange failed: {exc}") from exc
try:
payload = resp.json()
except ValueError as exc:
raise ProcessingError(
"SAP OAuth2 token endpoint did not return JSON."
) from exc
token = payload.get("access_token")
if not token:
raise ProcessingError("SAP OAuth2 token response missing 'access_token'.")
return str(token)
def close(self) -> None:
"""Close the underlying :mod:`requests` session."""
self.session.close()
class SAPIngestor:
"""Ingest an Entity Set from a SAP OData service.
Example usage::
>>> from semantica.ingest import SAPIngestor
>>> ing = SAPIngestor(
... base_url="https://host/sap/opu/odata/sap/",
... username="u", password="p", # or client_id/client_secret/token_url
... )
>>> sets = ing.discover_service()
... # -> [{"name": "SalesOrderSet", "fields": [...]}, ...]
>>> docs = ing.export_as_documents(
... ing.ingest_entity_set(entity_set="SalesOrderSet", expand="to_Item"))
"""
def __init__(
self,
base_url: Optional[str] = None,
connector: Optional[SAPODataConnector] = None,
**config: Any,
) -> None:
"""Initialize the SAP ingestor.
Args:
base_url: Base OData service URL. Mutually exclusive with
``connector``; ignored if a connector is given.
connector: An existing :class:`SAPODataConnector`. When provided,
its session and base URL are reused.
**config: Passed to :class:`SAPODataConnector` when one is created.
"""
self.logger = _logger
self.connector = connector or SAPODataConnector(base_url=base_url, **config)
# urljoin() replaces the last path segment unless the base ends in '/',
# so normalize once here: .../API_BUSINESS_PARTNER -> .../$metadata would
# silently drop the service segment.
self._base_url = self.connector.base_url
if not self._base_url.endswith("/"):
self._base_url += "/"
def discover_service(self, service: Optional[str] = None) -> List[Dict[str, Any]]:
"""Fetch and parse ``$metadata`` into the service's entity sets.
Args:
service: Service root URL (absolute) or path suffix resolved
against the base URL. Defaults to the base URL. ``$metadata``
is appended automatically same meaning as in
:meth:`ingest_entity_set`.
Returns:
List of dicts, one per EntitySet, each with ``name`` and ``fields``
(a list of ``{name, type, nullable}`` parsed from the CSDL).
"""
metadata_url = self._metadata_url(service)
session = self.connector.get_session()
resp = request_with_ssrf_guard(
"GET",
metadata_url,
session=session,
allow_private_ips=self.connector.allow_private_ips,
headers={"Accept": "application/xml"},
timeout=self.connector.config.get("timeout", 30),
)
try:
resp.raise_for_status()
except requests.exceptions.RequestException as exc:
self.logger.error("Failed to fetch SAP metadata %s: %s", metadata_url, exc)
raise ProcessingError(f"Failed to fetch SAP $metadata: {exc}") from exc
return self._parse_metadata(resp.text)
def _metadata_url(self, service: Optional[str]) -> str:
"""Build the ``$metadata`` URL for a service root.
``service`` has the same meaning as in :meth:`ingest_entity_set`
a service root (absolute URL or path suffix resolved against the
base URL). ``$metadata`` is appended here, so callers pass the root
the same way for both discovery and ingestion. A value already
ending in ``$metadata`` is used as-is.
"""
if not service:
return urljoin(self._base_url, "$metadata")
if "://" not in service:
service = urljoin(self._base_url, service)
if service.endswith("$metadata"):
return service
if not service.endswith("/"):
service += "/"
return urljoin(service, "$metadata")
def _parse_metadata(self, metadata_xml: str) -> List[Dict[str, Any]]:
"""Minimal CSDL/EDMX parser -> entity set name + property fields.
Element local names (``Schema``/``EntitySet``/``EntityType``/
``Property``) are stable across OData v2 (Microsoft ns) and v4 (OASIS
ns), so we match them by local name instead of hard-coding one
namespace. Entity-Type references are resolved per-schema, so
same-named types in different schemas cannot bleed fields into each
other.
"""
try:
root = ET.fromstring(metadata_xml)
except ET.ParseError as exc:
raise ProcessingError(f"SAP $metadata is not valid XML: {exc}") from exc
# Index fully-qualified type name -> property fields, per schema.
schema_types: Dict[str, List[Dict[str, Any]]] = {}
for schema in (e for e in root.iter() if _match(e, "Schema")):
ns = (schema.get("Namespace") or schema.get("namespace") or "").rstrip(".")
for entity_type in (e for e in schema.iter() if _match(e, "EntityType")):
tname = entity_type.get("Name") or entity_type.get("name")
if not tname:
continue
fq = f"{ns}.{tname}" if ns else tname
schema_types[fq] = [
{
"name": prop.get("Name") or prop.get("name"),
"type": prop.get("Type") or prop.get("type"),
"nullable": _prop_is_nullable(prop),
}
for prop in (e for e in entity_type.iter() if _match(e, "Property"))
]
entity_sets: List[Dict[str, Any]] = []
for schema in (e for e in root.iter() if _match(e, "Schema")):
ns = (schema.get("Namespace") or schema.get("namespace") or "").rstrip(".")
for entity_set in (e for e in schema.iter() if _match(e, "EntitySet")):
name = entity_set.get("Name") or entity_set.get("name")
ref = entity_set.get("EntityType") or entity_set.get("entityType") or ""
qualified = ref if "." in ref else (f"{ns}.{ref}" if ns else ref)
fields = schema_types.get(qualified) or schema_types.get(ref) or []
entity_sets.append({"name": name, "fields": fields})
return entity_sets
def ingest_entity_set(
self,
service: Optional[str] = None,
entity_set: Optional[str] = None,
*,
select: Optional[str] = None,
filter: Optional[str] = None,
expand: Optional[str] = None,
top: Optional[int] = None,
skip: Optional[int] = None,
batch_size: int = 100,
) -> SAPODataEntity:
"""Fetch pages of *entity_set* from the OData service.
Args:
service: Service root URL (absolute) or path suffix resolved
against the base URL. Defaults to the base URL. Same meaning
as in :meth:`discover_service`.
entity_set: Entity set name, e.g. ``"SalesOrderSet"``.
select: Optional ``$select`` comma string.
filter: Optional ``$filter`` expression.
expand: Optional ``$expand`` expression (e.g. ``"to_Item"`` for
use case 2's sales-order headers -> line items).
top: Maximum number of rows to return.
skip: Number of leading rows to skip.
batch_size: ``$top`` pagination size per request.
Returns:
A single :class:`SAPODataEntity` holding every fetched record
(server-driven pagination is followed to completion).
"""
if service is None:
base = self._base_url
elif "://" in service:
base = service
else:
base = urljoin(self._base_url, service)
if not base.endswith("/"):
base += "/"
if not entity_set:
raise ValidationError("SAP 'entity_set' is required.")
session = self.connector.get_session()
records: List[Dict[str, Any]] = []
next_link: Optional[str] = urljoin(base, entity_set)
params = self._query_params(select, filter, expand, top, skip, batch_size)
if top is not None and top < 0:
raise ValidationError("SAP 'top' must be >= 0 (got %r)" % top)
if top == 0:
return SAPODataEntity(
records=[], entity_set=entity_set, count=0, service=base
)
original_host = (urlparse(base).hostname or "").lower()
while next_link:
# Server-provided next links may point anywhere; never send the
# session credentials (Basic/Bearer) to a different origin than
# the service root. Legit SAP pagination stays on the same host.
next_host = (urlparse(next_link).hostname or "").lower()
if next_host != original_host:
raise ProcessingError(
f"SAP next link '{next_link}' points to a different host "
f"than service root '{base}'"
)
resp = request_with_ssrf_guard(
"GET",
next_link,
session=session,
allow_private_ips=self.connector.allow_private_ips,
headers={"Accept": "application/json"},
params=params,
timeout=self.connector.config.get("timeout", 30),
)
try:
resp.raise_for_status()
except requests.exceptions.RequestException as exc:
self.logger.error(
"Failed to fetch SAP entity set %s: %s", entity_set, exc
)
raise ProcessingError(
f"Failed to fetch SAP entity set {entity_set}: {exc}"
) from exc
payload = self._parse_page(resp)
rows, next_link = payload["rows"], payload["next_link"]
for raw_row in rows:
records.append(self._flatten_row(raw_row))
self.logger.debug(
"Fetched %d rows from %s (next=%s)",
len(rows),
entity_set,
bool(next_link),
)
if top is not None and len(records) >= top:
break
params = None # query params already baked into the server next link
# Refresh next_link against base in case it's a relative pointer.
if next_link and not next_link.startswith("http"):
next_link = urljoin(resp.url, next_link)
return SAPODataEntity(
records=records,
entity_set=entity_set,
count=len(records),
service=base,
)
def _query_params(
self,
select: Optional[str],
filter: Optional[str],
expand: Optional[str],
top_value: Optional[int],
skip: Optional[int],
batch_size: int,
) -> Dict[str, str]:
params: Dict[str, str] = {}
if batch_size > 0:
if top_value is not None:
params["$top"] = str(min(batch_size, top_value))
else:
params["$top"] = str(batch_size)
if select:
params["$select"] = select
if filter:
params["$filter"] = filter
if expand:
params["$expand"] = expand
if skip is not None:
params["$skip"] = str(skip)
return params
def _parse_page(self, resp: requests.Response) -> Dict[str, Any]:
try:
payload = resp.json()
except ValueError as exc:
raise ProcessingError(f"SAP OData response is not JSON: {exc}") from exc
rows: Any
next_link: Optional[str] = None
if isinstance(payload, list):
rows = payload
elif isinstance(payload, dict):
d = payload.get("d")
if isinstance(d, dict):
# OData v2 atom: {"d": {"results": [...], "__next": ...}}
rows = d.get("results")
nxt = d.get("__next") or payload.get("@odata.nextLink")
else:
# OData v4 JSON: {"value": [...], "@odata.nextLink": ...}
rows = payload.get("value", d)
nxt = payload.get("@odata.nextLink")
if isinstance(nxt, dict):
nxt = nxt.get("__deferred", {}).get("uri")
next_link = nxt
else:
rows = None
if not isinstance(rows, list):
raise ProcessingError(
"SAP OData payload has no list of rows (got %s)" % type(rows).__name__
)
return {"rows": rows, "next_link": next_link}
def _flatten_row(self, row: Any) -> Dict[str, Any]:
if isinstance(row, dict):
# v2 wraps items in "__metadata"; keep it but expose plain keys.
return {k: v for k, v in row.items() if k != "__metadata"}
return {"value": row}
def export_as_documents(self, data: SAPODataEntity) -> List[Dict[str, Any]]:
"""Convert an ingested entity set to flat document dicts.
Normalizes every records held by ``data`` into a list of dicts with an
injected ``id``/``name``/``source``, ready to hand to ``GraphBuilder``.
"""
return data.to_documents()
def close(self) -> None:
"""Close the underlying connector's session."""
self.connector.close()
+63
View File
@@ -117,6 +117,8 @@ class PropertyGenerator:
data_properties = self._infer_data_properties(entities, classes, **options)
properties.extend(data_properties)
properties = self._coalesce_normalized_properties(properties)
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
@@ -193,6 +195,67 @@ class PropertyGenerator:
return properties
def _coalesce_normalized_properties(
self, properties: List[Dict[str, Any]]
) -> List[Dict[str, Any]]:
"""Merge same-kind properties that normalize to the same name."""
property_kinds = defaultdict(set)
for prop in properties:
property_kinds[prop["name"]].add(prop.get("type"))
collisions = {
name: sorted(kind for kind in kinds if kind is not None)
for name, kinds in property_kinds.items()
if len({kind for kind in kinds if kind is not None}) > 1
}
if collisions:
raise ValidationError(
"Normalized property names cannot be shared by object and "
"data properties.",
validation_context={"property_kind_collisions": collisions},
)
merged: Dict[tuple, Dict[str, Any]] = {}
result = []
for prop in properties:
key = (prop.get("type"), prop["name"])
existing = merged.get(key)
if existing is None:
merged[key] = prop
result.append(prop)
continue
existing["domain"] = self._merge_property_values(
existing.get("domain", []), prop.get("domain", [])
)
if prop.get("type") == "object":
existing["range"] = self._merge_property_values(
existing.get("range", []), prop.get("range", [])
)
existing_metadata = existing.setdefault("metadata", {})
existing_metadata["occurrence_count"] = (
existing_metadata.get("occurrence_count", 0)
+ prop.get("metadata", {}).get("occurrence_count", 0)
)
elif existing.get("range") != prop.get("range"):
existing["range"] = self._get_more_general_type(
existing["range"], prop["range"]
)
return result
@staticmethod
def _merge_property_values(current: Any, incoming: Any) -> List[Any]:
"""Merge scalar-or-list property values while preserving input order."""
values = list(current) if isinstance(current, list) else [current]
incoming_values = (
incoming if isinstance(incoming, list) else [incoming]
)
for value in incoming_values:
if value not in values:
values.append(value)
return [value for value in values if value is not None]
def _infer_data_properties(
self, entities: List[Dict[str, Any]], classes: List[Dict[str, Any]], **options
) -> List[Dict[str, Any]]:
@@ -70,7 +70,8 @@ sem:metadata a owl:AnnotationProperty ;
rdfs:label "metadata" ;
rdfs:comment """Free-form metadata carried through from extraction. An
annotation property because its value is an arbitrary structure rather than a
modelled one.""" ;
modelled one; in the JSON-LD export the whole mapping is written as one
rdf:JSON literal so caller keys never expand into this namespace (#1146).""" ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
# ── Relationship terms (JSON-LD export) ──────────────────────────────────────
@@ -96,10 +97,10 @@ sem:target a owl:ObjectProperty ;
sem:type a owl:DatatypeProperty ;
rdfs:label "type" ;
rdfs:comment """The relationship type as a label, as emitted in the JSON-LD
export. Distinct from rdf:type, which relates a node to a class rather than to
a string.""" ;
rdfs:domain sem:Relationship ;
rdfs:comment """The entity or relationship type as a label, as emitted in
the JSON-LD export. Distinct from rdf:type, which relates a node to a class
rather than to a string. Emitted for both entities and relationships, so the
domain is left open rather than tied to sem:Relationship.""" ;
rdfs:range xsd:string ;
rdfs:isDefinedBy <https://semantica.dev/ns> .
+369 -59
View File
@@ -32,8 +32,10 @@ Author: Semantica Contributors
License: MIT
"""
import copy
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
@@ -59,6 +61,14 @@ class PipelineStatus(Enum):
STOPPED = "stopped"
class _ParallelResultContractError(ProcessingError):
"""Raised when a parallel step violates the dict-result contract.
Contract violations are deterministic: re-running the handler cannot
change its return type, so the retry loop must be skipped entirely.
"""
@dataclass
class ExecutionResult:
"""Pipeline execution result."""
@@ -242,11 +252,67 @@ class ExecutionEngine:
return ExecutionResult(success=False, output=None, errors=[str(e)])
def _execute_steps(self, pipeline: Pipeline, data: Any, **options) -> Any:
"""Execute pipeline steps."""
# Sort steps by dependencies (topological sort)
sorted_steps = self._topological_sort(pipeline.steps)
"""
Execute pipeline steps.
Steps are grouped into dependency layers. When a pipeline is
configured with parallelism > 1, a layer whose steps are all marked
``parallel_safe`` and whose input is a dict is executed concurrently
(bounded by the effective parallelism); every other layer runs
sequentially, preserving the default serial behaviour.
"""
effective_parallelism = self._get_effective_parallelism(pipeline)
if effective_parallelism <= 1:
return self._execute_steps_sequential(pipeline, data, **options)
layers = self._group_steps_by_dependency_level(pipeline.steps)
current_data = data
for layer in layers:
if self.pipeline_status.get(pipeline.name) == PipelineStatus.STOPPED:
break
# Wait if paused
while self.pipeline_status.get(pipeline.name) == PipelineStatus.PAUSED:
time.sleep(0.1)
if self._can_run_layer_in_parallel(layer, current_data):
merged = self._execute_parallel_group(
layer,
current_data,
effective_parallelism,
pipeline_name=pipeline.name,
**options,
)
if merged is None:
# Input isolation failed before any handler started;
# run this layer sequentially instead.
current_data = self._execute_steps_sequential(
pipeline, current_data, steps=layer, **options
)
else:
current_data = merged
else:
current_data = self._execute_steps_sequential(
pipeline, current_data, steps=layer, **options
)
return current_data
def _execute_steps_sequential(
self,
pipeline: Pipeline,
data: Any,
steps: Optional[List[PipelineStep]] = None,
**options,
) -> Any:
"""Execute steps sequentially following dependency order."""
if steps is None:
sorted_steps = self._topological_sort(pipeline.steps)
else:
sorted_steps = list(steps)
# Execute steps
current_data = data
total_steps = len(sorted_steps)
@@ -258,76 +324,320 @@ class ExecutionEngine:
while self.pipeline_status.get(pipeline.name) == PipelineStatus.PAUSED:
time.sleep(0.1)
# Track step execution
step_tracking_id = self.progress_tracker.start_tracking(
module="pipeline",
submodule=step.step_type or step.name,
message=f"Step {step_idx + 1}/{total_steps}: {step.name}",
current_data = self._execute_step_with_retries(
step,
current_data,
step_label=f"Step {step_idx + 1}/{total_steps}: {step.name}",
pipeline_name=pipeline.name,
**options,
)
try:
# Execute step
step.status = StepStatus.RUNNING
step_result = self._execute_step(step, current_data, **options)
step.status = StepStatus.COMPLETED
step.result = step_result
current_data = step_result
return current_data
def _execute_step_with_retries(
self,
step: PipelineStep,
data: Any,
step_label: Optional[str] = None,
pipeline_name: Optional[str] = None,
require_dict_result: bool = False,
**options,
) -> Any:
"""
Execute a single step with retry handling.
Shared by the sequential and parallel execution paths so that retry
policies, step status tracking and progress reporting behave
identically. Returns the step result, or raises the final error
after retries are exhausted.
When ``require_dict_result`` is set (parallel layers), a handler
returning a non-dict raises ProcessingError before the step is
marked completed, so status and progress reporting stay consistent.
Such contract violations are never retried: the handler's return
type cannot change between attempts.
"""
step_tracking_id = self.progress_tracker.start_tracking(
module="pipeline",
# Pipeline identity + step name keep tracking IDs unique so
# concurrent steps of the same step_type cannot overwrite each
# other's progress records.
submodule=(
f"{pipeline_name or 'pipeline'}:"
f"{step.step_type or 'step'}:{step.name}"
),
message=step_label or f"Executing step: {step.name}",
)
try:
step.status = StepStatus.RUNNING
step_result = self._execute_step(step, data, **options)
if require_dict_result and not isinstance(step_result, dict):
raise _ParallelResultContractError(
f"Step '{step.name}' is marked parallel_safe and must "
f"return a dict so parallel results can be merged, got "
f"{type(step_result).__name__}"
)
step.status = StepStatus.COMPLETED
step.result = step_result
self.progress_tracker.stop_tracking(
step_tracking_id,
status="completed",
message=f"Completed step: {step.name}",
)
return step_result
except Exception as e:
step.status = StepStatus.FAILED
step.error = e
# Contract violations are deterministic failures: re-running
# the handler cannot change its return type, so never consult
# the retry policy for them.
if isinstance(e, _ParallelResultContractError):
self.progress_tracker.stop_tracking(
step_tracking_id, status="failed", message=str(e)
)
raise
# Retry loop respecting max_retries from the policy
retry_policy = self.failure_handler.get_retry_policy(step.step_type)
max_retries = retry_policy.max_retries if retry_policy else 0
retry_count = 0
success = False
while retry_count < max_retries:
recovery_result = self.failure_handler.handle_step_failure(step, e)
if not recovery_result.get("retry", False):
break
retry_delay = recovery_result.get("retry_delay", 0.0)
if retry_delay > 0:
time.sleep(retry_delay)
self.progress_tracker.update_tracking(
step_tracking_id,
status="running",
message=f"Retrying step: {step.name} (attempt {retry_count + 1})",
)
step.status = StepStatus.RUNNING
try:
step_result = self._execute_step(step, data, **options)
if require_dict_result and not isinstance(
step_result, dict
):
raise _ParallelResultContractError(
f"Step '{step.name}' is marked parallel_safe "
f"and must return a dict so parallel results "
f"can be merged, got "
f"{type(step_result).__name__}"
)
step.status = StepStatus.COMPLETED
step.result = step_result
success = True
break
except Exception as retry_e:
step.status = StepStatus.FAILED
step.error = retry_e
e = retry_e
retry_count += 1
if success:
self.progress_tracker.stop_tracking(
step_tracking_id,
status="completed",
message=f"Completed step: {step.name}",
message=f"Retry successful: {step.name}",
)
return step_result
else:
self.progress_tracker.stop_tracking(
step_tracking_id, status="failed", message=str(e)
)
raise e
except Exception as e:
step.status = StepStatus.FAILED
step.error = e
def _get_effective_parallelism(self, pipeline: Pipeline) -> int:
"""Return the parallelism actually used for this pipeline."""
configured = pipeline.config.get("parallelism", 1)
if not isinstance(configured, int) or configured <= 0:
return 1
return min(configured, self.parallelism_manager.max_workers)
# Retry loop respecting max_retries from the policy
retry_policy = self.failure_handler.get_retry_policy(step.step_type)
max_retries = retry_policy.max_retries if retry_policy else 0
retry_count = 0
success = False
def _group_steps_by_dependency_level(
self, steps: List[PipelineStep]
) -> List[List[PipelineStep]]:
"""
Group steps into dependency layers, preserving declaration order.
while retry_count < max_retries:
recovery_result = self.failure_handler.handle_step_failure(step, e)
if not recovery_result.get("retry", False):
break
retry_delay = recovery_result.get("retry_delay", 0.0)
if retry_delay > 0:
time.sleep(retry_delay)
self.progress_tracker.update_tracking(
step_tracking_id,
status="running",
message=f"Retrying step: {step.name} (attempt {retry_count + 1})",
Circular or unknown dependencies raise ValidationError so the
parallel path fails deterministically, matching the validation
behaviour of the serial topological sort.
"""
step_map = {step.name: step for step in steps}
levels: Dict[str, int] = {}
visiting: set = set()
for step in steps:
for dep in step.dependencies:
if dep not in step_map:
raise ValidationError(
f"Step '{step.name}' depends on unknown step '{dep}'"
)
step.status = StepStatus.RUNNING
try:
step_result = self._execute_step(step, current_data, **options)
step.status = StepStatus.COMPLETED
step.result = step_result
current_data = step_result
success = True
break
except Exception as retry_e:
step.status = StepStatus.FAILED
step.error = retry_e
e = retry_e
retry_count += 1
if success:
self.progress_tracker.stop_tracking(
step_tracking_id,
status="completed",
message=f"Retry successful: {step.name}",
)
def get_level(step_name: str) -> int:
if step_name in levels:
return levels[step_name]
if step_name in visiting:
raise ValidationError(
"Circular dependency detected in pipeline "
f"(cycle passes through step '{step_name}')"
)
visiting.add(step_name)
step = step_map[step_name]
if not step.dependencies:
level = 0
else:
level = max(get_level(dep) for dep in step.dependencies) + 1
visiting.discard(step_name)
levels[step_name] = level
return level
for step in steps:
get_level(step.name)
grouped: Dict[int, List[PipelineStep]] = {}
for step in steps:
grouped.setdefault(levels[step.name], []).append(step)
return [grouped[level] for level in sorted(grouped)]
def _can_run_layer_in_parallel(self, layer: List[PipelineStep], data: Any) -> bool:
"""Check whether a dependency layer can safely run in parallel."""
if len(layer) <= 1:
return False
if not isinstance(data, dict):
return False
for step in layer:
# Strict boolean check: truthy non-bool values (e.g. the
# string "false") must never opt a step into concurrency.
if getattr(step, "parallel_safe", False) is not True:
return False
if getattr(step, "delta_mode", False):
return False
return True
def _execute_parallel_group(
self,
layer: List[PipelineStep],
data: Any,
effective_parallelism: int,
pipeline_name: Optional[str] = None,
**options,
) -> Optional[Any]:
"""
Execute a dependency layer concurrently.
Per-step inputs are deep-copied before any handler starts so that
parallel steps do not share mutable state. Returns the merged dict
result, or None when input isolation failed (before any handler
ran) and the layer should fall back to sequential execution.
"""
# Isolate per-step inputs before starting any handler
try:
step_inputs = {step.name: copy.deepcopy(data) for step in layer}
except Exception as e:
self.logger.warning(
f"Falling back to sequential execution: input for parallel "
f"layer could not be isolated ({e})"
)
return None
step_results: Dict[str, Any] = {}
failure: Optional[BaseException] = None
max_workers = min(effective_parallelism, len(layer))
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {
executor.submit(
self._execute_step_with_retries,
step,
step_inputs[step.name],
# Validate the dict-result contract inside the shared
# lifecycle path, before completion is reported.
pipeline_name=pipeline_name,
require_dict_result=True,
**options,
): step
for step in layer
}
for future in as_completed(futures):
step = futures[future]
try:
step_result = future.result()
except Exception as e:
if failure is None:
failure = e
# Cancel steps that have not started yet
for pending in futures:
pending.cancel()
else:
self.progress_tracker.stop_tracking(
step_tracking_id, status="failed", message=str(e)
step_results[step.name] = step_result
if failure is not None:
raise failure
return self._merge_parallel_results(data, layer, step_results)
def _merge_parallel_results(
self,
base: Dict[str, Any],
layer: List[PipelineStep],
step_results: Dict[str, Any],
) -> Dict[str, Any]:
"""
Merge the results of a parallel layer into a single dict.
Steps are processed in declaration order (never by completion
order). Keys whose values are unchanged from the shared base input
are skipped (handlers commonly return complete dicts such as
``{**data, ...}``); only added or changed keys count as branch
writes. Keys written with equal values by multiple steps are
allowed; conflicting values for the same key raise a
ProcessingError naming the key and both steps. Ambiguous equality
comparisons count as changed, never as unchanged.
"""
merged = dict(base)
key_sources: Dict[str, str] = {}
for step in layer:
step_result = step_results.get(step.name)
if step_result is None:
continue
for key, value in step_result.items():
if key in base and self._values_equal(base[key], value):
# Echo of the shared input: not a branch write, so it
# cannot conflict with a sibling that changes the key.
continue
if key in key_sources and not self._values_equal(
merged.get(key), value
):
raise ProcessingError(
f"Conflicting values for key '{key}' in parallel step "
f"results: step '{step.name}' produced {value!r}, but "
f"step '{key_sources[key]}' previously produced "
f"{merged.get(key)!r}"
)
raise e
key_sources[key] = step.name
merged[key] = value
return merged
@staticmethod
def _values_equal(left: Any, right: Any) -> bool:
"""Safely compare two values; ambiguous comparisons count as conflicts."""
try:
return bool(left == right)
except (TypeError, ValueError):
return False
return current_data
def _execute_step(self, step: PipelineStep, data: Any, **options) -> Any:
"""
+35
View File
@@ -67,6 +67,7 @@ class PipelineStep:
delta_mode: bool = False
base_version_id: Optional[str] = None
target_version_id: Optional[str] = None
parallel_safe: bool = False
@dataclass
@@ -133,6 +134,12 @@ class PipelineBuilder:
target_version_id = config.pop("target_version_id", None)
dependencies = config.pop("dependencies", [])
handler = config.pop("handler", None)
parallel_safe = config.pop("parallel_safe", False)
if not isinstance(parallel_safe, bool):
raise ValidationError(
f"parallel_safe must be a boolean, got "
f"{type(parallel_safe).__name__} for step '{step_name}'"
)
if handler is None:
handler = self.step_registry.get(step_type)
@@ -145,6 +152,7 @@ class PipelineBuilder:
delta_mode = delta_mode,
base_version_id=base_version_id,
target_version_id=target_version_id,
parallel_safe=parallel_safe,
)
self.steps.append(step)
@@ -186,6 +194,14 @@ class PipelineBuilder:
Returns:
Self for method chaining
"""
if (
isinstance(level, bool)
or not isinstance(level, int)
or level <= 0
):
raise ValidationError(
f"Parallelism level must be a positive integer, got {level!r}"
)
self.pipeline_config["parallelism"] = level
return self
@@ -289,6 +305,16 @@ class PipelineBuilder:
step.target_version_id = step_config.get(
"target_version_id", step.target_version_id
)
raw_parallel_safe = step_config.get(
"parallel_safe", step.parallel_safe
)
if not isinstance(raw_parallel_safe, bool):
raise ValidationError(
"parallel_safe must be a boolean for step "
f"'{step_name}', got "
f"{type(raw_parallel_safe).__name__}"
)
step.parallel_safe = raw_parallel_safe
# Set parallelism if specified
if "parallelism" in pipeline_config:
@@ -344,6 +370,7 @@ class PipelineBuilder:
"type": step.step_type,
"config": step.config,
"dependencies": step.dependencies,
"parallel_safe": step.parallel_safe,
}
for step in self.steps
],
@@ -425,6 +452,7 @@ class PipelineSerializer:
"delta_mode",
"base_version_id",
"target_version_id",
"parallel_safe",
}
pipeline_data = {
"name": pipeline.name,
@@ -441,6 +469,7 @@ class PipelineSerializer:
"delta_mode": getattr(step, "delta_mode", False),
"base_version_id": getattr(step, "base_version_id", None),
"target_version_id": getattr(step, "target_version_id", None),
"parallel_safe": getattr(step, "parallel_safe", False),
}
for step in pipeline.steps
],
@@ -488,6 +517,12 @@ class PipelineSerializer:
sanitized_steps.append(sanitized_step)
pipeline_data["steps"] = sanitized_steps
# Reapply pipeline-level config (e.g. parallelism) at the top level
# so build_pipeline picks it up
serialized_config = pipeline_data.pop("config", None) or {}
for key, value in serialized_config.items():
pipeline_data.setdefault(key, value)
# Reconstruct pipeline
builder = PipelineBuilder(**self.config)
pipeline = builder.build_pipeline(pipeline_data, **options)
+1 -1
View File
@@ -67,7 +67,7 @@ def test_merging_repeated_exports_yields_one_graph_node(tmp_path):
assert len(kg_nodes) == 1
entity_nodes = set(
merged.subjects(RDF.type, URIRef("https://semantica.dev/vocab/ORG"))
merged.subjects(RDF.type, URIRef("https://semantica.dev/ns#Entity"))
)
assert len(entity_nodes) == 1
+120
View File
@@ -0,0 +1,120 @@
"""Caller data must never expand into the Semantica namespace (#1146).
``@vocab`` used to sit in every JSON-LD context pointing at ``ns#``, so every
bare term in caller data expanded into it: an extracted type like ``"ORG"``
became ``ns#ORG`` (a term the vocabulary does not define), and a metadata key
like ``"source"`` collided with the real ``sem:source`` object property,
attaching a plain string to a property whose range is a resource. The fix
removes ``@vocab`` outright: only explicit ``semantica:``-prefixed terms
resolve, caller type labels travel as ``semantica:type`` strings, and caller
metadata survives as one ``rdf:JSON`` literal.
"""
import json
from rdflib import RDF, Graph, Literal, URIRef
from semantica.export.json_exporter import JSONExporter
from semantica.export.rdf_exporter import RDFExporter, SEMANTICA_NS
NS = SEMANTICA_NS
E1 = "https://example.org/e1"
KG = {
"entities": [
{
"id": E1,
"text": "Acme",
"type": "ORG",
"metadata": {"source": "crm_export_2024"},
}
],
"relationships": [
{
"source_id": E1,
"target_id": "https://example.org/e2",
"type": "employs",
}
],
}
def _jsonld_file(exporter, kind, tmp_path, name):
path = tmp_path / name
if kind == "knowledge_graph":
exporter.export_knowledge_graph(KG, path, format="json-ld")
elif kind == "entities":
exporter.export_entities(KG["entities"], path, format="json-ld")
elif kind == "relationships":
exporter.export_relationships(KG["relationships"], path, format="json-ld")
elif kind == "generic":
exporter.export({"note": "plain payload, no @id"}, path, format="json-ld")
else:
raise AssertionError(kind)
return json.loads(path.read_text())
def test_no_jsonld_context_declares_a_vocab(tmp_path):
exporter = JSONExporter()
for kind in ("knowledge_graph", "entities", "relationships", "generic"):
context = _jsonld_file(exporter, kind, tmp_path, f"{kind}.jsonld")[
"@context"
]
assert "@vocab" not in context, f"{kind}: @vocab expands caller data"
context = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))[
"@context"
]
assert "@vocab" not in context
def test_extracted_type_labels_stay_out_of_the_namespace(tmp_path):
path = tmp_path / "kg.jsonld"
JSONExporter().export_knowledge_graph(KG, path, format="json-ld")
graph = Graph()
graph.parse(str(path), format="json-ld")
assert (None, RDF.type, URIRef(NS + "ORG")) not in graph, (
"the caller's type label was minted as a class in ns#"
)
assert (URIRef(E1), RDF.type, URIRef(NS + "Entity")) in graph
assert (URIRef(E1), URIRef(NS + "type"), Literal("ORG")) in graph, (
"the label itself must survive, as a string"
)
def test_metadata_keys_stay_out_of_the_namespace(tmp_path):
path = tmp_path / "kg.jsonld"
JSONExporter().export_knowledge_graph(KG, path, format="json-ld")
graph = Graph()
graph.parse(str(path), format="json-ld")
assert (None, URIRef(NS + "source"), Literal("crm_export_2024")) not in (
graph
), "caller metadata value attached to the real sem:source object property"
for _, _, o in graph.triples((None, URIRef(NS + "source"), None)):
assert not isinstance(o, Literal), (
"sem:source has a resource range but received a plain literal"
)
literals = [
o
for o in graph.objects(None, URIRef(NS + "metadata"))
if isinstance(o, Literal)
]
assert literals, "the metadata dict was dropped instead of preserved"
assert literals[0].datatype == RDF.JSON
assert json.loads(str(literals[0])) == {"source": "crm_export_2024"}
def test_rdf_exporter_jsonld_keeps_type_labels_out_of_the_namespace():
graph = Graph()
graph.parse(
data=RDFExporter().export_to_rdf(KG, format="jsonld"), format="json-ld"
)
assert (None, RDF.type, URIRef(NS + "ORG")) not in graph
assert (URIRef(E1), RDF.type, URIRef(NS + "Entity")) in graph
assert (URIRef(E1), URIRef(NS + "type"), Literal("ORG")) in graph
+663
View File
@@ -0,0 +1,663 @@
"""Tests for the SAP OData ingestor.
The SAP connector never touches a live SAP system: every outbound request goes
through ``semantica.ingest.ssrf.request_with_ssrf_guard`` (see
``sap_ingestor.py``), so these tests patch that single entry point and drive
the parser / connector / ingestor with canned responses.
Covered:
- ``$metadata`` (CSDL XML) -> entity set discovery & property fields
- OData v2 atom (``__next``) and OData v4 (``@odata.nextLink``) pagination
- Basic and OAuth2 credential payloads propagated on the request
- SSRF guard is used for data *and* token-exchange requests
- ``export_as_documents`` yields the flat document shape GraphBuilder expects
"""
from unittest.mock import MagicMock, patch
import pytest
import requests
from semantica.ingest import SAPIngestor, SAPODataConnector, SAPODataEntity
from semantica.utils.exceptions import ProcessingError, ValidationError
METADATA_V2 = """<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx">
<edmx:DataServices
xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
<Schema Namespace="SAPService" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
<EntityContainer Name="Default" m:IsDefaultEntityContainer="true">
<EntitySet Name="SalesOrderSet" EntityType="SAPService.SalesOrder"/>
<EntitySet Name="CustomerSet" EntityType="SAPService.Customer"/>
</EntityContainer>
<EntityType Name="SalesOrder">
<Key><PropertyRef Name="SalesOrderID"/></Key>
<Property Name="SalesOrderID" Type="Edm.String" Nullable="false"/>
<Property Name="CustomerID" Type="Edm.String"/>
<Property Name="GrossAmount" Type="Edm.Decimal"/>
</EntityType>
<EntityType Name="Customer">
<Key><PropertyRef Name="CustomerID"/></Key>
<Property Name="CustomerID" Type="Edm.String" Nullable="false"/>
<Property Name="Name" Type="Edm.String"/>
</EntityType>
</Schema>
</edmx:DataServices>
</edmx:Edmx>
"""
METADATA_V4 = """<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
<Schema Namespace="SalesNs" xmlns="http://docs.oasis-open.org/odata/ns/edm">
<EntityContainer Name="Svc">
<EntitySet Name="SalesOrders" EntityType="SalesNs.SalesOrder"/>
<EntitySet Name="Customers" EntityType="SalesNs.Customer"/>
</EntityContainer>
<EntityType Name="SalesOrder">
<Property Name="ID" Type="Edm.String"/>
<Property Name="Total" Type="Edm.Decimal" Nullable="false"/>
</EntityType>
<EntityType Name="Customer">
<Property Name="ID" Type="Edm.String"/>
<Property Name="Name" Type="Edm.String"/>
</EntityType>
</Schema>
</edmx:DataServices>
</edmx:Edmx>
"""
METADATA_MULTI_SCHEMA = """<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx">
<edmx:DataServices>
<Schema Namespace="OrdersNs" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
<EntityContainer Name="OrdersSvc">
<EntitySet Name="OrdersNs" EntityType="OrdersNs.Row"/>
</EntityContainer>
<EntityType Name="Row">
<Property Name="A" Type="Edm.String"/>
</EntityType>
</Schema>
<Schema Namespace="InvoicesNs" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
<EntityContainer Name="InvoicesSvc">
<EntitySet Name="InvoicesNs" EntityType="InvoicesNs.Row"/>
</EntityContainer>
<EntityType Name="Row">
<Property Name="B" Type="Edm.String"/>
</EntityType>
</Schema>
</edmx:DataServices>
</edmx:Edmx>
"""
def _fake_response(status_code=200, json_payload=None, text="", headers=None):
"""requests.Response-like stand-in returned by the mocked guard."""
resp = MagicMock()
resp.status_code = status_code
resp.headers = headers or {}
resp.text = text
resp.url = "https://sap.example/odata/$metadata"
if json_payload is not None:
resp.json.return_value = json_payload
else:
resp.json.side_effect = ValueError("not json")
if status_code >= 400:
resp.raise_for_status.side_effect = requests.exceptions.HTTPError(
f"{status_code} error"
)
return resp
def _json_resp(payload, url):
r = MagicMock()
r.status_code = 200
r.headers = {"content-type": "application/json"}
r.text = ""
r.url = url
r.json.return_value = payload
return r
class TestDiscoverService:
def test_metadata_parses_entity_sets_and_fields(self):
resp = MagicMock()
resp.status_code = 200
resp.headers = {"content-type": "application/xml"}
resp.text = METADATA_V2
resp.json.side_effect = ValueError("xml not json")
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/", username="u", password="p"
)
sets = ing.discover_service()
assert guard.call_count == 1
assert [s["name"] for s in sets] == ["SalesOrderSet", "CustomerSet"]
sales = next(s for s in sets if s["name"] == "SalesOrderSet")
assert {f["name"] for f in sales["fields"]} == {
"SalesOrderID",
"CustomerID",
"GrossAmount",
}
assert sales["fields"][0]["type"] == "Edm.String"
def test_discover_flow_routes_via_ssrf(self):
resp = MagicMock()
resp.status_code = 200
resp.headers = {}
resp.text = METADATA_V2
resp.json.side_effect = ValueError()
with patch("semantica.ingest.sap_ingestor.request_with_ssrf_guard") as guard:
guard.return_value = resp
ing = SAPIngestor(
base_url="https://sap.example/odata/", username="u", password="p"
)
ing.discover_service()
method, url = guard.call_args[0]
assert method == "GET"
assert url.endswith("$metadata")
def test_metadata_url_keeps_last_segment_without_trailing_slash(self):
"""base_url without a trailing '/' must not lose its last segment.
urljoin() replaces the final path segment when the base has no
trailing slash, which would silently turn .../API_BUSINESS_PARTNER
into .../$metadata against the wrong service root.
"""
resp = MagicMock()
resp.status_code = 200
resp.headers = {}
resp.text = METADATA_V2
resp.json.side_effect = ValueError()
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/API_BUSINESS_PARTNER",
username="u",
password="p",
)
ing.discover_service()
method, url = guard.call_args[0]
assert url == "https://sap.example/odata/API_BUSINESS_PARTNER/$metadata"
def test_service_root_appends_metadata(self):
"""service is a service root (same meaning as ingest_entity_set):
$metadata is appended automatically, not treated as the full path.
"""
resp = MagicMock()
resp.status_code = 200
resp.headers = {}
resp.text = METADATA_V2
resp.json.side_effect = ValueError()
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/sap/",
username="u",
password="p",
)
ing.discover_service("API_BUSINESS_PARTNER")
_, url = guard.call_args[0]
assert url == "https://sap.example/odata/sap/API_BUSINESS_PARTNER/$metadata"
def test_absolute_service_root_appends_metadata(self):
resp = MagicMock()
resp.status_code = 200
resp.headers = {}
resp.text = METADATA_V2
resp.json.side_effect = ValueError()
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/sap/",
username="u",
password="p",
)
ing.discover_service("https://other.example/services/sap/")
_, url = guard.call_args[0]
assert url == "https://other.example/services/sap/$metadata"
def test_v4_oasis_metadata_parses_with_nullable_defaults(self):
resp = _fake_response(
text=METADATA_V4, headers={"content-type": "application/xml"}
)
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
sets = ing.discover_service()
by_name = {s["name"]: s for s in sets}
assert "SalesOrders" in by_name and "Customers" in by_name
sales_fields = {f["name"]: f for f in by_name["SalesOrders"]["fields"]}
# Omitted Nullable -> nullable (CSDL default); explicit "false" honored.
assert sales_fields["ID"]["nullable"] is True
assert sales_fields["Total"]["nullable"] is False
def test_same_named_types_in_different_schemas_do_not_merge(self):
resp = _fake_response(
text=METADATA_MULTI_SCHEMA, headers={"content-type": "application/xml"}
)
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
sets = ing.discover_service()
by_name = {s["name"]: s for s in sets}
assert [f["name"] for f in by_name["OrdersNs"]["fields"]] == ["A"]
assert [f["name"] for f in by_name["InvoicesNs"]["fields"]] == ["B"]
class TestAuth:
def test_basic_auth_payload_attached(self):
conn = SAPODataConnector(
base_url="https://sap.example/odata/",
username="erp_user",
# Non-functional test placeholder; the low-entropy value keeps the
# secret scanner from treating it as a hardcoded credential.
password="test",
)
session = conn.get_session()
assert session.headers["Authorization"].startswith("Basic ")
# base64("erp_user:test")
assert session.headers["Authorization"].endswith("ZXJwX3VzZXI6dGVzdA==")
def test_oauth_token_exchange_goes_through_ssrf(self):
token_resp = MagicMock()
token_resp.status_code = 200
token_resp.headers = {}
token_resp.json.return_value = {"access_token": "tok-123"}
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=token_resp,
) as guard:
conn = SAPODataConnector(
base_url="https://sap.example/odata/",
token_url="https://auth.example/oauth/token",
client_id="cid",
client_secret="secret",
)
session = conn.get_session()
assert guard.call_count == 1
method_tok, url = guard.call_args[0]
assert method_tok == "POST"
assert url == "https://auth.example/oauth/token"
assert session.headers["Authorization"] == "Bearer tok-123"
body = guard.call_args.kwargs.get("data", {})
assert body["grant_type"] == "client_credentials"
def test_requires_auth(self):
with pytest.raises(ValidationError):
SAPODataConnector(base_url="https://sap.example/odata/")
def test_requires_base_url(self):
with pytest.raises(ValidationError):
SAPODataConnector(username="u", password="p")
class TestIngestPagination:
V4_PAGE = {
"value": [{"SalesOrderID": "SO-1"}, {"SalesOrderID": "SO-2"}],
"@odata.nextLink": "https://sap.example/odata/SalesOrderSet?$skiptoken=abc",
}
V4_LAST = {"value": [{"SalesOrderID": "SO-3"}]}
def test_v4_nextlink_paginates(self):
calls = []
def fake_guard(method, url, **kw):
page1 = _json_resp(self.V4_PAGE, url)
page2 = _json_resp(self.V4_LAST, url)
calls.append(url)
return page1 if method == "GET" and len(calls) == 1 else page2
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
side_effect=fake_guard,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
assert result.count == 3
assert [r["SalesOrderID"] for r in result.records] == [
"SO-1",
"SO-2",
"SO-3",
]
# Reached the second page's next link then stopped.
assert len(calls) == 2
def test_v4_nextlink_is_followed_past_first_page(self):
calls = []
def fake_guard(method, url, **kw):
calls.append(url)
if len(calls) == 1:
return _json_resp(self.V4_PAGE, url)
return _json_resp(self.V4_LAST, url)
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
side_effect=fake_guard,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
ing.ingest_entity_set(entity_set="SalesOrderSet")
assert len(calls) == 2
def test_v2_atom_next_pagination(self):
v2_first = {
"d": {
"results": [{"SalesOrderID": "A"}],
"__next": {"__deferred": {"uri": "https://sap.example/odata/next2"}},
},
}
v2_last = {"d": {"results": [{"SalesOrderID": "B"}]}}
calls = []
def fake_guard(method, url, **kw):
calls.append(url)
return (
_json_resp(v2_first, url)
if len(calls) == 1
else _json_resp(v2_last, url)
)
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
side_effect=fake_guard,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
assert [r["SalesOrderID"] for r in result.records] == ["A", "B"]
assert len(calls) == 2
def test_v2_next_as_plain_string_paginates(self):
"""Canonical OData v2 JSON: ``__next`` is a plain string URL."""
v2_first = {
"d": {
"results": [{"SalesOrderID": "A"}],
"__next": "https://sap.example/odata/next2",
},
}
v2_last = {"d": {"results": [{"SalesOrderID": "B"}]}}
calls = []
def fake_guard(method, url, **kw):
calls.append(url)
if len(calls) == 1:
return _json_resp(v2_first, url)
return _json_resp(v2_last, url)
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
side_effect=fake_guard,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
assert [r["SalesOrderID"] for r in result.records] == ["A", "B"]
assert len(calls) == 2
def test_relative_service_resolves_against_base(self):
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=_json_resp({"value": []}, "https://x/"),
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/sap/",
username="u",
password="p",
)
ing.ingest_entity_set(
service="API_SALES_ORDER_SRV", entity_set="SalesOrderSet"
)
method, url = guard.call_args[0]
assert url == "https://sap.example/odata/sap/API_SALES_ORDER_SRV/SalesOrderSet"
def test_expand_passthrough_for_line_items(self):
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=_json_resp(
{"value": [{"SalesOrderID": "SO-9"}]}, "https://x/"
),
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/", username="u", password="p"
)
ing.ingest_entity_set(entity_set="SalesOrderSet", expand="to_Item")
params = guard.call_args_list[0].kwargs.get("params") or {}
assert params.get("$expand") == "to_Item"
def test_select_filter_top_skip_objects_passthrough(self):
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=_json_resp({"value": []}, "https://x/"),
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/", username="u", password="p"
)
ing.ingest_entity_set(
entity_set="SalesOrderSet",
select="SalesOrderID,GrossAmount",
filter="GrossAmount gt 100",
top=5,
skip=2,
)
params = guard.call_args_list[0].kwargs.get("params") or {}
assert params["$select"] == "SalesOrderID,GrossAmount"
assert params["$filter"] == "GrossAmount gt 100"
assert params["$top"] == "5"
assert params["$skip"] == "2"
class TestErrorPaths:
def test_http_error_on_entity_set_raises_processing_error(self):
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=_fake_response(status_code=403),
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
with pytest.raises(ProcessingError, match="403"):
ing.ingest_entity_set(entity_set="SalesOrderSet")
def test_invalid_metadata_xml_raises_processing_error(self):
resp = _fake_response(text="<not-xml")
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
with pytest.raises(ProcessingError, match="not valid XML"):
ing.discover_service()
def test_non_json_entity_response_raises_processing_error(self):
resp = _fake_response(text="plain text")
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
with pytest.raises(ProcessingError, match="not JSON"):
ing.ingest_entity_set(entity_set="SalesOrderSet")
def test_cross_host_next_link_rejected(self):
first = _json_resp(
{
"value": [{"SalesOrderID": "A"}],
"@odata.nextLink": "https://evil.example/next",
},
"https://sap.example/odata/SalesOrderSet",
)
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=first,
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
with pytest.raises(ProcessingError, match="different host"):
ing.ingest_entity_set(entity_set="SalesOrderSet")
# The credential-bearing session is never sent to the evil host.
assert guard.call_count == 1
def test_top_zero_returns_no_rows_and_makes_no_request(self):
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
) as guard:
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
result = ing.ingest_entity_set(entity_set="SalesOrderSet", top=0)
assert result.count == 0
assert result.records == []
guard.assert_not_called()
def test_negative_top_rejected(self):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
with pytest.raises(ValidationError, match="must be >= 0"):
ing.ingest_entity_set(entity_set="SalesOrderSet", top=-1)
def test_malformed_rows_container_rejected(self):
resp = _json_resp(
{"value": {"not": "a list"}},
"https://sap.example/odata/SalesOrderSet",
)
with patch(
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
return_value=resp,
):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
with pytest.raises(ProcessingError, match="no list of rows"):
ing.ingest_entity_set(entity_set="SalesOrderSet")
class TestExport:
def test_export_as_documents_flattens_to_graph_builder_shape(self):
ing = SAPIngestor(
base_url="https://sap.example/odata/",
username="u",
password="p",
)
entity = SAPODataEntity(
records=[{"SalesOrderID": "SO-1", "CustomerID": "C-1"}],
entity_set="SalesOrderSet",
count=1,
service="https://sap.example/odata/",
)
docs = ing.export_as_documents(entity)
assert docs[0]["SalesOrderID"] == "SO-1"
assert docs[0]["id"] == "SO-1"
assert docs[0]["source"] == "https://sap.example/odata/"
class TestEntityDocument:
def test_to_document_adds_source(self):
ent = SAPODataEntity(
records=[{"k": "v"}],
entity_set="S",
count=1,
service="https://sap.example/odata/",
)
assert ent.to_documents()[0]["source"] == "https://sap.example/odata/"
def test_to_document_injects_graph_builder_identifier(self):
ent = SAPODataEntity(
records=[{"SalesOrderID": "SO-1", "CustomerID": "C-1"}],
entity_set="SalesOrderSet",
count=1,
service="https://sap.example/odata/",
)
doc = ent.to_documents()[0]
assert doc["id"] == "SO-1"
assert doc["name"] == "SO-1"
assert doc["SalesOrderID"] == "SO-1" # original field preserved
def test_to_document_falls_back_to_entity_set_index(self):
ent = SAPODataEntity(
records=[{"GrossAmount": "10.0"}],
entity_set="SalesOrderSet",
count=1,
service="https://sap.example/odata/",
)
assert ent.to_documents()[0]["id"] == "SalesOrderSet:0"
def test_public_import_through_lazy_export(self):
# __getattr__ should resolve the lazy export once the object is created
assert callable(SAPODataConnector)
@@ -0,0 +1,51 @@
import pytest
from semantica.ontology.class_inferrer import ClassInferrer
from semantica.ontology.property_generator import PropertyGenerator
from semantica.utils.exceptions import ValidationError
def test_same_kind_normalized_object_properties_are_merged():
entities = [
{"id": "p1", "type": "Person", "name": "Alice"},
{"id": "o1", "type": "Organization", "name": "Acme"},
]
classes = ClassInferrer(min_occurrences=1).infer_classes(entities)
relationships = [
{
"source_type": "Person",
"target_type": "Organization",
"type": "works_for",
},
{
"source_type": "Person",
"target_type": "Organization",
"type": "worksFor",
},
]
properties = PropertyGenerator().infer_properties(
entities, relationships, classes, min_occurrences=1
)
works_for = [prop for prop in properties if prop["name"] == "worksFor"]
assert len(works_for) == 1
assert works_for[0]["domain"] == ["Person"]
assert works_for[0]["range"] == ["Organization"]
def test_normalized_name_cannot_be_both_object_and_data_property():
entities = [
{"id": "p1", "type": "Person", "value": "Alice"},
{"id": "p2", "type": "Person", "value": "Bob"},
]
classes = ClassInferrer(min_occurrences=1).infer_classes(entities)
relationships = [
{"source_type": "Person", "target_type": "Person", "type": "value"},
{"source_type": "Person", "target_type": "Person", "type": "value"},
]
with pytest.raises(ValidationError, match="object and data"):
PropertyGenerator().infer_properties(
entities, relationships, classes, min_occurrences=1
)
+785
View File
@@ -0,0 +1,785 @@
"""Focused tests for pipeline parallel execution (issue #1223).
Covers:
- ``PipelineBuilder.set_parallelism()`` validation
- dependency-layer parallel execution gated on ``parallel_safe``
- input isolation via deep copies
- incremental dict merging of parallel outputs
- failure handling and cancellation semantics
- ``parallel_safe`` not leaking into handler kwargs
- ``parallel_safe`` surviving serialization round-trips
"""
import threading
import time
import unittest
from unittest.mock import MagicMock, patch
from semantica.pipeline.execution_engine import ExecutionEngine
from semantica.pipeline.failure_handler import RetryPolicy, RetryStrategy
from semantica.pipeline.pipeline_builder import (
Pipeline,
PipelineBuilder,
PipelineSerializer,
PipelineStep,
StepStatus,
)
from semantica.utils.exceptions import ProcessingError, ValidationError
class ConcurrencyProbe:
"""Thread-safe tracker of handler invocations and concurrency level."""
def __init__(self):
self._lock = threading.Lock()
self.active = 0
self.max_active = 0
self.calls = 0
def __enter__(self):
with self._lock:
self.active += 1
self.calls += 1
self.max_active = max(self.max_active, self.active)
return self
def __exit__(self, *exc):
with self._lock:
self.active -= 1
return False
def branch_handler(probe, key, value=True, delay=0.0):
"""Build a handler that records concurrency and adds one output key."""
def handler(data, **kwargs):
with probe:
if delay:
time.sleep(delay)
return {**data, key: value}
return handler
class Undeepcopyable:
"""Object whose deepcopy always fails."""
def __deepcopy__(self, memo):
raise TypeError("cannot deepcopy this object")
class TestSetParallelismValidation(unittest.TestCase):
"""set_parallelism() must only accept positive integers."""
def test_rejects_invalid_levels(self):
builder = PipelineBuilder()
for invalid in (0, -1, 1.5, True, False, "2", None):
with self.assertRaises(ValidationError):
builder.set_parallelism(invalid)
def test_accepts_positive_integers(self):
builder = PipelineBuilder()
builder.set_parallelism(4)
self.assertEqual(builder.pipeline_config["parallelism"], 4)
class TestPipelineParallelExecution(unittest.TestCase):
def setUp(self):
self.mock_tracker_patcher = patch(
"semantica.utils.progress_tracker.get_progress_tracker"
)
self.mock_get_tracker = self.mock_tracker_patcher.start()
self.mock_get_tracker.return_value = MagicMock()
def tearDown(self):
self.mock_tracker_patcher.stop()
def _build(self, steps, parallelism=None, name="parallel_pipeline"):
"""steps: list of (step_name, step_type, handler, parallel_safe)."""
builder = PipelineBuilder()
for step_name, step_type, handler, parallel_safe in steps:
builder.add_step(
step_name, step_type, handler=handler, parallel_safe=parallel_safe
)
if parallelism is not None:
builder.set_parallelism(parallelism)
return builder.build(name)
def test_unconfigured_pipeline_stays_serial(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", branch_handler(probe, "b"), True),
]
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertEqual(probe.max_active, 1)
def test_parallelism_one_stays_serial(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", branch_handler(probe, "b"), True),
],
parallelism=1,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertEqual(probe.max_active, 1)
def test_parallel_safe_steps_overlap_with_barrier(self):
barrier = threading.Barrier(2)
done = {"a": False, "b": False}
def handler(key):
def inner(data, **kwargs):
barrier.wait(timeout=5)
done[key] = True
return {**data, key: True}
return inner
pipeline = self._build(
[
("a", "branch", handler("a"), True),
("b", "branch", handler("b"), True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success, msg=str(result.errors))
self.assertTrue(done["a"])
self.assertTrue(done["b"])
def test_active_workers_do_not_exceed_parallelism(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a", delay=0.15), True),
("b", "branch", branch_handler(probe, "b", delay=0.15), True),
("c", "branch", branch_handler(probe, "c", delay=0.15), True),
("d", "branch", branch_handler(probe, "d", delay=0.15), True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertLessEqual(probe.max_active, 2)
self.assertGreaterEqual(probe.max_active, 2)
def test_unmarked_steps_stay_serial(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), False),
("b", "branch", branch_handler(probe, "b"), False),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertEqual(probe.max_active, 1)
def test_layer_with_one_unsafe_step_is_serial(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", branch_handler(probe, "b"), False),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertEqual(probe.max_active, 1)
def test_pure_dependency_chain_stays_serial(self):
probe = ConcurrencyProbe()
def chain_handler(key):
def inner(data, **kwargs):
with probe:
return {**data, key: True}
return inner
builder = PipelineBuilder()
builder.add_step(
"a", "branch", handler=chain_handler("a"), parallel_safe=True
)
builder.add_step(
"b",
"branch",
handler=chain_handler("b"),
parallel_safe=True,
dependencies=["a"],
)
builder.add_step(
"c",
"branch",
handler=chain_handler("c"),
parallel_safe=True,
dependencies=["b"],
)
builder.set_parallelism(4)
pipeline = builder.build("chain_pipeline")
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertEqual(probe.max_active, 1)
self.assertEqual(result.output, {"text": "hi", "a": True, "b": True, "c": True})
def test_non_dict_input_falls_back_to_serial(self):
probe = ConcurrencyProbe()
def passthrough(data, **kwargs):
with probe:
return data
pipeline = self._build(
[
("a", "branch", passthrough, True),
("b", "branch", passthrough, True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(
pipeline, data="plain-text-input"
)
self.assertTrue(result.success)
self.assertEqual(probe.calls, 2)
self.assertEqual(probe.max_active, 1)
self.assertEqual(result.output, "plain-text-input")
def test_deepcopy_failure_falls_back_to_serial(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", branch_handler(probe, "b"), True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(
pipeline, data={"text": "hi", "obj": Undeepcopyable()}
)
self.assertTrue(result.success, msg=str(result.errors))
self.assertEqual(probe.calls, 2)
self.assertEqual(probe.max_active, 1)
self.assertIn("a", result.output)
self.assertIn("b", result.output)
def test_parallel_results_merge_different_fields(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "entities", ["Alice"]), True),
("b", "branch", branch_handler(probe, "triplets", [(1, 2)]), True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(
pipeline, data={"text": "Alice works at Acme"}
)
self.assertTrue(result.success)
self.assertEqual(
result.output,
{
"text": "Alice works at Acme",
"entities": ["Alice"],
"triplets": [(1, 2)],
},
)
def test_parallel_results_same_key_same_value_allowed(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "shared", [1, 2]), True),
("b", "branch", branch_handler(probe, "shared", [1, 2]), True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success, msg=str(result.errors))
self.assertEqual(result.output["shared"], [1, 2])
def test_parallel_results_same_key_different_values_raises(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "shared", 1), True),
("b", "branch", branch_handler(probe, "shared", 2), True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertFalse(result.success)
error_text = result.errors[0] if result.errors else ""
self.assertIn("shared", error_text)
self.assertIn("a", error_text)
self.assertIn("b", error_text)
def test_parallel_handler_non_dict_return_fails(self):
probe = ConcurrencyProbe()
def non_dict_handler(data, **kwargs):
with probe:
return "not-a-dict"
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", non_dict_handler, True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertFalse(result.success)
error_text = result.errors[0] if result.errors else ""
self.assertIn("b", error_text)
self.assertIn("str", error_text)
def test_non_dict_handler_executed_only_once(self):
probe = ConcurrencyProbe()
def non_dict_handler(data, **kwargs):
with probe:
return "not-a-dict"
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", non_dict_handler, True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertFalse(result.success)
self.assertEqual(probe.calls, 2) # one invocation per handler, no re-runs
def test_parallel_step_retry_succeeds(self):
engine = ExecutionEngine()
engine.failure_handler.set_retry_policy(
"flaky",
RetryPolicy(
max_retries=2, initial_delay=0.0, strategy=RetryStrategy.FIXED
),
)
probe = ConcurrencyProbe()
attempts = {"flaky": 0}
def flaky_handler(data, **kwargs):
with probe:
attempts["flaky"] += 1
if attempts["flaky"] == 1:
raise RuntimeError("transient failure")
return {**data, "flaky": True}
pipeline = self._build(
[
("a", "flaky", flaky_handler, True),
("b", "branch", branch_handler(probe, "b"), True),
],
parallelism=2,
)
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success, msg=str(result.errors))
self.assertEqual(attempts["flaky"], 2)
def test_failed_branch_skips_downstream_layer(self):
engine = ExecutionEngine()
engine.failure_handler.set_retry_policy(
"always_fail", RetryPolicy(max_retries=0)
)
probe = ConcurrencyProbe()
def failing_handler(data, **kwargs):
with probe:
raise RuntimeError("permanent failure")
downstream_calls = {"c": 0}
def downstream_handler(data, **kwargs):
downstream_calls["c"] += 1
return {**data, "c": True}
builder = PipelineBuilder()
builder.add_step("a", "always_fail", handler=failing_handler, parallel_safe=True)
builder.add_step("b", "branch", handler=branch_handler(probe, "b"), parallel_safe=True)
builder.add_step(
"c", "branch", handler=downstream_handler, dependencies=["a", "b"]
)
builder.set_parallelism(2)
pipeline = builder.build("failure_pipeline")
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
self.assertFalse(result.success)
self.assertEqual(downstream_calls["c"], 0)
failed_step = next(s for s in pipeline.steps if s.name == "a")
self.assertEqual(failed_step.status, StepStatus.FAILED)
self.assertIsNotNone(failed_step.error)
def test_parallel_safe_not_passed_to_handler_kwargs(self):
received_kwargs = {}
def capturing_handler(data, **kwargs):
received_kwargs.update(kwargs)
return data
builder = PipelineBuilder()
builder.add_step(
"a", "branch", handler=capturing_handler, parallel_safe=True, batch_size=2
)
pipeline = builder.build("kwargs_pipeline")
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertIn("batch_size", received_kwargs)
self.assertNotIn("parallel_safe", received_kwargs)
self.assertNotIn("parallel_safe", pipeline.steps[0].config)
def test_unchanged_echoed_key_does_not_conflict(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
# "a" echoes the base value of "shared" (unchanged),
# "b" legitimately changes it: no false conflict.
("a", "branch", branch_handler(probe, "shared", 1), True),
("b", "branch", branch_handler(probe, "shared", 2), True),
],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(
pipeline, data={"text": "hi", "shared": 1}
)
self.assertTrue(result.success, msg=str(result.errors))
self.assertEqual(result.output["shared"], 2)
def test_ambiguous_equality_counts_as_changed(self):
sentinel = Uncomparable()
def echo(data, **kwargs):
return {**data, "shared": data["shared"]}
def change(data, **kwargs):
return {**data, "shared": "changed"}
pipeline = self._build(
[("a", "branch", echo, True), ("b", "branch", change, True)],
parallelism=2,
)
result = ExecutionEngine().execute_pipeline(
pipeline, data={"text": "hi", "shared": sentinel}
)
# Ambiguous equality must not be treated as unchanged, so both
# branches count as writes and the merge reports the conflict.
self.assertFalse(result.success)
error_text = result.errors[0] if result.errors else ""
self.assertIn("shared", error_text)
def test_non_boolean_parallel_safe_stays_serial(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", branch_handler(probe, "b"), True),
],
parallelism=2,
)
# Simulate a truthy non-bool value reaching the engine (e.g. set
# directly on the step attribute): must not enable concurrency.
for step in pipeline.steps:
step.parallel_safe = "false"
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success)
self.assertEqual(probe.max_active, 1)
def test_non_dict_result_reports_failure_not_completion(self):
probe = ConcurrencyProbe()
tracking_ids = {}
counter = {"n": 0}
def fake_start(*args, **kwargs):
tid = f"tid_{counter['n']}"
counter["n"] += 1
tracking_ids[kwargs.get("submodule")] = tid
return tid
with patch(
"semantica.pipeline.execution_engine.get_progress_tracker"
) as mock_get:
tracker = MagicMock()
tracker.start_tracking.side_effect = fake_start
mock_get.return_value = tracker
engine = ExecutionEngine()
def non_dict_handler(data, **kwargs):
with probe:
return "not-a-dict"
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", non_dict_handler, True),
],
parallelism=2,
name="progress_pipeline",
)
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
self.assertFalse(result.success)
failed_step = next(s for s in pipeline.steps if s.name == "b")
self.assertEqual(failed_step.status, StepStatus.FAILED)
self.assertIsInstance(failed_step.error, ProcessingError)
# Handler ran exactly once: never re-run serially.
self.assertEqual(probe.calls, 2)
b_tid = tracking_ids.get("progress_pipeline:branch:b")
self.assertIsNotNone(
b_tid, f"expected tracking for step b, got {tracking_ids}"
)
b_stops = [
c
for c in tracker.stop_tracking.call_args_list
if (c.args[0] if c.args else c.kwargs.get("tracking_id")) == b_tid
]
self.assertTrue(b_stops)
for call in b_stops:
status = call.kwargs.get("status")
self.assertEqual(status, "failed")
self.assertNotEqual(status, "completed")
def test_same_type_steps_get_distinct_tracking_records(self):
probe = ConcurrencyProbe()
pipeline = self._build(
[
("a", "branch", branch_handler(probe, "a"), True),
("b", "branch", branch_handler(probe, "b"), True),
],
parallelism=2,
name="tracking_pipeline",
)
with patch(
"semantica.pipeline.execution_engine.get_progress_tracker"
) as mock_get:
tracker = MagicMock()
mock_get.return_value = tracker
engine = ExecutionEngine()
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
self.assertTrue(result.success, msg=str(result.errors))
submodules = [
call.kwargs.get("submodule")
for call in tracker.start_tracking.call_args_list
if call.kwargs.get("module") == "pipeline"
]
# Concurrent steps of the same step_type get distinct submodules
# (and therefore distinct tracking IDs), including pipeline identity.
self.assertIn("tracking_pipeline:branch:a", submodules)
self.assertIn("tracking_pipeline:branch:b", submodules)
class TestParallelSafeSerialization(unittest.TestCase):
"""parallel_safe must survive dict and JSON round-trips."""
def _build_pipeline(self):
builder = PipelineBuilder()
builder.add_step("extract", "source", parallel_safe=True, batch_size=10)
builder.add_step("index", "sink", dependencies=["extract"])
builder.set_parallelism(3)
return builder.build("parallel-serialization")
def test_serializer_roundtrip_preserves_parallel_safe(self):
pipeline = self._build_pipeline()
serializer = PipelineSerializer()
serialized = serializer.serialize_pipeline(pipeline, format="dict")
self.assertTrue(serialized["steps"][0]["parallel_safe"])
self.assertFalse(serialized["steps"][1]["parallel_safe"])
restored = serializer.deserialize_pipeline(serialized)
self.assertTrue(restored.steps[0].parallel_safe)
self.assertFalse(restored.steps[1].parallel_safe)
self.assertEqual(restored.config.get("parallelism"), 3)
def test_serializer_json_roundtrip_preserves_parallel_safe(self):
pipeline = self._build_pipeline()
serializer = PipelineSerializer()
serialized = serializer.serialize_pipeline(pipeline, format="json")
restored = serializer.deserialize_pipeline(serialized)
self.assertTrue(restored.steps[0].parallel_safe)
self.assertFalse(restored.steps[1].parallel_safe)
def test_builder_serialize_outputs_parallel_safe(self):
builder = PipelineBuilder()
builder.add_step("extract", "source", parallel_safe=True)
builder.add_step("index", "sink", dependencies=["extract"])
builder.set_parallelism(2)
serialized = builder.serialize(format="dict")
self.assertTrue(serialized["steps"][0]["parallel_safe"])
self.assertFalse(serialized["steps"][1]["parallel_safe"])
self.assertEqual(serialized["config"]["parallelism"], 2)
class Uncomparable:
"""Object whose equality comparison always raises TypeError."""
def __eq__(self, other):
raise TypeError("cannot compare")
class TestParallelDependencyValidation(unittest.TestCase):
"""Cyclic/unknown dependencies must fail with ValidationError in the
parallel path, not RecursionError/KeyError (review finding #1)."""
def _engine(self):
with patch(
"semantica.pipeline.execution_engine.get_progress_tracker"
) as mock_get:
mock_get.return_value = MagicMock()
engine = ExecutionEngine()
return engine
def _handler(self):
return lambda data, **kwargs: data
def test_cyclic_dependencies_raise_validation_error(self):
# Construct the pipeline directly: the builder already rejects
# cycles at build time, so bypassing it exercises the engine-level
# grouping guard (which otherwise hits RecursionError).
steps = [
PipelineStep(
name="a",
step_type="branch",
handler=self._handler(),
dependencies=["b"],
parallel_safe=True,
),
PipelineStep(
name="b",
step_type="branch",
handler=self._handler(),
dependencies=["a"],
parallel_safe=True,
),
]
pipeline = Pipeline(
name="cyclic_pipeline",
steps=steps,
config={"parallelism": 4},
)
result = self._engine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertFalse(result.success)
error_text = result.errors[0] if result.errors else ""
self.assertIn("Circular dependency", error_text)
def test_unknown_dependency_raises_validation_error(self):
# Construct the pipeline directly so the engine-level unknown
# dependency guard is exercised (instead of a builder-time
# KeyError from the grouping DFS).
steps = [
PipelineStep(
name="a",
step_type="branch",
handler=self._handler(),
dependencies=["missing"],
parallel_safe=True,
),
]
pipeline = Pipeline(
name="unknown_dep_pipeline",
steps=steps,
config={"parallelism": 4},
)
result = self._engine().execute_pipeline(pipeline, data={"text": "hi"})
self.assertFalse(result.success)
error_text = result.errors[0] if result.errors else ""
self.assertIn("unknown step 'missing'", error_text)
self.assertIn("'a'", error_text)
class TestParallelSafeMustBeBoolean(unittest.TestCase):
"""parallel_safe must be an explicit boolean everywhere (review #4)."""
def test_add_step_rejects_non_boolean_values(self):
builder = PipelineBuilder()
for invalid in ("false", "true", 1, 0, None, [True]):
with self.assertRaises(ValidationError):
builder.add_step(
"a",
"branch",
handler=lambda data, **kwargs: data,
parallel_safe=invalid,
)
def test_build_pipeline_rejects_non_boolean_values(self):
builder = PipelineBuilder()
config = {
"name": "invalid_parallel_safe",
"steps": [
{"name": "a", "type": "branch", "parallel_safe": "false"}
],
}
with self.assertRaises(ValidationError):
builder.build_pipeline(config)
if __name__ == "__main__":
unittest.main()