mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Compare commits
6
Commits
e12eec40a1
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
da642f12fa | ||
|
|
5376f046ca | ||
|
|
56d9e9a857 | ||
|
|
ecb33a5b7d | ||
|
|
100e95a098 | ||
|
|
cce5ea177c |
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.6.7] - 2026-08-28
|
||||
|
||||
### Added
|
||||
|
||||
- **First-class LangChain integration** (closes #963; recreates #969)
|
||||
@@ -20,6 +22,98 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- All adapters remain importable without langchain-core (`LANGCHAIN_AVAILABLE` flag)
|
||||
- Docs: `docs/integrations/langchain.md`, README native-integration matrix, and `docs.json` nav entry
|
||||
|
||||
- **SAP OData ingestor** (#1234, closes #1228) by @pkupt
|
||||
- New `SAPODataEntity` / `SAPODataConnector` / `SAPIngestor` (`semantica.ingest`, lazy exports), following the three-layer connector pattern already used for Snowflake/Databricks, to pull master/transactional data (Business Partners, Sales Orders) from SAP OData v2/v4 services into the Context Graph
|
||||
- Dual auth (OAuth2 client-credentials for BTP/S4HANA Cloud, Basic for on-prem NetWeaver); every outbound request, including the token exchange, routes through `request_with_ssrf_guard`
|
||||
- `$metadata` (CSDL XML) is parsed with a hand-rolled `xml.etree` reader rather than pulling in `pyodata`; pagination follows OData v2 `__next`/`__deferred` and v4 `@odata.nextLink`
|
||||
- New `pip install semantica[ingest-sap]` extra (`requests>=2.28.0`)
|
||||
- **Known phase-1 limits** (documented in docstrings): the OAuth2 token is cached but never refreshed, and pagination has no `max_pages` fuse (`top` bounds it when supplied)
|
||||
- New `tests/ingest/test_sap_ingestor.py`: 22 tests (auth, EDMX parsing, v2/v4 pagination, SSRF routing, error paths, service-root normalization)
|
||||
|
||||
- **`ContextGraph` gains deterministic, human-editable Markdown round-trip persistence** (#852) by @SaurabhScripts
|
||||
- `save_to_markdown()`/`load_from_markdown()` write one file per node plus a graph manifest, so a graph can be reviewed and hand-edited outside the application without giving up the existing JSON API or its default behavior
|
||||
- An existing destination is validated as a complete, canonical managed export before atomic replacement, so the loader can't silently clobber an unrelated or manually-extended directory
|
||||
- Import/export paths and their ancestors reject symlinks, Windows junctions, and other reparse points, with pre-open and post-open validation — the same hardening applied to `AgentMemory`'s existing Markdown import in the companion fix below
|
||||
- Dangling edge endpoints import as JSON-compatible entity stubs rather than being rejected outright (matching what the JSON loader already accepts); node/edge indexes, adjacency, and analytics/retraction/tombstone state are rebuilt after a Markdown load, and granular node/edge events are still emitted so temporal audit history stays useful
|
||||
- New `tests/context/test_context_graph_markdown.py`: 29 passed, 1 skipped (the skipped case creates a real Windows junction and runs on Windows CI); full `tests/context/` suite: 614 passed, 1 skipped
|
||||
|
||||
- **Explorer graph inspector gains a read-only Markdown content viewer** (#1078, closes #900) by @sakshi04-ui — Preview (rendered GFM) and Source (exact, whitespace-preserving) tabs for node content, with a copy-to-clipboard action. A URL allowlist restricts links to `http:`/`https:`/`mailto:`/in-document anchors, raw HTML execution is disabled, and external links carry `rel="noopener noreferrer"`. A first, focused step toward human-editable memory (#765); no write path yet. New `explorer/tests/markdownContentViewer.test.ts`: 8 tests
|
||||
- **Follow-up (perf)** (#1195, addresses #1118) by @pravit-amp: `remarkPlugins` and the ~20-entry renderer `components` map were inline literals, so every unrelated re-render (e.g. clicking Copy) re-ran the full remark parse and remounted the whole subtree — up to 1.1s of main-thread block on a 2000-row GFM table. Both are now hoisted to module scope and the rendered element is memoized on content, cutting re-render cost from as much as 1121ms to ~0.1ms across all measured fixtures with no change to rendered output. A separate, upstream `remark-gfm` table-parse cost (~O(n^1.9), not fixed here) is left open on the issue as a product decision
|
||||
- **Follow-up (cleanup)** (#1194, closes #1119) by @pravit-amp: the pure `isSafeUrl` URL-safety helper is extracted out of `MarkdownContentViewer.tsx` into its own `markdownUrlSafety.ts` module (behavior-preserving — moved verbatim), so the component module exports only components and stops tripping `react-refresh/only-export-components`
|
||||
|
||||
- **`reasoning` gains a structured Action layer — rule-driven side effects with optional provenance** (#1096, closes #1095) by @cxzg007 — `AssertAction`/`RetractAction`/`CallAction`/`EmitEventAction` let a matched rule write facts back to a `KnowledgeGraph`, retract facts, call a structured handler (replacing the previously-unused `Rule.handler`), or emit to a sink registered via `Reasoner.on_event`, turning the reasoner from a pure inference engine into a production-rule system. With `provenance=True`, fired actions are recorded to `Reasoner.action_log`. Fully additive — rules without `actions` are unaffected, and the legacy `handler` field still fires (now wrapped internally as a `CallAction`). Also fixes a latent dangling import in `reasoning_provenance.py` (`ReasoningEngine`/`infer` → `Reasoner`/`infer_facts`). New `tests/reasoning/test_rule_actions.py`: 9 tests; full `tests/reasoning/` suite: 54 passed
|
||||
|
||||
- **`run_shacl_validation` is now a public, documented entry point** (#1189, closes #1186) by @mikemikimike — the SHACL guide had documented the private `_run_pyshacl` helper as the canonical API; it's now exposed through `semantica.ontology`, with `_run_pyshacl` kept as a compatibility alias over the same implementation. `tests/ontology/test_ontology_advanced.py`: 33 passed (also fixes a flaky comparison against pySHACL's non-deterministic blank-node shape identifiers by comparing stable report fields instead)
|
||||
|
||||
- **`docs/storage-backends.md`: adapter inventory and RDF/LPG feature matrix** (#899, addresses #888) by @yulinlina — which graph storage backends are built-in vs. bring-your-own, and where provenance/context support is partial
|
||||
- **`docs/guides/shacl-validation.md`: documented that `rdfs:range` + RDFS entailment makes `sh:class` unfalsifiable** (#1182, fixes #1130) by @ALDRIN121 — with entailment on, pyshacl infers the declared range class onto every object, so a `sh:class` constraint can never fail and reports `conforms: True` on non-conforming data; added to Common Pitfalls with the `inference="none"` vs `inference="rdfs"` contrast and guidance to re-run `sh:class` shape sets with entailment off before trusting a pass
|
||||
- **Cookbook: four new module notebooks** — `22_Provenance_Tracking.ipynb` (#989, lineage walks, revision history, invalidation, checksums), `23_Reasoning.ipynb` (#990, `Reasoner`/`DatalogReasoner`/`ExplanationGenerator`), `24_Change_Management.ipynb` (#991, versioned snapshots, named tags, checksum tamper-detection), and `25_Seed_Data.ipynb` (#992, bootstrapping a foundation graph from a trusted CSV source) — all by @LeonSGP43, filling gaps where the corresponding module shipped a usage doc but no runnable tutorial; every cell verified against current module source. `docs/cookbook.md` index entries for all four added in #1225
|
||||
- **README "Cite Us" section and `docs/citation.md` cross-link** (#1210) by @KaifAhmad1 — BibTeX/APA/MLA/Chicago/IEEE citation forms; also corrects the copyright holder in `LICENSE`/`docs/project-license.md` from the stale "Hawksight AI" to "Semantica" and replaces the retired `Hawksight-AI` GitHub org slug with `semantica-agi` across ~40 files (READMEs, issue templates, plugin manifests, cookbook notebooks, docs)
|
||||
|
||||
### Changed
|
||||
|
||||
- **A registered custom method can now refuse, instead of being silently overridden by the default implementation** (#1127, closes #1108) by @fabio-rovai — every module supporting custom methods wrapped the registered callable in a `try`/`except` that logged a warning and ran the built-in default on *any* exception, including one a validator or policy gate raised on purpose to say "do not produce this output." That made every registered gate advisory rather than authoritative. `semantica/utils/custom_methods.py` now centralizes the policy: an exception from a registered method propagates to the caller by default; `fallback_on_custom_error=True` restores the previous warn-and-continue behavior per call. Applied mechanically across all 58 call sites in `export/`, `ingest/`, `normalize/`, `parse/`, `embeddings/`, and `kg/` methods modules. New `tests/utils/test_custom_method_can_refuse.py`: 13 tests, including the reported gate-deletes-and-raises scenario and a guard that no call site still swallows
|
||||
- **Removed 13 confirmed-dead symbols across 9 files** (#1176, closes #1174) by @Vinv-AI — private helpers and Explorer app-layer code with zero callers in code, tests, or docs, none part of the public API or a FastAPI `response_model`; 289 deletions, no behavior change
|
||||
- **Consolidated the two duplicate Turtle/N-Triples literal escapers in `rdf_exporter.py`** (#1221, closes #1218) by @pkupt — `_escape_turtle_literal` (added in #1148) escaped the same five characters in the same order as the older module-level `_escape_literal`; the redundant one is dropped and all four call sites route through the original. Behavior no-op, verified against the full export suite (301 passed, 1 skipped)
|
||||
- **Removed the unreachable `_extract_with_spacy()` method and the unused `self.nlp` attribute from `NERExtractor`** (#1220, fixes #1058) by @yunaremaia — the ML dispatch path has always gone through `methods.py`'s process-level model cache instead; `__init__` still validates the spaCy runtime up front but no longer eagerly loads a model nothing on the instance reads
|
||||
- **Cleaned up an unused `sys` import and import ordering in `semantica/worker.py`** (#1061) by @aoright
|
||||
- **Test-only contributions**: isolated `sys.modules` mock leakage between `tests/visualization/` files so the suite passes in any collection order (#897, closes #859, by @luantaraschi); added coverage for 4 previously-untested `ConflictResolver` strategies and 3 `ConflictDetector` conflict types (#902, fixes #865, by @Devansh070); added a regression test tracking relationship provenance through `ProvenanceManager` (#1071, closes #1055, by @dex0shubham); added `max_tokens`-propagation regression coverage for LLM extraction methods, later folded into the cache-key fix below (#925, by @saiganesh47)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`SPARQLReasoner.execute_query()` claimed to run a query but always returned an empty result** (#1087, fixes #1083) by @ALDRIN121 — both the store-configured and unconfigured branches returned an empty `SPARQLQueryResult` with no real execution behind it, so a caller trusting "no matches" (e.g. a compliance check) could draw a false-negative conclusion from a method that never actually queried anything. Until a real triplet-store execution path lands, it now raises `NotImplementedError` explaining why, and the dead cache/inference scaffolding after the unreachable execution point is removed. 3 new regression tests
|
||||
- **`DuplicateDetector` merged entities that share no identifier, type, or name** (#1149, fixes #1137) by @pkupt — `_create_duplicate_candidate()` only ever boosted confidence for matching types and never penalized a mismatch, so two sparse, differently-typed entities (e.g. a `Person` and an `Organization`) could land above the merge threshold and collapse into one node, silently dropping the second. Two non-empty, differing types are now never a duplicate candidate. `tests/deduplication/`: 92 passed
|
||||
- **`TemporalGraphQuery.analyze_evolution()`'s `stability` metric was a hardcoded placeholder** (#1143, closes #1142) by @cxzg007 — every bounded relationship contributed a constant `1`, so `stability` was always `1.0` or `0` regardless of how long relationships actually stayed valid. Now computes the mean valid-time duration in seconds across relationships with both `valid_from`/`valid_until` set; unbounded/half-open intervals are skipped and negative intervals clamp to zero. 3 new tests in `tests/kg/test_kg.py`
|
||||
- **CodeQL false-positive on a JSON-LD test's URL check** (#1183) by @KaifAhmad1 — `"https://schema.org/" in flattened` pattern-matched CodeQL's substring-sanitization heuristic even though `flattened` is always a `list` (exact membership, no sanitization or SSRF path involved); rewritten as an explicit `any(entry == ... for entry in flattened)` with identical behavior
|
||||
- **HuggingFace NER extraction crashed on `huggingface_model` being forwarded as an unexpected pipeline loader kwarg** (#1188, fixes #1063) by @shahzaib-ahmadcs — while preserving genuinely supported pipeline kwargs like `aggregation_strategy`. 5 tests pass
|
||||
- **JSON-LD document/graph `@id` was minted from the wall clock, so re-exporting an unchanged graph produced a new subject every time** (#1181, closes #1147) by @reddynitish — merging repeated exports duplicated graph identity instead of recognizing them as the same graph. The `@id` is now content-derived, with optional `graph_uri`/`document_uri` overrides for callers with a stable graph name; `semantica:exportedAt` still records export time separately. Applies to both JSON-LD export paths
|
||||
- **`ContextGraph.get_causal_chain()` only matched the canonical uppercase causal-edge spellings, silently missing edges recorded in `CausalChainAnalyzer`'s present-tense vocabulary** (#1187, fixes #1184) by @ALDRIN121 — `causes`/`influences`/`precedes` differ from `CAUSED`/`INFLUENCED`/`PRECEDENT_FOR` in word form, not just case, so an edge recorded with the analyzer's spelling produced an empty audit chain — silent, and in the dangerous direction for a compliance trace. `add_causal_relationship()` now normalizes through an alias map before storing the canonical form; traversal accepts the union vocabulary. 2 new regression tests, full `tests/context/` suite: 587 passed
|
||||
- **`semantica embed generate` corrupted its own output and could recurse into a stack overflow** (#996/#1004/#1005, closes #994) by @varunsahni18, @yzxcj797 — three compounding defects in one pipeline. (1) `generate_embeddings`/`embed_text`/`calculate_similarity`/`pool_embeddings` all registered themselves as their own custom-method-registry default, so an unqualified call (exactly what the CLI does) re-entered the same wrapper until Python's recursion limit; each of the four dispatch sites now guards on registry identity before recursing (#996, #1005). A second self-recursion in `EmbeddingGeneratorWithProvenance.__getattr__` (re-entering itself when `_generator` is unset, e.g. during a `deepcopy` probe) now raises a normal `AttributeError` for private names instead (#1005). (2) `--output embeddings.parquet` wrote `json.dumps(result, default=str)` regardless of extension, turning a numpy array into its plain-text `repr()` — a file `embed index` then failed to open as Parquet; the writer now detects `.parquet`/`.json`/`.jsonl` and produces real Parquet/JSON, rejecting any other extension with a clear message (#996, #1004). (3) `pyarrow` was only in optional extras despite being required by the documented quick-start flow; promoted to a core dependency (#996)
|
||||
- **`AgentMemory`'s existing Markdown import accepted symbolic links, NTFS junctions, and other Windows reparse points** (#851) by @SaurabhScripts — a direct linked import path is now rejected with an actionable error, and a linked entry found inside an otherwise-valid directory is skipped rather than aborting the whole import; hardened with pre-open/post-open checks, `O_NOFOLLOW` where available, and `fstat`-based regular-file validation. `tests/context/`: 595 passed, 1 skipped (Windows-junction test, runs on Windows CI)
|
||||
- **A caught vector-similarity scoring exception left stale partial state behind, risking a misleading match on the next call** (#885, fixes #875) by @ArmanGrewal007 — the exception is now logged at debug level and `vector_score`/`vector_idx` reset to neutral values before the remaining matching stages continue
|
||||
- **`RDFExporter` could write invalid or unintended relative IRIs for `GraphBuilder`-default entity/relationship identifiers** (#1112, closes #1099) by @mikemikimike — normalization is now applied at the RDF export boundary across Turtle (including temporal Turtle), RDF/XML, and N-Triples: bare/relative identifiers are minted under the Semantica namespace with safe percent-encoding, absolute IRIs pass through unchanged, and configured/input-context prefixes expand through the effective namespace mapping. 38 focused regression tests; 175 export tests plus 46 subtests pass
|
||||
- **`RDF4JStore`'s `repository_id` constructor argument had no effect** (#1192, closes #1191) by @Freakz2z — the explicit id is now honored when selecting the repository; stale documentation caveats claiming otherwise are removed. 65 tests pass across the affected triplet-store suites
|
||||
- **Non-interactive stdout (piped/redirected output, CI logs) was flooded with progress-bar escape sequences** (#1193, fixes #1185) by @ALDRIN121 — a plain `python demo.py > out.txt` captured 173 bytes of progress noise around 10 bytes of real output. `ProgressTracker` now attaches its console display only for an interactive terminal, Jupyter, or the new `SEMANTICA_FORCE_PROGRESS` opt-in (following the `NO_COLOR`/`FORCE_COLOR` convention); file-based progress logging is untouched. Both switches are now documented in the README and `docs/reference/utils.md`. 11 tests pass (6 new)
|
||||
- **Entity `metadata` was dropped by every RDF serializer except the JSON-LD path**, so an entity kept its confidence but lost its source document, page, extractor, and reviewer on Turtle/N-Triples/RDF/XML/`RDFExporter`'s own JSON-LD (#1165, closes #1154) by @fabio-rovai — Semantica's own metadata keys (`num_entities`, `snapshot_time`, Neo4j loader fields, etc.) are now mapped to declared vocabulary terms and carried through on every path; a caller-supplied key with no mapped term is skipped with an explicit warning (rather than silently vanishing) naming the override needed, pending the caller-key namespace decision tracked in #1146. 21 new tests in `tests/export/test_metadata_passthrough.py`; `tests/export`+`tests/ontology`: 274 pass
|
||||
- **`extract_relations_llm` silently dropped caller-supplied generation parameters** (`max_tokens`, `top_p`, `seed`, etc.), and the extraction cache didn't distinguish calls made with different generation settings (#1213, with test coverage from #925) by @Sameer6305 — a small hardcoded allowlist forwarded only `temperature`/`verbose` to `generate_typed`, discarding the rest; fixed by forwarding all caller kwargs. Once forwarded, those parameters also needed to enter the cache key, since two calls differing only in `max_tokens` previously shared one cache entry and the second could silently reuse a result generated under the first's settings — now applied consistently across entity, relation, and triplet LLM extraction. New regression tests for cache bypass/reuse under differing `max_tokens`/`temperature`
|
||||
- **`OxigraphStore` silently ignored the `storage_path` constructor argument and never flushed writes before a reopen**, both causing silent on-disk data loss (#970) by @logan-jl-cc — `__init__`'s parameter is named `path`, so the project-conventional `storage_path` landed in `**config` and was ignored, degrading a supposedly-persistent store to in-memory with no error; `storage_path` is now accepted as an alias. Separately, pyoxigraph's background flush can lag behind a write, so a reopen immediately after `add_triplets` could observe fewer triples than were written; writes to an on-disk store now call `flush()` explicitly. 2 new regression tests, full suite: 9 passed
|
||||
- **MCP server's `export_graph` tool was broken on every output format** (#1151) by @Arasz — the `json` branch called `JSONExporter().export()` without the `file_path` it requires, and every RDF branch passed a `ContextGraph` object where the exporters expect the canonical kg dict, both surfacing as a raw exception string. A third bug compounded both: the RDF export path's progress bar wrote to stdout, which over stdio MCP *is* the JSON-RPC framing, corrupting the protocol and hanging the client (a 300s timeout on an empty graph). Fixed by converting through `ContextGraph.to_kg_dict()`, serializing the JSON branch to match the RDF branches' string contract, and forcing `SEMANTICA_DISABLE_PROGRESS=1` for the server process. 5 new tests, verified failing against 0.6.6 beforehand
|
||||
- **`OntologyIngestor` dropped every class and property from a JSON-LD document using a named graph** (#1156, fixes #1129) by @13g4d0 — a top-level `@id` beside `@graph` names the graph, and `rdflib.Graph.parse()` silently loads only the default graph, discarding the rest; `POST /api/ontology/load` returned `status: "success"` with `class_count: 0`. Now parses into a `Dataset` and flattens all quads into the working graph (the same `Graph`→`Dataset` migration #757 made for `JenaStore`, extended to the ingest path). On the PR's real-world reproduction: 25 triples/1 subject before, 719 triples/45 classes/40 object properties after. 4 new tests including a default-graph canary so the fix can't trade one blind spot for another
|
||||
- **Turtle and N-Triples RDF export interpolated entity `text` into string literals with no escaping**, so a `"`, backslash, newline, CR, or tab in the source text emitted invalid RDF other parsers rejected (#1148, closes #1098) by @pkupt — a shared `_escape_turtle_literal()` (later consolidated in #1221) now escapes per the RDF 1.1 Turtle grammar and is reused for the N-Triples path, which previously escaped only quotes and newlines. `tests/export/`: 161 passed, 1 skipped
|
||||
- **`PipelineSerializer` round trips dropped step dependencies and delta-processing metadata, and could rehydrate a legacy stringified handler as a non-callable string** (#1217, fixes #1216) by @cxzg007 — step dependencies, delta mode, and base/target version IDs are now restored from the serialized schema; runtime handler callables are treated as process-local state and excluded from serialized business configuration rather than (mis)serialized. 52 tests pass
|
||||
- **`PipelineBuilder` never actually dispatched to a handler registered by `step_type`**, and a serialize/deserialize round trip could leak `handler`/`dependencies` into a step's business config (#1215, fixes #1214) by @cxzg007 — a registered handler is now resolved by `step_type` when no explicit `handler=` is supplied (explicit handlers still take precedence), and the two builder-control fields are kept out of `PipelineStep.config` so a strict handler signature can't receive them as unexpected kwargs. `tests/core`+`tests/pipeline`: 50 passed
|
||||
- **`PipelineBuilder.set_parallelism()` was accepted and stored but never read — pipeline steps always ran strictly sequentially**, and the setting didn't survive a serialize/deserialize round trip (#1226, fixes #1223) by @cxzg007 — wired through builder → serializer → execution engine, plus a new opt-in `PipelineStep.parallel_safe` flag. A dependency layer now runs in parallel only when every step in it is marked `parallel_safe`, the layer has more than one step, the input is dict-typed, and no step is in delta mode; otherwise it falls back to sequential execution. Each parallel step's input is deep-copied for isolation, execution is bounded by `ThreadPoolExecutor(max_workers=min(configured parallelism, max_workers))`, a failure cancels pending futures in the layer, and layer results merge back in declaration order (a same-key conflict raises `ProcessingError`). 22 new tests in `tests/pipeline/test_pipeline_parallel.py`
|
||||
- **`Config.get()` silently dropped boolean environment-variable overrides** (#1038, fixes #1035) by @Kyou12138 — the type dispatch checked `isinstance(default, int)` before `isinstance(default, bool)`, and since `bool` subclasses `int` in Python, the bool branch was unreachable: `CONFLICT_ZZTESTFLAG=true` with a `False` default returned `False`, and `=1` returned the int `1` rather than `True`. Bool is now checked first (with whitespace stripped before parsing truthy/falsy spellings), fixed across all ten affected config modules (`conflicts`, `deduplication`, `split`, `embeddings`, `export`, `ingest`, `kg`, `parse`, `ontology`, `normalize`). 12 new tests plus 6 existing conflicts tests and 131 related module tests pass
|
||||
- **Scanned (image-only) PDFs parsed with no error and no warning, returning empty text with a "completed" status** (#1021, closes #1020) by @shanyu910 — `PDFParser._parse_page` swallowed a missing text layer via `page.extract_text() or ""`, so the failure only surfaced far downstream as zero extracted entities. A warning now fires when every parsed page yields no text with `extract_text` enabled, pointing at `parse_pdf(..., method="docling", enable_ocr=True)`. Also fixes a separate `import semantica.parse` failure on a fresh interpreter (`email_parser.py` used `email.message.Message` without importing `email.message`) that was blocking the parse test suite from even collecting. 25 tests pass in `tests/parse/`
|
||||
- **`GET /api/decisions` returned HTTP 422 for any graph containing real decisions**, breaking the Explorer Decisions workspace entirely (#937) by @logan-jl-cc — `record_decision()` stores the timestamp as a POSIX float, but `DecisionResponse.timestamp` is typed `Optional[str]` and Pydantic's strict mode rejected the coercion. Fixed by coercing to `str` (preserving `None`) at the response-adapter boundary
|
||||
- **Decision persistence/query bugs, CJK text handling, and three missing MCP graph tools** (#967) by @toratto — `mcp_server`'s `_get_graph` called a non-existent `graph.load` instead of `load_from_file`, so `SEMANTICA_KG_PATH` was silently ignored and the server always started with an empty graph; `query_decisions` read `category` from the wrong field, always returning nothing for a category filter; `find_precedents`/`query_decisions(query=)`'s similarity threshold was too high for short CJK queries, which also failed outright because `_calculate_decision_content_similarity`'s whitespace-Jaccard fallback is always zero for languages with no whitespace tokenization (now falls back further to a character-bigram overlap coefficient); `load_from_file` didn't rebuild the in-memory decision/entity/temporal indexes after loading, breaking `find_precedents_by_scenario` and decision counts post-reload; `extract_entities`/`extract_relations` returned the spaCy type label as `text` and dropped the actual entity text, and had no way to select a non-English NER model. Also adds three new MCP tools (`query_graph`, `update_node`, `delete_node`, the latter two persisting back to `SEMANTICA_KG_PATH`)
|
||||
- **`sqlalchemy.text` was used but never imported in two `DBIngestor`/`DataExporter` methods**, raising `NameError` on every call before any query reached the database (#1017, closes #1015) by @pravit-amp — `connect()`/`test_connection()` imported `text` function-locally, so the binding never reached `export_table_data()` or `execute_query()`, which called it anyway; both raised immediately, re-wrapped by an `except Exception` into a `ProcessingError` that read like a database fault rather than a missing import. `docs/guides/ontology.md` documents `DBIngestor().execute_query()` as a supported entry point, so documented usage walked straight into it. 5 new tests against a temporary SQLite database, also repairing a previously-failing `tests/ingest/test_notebook_02.py` case
|
||||
- **Ontology generation resolved relationship endpoint types incorrectly, producing wrong object-property domains/ranges** (#1170, closes #1168) by @T1mn — endpoint types are now resolved from the canonical `source_id`/`target_id` fields and supported aliases instead of defaulting to the first entity when a field was missing, preventing e.g. a `Person -> Organization` relationship from generating a `Person -> Person` property. 80 tests pass, 1 skipped
|
||||
- **Ontology property generation dropped data properties when a raw entity type was normalized into a class name** (#1171, closes #1169) by @T1mn — e.g. `software engineer` → `SoftwareEngineer` lost its `email` property; attributes are now grouped by matching raw, normalized, and recorded class names, so the normalized class stays each property's domain. 79 tests pass, 1 skipped
|
||||
- **`flatten_dict()` silently dropped data when a top-level key already containing the separator collided with a key produced by flattening a nested dict** (#1012, fixes #1010) by @yzxcj797 — `{"a.b": 1, "a": {"b": 2}}` flattened to `{"a.b": 2}` with no error, the `1` simply gone; collisions are now detected (unique-key count vs. item count) and raise `ValueError` naming the colliding key before data is lost. 6 new tests
|
||||
- **Creating relationships after `GraphStore.add_edges`/`build_from_entities_and_relationships` silently produced zero edges against ID-minting backends** (#1173, fixes #1136) by @yzxcj797 — an id-space mismatch across three layers: `add_edges` reads application-level string ids and passes them to `create_relationship`, which is a pure passthrough into `Neo4jStore.create_relationship`'s `MATCH ... WHERE id(a) = $start_id` — a Neo4j-internal integer id. Every node was created and every relationship silently failed with one easily-missed warning per edge. `GraphStore` now keeps an application-id→internal-id map, populated by `add_nodes`/`create_node` from the backend's own creation results and consulted by `create_relationship`; unknown ids and identity-mapped backends are unaffected. `tests/graph_store/`: 100 passed
|
||||
- **RDF export left `semantica:text`/`rdfs:label` empty for entities that only carry a `name` field**, across all four RDF formats (#1113, fixes #1097) by @cxzg007 — `RDFSerializer.convert_kg_to_rdf()` already implemented the `name`→`label`/`text` normalization, but `export_to_rdf()` never called it. Now called once at the export boundary (idempotent, non-destructive, falls back to a label derived from the id suffix). 7 new tests, `tests/export/test_rdf_exporter.py`: 17 passed
|
||||
- **Docker Explorer image failed to build on Python 3.14** — `gensim` has no prebuilt wheel for it and the slim base has no `gcc` to build from source (#1172, closes #1025) by @DwitiThaker — runtime pinned to `python:3.13-slim`, where `gensim` installs from a prebuilt wheel
|
||||
- **Unit normalization rejected common aliases before conversion** — `kg`, `g`, and other abbreviated/plural unit spellings failed category validation and the conversion-factor lookup ahead of it (#939) by @Mr-Neutr0n — aliases now normalize first; canonical aliases added for feet, yards, miles, and gallons. 7 tests pass
|
||||
- **An oversized, caller-controlled mapping key could blow up a `ValidationError` message to megabyte scale**, and equally inflate application logs on repeated malformed input (#1088, fixes #1001) by @ALDRIN121 — follow-up to the graph-payload validation added in #958. The displayed key is now truncated at 64 characters with an ellipsis; the underlying input and validation decisions are unchanged. 4 new tests
|
||||
- **`SeedDataManager.load_from_api()` mislabeled genuine connection failures as a missing `requests` dependency** (#972, closes #949) by @pravit-amp — `requests.exceptions.RequestException` (connection errors, timeouts, `raise_for_status()` failures) subclasses `OSError`, so an `except (ImportError, OSError)` block written to guard a lazy import that no longer existed (`requests` is a core dependency) caught real failures too and told users to reinstall an already-installed library while dropping the original exception chain. The block is removed; genuine failures now surface through the existing `Failed to load from API: {e}` path with `from e` intact. 5 new regression tests
|
||||
- **`SHACLGenerator` produced shapes that matched nothing, and pySHACL reported `conforms: True` on data that plainly violated them** (#1124, closes #1104, closes #1105) by @fabio-rovai — `base_uri` was used both as where shape resources live and to expand every `sh:targetClass`/`sh:path`, so with the default shapes namespace, generated shapes targeted classes no data graph in the package actually uses; a shape with zero matching focus nodes is vacuously satisfied, so validation silently passed regardless of real violations. The target namespace now resolves independently (explicit argument → ontology's declared namespace → an existing absolute class/property IRI → ontology `uri` → the vocabulary namespace), never the shapes namespace. Separately, `_attach_property_shapes` attached a domain-less property's constraint to *every* shape ("no domain declared, attach to all"), asserting a constraint the ontology never stated; a domain-less property is now left unattached by default, with `attach_domainless_properties=True` to restore the old behavior. 17 new tests validate real data through pySHACL rather than reading shape text; `tests/ontology`+`tests/export`: 239 passed
|
||||
- **OWL export dropped every generated property and collapsed distinct classes onto one node** (#1123, closes #1103) by @fabio-rovai — `OWLExporter` reads `object_properties`/`data_properties`, but `OntologyGenerator` emits one combined `properties` list, so every property was silently discarded; separately, a class built without a namespace manager gets `"uri": None`, which a `"uri" not in cls"` guard never catches (the key is present), so the exporter wrote a relative `<>` IRI for it — resolved by rdflib against the current working directory, meaning two classes could collapse onto one subject and that subject's identity changed with the export's working directory. Both dict shapes are now merged and classified correctly, and a class/property IRI resolves through `uri`→`iri`→`id`→a name joined onto the ontology base, skipping (with a warning) a term with none of those instead of minting `<>`. 10 new regression tests parse the real output with rdflib and Oxigraph; `tests/export`+`tests/ontology`: 231 passed
|
||||
- **Confidence scores serialized as four different, mutually-disagreeing RDF terms depending on export format, and one non-numeric confidence value could break an entire Turtle export** (#1125, closes #1100, closes #1102) by @fabio-rovai — Turtle wrote a bare `xsd:decimal`, N-Triples an explicit `xsd:float`, RDF/XML an untyped plain literal, and JSON-LD's native number expanded to `xsd:double`; loading a Turtle and an N-Triples export of the same graph into one store gave the same entity two different confidence values. Separately, an unparseable confidence (e.g. the string `"high"`) was interpolated into Turtle with no validation, producing a syntax error that dropped every entity from the export. All four paths now write one canonical `xsd:decimal` lexical form (matching the pre-existing Turtle behavior and the only exact representation of the four); an unusable value is omitted with a warning instead of corrupting the document. The vocabulary's `sem:confidence` now declares `xsd:decimal` (previously left undeclared to avoid contradicting the disagreeing exporters). 20 new tests compare parsed graphs across all four formats; `tests/export`+`tests/ontology`: 240 passed
|
||||
- **An OWL-Time validity interval was reified onto a relationship IRI the graph never actually referenced**, making it unreachable from the edge it described (#1126, closes #1106) by @fabio-rovai — a relationship serializes as a single triple with no node of its own, so `include_temporal=True` minted a well-formed `time:Interval` with zero inbound arcs to its subject. Turtle now also emits the `sem:Relationship`/`sem:source`/`sem:target`/`sem:type` reification the JSON-LD path already produced, but only when there's temporal data to attach — default and `include_temporal=False` output are byte-for-byte unchanged. 7 new tests include a SPARQL walk from the edge to its interval, the path the dangling node made impossible; `tests/export`+`tests/ontology`: 228 passed
|
||||
- **JSON-LD exports were unreadable by Semantica's own default parser** (#1145, fixes #1144) by @fabio-rovai — every export was written as a named graph (a top-level `@id` beside `@graph`), which a plain `rdflib.Graph.parse()` silently discards in favor of the (empty) default graph; a two-entity graph parsed as 2 triples instead of 20. Compounded by `export_knowledge_graph` converting its payload to JSON-LD and then handing the *already-converted* document to `export()`, which converted it again, producing two `@context` blocks and two document nodes. Metadata now attaches beside `@graph` rather than naming it, and a payload that already declares `@context` is merged rather than re-wrapped. 9 new tests parse with both `Graph()` and `Dataset()` and assert identical counts; full-suite failure set unchanged before/after (539/539)
|
||||
- **`GraphBuilder` didn't propagate entity-resolution's merged ids into the `source_id`/`target_id` relationship aliases**, only `source`/`target` (#1115, closes #1110) by @T1mn — a relationship's alias fields could still point at a pre-merge id after resolution. Both alias pairs are now kept in sync. 9 tests pass
|
||||
- **`GraphValidator` indexed entities only by `id`, rejecting graphs that use the `entity_id` alias as invalid even when their relationships were fine** (#1116, closes #1111) by @T1mn — validation and endpoint checks now go through the shared `get_entity_id()` helper, accepting both fields consistently. 5 tests pass
|
||||
- **Broken star history chart in README** (#1057) by @OctoBored — the embedded chart used the GitHub stargazer API, now access-restricted; switched to a token-free alternative data source
|
||||
|
||||
### Security
|
||||
|
||||
- **Agno's `AgnoKnowledgeGraph.load_urls()` made outbound requests with no SSRF protection beyond a scheme check** (#1212) by @Sameer6305 — caller-supplied URLs went straight to `urllib.request.urlopen()`, unguarded against loopback/private addresses, cloud metadata endpoints (`169.254.169.254`), IPv6-internal addresses, hostnames resolving to private space, or redirects into any of the above. Found during a project-wide SSRF audit following #936/#959. Now routed through the shared `request_with_ssrf_guard()`; an unsafe URL is skipped rather than aborting the rest of the ingestion batch. `OpenClawKGTool` (operator-configured, intentionally allowed to target `localhost` for local deployments) gains scheme/malformed-URL validation as defense in depth, without restricting its legitimate private-network use case. 29 new Agno tests, 26 new OpenClaw tests, all passing alongside the 15 pre-existing Agno integration tests
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Routine version bumps with no application-facing behavior change: `anthropic` 0.121.0→0.122.0 (#1045), `botocore` 1.43.69→1.43.73 (#1047), `agno` 2.8.7→2.9.0 (#1050), `google-genai` 2.17.0→2.18.1→2.19.0 (#1163, #1205), `lxml` 6.1.1→6.1.2 (#1197), `charset-normalizer` 3.5.0→3.5.1 (#1201), `pypickle` 2.0.1→2.0.2 (#1203)
|
||||
|
||||
## [0.6.6] - 2026-08-20
|
||||
|
||||
### Added
|
||||
|
||||
@@ -142,7 +142,7 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
|
||||
```bash
|
||||
semantica doctor
|
||||
# Python 3.11.9 pass
|
||||
# semantica 0.6.6 pass
|
||||
# semantica 0.6.7 pass
|
||||
# faiss vector store pass
|
||||
# Config file pass ~/.semantica/config.yaml
|
||||
```
|
||||
@@ -1463,18 +1463,18 @@ For contributor / dev-server setup: **[explorer/README.md: Local Setup Guide](ex
|
||||
|
||||
---
|
||||
|
||||
## What's New in v0.6.6
|
||||
## What's New in v0.6.7
|
||||
|
||||
**Security release — upgrading is strongly recommended.** Fixes for a privately disclosed batch of vulnerabilities spanning backup/restore, database export, outbound requests, and triplet-store backends, plus SSRF hardening across ingestion:
|
||||
**Feature release**, plus one SSRF hardening fix and a large batch of correctness fixes across the RDF/ontology export pipeline:
|
||||
|
||||
- **Tarball restore path traversal**: `semantica backup restore` now validates every archive member for path containment and rejects symlink/hardlink escapes before extraction
|
||||
- **Latent SQL injection in `DataExporter.export_table_data()`**: table/schema names are now identifier-allowlisted and `where`/`order_by` fragments are blocklist-checked
|
||||
- **DNS-rebinding TOCTOU in the shared SSRF guard**: the resolved IP that passes validation is now the one the connection is pinned to, closing the check-then-use race (also closes the `100.64.0.0/10` CGNAT gap)
|
||||
- **Stored XSS in HTML report generation** and **unvalidated SPARQL object IRIs in AnzoStore** (SPARQL injection): both now escape/validate before interpolation
|
||||
- **`Authorization`/`Proxy-Authorization` credential leakage across redirects**, plus **SSRF gaps in `FeedIngestor`/`FeedMonitor`, `RepoIngestor`, and the MCP/public-API ingest paths**: all now route through the shared, redirect-safe SSRF guard
|
||||
- **HTTP response header injection and an unbounded-memory DoS** in the Explorer API, and a **`fastapi`/`python-multipart` ReDoS** (PYSEC-2024-38): floors raised, inputs sanitized, candidate pools capped
|
||||
- **First-class LangChain integration** (`semantica[langchain]`): a `BaseRetriever` and `VectorStore` over `HybridSearch`, plus graph/decision-query tools
|
||||
- **SAP OData ingestor** (`semantica[ingest-sap]`): OAuth2/Basic-auth, SSRF-guarded ingestion for Business Partners and Sales Orders, following the existing Snowflake/Databricks connector pattern
|
||||
- **`ContextGraph` gains deterministic, human-editable Markdown round-trip persistence** alongside the existing JSON API, and the Explorer graph inspector gains a read-only Markdown content viewer
|
||||
- **`reasoning` gains a structured Action layer**: rule-driven `Assert`/`Retract`/`Call`/`EmitEvent` actions with optional provenance, turning the reasoner into a production-rule system
|
||||
- **`run_shacl_validation` is now a public, documented API**, and a dozen ontology/RDF export correctness fixes land: OWL property/class export, SHACL target-namespace resolution, one canonical confidence datatype across all four RDF formats, reachable OWL-Time reification, JSON-LD default-graph and content-derived document identity, and full metadata passthrough on every RDF serializer
|
||||
- **Security**: Agno's `AgnoKnowledgeGraph.load_urls()` and OpenClaw's MCP tool now route outbound requests through the shared SSRF guard
|
||||
|
||||
Also ships: **first-class CrewAI integration** (`semantica[crewai]`, extraction/decision tools + a knowledge source), **`ContextGraph` retraction and purge** (GDPR-style erasure without a full `clear()`), a declared **Semantica RDF vocabulary with deterministic entity/relationship IRIs** (stable, diffable exports), and **timezone-aware timestamps** across `export/` and `provenance/`.
|
||||
Also fixes: `PipelineBuilder.set_parallelism()` now actually parallelizes independent pipeline steps, `flatten_dict()` no longer silently drops data on a key collision, `Config.get()` honors boolean environment overrides, and the MCP server's `export_graph` tool works again on every format.
|
||||
|
||||
→ [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md)
|
||||
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ icon: "circle-question"
|
||||
| API key required? | Optional: pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| Latest version? | **v0.6.6** (August 2026) |
|
||||
| Latest version? | **v0.6.7** (August 2026) |
|
||||
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ icon: "rocket"
|
||||
Verify installation:
|
||||
```python
|
||||
import semantica
|
||||
print(semantica.__version__) # 0.6.6
|
||||
print(semantica.__version__) # 0.6.7
|
||||
```
|
||||
</Check>
|
||||
</Step>
|
||||
|
||||
@@ -382,6 +382,45 @@ For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-p
|
||||
|
||||
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
|
||||
|
||||
## Source 7 — SAP OData
|
||||
|
||||
`SAPIngestor` ingests an Entity Set from a SAP OData service (S/4HANA Cloud, SuccessFactors, or an on-prem NetWeaver Gateway over its REST surface). It speaks OData v2 and v4, follows server-driven pagination automatically, and flattens each record into a document dict via `export_as_documents()` — the same structured "transform to text, then store" pattern as the other sources.
|
||||
|
||||
```python
|
||||
from semantica.ingest import SAPIngestor
|
||||
|
||||
ing = SAPIngestor(
|
||||
base_url="https://my-sap.example.com/sap/opu/odata/sap/API_BUSINESS_PARTNER",
|
||||
client_id="...", client_secret="...",
|
||||
token_url="https://my-sap.example.com/oauth/token", # OAuth2 client-credentials (BTP/S/4HANA Cloud)
|
||||
# On-prem NetWeaver often uses Basic auth instead — swap the block above for:
|
||||
# username="erp_user", password="...",
|
||||
)
|
||||
|
||||
# 1. Discover an unfamiliar service: entity sets + field types from $metadata
|
||||
sets = ing.discover_service() # [{"name": "A_BusinessPartnerSet", "fields": [...]}, ...]
|
||||
|
||||
# 2. Pull a page-walked Entity Set (v2/v4 next links handled for you)
|
||||
partners = ing.ingest_entity_set(
|
||||
entity_set="A_BusinessPartnerSet",
|
||||
select="BusinessPartner,BusinessPartnerFullName", # $select
|
||||
top=1000, # cap on total rows
|
||||
)
|
||||
|
||||
# 3. Flatten to document dicts, then build text for the graph
|
||||
docs = ing.export_as_documents(partners)
|
||||
partner_texts = [
|
||||
f"Business Partner {d['BusinessPartner']}: {d['BusinessPartnerFullName']}"
|
||||
for d in docs
|
||||
]
|
||||
```
|
||||
|
||||
- Use `expand="to_Item"` (e.g. on a sales-order header set) to pull nested line items in one request — handy for modeling order → line-item → material relationships.
|
||||
- Every outbound request, including the OAuth2 token exchange, is routed through the SSRF guard, so a user-supplied SAP URL can never reach private/loopback/link-local address space.
|
||||
- Install with `pip install 'semantica[ingest-sap]'`.
|
||||
|
||||
> **Security Note:** Never hardcode credentials (`client_secret`, `password`) in code; pass them via environment variables (e.g., `SAP_CLIENT_SECRET`, `SAP_PASSWORD`) or a secrets manager.
|
||||
|
||||
## Combining All Five Sources
|
||||
|
||||
Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly.
|
||||
|
||||
+12
-1
@@ -241,7 +241,18 @@ engine = ExecutionEngine(max_workers=2, retry_on_failure=True)
|
||||
result = engine.execute_pipeline(pipeline)
|
||||
```
|
||||
|
||||
`set_parallelism(n)` tells the engine how many steps it may run simultaneously. The topological sort guarantees that only steps whose dependencies are all completed are eligible for concurrent execution — you cannot accidentally run a step before its inputs are ready.
|
||||
`set_parallelism(n)` tells the engine how many steps it may run simultaneously; `n` must be a positive integer. The topological sort guarantees that only steps whose dependencies are all completed are eligible for concurrent execution — you cannot accidentally run a step before its inputs are ready. The effective concurrency is capped at `min(n, max_workers)`, so the engine's `max_workers` setting remains a hard resource ceiling.
|
||||
|
||||
Concurrency is opt-in per step. A dependency layer only runs in parallel when every step in that layer is marked `parallel_safe`, the layer has more than one step, and the data flowing into the layer is a dict:
|
||||
|
||||
```python
|
||||
builder.add_step("ner", "ner_extract", parallel_safe=True, confidence_threshold=0.75)
|
||||
builder.add_step("triplets", "triplet_extract", parallel_safe=True, include_temporal=True)
|
||||
```
|
||||
|
||||
If any step in a layer is not marked `parallel_safe`, or if a step runs in delta mode, the entire layer falls back to sequential execution — parallelism never silently bypasses a step that was not declared safe. `parallel_safe` is a control field: like `dependencies`, it is consumed by the builder and never reaches your handler's config.
|
||||
|
||||
Parallel-safe handlers must return a dict. Each step in a parallel layer receives an isolated deep copy of the layer's input, so steps cannot see each other's mutations. The per-step results are merged key by key in step declaration order: a key written by one step is added to the merged output, a key written by several steps with equal values is kept, and two steps writing different values for the same key fail the pipeline with a `ProcessingError` naming the conflicting key and both steps. Handlers that touch shared mutable resources — database connections, in-memory stores, global caches — should not be marked `parallel_safe`.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
|
||||
@@ -28,6 +28,7 @@ icon: "database"
|
||||
| `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries |
|
||||
| `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
|
||||
| `DatabricksIngestor` | Databricks Unity Catalog metadata, Delta table queries, and lineage |
|
||||
| `SAPIngestor` | SAP OData services (S/4HANA Cloud, SuccessFactors, NetWeaver Gateway): entity-set discovery and ingestion with v2/v4 pagination |
|
||||
| `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
|
||||
| `ArrowIngestor` | Apache Arrow IPC and Feather file processing |
|
||||
| `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |
|
||||
|
||||
@@ -41,6 +41,7 @@ import {
|
||||
} from "./plugins";
|
||||
import { explorationEffectsShouldLoad, neighborhoodPanelShouldLoad, temporalOverlayShouldLoad } from "./pluginRegistryPredicates";
|
||||
import { shouldFetchTemporalBounds, shouldFetchTemporalSnapshot } from "./temporalLifecyclePredicates";
|
||||
import { createTemporalSnapshotGuards, type TemporalSnapshotResponse } from "./temporalSnapshotGuards";
|
||||
import type { LinkPrediction, PathResponse } from "./GraphInspectorPanel";
|
||||
import type { GraphSceneHandle, GraphSceneRuntime } from "./scene";
|
||||
import type {
|
||||
@@ -1479,6 +1480,23 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
|
||||
summary?.edgeCount,
|
||||
]);
|
||||
|
||||
// Guards the snapshot lifecycle: at most one in-flight request per scrubber
|
||||
// position (identical-`at` polls are deduplicated, breaking the idle/play
|
||||
// polling loop), applied snapshots are cached and re-applied on revisit, and
|
||||
// a response applies only while the scrubber is still on its position
|
||||
// (out-of-order responses cannot clobber the active-node count).
|
||||
const temporalSnapshotGuardsRef = useRef<ReturnType<typeof createTemporalSnapshotGuards> | null>(null);
|
||||
if (temporalSnapshotGuardsRef.current === null) {
|
||||
temporalSnapshotGuardsRef.current = createTemporalSnapshotGuards();
|
||||
}
|
||||
const temporalSnapshotGuards = temporalSnapshotGuardsRef.current;
|
||||
|
||||
// A new graph summary means the graph data was replaced (reload/retry);
|
||||
// snapshots cached against the previous graph are stale, so reset all state.
|
||||
useEffect(() => {
|
||||
temporalSnapshotGuards.reset();
|
||||
}, [summary]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!canFetchTemporalSnapshot) {
|
||||
return;
|
||||
@@ -1488,37 +1506,67 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
|
||||
return;
|
||||
}
|
||||
|
||||
const atMs = debouncedTime.getTime();
|
||||
const { seq, cached } = temporalSnapshotGuards.begin(atMs);
|
||||
if (seq === null) {
|
||||
// An identical request is already in flight: one request per position.
|
||||
return;
|
||||
}
|
||||
|
||||
let cancelled = false;
|
||||
|
||||
const applyData = (data: TemporalSnapshotResponse) => {
|
||||
const nextActiveIds = new Set(data.active_node_ids);
|
||||
requestAnimationFrame(() => {
|
||||
if (cancelled) return;
|
||||
if (!temporalSnapshotGuards.shouldApply(atMs, seq)) {
|
||||
// The scrubber moved on (or this request was superseded): release the
|
||||
// position so a return to it refetches instead of stalling.
|
||||
temporalSnapshotGuards.finish(atMs, seq);
|
||||
return;
|
||||
}
|
||||
const previous = prevActiveIdsRef.current;
|
||||
previous.forEach((id) => {
|
||||
if (!nextActiveIds.has(id) && graph.hasNode(id)) {
|
||||
graph.setNodeAttribute(id, "hidden", true);
|
||||
}
|
||||
});
|
||||
nextActiveIds.forEach((id) => {
|
||||
if (graph.hasNode(id)) {
|
||||
graph.setNodeAttribute(id, "hidden", false);
|
||||
}
|
||||
});
|
||||
prevActiveIdsRef.current = nextActiveIds;
|
||||
setActiveNodeCount(data.active_node_count);
|
||||
setGraphVersion((current) => current + 1);
|
||||
sceneRef.current?.getRuntime()?.requestRender();
|
||||
temporalSnapshotGuards.apply(atMs, seq, data);
|
||||
});
|
||||
};
|
||||
|
||||
if (cached) {
|
||||
// Returning to a position whose snapshot was already applied: re-apply
|
||||
// the cached result without a network request.
|
||||
applyData(cached);
|
||||
return;
|
||||
}
|
||||
|
||||
const applySnapshot = async () => {
|
||||
try {
|
||||
const at = debouncedTime.toISOString();
|
||||
const response = await fetch(`/api/temporal/snapshot?at=${encodeURIComponent(at)}`);
|
||||
if (!response.ok || cancelled) return;
|
||||
|
||||
const data: { active_node_ids: string[]; active_node_count: number } = await response.json();
|
||||
if (!response.ok) {
|
||||
// A failed request must be retryable if the scrubber returns.
|
||||
if (!cancelled) temporalSnapshotGuards.finish(atMs, seq);
|
||||
return;
|
||||
}
|
||||
if (cancelled) return;
|
||||
|
||||
const nextActiveIds = new Set(data.active_node_ids);
|
||||
requestAnimationFrame(() => {
|
||||
if (cancelled) return;
|
||||
const previous = prevActiveIdsRef.current;
|
||||
previous.forEach((id) => {
|
||||
if (!nextActiveIds.has(id) && graph.hasNode(id)) {
|
||||
graph.setNodeAttribute(id, "hidden", true);
|
||||
}
|
||||
});
|
||||
nextActiveIds.forEach((id) => {
|
||||
if (graph.hasNode(id)) {
|
||||
graph.setNodeAttribute(id, "hidden", false);
|
||||
}
|
||||
});
|
||||
prevActiveIdsRef.current = nextActiveIds;
|
||||
setActiveNodeCount(data.active_node_count);
|
||||
setGraphVersion((current) => current + 1);
|
||||
sceneRef.current?.getRuntime()?.requestRender();
|
||||
});
|
||||
const data: TemporalSnapshotResponse = await response.json();
|
||||
if (cancelled) return;
|
||||
applyData(data);
|
||||
} catch (fetchError) {
|
||||
temporalSnapshotGuards.finish(atMs, seq);
|
||||
if (!cancelled) {
|
||||
console.error("[Temporal] Snapshot fetch failed", fetchError);
|
||||
}
|
||||
@@ -1528,6 +1576,8 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
|
||||
applySnapshot();
|
||||
return () => {
|
||||
cancelled = true;
|
||||
// A cancelled request must be retryable when its position is revisited.
|
||||
temporalSnapshotGuards.finish(atMs, seq);
|
||||
};
|
||||
}, [
|
||||
canFetchTemporalSnapshot,
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
/**
|
||||
* Guards for the temporal snapshot fetch/apply lifecycle.
|
||||
*
|
||||
* The snapshot effect previously fetched /api/temporal/snapshot with no
|
||||
* idempotency or ordering protection. Upstream churn (timeline recreation
|
||||
* while bounds settle, play ticks resetting the playhead, drag events) could
|
||||
* re-request the same `at` repeatedly, and responses could arrive after the
|
||||
* scrubber had moved on.
|
||||
*
|
||||
* The guards enforce:
|
||||
* - at most one in-flight request per scrubber position (identical `at`
|
||||
* values are deduplicated while a request is pending, breaking the
|
||||
* idle/play polling loop);
|
||||
* - successful snapshots are cached per position and re-applied when the
|
||||
* scrubber returns (play wrap-around, back-scrubbing) without a refetch;
|
||||
* - a response is applied only while the scrubber is still on its position,
|
||||
* so out-of-order responses cannot clobber a newer position's count;
|
||||
* - failed, cancelled, or superseded requests release their position so it
|
||||
* can be fetched again on the next visit;
|
||||
* - `reset()` drops all state when the underlying graph data is replaced
|
||||
* (reload/retry), because cached snapshots describe the previous graph.
|
||||
*
|
||||
* `createTemporalSnapshotGuards()` is stateful by design.
|
||||
*/
|
||||
|
||||
export interface TemporalSnapshotResponse {
|
||||
active_node_ids: string[];
|
||||
active_node_count: number;
|
||||
}
|
||||
|
||||
export interface TemporalSnapshotRequest {
|
||||
/** null when the request was deduplicated because one is already in flight. */
|
||||
seq: number | null;
|
||||
/** The snapshot previously applied for this position, when revisiting it. */
|
||||
cached: TemporalSnapshotResponse | null;
|
||||
}
|
||||
|
||||
export interface TemporalSnapshotGuards {
|
||||
/** Begin (or dedupe) a request for `atMs`; marks it as the current position. */
|
||||
begin(atMs: number): TemporalSnapshotRequest;
|
||||
/** True when the response for `atMs`/`seq` may be applied (scrubber still on `atMs`). */
|
||||
shouldApply(atMs: number, seq: number): boolean;
|
||||
/** Record a successful application and cache its snapshot for revisits. */
|
||||
apply(atMs: number, seq: number, data: TemporalSnapshotResponse): void;
|
||||
/** Release a position whose request failed, was cancelled, or was superseded. */
|
||||
finish(atMs: number, seq: number): void;
|
||||
/** Drop all state; call when the underlying graph data is replaced (reload). */
|
||||
reset(): void;
|
||||
}
|
||||
|
||||
interface SnapshotEntry {
|
||||
seq: number;
|
||||
/** null while the request is in flight (or before the first success). */
|
||||
data: TemporalSnapshotResponse | null;
|
||||
}
|
||||
|
||||
/** Upper bound on cached positions so long scrubbing sessions stay bounded. */
|
||||
const MAX_CACHED_POSITIONS = 256;
|
||||
|
||||
export function createTemporalSnapshotGuards(): TemporalSnapshotGuards {
|
||||
const entries = new Map<number, SnapshotEntry>();
|
||||
let latestRequestSeq = 0;
|
||||
let currentAtMs: number | null = null;
|
||||
|
||||
const evictOldest = () => {
|
||||
while (entries.size > MAX_CACHED_POSITIONS) {
|
||||
const oldestAtMs = entries.keys().next().value;
|
||||
if (oldestAtMs === undefined) return;
|
||||
entries.delete(oldestAtMs);
|
||||
}
|
||||
};
|
||||
|
||||
return {
|
||||
begin(atMs) {
|
||||
const existing = entries.get(atMs);
|
||||
if (existing && existing.data === null) {
|
||||
// Identical request already in flight: dedupe, but the scrubber is here now.
|
||||
currentAtMs = atMs;
|
||||
return { seq: null, cached: null };
|
||||
}
|
||||
latestRequestSeq += 1;
|
||||
const seq = latestRequestSeq;
|
||||
entries.set(atMs, { seq, data: existing?.data ?? null });
|
||||
currentAtMs = atMs;
|
||||
evictOldest();
|
||||
return { seq, cached: existing?.data ?? null };
|
||||
},
|
||||
|
||||
shouldApply(atMs, seq) {
|
||||
return atMs === currentAtMs && entries.get(atMs)?.seq === seq;
|
||||
},
|
||||
|
||||
apply(atMs, seq, data) {
|
||||
const entry = entries.get(atMs);
|
||||
if (entry && entry.seq === seq) {
|
||||
entry.data = data;
|
||||
}
|
||||
},
|
||||
|
||||
finish(atMs, seq) {
|
||||
const entry = entries.get(atMs);
|
||||
if (entry && entry.seq === seq && entry.data === null) {
|
||||
entries.delete(atMs);
|
||||
}
|
||||
},
|
||||
|
||||
reset() {
|
||||
entries.clear();
|
||||
latestRequestSeq = 0;
|
||||
currentAtMs = null;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import { createTemporalSnapshotGuards } from "../src/workspaces/GraphWorkspace/temporalSnapshotGuards.ts";
|
||||
|
||||
const POSITION_1 = new Date("2023-07-02T00:00:00Z").getTime();
|
||||
const POSITION_2 = new Date("2024-01-02T00:00:00Z").getTime();
|
||||
const POSITION_3 = new Date("2024-07-02T00:00:00Z").getTime();
|
||||
|
||||
const SNAPSHOT = { active_node_ids: ["n1", "n2"], active_node_count: 2 };
|
||||
|
||||
// ── begin: one request per scrubber position ─────────────────────────────────
|
||||
|
||||
test("begin: a new position returns a fresh request sequence", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
assert.deepEqual(guards.begin(POSITION_1), { seq: 1, cached: null });
|
||||
});
|
||||
|
||||
test("begin: an identical in-flight request is deduplicated (no duplicate fetch)", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
guards.begin(POSITION_1);
|
||||
assert.deepEqual(guards.begin(POSITION_1), { seq: null, cached: null });
|
||||
});
|
||||
|
||||
test("begin: distinct positions request independently", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
assert.equal(guards.begin(POSITION_1).seq, 1);
|
||||
assert.equal(guards.begin(POSITION_2).seq, 2);
|
||||
});
|
||||
|
||||
test("begin: revisiting an applied position returns its cached snapshot", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
guards.apply(POSITION_1, seq, SNAPSHOT);
|
||||
const revisit = guards.begin(POSITION_1);
|
||||
assert.equal(revisit.seq, 2);
|
||||
assert.deepEqual(revisit.cached, SNAPSHOT);
|
||||
});
|
||||
|
||||
test("begin: a failed position (finished) can be requested again", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
guards.finish(POSITION_1, seq);
|
||||
const retry = guards.begin(POSITION_1);
|
||||
assert.equal(retry.seq, 2);
|
||||
assert.equal(retry.cached, null);
|
||||
});
|
||||
|
||||
test("finish: does not clear a position whose snapshot was already applied", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
guards.apply(POSITION_1, seq, SNAPSHOT);
|
||||
guards.finish(POSITION_1, seq);
|
||||
assert.deepEqual(guards.begin(POSITION_1).cached, SNAPSHOT);
|
||||
});
|
||||
|
||||
test("finish: a stale sequence cannot release a newer request's position", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const first = guards.begin(POSITION_1);
|
||||
guards.finish(POSITION_1, first.seq);
|
||||
guards.begin(POSITION_1); // seq 2, in flight again
|
||||
guards.finish(POSITION_1, first.seq); // stale seq: must not release seq 2
|
||||
assert.deepEqual(guards.begin(POSITION_1), { seq: null, cached: null });
|
||||
});
|
||||
|
||||
// ── shouldApply: applied only while the scrubber is on that position ─────────
|
||||
|
||||
test("shouldApply: the current position's response is applied", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
assert.equal(guards.shouldApply(POSITION_1, seq), true);
|
||||
});
|
||||
|
||||
test("shouldApply: a response for a position the scrubber left is discarded", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq: seq1 } = guards.begin(POSITION_1);
|
||||
guards.begin(POSITION_2);
|
||||
assert.equal(guards.shouldApply(POSITION_1, seq1), false);
|
||||
assert.equal(guards.shouldApply(POSITION_2, 2), true);
|
||||
});
|
||||
|
||||
test("shouldApply: a late response for the position the scrubber returned to is applied", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq: seq1 } = guards.begin(POSITION_1);
|
||||
const { seq: seq2 } = guards.begin(POSITION_2);
|
||||
guards.begin(POSITION_1); // back to 1: deduplicated, no new request
|
||||
assert.equal(guards.shouldApply(POSITION_1, seq1), true);
|
||||
assert.equal(guards.shouldApply(POSITION_2, seq2), false);
|
||||
});
|
||||
|
||||
test("shouldApply: an unknown sequence is discarded", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
guards.begin(POSITION_1);
|
||||
assert.equal(guards.shouldApply(POSITION_1, 99), false);
|
||||
});
|
||||
|
||||
test("shouldApply: after a reset no pre-reset response applies", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
guards.reset();
|
||||
assert.equal(guards.shouldApply(POSITION_1, seq), false);
|
||||
});
|
||||
|
||||
// ── apply: caching for revisits ─────────────────────────────────────────────
|
||||
|
||||
test("apply: stores the snapshot so a revisit re-applies it without a request", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
guards.apply(POSITION_1, seq, SNAPSHOT);
|
||||
guards.begin(POSITION_2);
|
||||
assert.deepEqual(guards.begin(POSITION_1).cached, SNAPSHOT);
|
||||
});
|
||||
|
||||
test("apply: play wrap-around re-applies the wrapped-to position's snapshot", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
guards.apply(POSITION_1, seq, SNAPSHOT);
|
||||
guards.begin(POSITION_2);
|
||||
guards.begin(POSITION_3);
|
||||
const wrap = guards.begin(POSITION_1);
|
||||
assert.deepEqual(wrap.cached, SNAPSHOT);
|
||||
assert.equal(guards.shouldApply(POSITION_1, wrap.seq), true);
|
||||
});
|
||||
|
||||
// ── reset: graph reload ─────────────────────────────────────────────────────
|
||||
|
||||
test("reset: clears requested and cached state so positions refetch", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const { seq } = guards.begin(POSITION_1);
|
||||
guards.apply(POSITION_1, seq, SNAPSHOT);
|
||||
guards.reset();
|
||||
const fresh = guards.begin(POSITION_1);
|
||||
assert.equal(fresh.seq, 1);
|
||||
assert.equal(fresh.cached, null);
|
||||
});
|
||||
|
||||
// ── cache bound ─────────────────────────────────────────────────────────────
|
||||
|
||||
test("cache: oldest positions are evicted when the cache is full", () => {
|
||||
const guards = createTemporalSnapshotGuards();
|
||||
const count = 300;
|
||||
for (let i = 0; i < count; i++) {
|
||||
const { seq } = guards.begin(POSITION_1 + i * 1000);
|
||||
guards.apply(POSITION_1 + i * 1000, seq, SNAPSHOT);
|
||||
}
|
||||
const oldest = guards.begin(POSITION_1);
|
||||
assert.equal(oldest.cached, null); // evicted: must refetch on revisit
|
||||
const newest = guards.begin(POSITION_1 + (count - 1) * 1000);
|
||||
assert.deepEqual(newest.cached, SNAPSHOT); // still cached
|
||||
});
|
||||
+2
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "semantica"
|
||||
version = "0.6.6"
|
||||
version = "0.6.7"
|
||||
description = "Graph-Native Infrastructure for Context and Accountable AI Systems: context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
@@ -126,6 +126,7 @@ db-databricks = ["databricks-sdk>=0.60.0", "databricks-sql-connector>=4.0.0"]
|
||||
db-arrow = ["pyarrow>=24.0.0"]
|
||||
ingest-parquet = ["pyarrow>=24.0.0"]
|
||||
ingest-arrow = ["pyarrow>=24.0.0"]
|
||||
ingest-sap = ["requests>=2.28.0"]
|
||||
|
||||
db-all = [
|
||||
"semantica[db-snowflake,db-databricks,db-arrow]"
|
||||
|
||||
@@ -10,7 +10,7 @@ Main exports:
|
||||
- Config: Configuration management
|
||||
"""
|
||||
|
||||
__version__ = "0.6.6"
|
||||
__version__ = "0.6.7"
|
||||
__author__ = "Semantica Contributors"
|
||||
__license__ = "MIT"
|
||||
|
||||
|
||||
@@ -306,10 +306,10 @@ class JSONExporter:
|
||||
|
||||
self.logger.debug(f"Exporting {len(entities)} entity(ies) to JSON")
|
||||
|
||||
# Build JSON data with JSON-LD context
|
||||
# Build JSON data with JSON-LD context. No @vocab: it would expand
|
||||
# every bare key in the caller's entity dicts into ns# (#1146).
|
||||
json_data = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"entities": {"@id": "semantica:entities", "@container": "@list"},
|
||||
},
|
||||
@@ -339,7 +339,6 @@ class JSONExporter:
|
||||
"""
|
||||
json_data = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"relationships": {
|
||||
"@id": "semantica:relationships",
|
||||
@@ -434,11 +433,14 @@ class JSONExporter:
|
||||
Returns:
|
||||
Dictionary in JSON-LD format with @context, @graph/@value, and metadata
|
||||
"""
|
||||
# Initialize JSON-LD structure with context
|
||||
# Initialize JSON-LD structure with context. No @vocab: for a generic
|
||||
# payload it turned whatever bare keys the caller happened to use into
|
||||
# ns# terms (#1146). Undeclared terms now simply expand to nothing,
|
||||
# which is standard JSON-LD behaviour for a context that does not
|
||||
# know them; the raw payload is still in the document.
|
||||
jsonld = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": "https://semantica.dev/ns#",
|
||||
"semantica": SEMANTICA_NS,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -598,13 +600,21 @@ class JSONExporter:
|
||||
Returns:
|
||||
Dictionary in JSON-LD format with @context, @id, @type, and graph data
|
||||
"""
|
||||
# Initialize JSON-LD structure with RDF context
|
||||
# Initialize JSON-LD structure with RDF context. No @vocab: it applied
|
||||
# to every bare term in caller data, so an extracted type like "ORG"
|
||||
# became ns#ORG and a metadata key like "source" collided with the
|
||||
# real sem:source object property (#1146). Only explicit semantica:
|
||||
# terms resolve now, and the caller's metadata dict is typed @json so
|
||||
# it survives as one rdf:JSON literal instead of expanding its keys.
|
||||
jsonld = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": "https://semantica.dev/ns#",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
|
||||
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
|
||||
"semantica:metadata": {
|
||||
"@id": "semantica:metadata",
|
||||
"@type": "@json",
|
||||
},
|
||||
},
|
||||
# Minted from the graph's own content rather than the wall clock
|
||||
# (#1147): re-exporting an unchanged graph must produce the same
|
||||
@@ -664,14 +674,23 @@ class JSONExporter:
|
||||
entity_text = entity.get("text") or entity.get("label", "unknown")
|
||||
entity_id = entity.get("id") or mint_entity_iri(entity_text)
|
||||
|
||||
# The caller's type label is data, not a class we define: minting it
|
||||
# into @type expanded it through @vocab into ns#ORG and friends, terms
|
||||
# that look official but do not exist (#1146). The node is always a
|
||||
# semantica:Entity and the label travels as semantica:type, exactly
|
||||
# how _relationship_to_jsonld has always carried the relationship type.
|
||||
jsonld = {
|
||||
"@id": entity_id,
|
||||
"@type": entity.get("type") or "semantica:Entity",
|
||||
"@type": "semantica:Entity",
|
||||
"semantica:text": entity.get("text") or entity.get("label", ""),
|
||||
"semantica:confidence": entity.get("confidence", 1.0),
|
||||
}
|
||||
entity_type = entity.get("type")
|
||||
if entity_type:
|
||||
jsonld["semantica:type"] = entity_type
|
||||
|
||||
# Add metadata if present
|
||||
# Add metadata if present. The @json term definition on
|
||||
# semantica:metadata keeps the whole dict one rdf:JSON literal.
|
||||
if "metadata" in entity:
|
||||
jsonld["semantica:metadata"] = entity["metadata"]
|
||||
|
||||
|
||||
@@ -1226,11 +1226,14 @@ class RDFSerializer:
|
||||
metadata_terms = _resolve_metadata_terms(options.pop("metadata_terms", None))
|
||||
graph_uri: Optional[str] = options.pop("graph_uri", None)
|
||||
|
||||
# Initialize JSON-LD structure with context
|
||||
# Initialize JSON-LD structure with context. No @vocab: it applied to
|
||||
# every bare term in caller data, so an extracted type like "ORG"
|
||||
# became ns#ORG and a metadata key like "source" collided with the
|
||||
# real sem:source object property (#1146). Only explicit semantica:
|
||||
# terms resolve now.
|
||||
jsonld = {
|
||||
"@context": {
|
||||
"@vocab": "https://semantica.dev/vocab/",
|
||||
"semantica": "https://semantica.dev/ns#",
|
||||
"semantica": SEMANTICA_NS,
|
||||
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
|
||||
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
|
||||
},
|
||||
@@ -1252,11 +1255,20 @@ class RDFSerializer:
|
||||
# and was dropped in full by a JSON-LD parser, silently.
|
||||
entity_id = entity.get("id") or mint_entity_iri(entity.get("text", ""))
|
||||
|
||||
# The caller's type label is data, not a class we define: minting
|
||||
# it into @type expanded it through @vocab into ns#ORG and
|
||||
# friends, terms that look official but do not exist (#1146).
|
||||
# The node is always a semantica:Entity and the label travels as
|
||||
# semantica:type, matching the relationship node below and
|
||||
# JSONExporter._entity_to_jsonld.
|
||||
node = {
|
||||
"@id": entity_id,
|
||||
"@type": entity.get("type", "semantica:Entity"),
|
||||
"@type": "semantica:Entity",
|
||||
"semantica:text": entity.get("text") or entity.get("label", ""),
|
||||
}
|
||||
entity_type = entity.get("type")
|
||||
if entity_type:
|
||||
node["semantica:type"] = entity_type
|
||||
confidence = normalize_confidence(entity.get("confidence", 1.0))
|
||||
if confidence is None:
|
||||
self.logger.warning(
|
||||
|
||||
@@ -218,6 +218,10 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
|
||||
"SnowflakeIngestor": (".snowflake_ingestor", "SnowflakeIngestor"),
|
||||
"SnowflakeData": (".snowflake_ingestor", "SnowflakeData"),
|
||||
"SnowflakeConnector": (".snowflake_ingestor", "SnowflakeConnector"),
|
||||
# SAP OData ingestion
|
||||
"SAPIngestor": (".sap_ingestor", "SAPIngestor"),
|
||||
"SAPODataEntity": (".sap_ingestor", "SAPODataEntity"),
|
||||
"SAPODataConnector": (".sap_ingestor", "SAPODataConnector"),
|
||||
# Databricks ingestion
|
||||
"DatabricksIngestor": (".databricks_ingestor", "DatabricksIngestor"),
|
||||
"DatabricksData": (".databricks_ingestor", "DatabricksData"),
|
||||
@@ -345,6 +349,10 @@ __all__ = [
|
||||
"SnowflakeIngestor",
|
||||
"SnowflakeData",
|
||||
"SnowflakeConnector",
|
||||
# SAP OData ingestion
|
||||
"SAPIngestor",
|
||||
"SAPODataEntity",
|
||||
"SAPODataConnector",
|
||||
# Databricks ingestion
|
||||
"DatabricksIngestor",
|
||||
"DatabricksData",
|
||||
|
||||
@@ -875,6 +875,57 @@ schema = connector.get_schema(engine)
|
||||
print(f" {table_name}: {[col['name'] for col in columns]}")
|
||||
```
|
||||
|
||||
## SAP OData Ingestion
|
||||
|
||||
`SAPIngestor` reads an Entity Set from a SAP OData service — S/4HANA Cloud,
|
||||
SuccessFactors, or an on-prem NetWeaver Gateway over its REST surface. It
|
||||
follows OData v2/v4 server-driven pagination and flattens each record into a
|
||||
document dict via `export_as_documents()`.
|
||||
|
||||
Install with `pip install 'semantica[ingest-sap]'`.
|
||||
|
||||
### Connector Construction & Authentication
|
||||
|
||||
```python
|
||||
from semantica.ingest import SAPIngestor
|
||||
|
||||
# OAuth2 client-credentials (BTP / S/4HANA Cloud)
|
||||
ing = SAPIngestor(
|
||||
base_url="https://my-sap.example.com/sap/opu/odata/sap/API_BUSINESS_PARTNER",
|
||||
client_id="...", client_secret="...",
|
||||
token_url="https://my-sap.example.com/oauth/token",
|
||||
)
|
||||
# On-prem NetWeaver often uses Basic auth instead — swap the block above for:
|
||||
# ing = SAPIngestor(base_url="...", username="erp_user", password="...")
|
||||
```
|
||||
|
||||
### Entity-Set Ingestion & Document Export
|
||||
|
||||
```python
|
||||
# 1. Discover entity sets + field types from $metadata
|
||||
sets = ing.discover_service()
|
||||
|
||||
# 2. Page-walk an Entity Set (v2/v4 next links handled automatically)
|
||||
partners = ing.ingest_entity_set(
|
||||
entity_set="A_BusinessPartnerSet",
|
||||
select="BusinessPartner,BusinessPartnerFullName",
|
||||
top=1000,
|
||||
)
|
||||
|
||||
# 3. Flatten to document dicts that GraphBuilder can consume directly
|
||||
docs = ing.export_as_documents(partners)
|
||||
```
|
||||
|
||||
- Use `expand="to_Item"` on a sales-order header set to pull nested line items
|
||||
in one request — handy for modeling order → line-item → material relations.
|
||||
- Every outbound request, including the OAuth2 token exchange, is routed through
|
||||
the SSRF guard, and pagination never follows a next link that points to a
|
||||
different host than the service root.
|
||||
|
||||
> **Security Note:** Never hardcode credentials (`client_secret`, `password`);
|
||||
> pass them via environment variables (`SAP_CLIENT_SECRET`, `SAP_PASSWORD`) or a
|
||||
> secrets manager.
|
||||
|
||||
## MCP Server Ingestion
|
||||
|
||||
**IMPORTANT**: This implementation supports **ONLY Python-based MCP servers and FastMCP servers**. Users can bring their own Python or FastMCP MCP servers via URL connections. JavaScript, TypeScript, C#, Java, and other language implementations are **NOT supported**.
|
||||
|
||||
@@ -0,0 +1,617 @@
|
||||
"""SAP OData ingestion module.
|
||||
|
||||
Pulls an Entity Set from a SAP OData service (S/4HANA and on-prem NetWeaver
|
||||
REST surfaces) and flattens it into document dicts that the pipeline can feed
|
||||
to ``GraphBuilder``.
|
||||
|
||||
Why this exists
|
||||
---------------
|
||||
Semantica ingests from many sources; SAP is the ERP backbone of finance and
|
||||
regulated industries, and its master/transactional data (customers, vendors,
|
||||
sales orders) is exactly the "context" a Context Graph wants. SAP exposes that
|
||||
data over OData (v2 on many on-prem NetWeaver systems, v4 on BTP / S/4HANA
|
||||
Cloud). This connector speaks the REST surface of OData only.
|
||||
|
||||
Design notes
|
||||
------------
|
||||
Three classes, matching the Snowflake/Databricks ingestors:
|
||||
- ``SAPODataEntity``: a collection fetch from one Entity Set (``records``,
|
||||
``count``, ``service``, ``metadata``), flattened to document dicts for
|
||||
``GraphBuilder`` via ``export_as_documents``.
|
||||
- ``SAPODataConnector``: auth (OAuth2 client-credentials or Basic) + the
|
||||
shared, SSRF-guarded :mod:`requests` session. *Every* outbound request,
|
||||
including the OAuth2 token exchange, goes through
|
||||
``request_with_ssrf_guard`` so user-supplied endpoints can not reach
|
||||
private/loopback/link-local address space.
|
||||
- ``SAPIngestor``: the three methods the issue requested —
|
||||
``discover_service``, ``ingest_entity_set`` and ``export_as_documents``
|
||||
(plus ``close`` for symmetry with the SQL connectors).
|
||||
|
||||
EDMX
|
||||
----
|
||||
``$metadata`` is plain CSDL XML in *both* OData v2 and v4, so we hand-roll a
|
||||
minimal parser with :mod:`xml.etree` instead of pulling in ``pyodata``. That
|
||||
keeps phase 1 ``requests``-only, exactly as scoped in the issue.
|
||||
|
||||
Pagination
|
||||
----------
|
||||
OData uses a server-driven "next link": OData v2 surfaces it as the atom
|
||||
``__next`` element, OData v4 as the ``@odata.nextLink`` field on the JSON
|
||||
payload. ``ingest_entity_set`` follows whichever it sees until the set is
|
||||
exhausted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import os
|
||||
import xml.etree.ElementTree as ET
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import requests
|
||||
from requests.adapters import HTTPAdapter
|
||||
|
||||
try:
|
||||
from urllib3.util.retry import Retry
|
||||
except (ImportError, OSError): # pragma: no cover - old urllib3 layout
|
||||
from requests.packages.urllib3.util.retry import Retry # type: ignore
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from .ssrf import parse_bool, request_with_ssrf_guard
|
||||
|
||||
__all__ = [
|
||||
"SAPODataEntity",
|
||||
"SAPODataConnector",
|
||||
"SAPIngestor",
|
||||
]
|
||||
|
||||
_logger = get_logger("sap_ingestor")
|
||||
|
||||
|
||||
def _prop_is_nullable(prop: Any) -> bool:
|
||||
"""CSDL structural properties default to nullable=True when omitted."""
|
||||
val = prop.get("Nullable", prop.get("nullable"))
|
||||
return True if val is None else val.strip().lower() == "true"
|
||||
|
||||
|
||||
def _match(elem: Any, localname: str) -> bool:
|
||||
"""True if *elem* has the given local name in any namespace."""
|
||||
return elem.tag.rsplit("}", 1)[-1] == localname
|
||||
|
||||
|
||||
@dataclass
|
||||
class SAPODataEntity:
|
||||
"""A collection fetch from a SAP OData Entity Set.
|
||||
|
||||
Holds the rows pulled from one Entity Set (all paging fan-in'd), with the
|
||||
shape the issue specifies: ``records`` (the row data), ``count``,
|
||||
``service`` (the resolved service root), optional ``metadata`` (entity-set
|
||||
schema from ``$metadata``) and ``ingested_at``.
|
||||
"""
|
||||
|
||||
records: List[Dict[str, Any]]
|
||||
entity_set: str
|
||||
count: int
|
||||
service: str
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
ingested_at: datetime = field(default_factory=datetime.now)
|
||||
|
||||
def to_documents(self) -> List[Dict[str, Any]]:
|
||||
"""Flatten each record to a document dict ``GraphBuilder`` can consume.
|
||||
|
||||
GraphBuilder only treats a dict as an entity when it carries
|
||||
``id``/``entity_id``/``name`` (or ``text``+``type``); SAP records have
|
||||
none of those, so they would be silently dropped. We inject an
|
||||
identifier resolved from each record's primary-key-like field, falling
|
||||
back to ``entity_set:index``, and expose it under both ``id`` and
|
||||
``name``.
|
||||
"""
|
||||
docs: List[Dict[str, Any]] = []
|
||||
for index, record in enumerate(self.records):
|
||||
doc = dict(record)
|
||||
key_value = self._id_value(record)
|
||||
doc.setdefault("id", key_value or f"{self.entity_set}:{index}")
|
||||
doc.setdefault("name", key_value or self.entity_set)
|
||||
doc.setdefault("source", self.service)
|
||||
docs.append(doc)
|
||||
return docs
|
||||
|
||||
@staticmethod
|
||||
def _id_value(record: Dict[str, Any]) -> str:
|
||||
for key, value in record.items():
|
||||
if "id" in key.lower() and value not in (None, ""):
|
||||
return str(value)
|
||||
return ""
|
||||
|
||||
|
||||
class SAPODataConnector:
|
||||
"""Connection + authentication management for a SAP OData REST service.
|
||||
|
||||
Supports the two auth landscapes called out in the issue:
|
||||
|
||||
- **OAuth2 client-credentials** (BTP / S/4HANA Cloud). The token URL is
|
||||
user supplied; both the token exchange *and* every subsequent data
|
||||
request are validated through the SSRF guard.
|
||||
- **Basic** (on-prem NetWeaver). Username/password passed through as an
|
||||
``Authorization: Basic`` header, also through the guard.
|
||||
|
||||
Example usage::
|
||||
|
||||
>>> connector = SAPODataConnector(
|
||||
... base_url="https://myhost/sap/opu/odata/sap/",
|
||||
... token_url="https://myhost/oauth/token",
|
||||
... client_id="cid", client_secret="secret",
|
||||
... )
|
||||
>>> session = connector.get_session()
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: Optional[str] = None,
|
||||
*,
|
||||
auth: Optional[str] = None,
|
||||
token_url: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None,
|
||||
username: Optional[str] = None,
|
||||
password: Optional[str] = None,
|
||||
allow_private_ips: bool = False,
|
||||
**config: Any,
|
||||
) -> None:
|
||||
"""Initialize the SAP OData connector.
|
||||
|
||||
Args:
|
||||
base_url: Base OData service URL, e.g. ``https://host/sap/opu
|
||||
/odata/sap/``. The issue's ``service`` value.
|
||||
auth: Explicit auth flow, ``"oauth2"`` or ``"basic"``. When
|
||||
omitted, the flow is inferred from which credentials are set.
|
||||
token_url: OAuth2 token endpoint. Required only for OAuth2 flow.
|
||||
client_id: OAuth2 client id (OAuth2 flow).
|
||||
client_secret: OAuth2 client secret (OAuth2 flow).
|
||||
username: Basic-auth username (on-prem flow).
|
||||
password: Basic-auth password (on-prem flow).
|
||||
allow_private_ips: Opt into private/loopback/link-local endpoints.
|
||||
Defaults to False (SSRF-safe).
|
||||
**config: Extra options, notably ``timeout``, ``max_retries``,
|
||||
``backoff_factor``, ``headers``.
|
||||
"""
|
||||
self.logger = _logger
|
||||
self.base_url = base_url or os.getenv("SAP_BASE_URL")
|
||||
self.auth = (auth or os.getenv("SAP_AUTH") or "").lower()
|
||||
self.token_url = token_url or os.getenv("SAP_TOKEN_URL")
|
||||
self.client_id = client_id or os.getenv("SAP_CLIENT_ID")
|
||||
self.client_secret = client_secret or os.getenv("SAP_CLIENT_SECRET")
|
||||
self.username = username or os.getenv("SAP_USERNAME")
|
||||
self.password = password or os.getenv("SAP_PASSWORD")
|
||||
self.allow_private_ips = parse_bool(
|
||||
config.pop("allow_private_ips", allow_private_ips), default=False
|
||||
)
|
||||
self.config = config
|
||||
|
||||
if not self.base_url:
|
||||
raise ValidationError(
|
||||
"SAP base_url is required. Provide via 'base_url' or "
|
||||
"SAP_BASE_URL environment variable."
|
||||
)
|
||||
oauth_configured = bool(self.client_id or self.client_secret or self.token_url)
|
||||
if self.auth in ("oauth2", "oauth"):
|
||||
if not (self.client_id and self.client_secret and self.token_url):
|
||||
raise ValidationError(
|
||||
"SAP OAuth2 flow requires client_id, client_secret and "
|
||||
"token_url all set."
|
||||
)
|
||||
elif self.auth == "basic":
|
||||
if not (self.username and self.password):
|
||||
raise ValidationError("SAP Basic flow requires username and password.")
|
||||
elif oauth_configured:
|
||||
if not (self.client_id and self.client_secret and self.token_url):
|
||||
raise ValidationError(
|
||||
"SAP OAuth2 flow requires client_id, client_secret and "
|
||||
"token_url all set."
|
||||
)
|
||||
elif not self.username:
|
||||
raise ValidationError(
|
||||
"SAP authentication requires either (username/password) or "
|
||||
"(client_id/client_secret + token_url)."
|
||||
)
|
||||
|
||||
self.session = requests.Session()
|
||||
retry_strategy = Retry(
|
||||
total=self.config.get("max_retries", 3),
|
||||
backoff_factor=self.config.get("backoff_factor", 1),
|
||||
status_forcelist=[429, 500, 502, 503, 504],
|
||||
)
|
||||
adapter = HTTPAdapter(max_retries=retry_strategy)
|
||||
self.session.mount("http://", adapter)
|
||||
self.session.mount("https://", adapter)
|
||||
default_headers = self.config.get("headers", {})
|
||||
if default_headers:
|
||||
self.session.headers.update(default_headers)
|
||||
|
||||
self._token: Optional[str] = None
|
||||
self.logger.debug(
|
||||
"SAP OData connector initialized (base_url=%s, allow_private_ips=%s)",
|
||||
self.base_url,
|
||||
self.allow_private_ips,
|
||||
)
|
||||
|
||||
def get_session(self) -> requests.Session:
|
||||
"""Return an authenticated session for data requests.
|
||||
|
||||
For the Basic flow the credentials are attached eagerly; for the
|
||||
OAuth2 flow a token is fetched (and cached) on first use. The token
|
||||
is never refreshed, so a job that runs past the token TTL (typically
|
||||
3600s on SAP) will fail with 401 -- re-create the connector instead.
|
||||
"""
|
||||
if self.username:
|
||||
self.session.headers["Authorization"] = "Basic " + self._basic_header()
|
||||
return self.session
|
||||
if self._token is None:
|
||||
self._token = self._fetch_token()
|
||||
self.session.headers["Authorization"] = "Bearer " + self._token
|
||||
return self.session
|
||||
|
||||
def _basic_header(self) -> str:
|
||||
pair = f"{self.username}:{self.password or ''}".encode("utf-8")
|
||||
return base64.b64encode(pair).decode("ascii")
|
||||
|
||||
def _fetch_token(self) -> str:
|
||||
"""Perform the OAuth2 client-credentials token exchange (SSRF-guarded)."""
|
||||
if not self.token_url or not self.client_id:
|
||||
raise ProcessingError("OAuth2 flow requires token_url and client_id.")
|
||||
body = {
|
||||
"grant_type": "client_credentials",
|
||||
"client_id": self.client_id,
|
||||
"client_secret": self.client_secret or "",
|
||||
}
|
||||
resp = request_with_ssrf_guard(
|
||||
"POST",
|
||||
self.token_url,
|
||||
session=self.session,
|
||||
allow_private_ips=self.allow_private_ips,
|
||||
data=body,
|
||||
timeout=self.config.get("timeout", 30),
|
||||
)
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
except requests.exceptions.RequestException as exc:
|
||||
raise ProcessingError(f"SAP OAuth2 token exchange failed: {exc}") from exc
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise ProcessingError(
|
||||
"SAP OAuth2 token endpoint did not return JSON."
|
||||
) from exc
|
||||
token = payload.get("access_token")
|
||||
if not token:
|
||||
raise ProcessingError("SAP OAuth2 token response missing 'access_token'.")
|
||||
return str(token)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the underlying :mod:`requests` session."""
|
||||
self.session.close()
|
||||
|
||||
|
||||
class SAPIngestor:
|
||||
"""Ingest an Entity Set from a SAP OData service.
|
||||
|
||||
Example usage::
|
||||
|
||||
>>> from semantica.ingest import SAPIngestor
|
||||
>>> ing = SAPIngestor(
|
||||
... base_url="https://host/sap/opu/odata/sap/",
|
||||
... username="u", password="p", # or client_id/client_secret/token_url
|
||||
... )
|
||||
>>> sets = ing.discover_service()
|
||||
... # -> [{"name": "SalesOrderSet", "fields": [...]}, ...]
|
||||
>>> docs = ing.export_as_documents(
|
||||
... ing.ingest_entity_set(entity_set="SalesOrderSet", expand="to_Item"))
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: Optional[str] = None,
|
||||
connector: Optional[SAPODataConnector] = None,
|
||||
**config: Any,
|
||||
) -> None:
|
||||
"""Initialize the SAP ingestor.
|
||||
|
||||
Args:
|
||||
base_url: Base OData service URL. Mutually exclusive with
|
||||
``connector``; ignored if a connector is given.
|
||||
connector: An existing :class:`SAPODataConnector`. When provided,
|
||||
its session and base URL are reused.
|
||||
**config: Passed to :class:`SAPODataConnector` when one is created.
|
||||
"""
|
||||
self.logger = _logger
|
||||
self.connector = connector or SAPODataConnector(base_url=base_url, **config)
|
||||
# urljoin() replaces the last path segment unless the base ends in '/',
|
||||
# so normalize once here: .../API_BUSINESS_PARTNER -> .../$metadata would
|
||||
# silently drop the service segment.
|
||||
self._base_url = self.connector.base_url
|
||||
if not self._base_url.endswith("/"):
|
||||
self._base_url += "/"
|
||||
|
||||
def discover_service(self, service: Optional[str] = None) -> List[Dict[str, Any]]:
|
||||
"""Fetch and parse ``$metadata`` into the service's entity sets.
|
||||
|
||||
Args:
|
||||
service: Service root URL (absolute) or path suffix resolved
|
||||
against the base URL. Defaults to the base URL. ``$metadata``
|
||||
is appended automatically — same meaning as in
|
||||
:meth:`ingest_entity_set`.
|
||||
|
||||
Returns:
|
||||
List of dicts, one per EntitySet, each with ``name`` and ``fields``
|
||||
(a list of ``{name, type, nullable}`` parsed from the CSDL).
|
||||
"""
|
||||
metadata_url = self._metadata_url(service)
|
||||
session = self.connector.get_session()
|
||||
resp = request_with_ssrf_guard(
|
||||
"GET",
|
||||
metadata_url,
|
||||
session=session,
|
||||
allow_private_ips=self.connector.allow_private_ips,
|
||||
headers={"Accept": "application/xml"},
|
||||
timeout=self.connector.config.get("timeout", 30),
|
||||
)
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
except requests.exceptions.RequestException as exc:
|
||||
self.logger.error("Failed to fetch SAP metadata %s: %s", metadata_url, exc)
|
||||
raise ProcessingError(f"Failed to fetch SAP $metadata: {exc}") from exc
|
||||
|
||||
return self._parse_metadata(resp.text)
|
||||
|
||||
def _metadata_url(self, service: Optional[str]) -> str:
|
||||
"""Build the ``$metadata`` URL for a service root.
|
||||
|
||||
``service`` has the same meaning as in :meth:`ingest_entity_set` —
|
||||
a service root (absolute URL or path suffix resolved against the
|
||||
base URL). ``$metadata`` is appended here, so callers pass the root
|
||||
the same way for both discovery and ingestion. A value already
|
||||
ending in ``$metadata`` is used as-is.
|
||||
"""
|
||||
if not service:
|
||||
return urljoin(self._base_url, "$metadata")
|
||||
if "://" not in service:
|
||||
service = urljoin(self._base_url, service)
|
||||
if service.endswith("$metadata"):
|
||||
return service
|
||||
if not service.endswith("/"):
|
||||
service += "/"
|
||||
return urljoin(service, "$metadata")
|
||||
|
||||
def _parse_metadata(self, metadata_xml: str) -> List[Dict[str, Any]]:
|
||||
"""Minimal CSDL/EDMX parser -> entity set name + property fields.
|
||||
|
||||
Element local names (``Schema``/``EntitySet``/``EntityType``/
|
||||
``Property``) are stable across OData v2 (Microsoft ns) and v4 (OASIS
|
||||
ns), so we match them by local name instead of hard-coding one
|
||||
namespace. Entity-Type references are resolved per-schema, so
|
||||
same-named types in different schemas cannot bleed fields into each
|
||||
other.
|
||||
"""
|
||||
try:
|
||||
root = ET.fromstring(metadata_xml)
|
||||
except ET.ParseError as exc:
|
||||
raise ProcessingError(f"SAP $metadata is not valid XML: {exc}") from exc
|
||||
|
||||
# Index fully-qualified type name -> property fields, per schema.
|
||||
schema_types: Dict[str, List[Dict[str, Any]]] = {}
|
||||
for schema in (e for e in root.iter() if _match(e, "Schema")):
|
||||
ns = (schema.get("Namespace") or schema.get("namespace") or "").rstrip(".")
|
||||
for entity_type in (e for e in schema.iter() if _match(e, "EntityType")):
|
||||
tname = entity_type.get("Name") or entity_type.get("name")
|
||||
if not tname:
|
||||
continue
|
||||
fq = f"{ns}.{tname}" if ns else tname
|
||||
schema_types[fq] = [
|
||||
{
|
||||
"name": prop.get("Name") or prop.get("name"),
|
||||
"type": prop.get("Type") or prop.get("type"),
|
||||
"nullable": _prop_is_nullable(prop),
|
||||
}
|
||||
for prop in (e for e in entity_type.iter() if _match(e, "Property"))
|
||||
]
|
||||
|
||||
entity_sets: List[Dict[str, Any]] = []
|
||||
for schema in (e for e in root.iter() if _match(e, "Schema")):
|
||||
ns = (schema.get("Namespace") or schema.get("namespace") or "").rstrip(".")
|
||||
for entity_set in (e for e in schema.iter() if _match(e, "EntitySet")):
|
||||
name = entity_set.get("Name") or entity_set.get("name")
|
||||
ref = entity_set.get("EntityType") or entity_set.get("entityType") or ""
|
||||
qualified = ref if "." in ref else (f"{ns}.{ref}" if ns else ref)
|
||||
fields = schema_types.get(qualified) or schema_types.get(ref) or []
|
||||
entity_sets.append({"name": name, "fields": fields})
|
||||
return entity_sets
|
||||
|
||||
def ingest_entity_set(
|
||||
self,
|
||||
service: Optional[str] = None,
|
||||
entity_set: Optional[str] = None,
|
||||
*,
|
||||
select: Optional[str] = None,
|
||||
filter: Optional[str] = None,
|
||||
expand: Optional[str] = None,
|
||||
top: Optional[int] = None,
|
||||
skip: Optional[int] = None,
|
||||
batch_size: int = 100,
|
||||
) -> SAPODataEntity:
|
||||
"""Fetch pages of *entity_set* from the OData service.
|
||||
|
||||
Args:
|
||||
service: Service root URL (absolute) or path suffix resolved
|
||||
against the base URL. Defaults to the base URL. Same meaning
|
||||
as in :meth:`discover_service`.
|
||||
entity_set: Entity set name, e.g. ``"SalesOrderSet"``.
|
||||
select: Optional ``$select`` comma string.
|
||||
filter: Optional ``$filter`` expression.
|
||||
expand: Optional ``$expand`` expression (e.g. ``"to_Item"`` for
|
||||
use case 2's sales-order headers -> line items).
|
||||
top: Maximum number of rows to return.
|
||||
skip: Number of leading rows to skip.
|
||||
batch_size: ``$top`` pagination size per request.
|
||||
|
||||
Returns:
|
||||
A single :class:`SAPODataEntity` holding every fetched record
|
||||
(server-driven pagination is followed to completion).
|
||||
"""
|
||||
if service is None:
|
||||
base = self._base_url
|
||||
elif "://" in service:
|
||||
base = service
|
||||
else:
|
||||
base = urljoin(self._base_url, service)
|
||||
if not base.endswith("/"):
|
||||
base += "/"
|
||||
if not entity_set:
|
||||
raise ValidationError("SAP 'entity_set' is required.")
|
||||
|
||||
session = self.connector.get_session()
|
||||
records: List[Dict[str, Any]] = []
|
||||
next_link: Optional[str] = urljoin(base, entity_set)
|
||||
params = self._query_params(select, filter, expand, top, skip, batch_size)
|
||||
|
||||
if top is not None and top < 0:
|
||||
raise ValidationError("SAP 'top' must be >= 0 (got %r)" % top)
|
||||
if top == 0:
|
||||
return SAPODataEntity(
|
||||
records=[], entity_set=entity_set, count=0, service=base
|
||||
)
|
||||
original_host = (urlparse(base).hostname or "").lower()
|
||||
|
||||
while next_link:
|
||||
# Server-provided next links may point anywhere; never send the
|
||||
# session credentials (Basic/Bearer) to a different origin than
|
||||
# the service root. Legit SAP pagination stays on the same host.
|
||||
next_host = (urlparse(next_link).hostname or "").lower()
|
||||
if next_host != original_host:
|
||||
raise ProcessingError(
|
||||
f"SAP next link '{next_link}' points to a different host "
|
||||
f"than service root '{base}'"
|
||||
)
|
||||
resp = request_with_ssrf_guard(
|
||||
"GET",
|
||||
next_link,
|
||||
session=session,
|
||||
allow_private_ips=self.connector.allow_private_ips,
|
||||
headers={"Accept": "application/json"},
|
||||
params=params,
|
||||
timeout=self.connector.config.get("timeout", 30),
|
||||
)
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
except requests.exceptions.RequestException as exc:
|
||||
self.logger.error(
|
||||
"Failed to fetch SAP entity set %s: %s", entity_set, exc
|
||||
)
|
||||
raise ProcessingError(
|
||||
f"Failed to fetch SAP entity set {entity_set}: {exc}"
|
||||
) from exc
|
||||
|
||||
payload = self._parse_page(resp)
|
||||
rows, next_link = payload["rows"], payload["next_link"]
|
||||
|
||||
for raw_row in rows:
|
||||
records.append(self._flatten_row(raw_row))
|
||||
|
||||
self.logger.debug(
|
||||
"Fetched %d rows from %s (next=%s)",
|
||||
len(rows),
|
||||
entity_set,
|
||||
bool(next_link),
|
||||
)
|
||||
if top is not None and len(records) >= top:
|
||||
break
|
||||
|
||||
params = None # query params already baked into the server next link
|
||||
# Refresh next_link against base in case it's a relative pointer.
|
||||
if next_link and not next_link.startswith("http"):
|
||||
next_link = urljoin(resp.url, next_link)
|
||||
|
||||
return SAPODataEntity(
|
||||
records=records,
|
||||
entity_set=entity_set,
|
||||
count=len(records),
|
||||
service=base,
|
||||
)
|
||||
|
||||
def _query_params(
|
||||
self,
|
||||
select: Optional[str],
|
||||
filter: Optional[str],
|
||||
expand: Optional[str],
|
||||
top_value: Optional[int],
|
||||
skip: Optional[int],
|
||||
batch_size: int,
|
||||
) -> Dict[str, str]:
|
||||
params: Dict[str, str] = {}
|
||||
if batch_size > 0:
|
||||
if top_value is not None:
|
||||
params["$top"] = str(min(batch_size, top_value))
|
||||
else:
|
||||
params["$top"] = str(batch_size)
|
||||
if select:
|
||||
params["$select"] = select
|
||||
if filter:
|
||||
params["$filter"] = filter
|
||||
if expand:
|
||||
params["$expand"] = expand
|
||||
if skip is not None:
|
||||
params["$skip"] = str(skip)
|
||||
return params
|
||||
|
||||
def _parse_page(self, resp: requests.Response) -> Dict[str, Any]:
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise ProcessingError(f"SAP OData response is not JSON: {exc}") from exc
|
||||
|
||||
rows: Any
|
||||
next_link: Optional[str] = None
|
||||
if isinstance(payload, list):
|
||||
rows = payload
|
||||
elif isinstance(payload, dict):
|
||||
d = payload.get("d")
|
||||
if isinstance(d, dict):
|
||||
# OData v2 atom: {"d": {"results": [...], "__next": ...}}
|
||||
rows = d.get("results")
|
||||
nxt = d.get("__next") or payload.get("@odata.nextLink")
|
||||
else:
|
||||
# OData v4 JSON: {"value": [...], "@odata.nextLink": ...}
|
||||
rows = payload.get("value", d)
|
||||
nxt = payload.get("@odata.nextLink")
|
||||
if isinstance(nxt, dict):
|
||||
nxt = nxt.get("__deferred", {}).get("uri")
|
||||
next_link = nxt
|
||||
else:
|
||||
rows = None
|
||||
|
||||
if not isinstance(rows, list):
|
||||
raise ProcessingError(
|
||||
"SAP OData payload has no list of rows (got %s)" % type(rows).__name__
|
||||
)
|
||||
return {"rows": rows, "next_link": next_link}
|
||||
|
||||
def _flatten_row(self, row: Any) -> Dict[str, Any]:
|
||||
if isinstance(row, dict):
|
||||
# v2 wraps items in "__metadata"; keep it but expose plain keys.
|
||||
return {k: v for k, v in row.items() if k != "__metadata"}
|
||||
return {"value": row}
|
||||
|
||||
def export_as_documents(self, data: SAPODataEntity) -> List[Dict[str, Any]]:
|
||||
"""Convert an ingested entity set to flat document dicts.
|
||||
|
||||
Normalizes every records held by ``data`` into a list of dicts with an
|
||||
injected ``id``/``name``/``source``, ready to hand to ``GraphBuilder``.
|
||||
"""
|
||||
return data.to_documents()
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the underlying connector's session."""
|
||||
self.connector.close()
|
||||
@@ -117,6 +117,8 @@ class PropertyGenerator:
|
||||
data_properties = self._infer_data_properties(entities, classes, **options)
|
||||
properties.extend(data_properties)
|
||||
|
||||
properties = self._coalesce_normalized_properties(properties)
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
@@ -193,6 +195,67 @@ class PropertyGenerator:
|
||||
|
||||
return properties
|
||||
|
||||
def _coalesce_normalized_properties(
|
||||
self, properties: List[Dict[str, Any]]
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Merge same-kind properties that normalize to the same name."""
|
||||
property_kinds = defaultdict(set)
|
||||
for prop in properties:
|
||||
property_kinds[prop["name"]].add(prop.get("type"))
|
||||
|
||||
collisions = {
|
||||
name: sorted(kind for kind in kinds if kind is not None)
|
||||
for name, kinds in property_kinds.items()
|
||||
if len({kind for kind in kinds if kind is not None}) > 1
|
||||
}
|
||||
if collisions:
|
||||
raise ValidationError(
|
||||
"Normalized property names cannot be shared by object and "
|
||||
"data properties.",
|
||||
validation_context={"property_kind_collisions": collisions},
|
||||
)
|
||||
|
||||
merged: Dict[tuple, Dict[str, Any]] = {}
|
||||
result = []
|
||||
for prop in properties:
|
||||
key = (prop.get("type"), prop["name"])
|
||||
existing = merged.get(key)
|
||||
if existing is None:
|
||||
merged[key] = prop
|
||||
result.append(prop)
|
||||
continue
|
||||
|
||||
existing["domain"] = self._merge_property_values(
|
||||
existing.get("domain", []), prop.get("domain", [])
|
||||
)
|
||||
if prop.get("type") == "object":
|
||||
existing["range"] = self._merge_property_values(
|
||||
existing.get("range", []), prop.get("range", [])
|
||||
)
|
||||
existing_metadata = existing.setdefault("metadata", {})
|
||||
existing_metadata["occurrence_count"] = (
|
||||
existing_metadata.get("occurrence_count", 0)
|
||||
+ prop.get("metadata", {}).get("occurrence_count", 0)
|
||||
)
|
||||
elif existing.get("range") != prop.get("range"):
|
||||
existing["range"] = self._get_more_general_type(
|
||||
existing["range"], prop["range"]
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _merge_property_values(current: Any, incoming: Any) -> List[Any]:
|
||||
"""Merge scalar-or-list property values while preserving input order."""
|
||||
values = list(current) if isinstance(current, list) else [current]
|
||||
incoming_values = (
|
||||
incoming if isinstance(incoming, list) else [incoming]
|
||||
)
|
||||
for value in incoming_values:
|
||||
if value not in values:
|
||||
values.append(value)
|
||||
return [value for value in values if value is not None]
|
||||
|
||||
def _infer_data_properties(
|
||||
self, entities: List[Dict[str, Any]], classes: List[Dict[str, Any]], **options
|
||||
) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -70,7 +70,8 @@ sem:metadata a owl:AnnotationProperty ;
|
||||
rdfs:label "metadata" ;
|
||||
rdfs:comment """Free-form metadata carried through from extraction. An
|
||||
annotation property because its value is an arbitrary structure rather than a
|
||||
modelled one.""" ;
|
||||
modelled one; in the JSON-LD export the whole mapping is written as one
|
||||
rdf:JSON literal so caller keys never expand into this namespace (#1146).""" ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
# ── Relationship terms (JSON-LD export) ──────────────────────────────────────
|
||||
@@ -96,10 +97,10 @@ sem:target a owl:ObjectProperty ;
|
||||
|
||||
sem:type a owl:DatatypeProperty ;
|
||||
rdfs:label "type" ;
|
||||
rdfs:comment """The relationship type as a label, as emitted in the JSON-LD
|
||||
export. Distinct from rdf:type, which relates a node to a class rather than to
|
||||
a string.""" ;
|
||||
rdfs:domain sem:Relationship ;
|
||||
rdfs:comment """The entity or relationship type as a label, as emitted in
|
||||
the JSON-LD export. Distinct from rdf:type, which relates a node to a class
|
||||
rather than to a string. Emitted for both entities and relationships, so the
|
||||
domain is left open rather than tied to sem:Relationship.""" ;
|
||||
rdfs:range xsd:string ;
|
||||
rdfs:isDefinedBy <https://semantica.dev/ns> .
|
||||
|
||||
|
||||
@@ -32,8 +32,10 @@ Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
import copy
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
@@ -59,6 +61,14 @@ class PipelineStatus(Enum):
|
||||
STOPPED = "stopped"
|
||||
|
||||
|
||||
class _ParallelResultContractError(ProcessingError):
|
||||
"""Raised when a parallel step violates the dict-result contract.
|
||||
|
||||
Contract violations are deterministic: re-running the handler cannot
|
||||
change its return type, so the retry loop must be skipped entirely.
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExecutionResult:
|
||||
"""Pipeline execution result."""
|
||||
@@ -242,11 +252,67 @@ class ExecutionEngine:
|
||||
return ExecutionResult(success=False, output=None, errors=[str(e)])
|
||||
|
||||
def _execute_steps(self, pipeline: Pipeline, data: Any, **options) -> Any:
|
||||
"""Execute pipeline steps."""
|
||||
# Sort steps by dependencies (topological sort)
|
||||
sorted_steps = self._topological_sort(pipeline.steps)
|
||||
"""
|
||||
Execute pipeline steps.
|
||||
|
||||
Steps are grouped into dependency layers. When a pipeline is
|
||||
configured with parallelism > 1, a layer whose steps are all marked
|
||||
``parallel_safe`` and whose input is a dict is executed concurrently
|
||||
(bounded by the effective parallelism); every other layer runs
|
||||
sequentially, preserving the default serial behaviour.
|
||||
"""
|
||||
effective_parallelism = self._get_effective_parallelism(pipeline)
|
||||
|
||||
if effective_parallelism <= 1:
|
||||
return self._execute_steps_sequential(pipeline, data, **options)
|
||||
|
||||
layers = self._group_steps_by_dependency_level(pipeline.steps)
|
||||
|
||||
current_data = data
|
||||
for layer in layers:
|
||||
if self.pipeline_status.get(pipeline.name) == PipelineStatus.STOPPED:
|
||||
break
|
||||
|
||||
# Wait if paused
|
||||
while self.pipeline_status.get(pipeline.name) == PipelineStatus.PAUSED:
|
||||
time.sleep(0.1)
|
||||
|
||||
if self._can_run_layer_in_parallel(layer, current_data):
|
||||
merged = self._execute_parallel_group(
|
||||
layer,
|
||||
current_data,
|
||||
effective_parallelism,
|
||||
pipeline_name=pipeline.name,
|
||||
**options,
|
||||
)
|
||||
if merged is None:
|
||||
# Input isolation failed before any handler started;
|
||||
# run this layer sequentially instead.
|
||||
current_data = self._execute_steps_sequential(
|
||||
pipeline, current_data, steps=layer, **options
|
||||
)
|
||||
else:
|
||||
current_data = merged
|
||||
else:
|
||||
current_data = self._execute_steps_sequential(
|
||||
pipeline, current_data, steps=layer, **options
|
||||
)
|
||||
|
||||
return current_data
|
||||
|
||||
def _execute_steps_sequential(
|
||||
self,
|
||||
pipeline: Pipeline,
|
||||
data: Any,
|
||||
steps: Optional[List[PipelineStep]] = None,
|
||||
**options,
|
||||
) -> Any:
|
||||
"""Execute steps sequentially following dependency order."""
|
||||
if steps is None:
|
||||
sorted_steps = self._topological_sort(pipeline.steps)
|
||||
else:
|
||||
sorted_steps = list(steps)
|
||||
|
||||
# Execute steps
|
||||
current_data = data
|
||||
total_steps = len(sorted_steps)
|
||||
|
||||
@@ -258,76 +324,320 @@ class ExecutionEngine:
|
||||
while self.pipeline_status.get(pipeline.name) == PipelineStatus.PAUSED:
|
||||
time.sleep(0.1)
|
||||
|
||||
# Track step execution
|
||||
step_tracking_id = self.progress_tracker.start_tracking(
|
||||
module="pipeline",
|
||||
submodule=step.step_type or step.name,
|
||||
message=f"Step {step_idx + 1}/{total_steps}: {step.name}",
|
||||
current_data = self._execute_step_with_retries(
|
||||
step,
|
||||
current_data,
|
||||
step_label=f"Step {step_idx + 1}/{total_steps}: {step.name}",
|
||||
pipeline_name=pipeline.name,
|
||||
**options,
|
||||
)
|
||||
|
||||
try:
|
||||
# Execute step
|
||||
step.status = StepStatus.RUNNING
|
||||
step_result = self._execute_step(step, current_data, **options)
|
||||
step.status = StepStatus.COMPLETED
|
||||
step.result = step_result
|
||||
current_data = step_result
|
||||
return current_data
|
||||
|
||||
def _execute_step_with_retries(
|
||||
self,
|
||||
step: PipelineStep,
|
||||
data: Any,
|
||||
step_label: Optional[str] = None,
|
||||
pipeline_name: Optional[str] = None,
|
||||
require_dict_result: bool = False,
|
||||
**options,
|
||||
) -> Any:
|
||||
"""
|
||||
Execute a single step with retry handling.
|
||||
|
||||
Shared by the sequential and parallel execution paths so that retry
|
||||
policies, step status tracking and progress reporting behave
|
||||
identically. Returns the step result, or raises the final error
|
||||
after retries are exhausted.
|
||||
|
||||
When ``require_dict_result`` is set (parallel layers), a handler
|
||||
returning a non-dict raises ProcessingError before the step is
|
||||
marked completed, so status and progress reporting stay consistent.
|
||||
Such contract violations are never retried: the handler's return
|
||||
type cannot change between attempts.
|
||||
"""
|
||||
step_tracking_id = self.progress_tracker.start_tracking(
|
||||
module="pipeline",
|
||||
# Pipeline identity + step name keep tracking IDs unique so
|
||||
# concurrent steps of the same step_type cannot overwrite each
|
||||
# other's progress records.
|
||||
submodule=(
|
||||
f"{pipeline_name or 'pipeline'}:"
|
||||
f"{step.step_type or 'step'}:{step.name}"
|
||||
),
|
||||
message=step_label or f"Executing step: {step.name}",
|
||||
)
|
||||
|
||||
try:
|
||||
step.status = StepStatus.RUNNING
|
||||
step_result = self._execute_step(step, data, **options)
|
||||
if require_dict_result and not isinstance(step_result, dict):
|
||||
raise _ParallelResultContractError(
|
||||
f"Step '{step.name}' is marked parallel_safe and must "
|
||||
f"return a dict so parallel results can be merged, got "
|
||||
f"{type(step_result).__name__}"
|
||||
)
|
||||
step.status = StepStatus.COMPLETED
|
||||
step.result = step_result
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
step_tracking_id,
|
||||
status="completed",
|
||||
message=f"Completed step: {step.name}",
|
||||
)
|
||||
return step_result
|
||||
|
||||
except Exception as e:
|
||||
step.status = StepStatus.FAILED
|
||||
step.error = e
|
||||
|
||||
# Contract violations are deterministic failures: re-running
|
||||
# the handler cannot change its return type, so never consult
|
||||
# the retry policy for them.
|
||||
if isinstance(e, _ParallelResultContractError):
|
||||
self.progress_tracker.stop_tracking(
|
||||
step_tracking_id, status="failed", message=str(e)
|
||||
)
|
||||
raise
|
||||
|
||||
# Retry loop respecting max_retries from the policy
|
||||
retry_policy = self.failure_handler.get_retry_policy(step.step_type)
|
||||
max_retries = retry_policy.max_retries if retry_policy else 0
|
||||
retry_count = 0
|
||||
success = False
|
||||
|
||||
while retry_count < max_retries:
|
||||
recovery_result = self.failure_handler.handle_step_failure(step, e)
|
||||
if not recovery_result.get("retry", False):
|
||||
break
|
||||
retry_delay = recovery_result.get("retry_delay", 0.0)
|
||||
if retry_delay > 0:
|
||||
time.sleep(retry_delay)
|
||||
self.progress_tracker.update_tracking(
|
||||
step_tracking_id,
|
||||
status="running",
|
||||
message=f"Retrying step: {step.name} (attempt {retry_count + 1})",
|
||||
)
|
||||
step.status = StepStatus.RUNNING
|
||||
try:
|
||||
step_result = self._execute_step(step, data, **options)
|
||||
if require_dict_result and not isinstance(
|
||||
step_result, dict
|
||||
):
|
||||
raise _ParallelResultContractError(
|
||||
f"Step '{step.name}' is marked parallel_safe "
|
||||
f"and must return a dict so parallel results "
|
||||
f"can be merged, got "
|
||||
f"{type(step_result).__name__}"
|
||||
)
|
||||
step.status = StepStatus.COMPLETED
|
||||
step.result = step_result
|
||||
success = True
|
||||
break
|
||||
except Exception as retry_e:
|
||||
step.status = StepStatus.FAILED
|
||||
step.error = retry_e
|
||||
e = retry_e
|
||||
retry_count += 1
|
||||
|
||||
if success:
|
||||
self.progress_tracker.stop_tracking(
|
||||
step_tracking_id,
|
||||
status="completed",
|
||||
message=f"Completed step: {step.name}",
|
||||
message=f"Retry successful: {step.name}",
|
||||
)
|
||||
return step_result
|
||||
else:
|
||||
self.progress_tracker.stop_tracking(
|
||||
step_tracking_id, status="failed", message=str(e)
|
||||
)
|
||||
raise e
|
||||
|
||||
except Exception as e:
|
||||
step.status = StepStatus.FAILED
|
||||
step.error = e
|
||||
def _get_effective_parallelism(self, pipeline: Pipeline) -> int:
|
||||
"""Return the parallelism actually used for this pipeline."""
|
||||
configured = pipeline.config.get("parallelism", 1)
|
||||
if not isinstance(configured, int) or configured <= 0:
|
||||
return 1
|
||||
return min(configured, self.parallelism_manager.max_workers)
|
||||
|
||||
# Retry loop respecting max_retries from the policy
|
||||
retry_policy = self.failure_handler.get_retry_policy(step.step_type)
|
||||
max_retries = retry_policy.max_retries if retry_policy else 0
|
||||
retry_count = 0
|
||||
success = False
|
||||
def _group_steps_by_dependency_level(
|
||||
self, steps: List[PipelineStep]
|
||||
) -> List[List[PipelineStep]]:
|
||||
"""
|
||||
Group steps into dependency layers, preserving declaration order.
|
||||
|
||||
while retry_count < max_retries:
|
||||
recovery_result = self.failure_handler.handle_step_failure(step, e)
|
||||
if not recovery_result.get("retry", False):
|
||||
break
|
||||
retry_delay = recovery_result.get("retry_delay", 0.0)
|
||||
if retry_delay > 0:
|
||||
time.sleep(retry_delay)
|
||||
self.progress_tracker.update_tracking(
|
||||
step_tracking_id,
|
||||
status="running",
|
||||
message=f"Retrying step: {step.name} (attempt {retry_count + 1})",
|
||||
Circular or unknown dependencies raise ValidationError so the
|
||||
parallel path fails deterministically, matching the validation
|
||||
behaviour of the serial topological sort.
|
||||
"""
|
||||
step_map = {step.name: step for step in steps}
|
||||
levels: Dict[str, int] = {}
|
||||
visiting: set = set()
|
||||
|
||||
for step in steps:
|
||||
for dep in step.dependencies:
|
||||
if dep not in step_map:
|
||||
raise ValidationError(
|
||||
f"Step '{step.name}' depends on unknown step '{dep}'"
|
||||
)
|
||||
step.status = StepStatus.RUNNING
|
||||
try:
|
||||
step_result = self._execute_step(step, current_data, **options)
|
||||
step.status = StepStatus.COMPLETED
|
||||
step.result = step_result
|
||||
current_data = step_result
|
||||
success = True
|
||||
break
|
||||
except Exception as retry_e:
|
||||
step.status = StepStatus.FAILED
|
||||
step.error = retry_e
|
||||
e = retry_e
|
||||
retry_count += 1
|
||||
|
||||
if success:
|
||||
self.progress_tracker.stop_tracking(
|
||||
step_tracking_id,
|
||||
status="completed",
|
||||
message=f"Retry successful: {step.name}",
|
||||
)
|
||||
def get_level(step_name: str) -> int:
|
||||
if step_name in levels:
|
||||
return levels[step_name]
|
||||
if step_name in visiting:
|
||||
raise ValidationError(
|
||||
"Circular dependency detected in pipeline "
|
||||
f"(cycle passes through step '{step_name}')"
|
||||
)
|
||||
visiting.add(step_name)
|
||||
step = step_map[step_name]
|
||||
if not step.dependencies:
|
||||
level = 0
|
||||
else:
|
||||
level = max(get_level(dep) for dep in step.dependencies) + 1
|
||||
visiting.discard(step_name)
|
||||
levels[step_name] = level
|
||||
return level
|
||||
|
||||
for step in steps:
|
||||
get_level(step.name)
|
||||
|
||||
grouped: Dict[int, List[PipelineStep]] = {}
|
||||
for step in steps:
|
||||
grouped.setdefault(levels[step.name], []).append(step)
|
||||
return [grouped[level] for level in sorted(grouped)]
|
||||
|
||||
def _can_run_layer_in_parallel(self, layer: List[PipelineStep], data: Any) -> bool:
|
||||
"""Check whether a dependency layer can safely run in parallel."""
|
||||
if len(layer) <= 1:
|
||||
return False
|
||||
if not isinstance(data, dict):
|
||||
return False
|
||||
for step in layer:
|
||||
# Strict boolean check: truthy non-bool values (e.g. the
|
||||
# string "false") must never opt a step into concurrency.
|
||||
if getattr(step, "parallel_safe", False) is not True:
|
||||
return False
|
||||
if getattr(step, "delta_mode", False):
|
||||
return False
|
||||
return True
|
||||
|
||||
def _execute_parallel_group(
|
||||
self,
|
||||
layer: List[PipelineStep],
|
||||
data: Any,
|
||||
effective_parallelism: int,
|
||||
pipeline_name: Optional[str] = None,
|
||||
**options,
|
||||
) -> Optional[Any]:
|
||||
"""
|
||||
Execute a dependency layer concurrently.
|
||||
|
||||
Per-step inputs are deep-copied before any handler starts so that
|
||||
parallel steps do not share mutable state. Returns the merged dict
|
||||
result, or None when input isolation failed (before any handler
|
||||
ran) and the layer should fall back to sequential execution.
|
||||
"""
|
||||
# Isolate per-step inputs before starting any handler
|
||||
try:
|
||||
step_inputs = {step.name: copy.deepcopy(data) for step in layer}
|
||||
except Exception as e:
|
||||
self.logger.warning(
|
||||
f"Falling back to sequential execution: input for parallel "
|
||||
f"layer could not be isolated ({e})"
|
||||
)
|
||||
return None
|
||||
|
||||
step_results: Dict[str, Any] = {}
|
||||
failure: Optional[BaseException] = None
|
||||
max_workers = min(effective_parallelism, len(layer))
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
futures = {
|
||||
executor.submit(
|
||||
self._execute_step_with_retries,
|
||||
step,
|
||||
step_inputs[step.name],
|
||||
# Validate the dict-result contract inside the shared
|
||||
# lifecycle path, before completion is reported.
|
||||
pipeline_name=pipeline_name,
|
||||
require_dict_result=True,
|
||||
**options,
|
||||
): step
|
||||
for step in layer
|
||||
}
|
||||
|
||||
for future in as_completed(futures):
|
||||
step = futures[future]
|
||||
try:
|
||||
step_result = future.result()
|
||||
except Exception as e:
|
||||
if failure is None:
|
||||
failure = e
|
||||
# Cancel steps that have not started yet
|
||||
for pending in futures:
|
||||
pending.cancel()
|
||||
else:
|
||||
self.progress_tracker.stop_tracking(
|
||||
step_tracking_id, status="failed", message=str(e)
|
||||
step_results[step.name] = step_result
|
||||
|
||||
if failure is not None:
|
||||
raise failure
|
||||
|
||||
return self._merge_parallel_results(data, layer, step_results)
|
||||
|
||||
def _merge_parallel_results(
|
||||
self,
|
||||
base: Dict[str, Any],
|
||||
layer: List[PipelineStep],
|
||||
step_results: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Merge the results of a parallel layer into a single dict.
|
||||
|
||||
Steps are processed in declaration order (never by completion
|
||||
order). Keys whose values are unchanged from the shared base input
|
||||
are skipped (handlers commonly return complete dicts such as
|
||||
``{**data, ...}``); only added or changed keys count as branch
|
||||
writes. Keys written with equal values by multiple steps are
|
||||
allowed; conflicting values for the same key raise a
|
||||
ProcessingError naming the key and both steps. Ambiguous equality
|
||||
comparisons count as changed, never as unchanged.
|
||||
"""
|
||||
merged = dict(base)
|
||||
key_sources: Dict[str, str] = {}
|
||||
|
||||
for step in layer:
|
||||
step_result = step_results.get(step.name)
|
||||
if step_result is None:
|
||||
continue
|
||||
for key, value in step_result.items():
|
||||
if key in base and self._values_equal(base[key], value):
|
||||
# Echo of the shared input: not a branch write, so it
|
||||
# cannot conflict with a sibling that changes the key.
|
||||
continue
|
||||
if key in key_sources and not self._values_equal(
|
||||
merged.get(key), value
|
||||
):
|
||||
raise ProcessingError(
|
||||
f"Conflicting values for key '{key}' in parallel step "
|
||||
f"results: step '{step.name}' produced {value!r}, but "
|
||||
f"step '{key_sources[key]}' previously produced "
|
||||
f"{merged.get(key)!r}"
|
||||
)
|
||||
raise e
|
||||
key_sources[key] = step.name
|
||||
merged[key] = value
|
||||
|
||||
return merged
|
||||
|
||||
@staticmethod
|
||||
def _values_equal(left: Any, right: Any) -> bool:
|
||||
"""Safely compare two values; ambiguous comparisons count as conflicts."""
|
||||
try:
|
||||
return bool(left == right)
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
|
||||
return current_data
|
||||
|
||||
def _execute_step(self, step: PipelineStep, data: Any, **options) -> Any:
|
||||
"""
|
||||
|
||||
@@ -67,6 +67,7 @@ class PipelineStep:
|
||||
delta_mode: bool = False
|
||||
base_version_id: Optional[str] = None
|
||||
target_version_id: Optional[str] = None
|
||||
parallel_safe: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -133,6 +134,12 @@ class PipelineBuilder:
|
||||
target_version_id = config.pop("target_version_id", None)
|
||||
dependencies = config.pop("dependencies", [])
|
||||
handler = config.pop("handler", None)
|
||||
parallel_safe = config.pop("parallel_safe", False)
|
||||
if not isinstance(parallel_safe, bool):
|
||||
raise ValidationError(
|
||||
f"parallel_safe must be a boolean, got "
|
||||
f"{type(parallel_safe).__name__} for step '{step_name}'"
|
||||
)
|
||||
if handler is None:
|
||||
handler = self.step_registry.get(step_type)
|
||||
|
||||
@@ -145,6 +152,7 @@ class PipelineBuilder:
|
||||
delta_mode = delta_mode,
|
||||
base_version_id=base_version_id,
|
||||
target_version_id=target_version_id,
|
||||
parallel_safe=parallel_safe,
|
||||
)
|
||||
|
||||
self.steps.append(step)
|
||||
@@ -186,6 +194,14 @@ class PipelineBuilder:
|
||||
Returns:
|
||||
Self for method chaining
|
||||
"""
|
||||
if (
|
||||
isinstance(level, bool)
|
||||
or not isinstance(level, int)
|
||||
or level <= 0
|
||||
):
|
||||
raise ValidationError(
|
||||
f"Parallelism level must be a positive integer, got {level!r}"
|
||||
)
|
||||
self.pipeline_config["parallelism"] = level
|
||||
return self
|
||||
|
||||
@@ -289,6 +305,16 @@ class PipelineBuilder:
|
||||
step.target_version_id = step_config.get(
|
||||
"target_version_id", step.target_version_id
|
||||
)
|
||||
raw_parallel_safe = step_config.get(
|
||||
"parallel_safe", step.parallel_safe
|
||||
)
|
||||
if not isinstance(raw_parallel_safe, bool):
|
||||
raise ValidationError(
|
||||
"parallel_safe must be a boolean for step "
|
||||
f"'{step_name}', got "
|
||||
f"{type(raw_parallel_safe).__name__}"
|
||||
)
|
||||
step.parallel_safe = raw_parallel_safe
|
||||
|
||||
# Set parallelism if specified
|
||||
if "parallelism" in pipeline_config:
|
||||
@@ -344,6 +370,7 @@ class PipelineBuilder:
|
||||
"type": step.step_type,
|
||||
"config": step.config,
|
||||
"dependencies": step.dependencies,
|
||||
"parallel_safe": step.parallel_safe,
|
||||
}
|
||||
for step in self.steps
|
||||
],
|
||||
@@ -425,6 +452,7 @@ class PipelineSerializer:
|
||||
"delta_mode",
|
||||
"base_version_id",
|
||||
"target_version_id",
|
||||
"parallel_safe",
|
||||
}
|
||||
pipeline_data = {
|
||||
"name": pipeline.name,
|
||||
@@ -441,6 +469,7 @@ class PipelineSerializer:
|
||||
"delta_mode": getattr(step, "delta_mode", False),
|
||||
"base_version_id": getattr(step, "base_version_id", None),
|
||||
"target_version_id": getattr(step, "target_version_id", None),
|
||||
"parallel_safe": getattr(step, "parallel_safe", False),
|
||||
}
|
||||
for step in pipeline.steps
|
||||
],
|
||||
@@ -488,6 +517,12 @@ class PipelineSerializer:
|
||||
sanitized_steps.append(sanitized_step)
|
||||
pipeline_data["steps"] = sanitized_steps
|
||||
|
||||
# Reapply pipeline-level config (e.g. parallelism) at the top level
|
||||
# so build_pipeline picks it up
|
||||
serialized_config = pipeline_data.pop("config", None) or {}
|
||||
for key, value in serialized_config.items():
|
||||
pipeline_data.setdefault(key, value)
|
||||
|
||||
# Reconstruct pipeline
|
||||
builder = PipelineBuilder(**self.config)
|
||||
pipeline = builder.build_pipeline(pipeline_data, **options)
|
||||
|
||||
@@ -67,7 +67,7 @@ def test_merging_repeated_exports_yields_one_graph_node(tmp_path):
|
||||
assert len(kg_nodes) == 1
|
||||
|
||||
entity_nodes = set(
|
||||
merged.subjects(RDF.type, URIRef("https://semantica.dev/vocab/ORG"))
|
||||
merged.subjects(RDF.type, URIRef("https://semantica.dev/ns#Entity"))
|
||||
)
|
||||
assert len(entity_nodes) == 1
|
||||
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Caller data must never expand into the Semantica namespace (#1146).
|
||||
|
||||
``@vocab`` used to sit in every JSON-LD context pointing at ``ns#``, so every
|
||||
bare term in caller data expanded into it: an extracted type like ``"ORG"``
|
||||
became ``ns#ORG`` (a term the vocabulary does not define), and a metadata key
|
||||
like ``"source"`` collided with the real ``sem:source`` object property,
|
||||
attaching a plain string to a property whose range is a resource. The fix
|
||||
removes ``@vocab`` outright: only explicit ``semantica:``-prefixed terms
|
||||
resolve, caller type labels travel as ``semantica:type`` strings, and caller
|
||||
metadata survives as one ``rdf:JSON`` literal.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
from rdflib import RDF, Graph, Literal, URIRef
|
||||
|
||||
from semantica.export.json_exporter import JSONExporter
|
||||
from semantica.export.rdf_exporter import RDFExporter, SEMANTICA_NS
|
||||
|
||||
NS = SEMANTICA_NS
|
||||
E1 = "https://example.org/e1"
|
||||
|
||||
KG = {
|
||||
"entities": [
|
||||
{
|
||||
"id": E1,
|
||||
"text": "Acme",
|
||||
"type": "ORG",
|
||||
"metadata": {"source": "crm_export_2024"},
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"source_id": E1,
|
||||
"target_id": "https://example.org/e2",
|
||||
"type": "employs",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def _jsonld_file(exporter, kind, tmp_path, name):
|
||||
path = tmp_path / name
|
||||
if kind == "knowledge_graph":
|
||||
exporter.export_knowledge_graph(KG, path, format="json-ld")
|
||||
elif kind == "entities":
|
||||
exporter.export_entities(KG["entities"], path, format="json-ld")
|
||||
elif kind == "relationships":
|
||||
exporter.export_relationships(KG["relationships"], path, format="json-ld")
|
||||
elif kind == "generic":
|
||||
exporter.export({"note": "plain payload, no @id"}, path, format="json-ld")
|
||||
else:
|
||||
raise AssertionError(kind)
|
||||
return json.loads(path.read_text())
|
||||
|
||||
|
||||
def test_no_jsonld_context_declares_a_vocab(tmp_path):
|
||||
exporter = JSONExporter()
|
||||
for kind in ("knowledge_graph", "entities", "relationships", "generic"):
|
||||
context = _jsonld_file(exporter, kind, tmp_path, f"{kind}.jsonld")[
|
||||
"@context"
|
||||
]
|
||||
assert "@vocab" not in context, f"{kind}: @vocab expands caller data"
|
||||
|
||||
context = json.loads(RDFExporter().export_to_rdf(KG, format="jsonld"))[
|
||||
"@context"
|
||||
]
|
||||
assert "@vocab" not in context
|
||||
|
||||
|
||||
def test_extracted_type_labels_stay_out_of_the_namespace(tmp_path):
|
||||
path = tmp_path / "kg.jsonld"
|
||||
JSONExporter().export_knowledge_graph(KG, path, format="json-ld")
|
||||
|
||||
graph = Graph()
|
||||
graph.parse(str(path), format="json-ld")
|
||||
|
||||
assert (None, RDF.type, URIRef(NS + "ORG")) not in graph, (
|
||||
"the caller's type label was minted as a class in ns#"
|
||||
)
|
||||
assert (URIRef(E1), RDF.type, URIRef(NS + "Entity")) in graph
|
||||
assert (URIRef(E1), URIRef(NS + "type"), Literal("ORG")) in graph, (
|
||||
"the label itself must survive, as a string"
|
||||
)
|
||||
|
||||
|
||||
def test_metadata_keys_stay_out_of_the_namespace(tmp_path):
|
||||
path = tmp_path / "kg.jsonld"
|
||||
JSONExporter().export_knowledge_graph(KG, path, format="json-ld")
|
||||
|
||||
graph = Graph()
|
||||
graph.parse(str(path), format="json-ld")
|
||||
|
||||
assert (None, URIRef(NS + "source"), Literal("crm_export_2024")) not in (
|
||||
graph
|
||||
), "caller metadata value attached to the real sem:source object property"
|
||||
for _, _, o in graph.triples((None, URIRef(NS + "source"), None)):
|
||||
assert not isinstance(o, Literal), (
|
||||
"sem:source has a resource range but received a plain literal"
|
||||
)
|
||||
|
||||
literals = [
|
||||
o
|
||||
for o in graph.objects(None, URIRef(NS + "metadata"))
|
||||
if isinstance(o, Literal)
|
||||
]
|
||||
assert literals, "the metadata dict was dropped instead of preserved"
|
||||
assert literals[0].datatype == RDF.JSON
|
||||
assert json.loads(str(literals[0])) == {"source": "crm_export_2024"}
|
||||
|
||||
|
||||
def test_rdf_exporter_jsonld_keeps_type_labels_out_of_the_namespace():
|
||||
graph = Graph()
|
||||
graph.parse(
|
||||
data=RDFExporter().export_to_rdf(KG, format="jsonld"), format="json-ld"
|
||||
)
|
||||
|
||||
assert (None, RDF.type, URIRef(NS + "ORG")) not in graph
|
||||
assert (URIRef(E1), RDF.type, URIRef(NS + "Entity")) in graph
|
||||
assert (URIRef(E1), URIRef(NS + "type"), Literal("ORG")) in graph
|
||||
@@ -0,0 +1,663 @@
|
||||
"""Tests for the SAP OData ingestor.
|
||||
|
||||
The SAP connector never touches a live SAP system: every outbound request goes
|
||||
through ``semantica.ingest.ssrf.request_with_ssrf_guard`` (see
|
||||
``sap_ingestor.py``), so these tests patch that single entry point and drive
|
||||
the parser / connector / ingestor with canned responses.
|
||||
|
||||
Covered:
|
||||
- ``$metadata`` (CSDL XML) -> entity set discovery & property fields
|
||||
- OData v2 atom (``__next``) and OData v4 (``@odata.nextLink``) pagination
|
||||
- Basic and OAuth2 credential payloads propagated on the request
|
||||
- SSRF guard is used for data *and* token-exchange requests
|
||||
- ``export_as_documents`` yields the flat document shape GraphBuilder expects
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from semantica.ingest import SAPIngestor, SAPODataConnector, SAPODataEntity
|
||||
from semantica.utils.exceptions import ProcessingError, ValidationError
|
||||
|
||||
METADATA_V2 = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx">
|
||||
<edmx:DataServices
|
||||
xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
|
||||
<Schema Namespace="SAPService" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
|
||||
<EntityContainer Name="Default" m:IsDefaultEntityContainer="true">
|
||||
<EntitySet Name="SalesOrderSet" EntityType="SAPService.SalesOrder"/>
|
||||
<EntitySet Name="CustomerSet" EntityType="SAPService.Customer"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="SalesOrder">
|
||||
<Key><PropertyRef Name="SalesOrderID"/></Key>
|
||||
<Property Name="SalesOrderID" Type="Edm.String" Nullable="false"/>
|
||||
<Property Name="CustomerID" Type="Edm.String"/>
|
||||
<Property Name="GrossAmount" Type="Edm.Decimal"/>
|
||||
</EntityType>
|
||||
<EntityType Name="Customer">
|
||||
<Key><PropertyRef Name="CustomerID"/></Key>
|
||||
<Property Name="CustomerID" Type="Edm.String" Nullable="false"/>
|
||||
<Property Name="Name" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
</edmx:DataServices>
|
||||
</edmx:Edmx>
|
||||
"""
|
||||
|
||||
METADATA_V4 = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
|
||||
<edmx:DataServices>
|
||||
<Schema Namespace="SalesNs" xmlns="http://docs.oasis-open.org/odata/ns/edm">
|
||||
<EntityContainer Name="Svc">
|
||||
<EntitySet Name="SalesOrders" EntityType="SalesNs.SalesOrder"/>
|
||||
<EntitySet Name="Customers" EntityType="SalesNs.Customer"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="SalesOrder">
|
||||
<Property Name="ID" Type="Edm.String"/>
|
||||
<Property Name="Total" Type="Edm.Decimal" Nullable="false"/>
|
||||
</EntityType>
|
||||
<EntityType Name="Customer">
|
||||
<Property Name="ID" Type="Edm.String"/>
|
||||
<Property Name="Name" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
</edmx:DataServices>
|
||||
</edmx:Edmx>
|
||||
"""
|
||||
|
||||
METADATA_MULTI_SCHEMA = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx">
|
||||
<edmx:DataServices>
|
||||
<Schema Namespace="OrdersNs" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
|
||||
<EntityContainer Name="OrdersSvc">
|
||||
<EntitySet Name="OrdersNs" EntityType="OrdersNs.Row"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="Row">
|
||||
<Property Name="A" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
<Schema Namespace="InvoicesNs" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
|
||||
<EntityContainer Name="InvoicesSvc">
|
||||
<EntitySet Name="InvoicesNs" EntityType="InvoicesNs.Row"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="Row">
|
||||
<Property Name="B" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
</edmx:DataServices>
|
||||
</edmx:Edmx>
|
||||
"""
|
||||
|
||||
|
||||
def _fake_response(status_code=200, json_payload=None, text="", headers=None):
|
||||
"""requests.Response-like stand-in returned by the mocked guard."""
|
||||
resp = MagicMock()
|
||||
resp.status_code = status_code
|
||||
resp.headers = headers or {}
|
||||
resp.text = text
|
||||
resp.url = "https://sap.example/odata/$metadata"
|
||||
if json_payload is not None:
|
||||
resp.json.return_value = json_payload
|
||||
else:
|
||||
resp.json.side_effect = ValueError("not json")
|
||||
if status_code >= 400:
|
||||
resp.raise_for_status.side_effect = requests.exceptions.HTTPError(
|
||||
f"{status_code} error"
|
||||
)
|
||||
return resp
|
||||
|
||||
|
||||
def _json_resp(payload, url):
|
||||
r = MagicMock()
|
||||
r.status_code = 200
|
||||
r.headers = {"content-type": "application/json"}
|
||||
r.text = ""
|
||||
r.url = url
|
||||
r.json.return_value = payload
|
||||
return r
|
||||
|
||||
|
||||
class TestDiscoverService:
|
||||
def test_metadata_parses_entity_sets_and_fields(self):
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {"content-type": "application/xml"}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError("xml not json")
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
sets = ing.discover_service()
|
||||
|
||||
assert guard.call_count == 1
|
||||
assert [s["name"] for s in sets] == ["SalesOrderSet", "CustomerSet"]
|
||||
sales = next(s for s in sets if s["name"] == "SalesOrderSet")
|
||||
assert {f["name"] for f in sales["fields"]} == {
|
||||
"SalesOrderID",
|
||||
"CustomerID",
|
||||
"GrossAmount",
|
||||
}
|
||||
assert sales["fields"][0]["type"] == "Edm.String"
|
||||
|
||||
def test_discover_flow_routes_via_ssrf(self):
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch("semantica.ingest.sap_ingestor.request_with_ssrf_guard") as guard:
|
||||
guard.return_value = resp
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
ing.discover_service()
|
||||
|
||||
method, url = guard.call_args[0]
|
||||
assert method == "GET"
|
||||
assert url.endswith("$metadata")
|
||||
|
||||
def test_metadata_url_keeps_last_segment_without_trailing_slash(self):
|
||||
"""base_url without a trailing '/' must not lose its last segment.
|
||||
|
||||
urljoin() replaces the final path segment when the base has no
|
||||
trailing slash, which would silently turn .../API_BUSINESS_PARTNER
|
||||
into .../$metadata against the wrong service root.
|
||||
"""
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/API_BUSINESS_PARTNER",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.discover_service()
|
||||
|
||||
method, url = guard.call_args[0]
|
||||
assert url == "https://sap.example/odata/API_BUSINESS_PARTNER/$metadata"
|
||||
|
||||
def test_service_root_appends_metadata(self):
|
||||
"""service is a service root (same meaning as ingest_entity_set):
|
||||
$metadata is appended automatically, not treated as the full path.
|
||||
"""
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/sap/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.discover_service("API_BUSINESS_PARTNER")
|
||||
|
||||
_, url = guard.call_args[0]
|
||||
assert url == "https://sap.example/odata/sap/API_BUSINESS_PARTNER/$metadata"
|
||||
|
||||
def test_absolute_service_root_appends_metadata(self):
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/sap/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.discover_service("https://other.example/services/sap/")
|
||||
|
||||
_, url = guard.call_args[0]
|
||||
assert url == "https://other.example/services/sap/$metadata"
|
||||
|
||||
def test_v4_oasis_metadata_parses_with_nullable_defaults(self):
|
||||
resp = _fake_response(
|
||||
text=METADATA_V4, headers={"content-type": "application/xml"}
|
||||
)
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
sets = ing.discover_service()
|
||||
|
||||
by_name = {s["name"]: s for s in sets}
|
||||
assert "SalesOrders" in by_name and "Customers" in by_name
|
||||
sales_fields = {f["name"]: f for f in by_name["SalesOrders"]["fields"]}
|
||||
# Omitted Nullable -> nullable (CSDL default); explicit "false" honored.
|
||||
assert sales_fields["ID"]["nullable"] is True
|
||||
assert sales_fields["Total"]["nullable"] is False
|
||||
|
||||
def test_same_named_types_in_different_schemas_do_not_merge(self):
|
||||
resp = _fake_response(
|
||||
text=METADATA_MULTI_SCHEMA, headers={"content-type": "application/xml"}
|
||||
)
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
sets = ing.discover_service()
|
||||
|
||||
by_name = {s["name"]: s for s in sets}
|
||||
assert [f["name"] for f in by_name["OrdersNs"]["fields"]] == ["A"]
|
||||
assert [f["name"] for f in by_name["InvoicesNs"]["fields"]] == ["B"]
|
||||
|
||||
|
||||
class TestAuth:
|
||||
def test_basic_auth_payload_attached(self):
|
||||
conn = SAPODataConnector(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="erp_user",
|
||||
# Non-functional test placeholder; the low-entropy value keeps the
|
||||
# secret scanner from treating it as a hardcoded credential.
|
||||
password="test",
|
||||
)
|
||||
session = conn.get_session()
|
||||
assert session.headers["Authorization"].startswith("Basic ")
|
||||
# base64("erp_user:test")
|
||||
assert session.headers["Authorization"].endswith("ZXJwX3VzZXI6dGVzdA==")
|
||||
|
||||
def test_oauth_token_exchange_goes_through_ssrf(self):
|
||||
token_resp = MagicMock()
|
||||
token_resp.status_code = 200
|
||||
token_resp.headers = {}
|
||||
token_resp.json.return_value = {"access_token": "tok-123"}
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=token_resp,
|
||||
) as guard:
|
||||
conn = SAPODataConnector(
|
||||
base_url="https://sap.example/odata/",
|
||||
token_url="https://auth.example/oauth/token",
|
||||
client_id="cid",
|
||||
client_secret="secret",
|
||||
)
|
||||
session = conn.get_session()
|
||||
|
||||
assert guard.call_count == 1
|
||||
method_tok, url = guard.call_args[0]
|
||||
assert method_tok == "POST"
|
||||
assert url == "https://auth.example/oauth/token"
|
||||
assert session.headers["Authorization"] == "Bearer tok-123"
|
||||
body = guard.call_args.kwargs.get("data", {})
|
||||
assert body["grant_type"] == "client_credentials"
|
||||
|
||||
def test_requires_auth(self):
|
||||
with pytest.raises(ValidationError):
|
||||
SAPODataConnector(base_url="https://sap.example/odata/")
|
||||
|
||||
def test_requires_base_url(self):
|
||||
with pytest.raises(ValidationError):
|
||||
SAPODataConnector(username="u", password="p")
|
||||
|
||||
|
||||
class TestIngestPagination:
|
||||
V4_PAGE = {
|
||||
"value": [{"SalesOrderID": "SO-1"}, {"SalesOrderID": "SO-2"}],
|
||||
"@odata.nextLink": "https://sap.example/odata/SalesOrderSet?$skiptoken=abc",
|
||||
}
|
||||
V4_LAST = {"value": [{"SalesOrderID": "SO-3"}]}
|
||||
|
||||
def test_v4_nextlink_paginates(self):
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
page1 = _json_resp(self.V4_PAGE, url)
|
||||
page2 = _json_resp(self.V4_LAST, url)
|
||||
calls.append(url)
|
||||
return page1 if method == "GET" and len(calls) == 1 else page2
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
assert result.count == 3
|
||||
assert [r["SalesOrderID"] for r in result.records] == [
|
||||
"SO-1",
|
||||
"SO-2",
|
||||
"SO-3",
|
||||
]
|
||||
# Reached the second page's next link then stopped.
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_v4_nextlink_is_followed_past_first_page(self):
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
calls.append(url)
|
||||
if len(calls) == 1:
|
||||
return _json_resp(self.V4_PAGE, url)
|
||||
return _json_resp(self.V4_LAST, url)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_v2_atom_next_pagination(self):
|
||||
v2_first = {
|
||||
"d": {
|
||||
"results": [{"SalesOrderID": "A"}],
|
||||
"__next": {"__deferred": {"uri": "https://sap.example/odata/next2"}},
|
||||
},
|
||||
}
|
||||
v2_last = {"d": {"results": [{"SalesOrderID": "B"}]}}
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
calls.append(url)
|
||||
return (
|
||||
_json_resp(v2_first, url)
|
||||
if len(calls) == 1
|
||||
else _json_resp(v2_last, url)
|
||||
)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
assert [r["SalesOrderID"] for r in result.records] == ["A", "B"]
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_v2_next_as_plain_string_paginates(self):
|
||||
"""Canonical OData v2 JSON: ``__next`` is a plain string URL."""
|
||||
v2_first = {
|
||||
"d": {
|
||||
"results": [{"SalesOrderID": "A"}],
|
||||
"__next": "https://sap.example/odata/next2",
|
||||
},
|
||||
}
|
||||
v2_last = {"d": {"results": [{"SalesOrderID": "B"}]}}
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
calls.append(url)
|
||||
if len(calls) == 1:
|
||||
return _json_resp(v2_first, url)
|
||||
return _json_resp(v2_last, url)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
assert [r["SalesOrderID"] for r in result.records] == ["A", "B"]
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_relative_service_resolves_against_base(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_json_resp({"value": []}, "https://x/"),
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/sap/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.ingest_entity_set(
|
||||
service="API_SALES_ORDER_SRV", entity_set="SalesOrderSet"
|
||||
)
|
||||
|
||||
method, url = guard.call_args[0]
|
||||
assert url == "https://sap.example/odata/sap/API_SALES_ORDER_SRV/SalesOrderSet"
|
||||
|
||||
def test_expand_passthrough_for_line_items(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_json_resp(
|
||||
{"value": [{"SalesOrderID": "SO-9"}]}, "https://x/"
|
||||
),
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet", expand="to_Item")
|
||||
|
||||
params = guard.call_args_list[0].kwargs.get("params") or {}
|
||||
assert params.get("$expand") == "to_Item"
|
||||
|
||||
def test_select_filter_top_skip_objects_passthrough(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_json_resp({"value": []}, "https://x/"),
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
ing.ingest_entity_set(
|
||||
entity_set="SalesOrderSet",
|
||||
select="SalesOrderID,GrossAmount",
|
||||
filter="GrossAmount gt 100",
|
||||
top=5,
|
||||
skip=2,
|
||||
)
|
||||
params = guard.call_args_list[0].kwargs.get("params") or {}
|
||||
assert params["$select"] == "SalesOrderID,GrossAmount"
|
||||
assert params["$filter"] == "GrossAmount gt 100"
|
||||
assert params["$top"] == "5"
|
||||
assert params["$skip"] == "2"
|
||||
|
||||
|
||||
class TestErrorPaths:
|
||||
def test_http_error_on_entity_set_raises_processing_error(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_fake_response(status_code=403),
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="403"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
def test_invalid_metadata_xml_raises_processing_error(self):
|
||||
resp = _fake_response(text="<not-xml")
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="not valid XML"):
|
||||
ing.discover_service()
|
||||
|
||||
def test_non_json_entity_response_raises_processing_error(self):
|
||||
resp = _fake_response(text="plain text")
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="not JSON"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
def test_cross_host_next_link_rejected(self):
|
||||
first = _json_resp(
|
||||
{
|
||||
"value": [{"SalesOrderID": "A"}],
|
||||
"@odata.nextLink": "https://evil.example/next",
|
||||
},
|
||||
"https://sap.example/odata/SalesOrderSet",
|
||||
)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=first,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="different host"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
# The credential-bearing session is never sent to the evil host.
|
||||
assert guard.call_count == 1
|
||||
|
||||
def test_top_zero_returns_no_rows_and_makes_no_request(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet", top=0)
|
||||
|
||||
assert result.count == 0
|
||||
assert result.records == []
|
||||
guard.assert_not_called()
|
||||
|
||||
def test_negative_top_rejected(self):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ValidationError, match="must be >= 0"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet", top=-1)
|
||||
|
||||
def test_malformed_rows_container_rejected(self):
|
||||
resp = _json_resp(
|
||||
{"value": {"not": "a list"}},
|
||||
"https://sap.example/odata/SalesOrderSet",
|
||||
)
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="no list of rows"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
|
||||
class TestExport:
|
||||
def test_export_as_documents_flattens_to_graph_builder_shape(self):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
entity = SAPODataEntity(
|
||||
records=[{"SalesOrderID": "SO-1", "CustomerID": "C-1"}],
|
||||
entity_set="SalesOrderSet",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
docs = ing.export_as_documents(entity)
|
||||
assert docs[0]["SalesOrderID"] == "SO-1"
|
||||
assert docs[0]["id"] == "SO-1"
|
||||
assert docs[0]["source"] == "https://sap.example/odata/"
|
||||
|
||||
|
||||
class TestEntityDocument:
|
||||
def test_to_document_adds_source(self):
|
||||
ent = SAPODataEntity(
|
||||
records=[{"k": "v"}],
|
||||
entity_set="S",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
assert ent.to_documents()[0]["source"] == "https://sap.example/odata/"
|
||||
|
||||
def test_to_document_injects_graph_builder_identifier(self):
|
||||
ent = SAPODataEntity(
|
||||
records=[{"SalesOrderID": "SO-1", "CustomerID": "C-1"}],
|
||||
entity_set="SalesOrderSet",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
doc = ent.to_documents()[0]
|
||||
assert doc["id"] == "SO-1"
|
||||
assert doc["name"] == "SO-1"
|
||||
assert doc["SalesOrderID"] == "SO-1" # original field preserved
|
||||
|
||||
def test_to_document_falls_back_to_entity_set_index(self):
|
||||
ent = SAPODataEntity(
|
||||
records=[{"GrossAmount": "10.0"}],
|
||||
entity_set="SalesOrderSet",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
assert ent.to_documents()[0]["id"] == "SalesOrderSet:0"
|
||||
|
||||
def test_public_import_through_lazy_export(self):
|
||||
# __getattr__ should resolve the lazy export once the object is created
|
||||
assert callable(SAPODataConnector)
|
||||
@@ -0,0 +1,51 @@
|
||||
import pytest
|
||||
|
||||
from semantica.ontology.class_inferrer import ClassInferrer
|
||||
from semantica.ontology.property_generator import PropertyGenerator
|
||||
from semantica.utils.exceptions import ValidationError
|
||||
|
||||
|
||||
def test_same_kind_normalized_object_properties_are_merged():
|
||||
entities = [
|
||||
{"id": "p1", "type": "Person", "name": "Alice"},
|
||||
{"id": "o1", "type": "Organization", "name": "Acme"},
|
||||
]
|
||||
classes = ClassInferrer(min_occurrences=1).infer_classes(entities)
|
||||
relationships = [
|
||||
{
|
||||
"source_type": "Person",
|
||||
"target_type": "Organization",
|
||||
"type": "works_for",
|
||||
},
|
||||
{
|
||||
"source_type": "Person",
|
||||
"target_type": "Organization",
|
||||
"type": "worksFor",
|
||||
},
|
||||
]
|
||||
|
||||
properties = PropertyGenerator().infer_properties(
|
||||
entities, relationships, classes, min_occurrences=1
|
||||
)
|
||||
|
||||
works_for = [prop for prop in properties if prop["name"] == "worksFor"]
|
||||
assert len(works_for) == 1
|
||||
assert works_for[0]["domain"] == ["Person"]
|
||||
assert works_for[0]["range"] == ["Organization"]
|
||||
|
||||
|
||||
def test_normalized_name_cannot_be_both_object_and_data_property():
|
||||
entities = [
|
||||
{"id": "p1", "type": "Person", "value": "Alice"},
|
||||
{"id": "p2", "type": "Person", "value": "Bob"},
|
||||
]
|
||||
classes = ClassInferrer(min_occurrences=1).infer_classes(entities)
|
||||
relationships = [
|
||||
{"source_type": "Person", "target_type": "Person", "type": "value"},
|
||||
{"source_type": "Person", "target_type": "Person", "type": "value"},
|
||||
]
|
||||
|
||||
with pytest.raises(ValidationError, match="object and data"):
|
||||
PropertyGenerator().infer_properties(
|
||||
entities, relationships, classes, min_occurrences=1
|
||||
)
|
||||
@@ -0,0 +1,785 @@
|
||||
"""Focused tests for pipeline parallel execution (issue #1223).
|
||||
|
||||
Covers:
|
||||
- ``PipelineBuilder.set_parallelism()`` validation
|
||||
- dependency-layer parallel execution gated on ``parallel_safe``
|
||||
- input isolation via deep copies
|
||||
- incremental dict merging of parallel outputs
|
||||
- failure handling and cancellation semantics
|
||||
- ``parallel_safe`` not leaking into handler kwargs
|
||||
- ``parallel_safe`` surviving serialization round-trips
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from semantica.pipeline.execution_engine import ExecutionEngine
|
||||
from semantica.pipeline.failure_handler import RetryPolicy, RetryStrategy
|
||||
from semantica.pipeline.pipeline_builder import (
|
||||
Pipeline,
|
||||
PipelineBuilder,
|
||||
PipelineSerializer,
|
||||
PipelineStep,
|
||||
StepStatus,
|
||||
)
|
||||
from semantica.utils.exceptions import ProcessingError, ValidationError
|
||||
|
||||
|
||||
class ConcurrencyProbe:
|
||||
"""Thread-safe tracker of handler invocations and concurrency level."""
|
||||
|
||||
def __init__(self):
|
||||
self._lock = threading.Lock()
|
||||
self.active = 0
|
||||
self.max_active = 0
|
||||
self.calls = 0
|
||||
|
||||
def __enter__(self):
|
||||
with self._lock:
|
||||
self.active += 1
|
||||
self.calls += 1
|
||||
self.max_active = max(self.max_active, self.active)
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
with self._lock:
|
||||
self.active -= 1
|
||||
return False
|
||||
|
||||
|
||||
def branch_handler(probe, key, value=True, delay=0.0):
|
||||
"""Build a handler that records concurrency and adds one output key."""
|
||||
|
||||
def handler(data, **kwargs):
|
||||
with probe:
|
||||
if delay:
|
||||
time.sleep(delay)
|
||||
return {**data, key: value}
|
||||
|
||||
return handler
|
||||
|
||||
|
||||
class Undeepcopyable:
|
||||
"""Object whose deepcopy always fails."""
|
||||
|
||||
def __deepcopy__(self, memo):
|
||||
raise TypeError("cannot deepcopy this object")
|
||||
|
||||
|
||||
class TestSetParallelismValidation(unittest.TestCase):
|
||||
"""set_parallelism() must only accept positive integers."""
|
||||
|
||||
def test_rejects_invalid_levels(self):
|
||||
builder = PipelineBuilder()
|
||||
for invalid in (0, -1, 1.5, True, False, "2", None):
|
||||
with self.assertRaises(ValidationError):
|
||||
builder.set_parallelism(invalid)
|
||||
|
||||
def test_accepts_positive_integers(self):
|
||||
builder = PipelineBuilder()
|
||||
builder.set_parallelism(4)
|
||||
self.assertEqual(builder.pipeline_config["parallelism"], 4)
|
||||
|
||||
|
||||
class TestPipelineParallelExecution(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
self.mock_tracker_patcher = patch(
|
||||
"semantica.utils.progress_tracker.get_progress_tracker"
|
||||
)
|
||||
self.mock_get_tracker = self.mock_tracker_patcher.start()
|
||||
self.mock_get_tracker.return_value = MagicMock()
|
||||
|
||||
def tearDown(self):
|
||||
self.mock_tracker_patcher.stop()
|
||||
|
||||
def _build(self, steps, parallelism=None, name="parallel_pipeline"):
|
||||
"""steps: list of (step_name, step_type, handler, parallel_safe)."""
|
||||
builder = PipelineBuilder()
|
||||
for step_name, step_type, handler, parallel_safe in steps:
|
||||
builder.add_step(
|
||||
step_name, step_type, handler=handler, parallel_safe=parallel_safe
|
||||
)
|
||||
if parallelism is not None:
|
||||
builder.set_parallelism(parallelism)
|
||||
return builder.build(name)
|
||||
|
||||
def test_unconfigured_pipeline_stays_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", branch_handler(probe, "b"), True),
|
||||
]
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
|
||||
def test_parallelism_one_stays_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", branch_handler(probe, "b"), True),
|
||||
],
|
||||
parallelism=1,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
|
||||
def test_parallel_safe_steps_overlap_with_barrier(self):
|
||||
barrier = threading.Barrier(2)
|
||||
done = {"a": False, "b": False}
|
||||
|
||||
def handler(key):
|
||||
def inner(data, **kwargs):
|
||||
barrier.wait(timeout=5)
|
||||
done[key] = True
|
||||
return {**data, key: True}
|
||||
|
||||
return inner
|
||||
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", handler("a"), True),
|
||||
("b", "branch", handler("b"), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success, msg=str(result.errors))
|
||||
self.assertTrue(done["a"])
|
||||
self.assertTrue(done["b"])
|
||||
|
||||
def test_active_workers_do_not_exceed_parallelism(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a", delay=0.15), True),
|
||||
("b", "branch", branch_handler(probe, "b", delay=0.15), True),
|
||||
("c", "branch", branch_handler(probe, "c", delay=0.15), True),
|
||||
("d", "branch", branch_handler(probe, "d", delay=0.15), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertLessEqual(probe.max_active, 2)
|
||||
self.assertGreaterEqual(probe.max_active, 2)
|
||||
|
||||
def test_unmarked_steps_stay_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), False),
|
||||
("b", "branch", branch_handler(probe, "b"), False),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
|
||||
def test_layer_with_one_unsafe_step_is_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", branch_handler(probe, "b"), False),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
|
||||
def test_pure_dependency_chain_stays_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
|
||||
def chain_handler(key):
|
||||
def inner(data, **kwargs):
|
||||
with probe:
|
||||
return {**data, key: True}
|
||||
|
||||
return inner
|
||||
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step(
|
||||
"a", "branch", handler=chain_handler("a"), parallel_safe=True
|
||||
)
|
||||
builder.add_step(
|
||||
"b",
|
||||
"branch",
|
||||
handler=chain_handler("b"),
|
||||
parallel_safe=True,
|
||||
dependencies=["a"],
|
||||
)
|
||||
builder.add_step(
|
||||
"c",
|
||||
"branch",
|
||||
handler=chain_handler("c"),
|
||||
parallel_safe=True,
|
||||
dependencies=["b"],
|
||||
)
|
||||
builder.set_parallelism(4)
|
||||
pipeline = builder.build("chain_pipeline")
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
self.assertEqual(result.output, {"text": "hi", "a": True, "b": True, "c": True})
|
||||
|
||||
def test_non_dict_input_falls_back_to_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
|
||||
def passthrough(data, **kwargs):
|
||||
with probe:
|
||||
return data
|
||||
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", passthrough, True),
|
||||
("b", "branch", passthrough, True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
pipeline, data="plain-text-input"
|
||||
)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(probe.calls, 2)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
self.assertEqual(result.output, "plain-text-input")
|
||||
|
||||
def test_deepcopy_failure_falls_back_to_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", branch_handler(probe, "b"), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
pipeline, data={"text": "hi", "obj": Undeepcopyable()}
|
||||
)
|
||||
|
||||
self.assertTrue(result.success, msg=str(result.errors))
|
||||
self.assertEqual(probe.calls, 2)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
self.assertIn("a", result.output)
|
||||
self.assertIn("b", result.output)
|
||||
|
||||
def test_parallel_results_merge_different_fields(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "entities", ["Alice"]), True),
|
||||
("b", "branch", branch_handler(probe, "triplets", [(1, 2)]), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
pipeline, data={"text": "Alice works at Acme"}
|
||||
)
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(
|
||||
result.output,
|
||||
{
|
||||
"text": "Alice works at Acme",
|
||||
"entities": ["Alice"],
|
||||
"triplets": [(1, 2)],
|
||||
},
|
||||
)
|
||||
|
||||
def test_parallel_results_same_key_same_value_allowed(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "shared", [1, 2]), True),
|
||||
("b", "branch", branch_handler(probe, "shared", [1, 2]), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success, msg=str(result.errors))
|
||||
self.assertEqual(result.output["shared"], [1, 2])
|
||||
|
||||
def test_parallel_results_same_key_different_values_raises(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "shared", 1), True),
|
||||
("b", "branch", branch_handler(probe, "shared", 2), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertFalse(result.success)
|
||||
error_text = result.errors[0] if result.errors else ""
|
||||
self.assertIn("shared", error_text)
|
||||
self.assertIn("a", error_text)
|
||||
self.assertIn("b", error_text)
|
||||
|
||||
def test_parallel_handler_non_dict_return_fails(self):
|
||||
probe = ConcurrencyProbe()
|
||||
|
||||
def non_dict_handler(data, **kwargs):
|
||||
with probe:
|
||||
return "not-a-dict"
|
||||
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", non_dict_handler, True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertFalse(result.success)
|
||||
error_text = result.errors[0] if result.errors else ""
|
||||
self.assertIn("b", error_text)
|
||||
self.assertIn("str", error_text)
|
||||
|
||||
def test_non_dict_handler_executed_only_once(self):
|
||||
probe = ConcurrencyProbe()
|
||||
|
||||
def non_dict_handler(data, **kwargs):
|
||||
with probe:
|
||||
return "not-a-dict"
|
||||
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", non_dict_handler, True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertFalse(result.success)
|
||||
self.assertEqual(probe.calls, 2) # one invocation per handler, no re-runs
|
||||
|
||||
def test_parallel_step_retry_succeeds(self):
|
||||
engine = ExecutionEngine()
|
||||
engine.failure_handler.set_retry_policy(
|
||||
"flaky",
|
||||
RetryPolicy(
|
||||
max_retries=2, initial_delay=0.0, strategy=RetryStrategy.FIXED
|
||||
),
|
||||
)
|
||||
|
||||
probe = ConcurrencyProbe()
|
||||
attempts = {"flaky": 0}
|
||||
|
||||
def flaky_handler(data, **kwargs):
|
||||
with probe:
|
||||
attempts["flaky"] += 1
|
||||
if attempts["flaky"] == 1:
|
||||
raise RuntimeError("transient failure")
|
||||
return {**data, "flaky": True}
|
||||
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "flaky", flaky_handler, True),
|
||||
("b", "branch", branch_handler(probe, "b"), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success, msg=str(result.errors))
|
||||
self.assertEqual(attempts["flaky"], 2)
|
||||
|
||||
def test_failed_branch_skips_downstream_layer(self):
|
||||
engine = ExecutionEngine()
|
||||
engine.failure_handler.set_retry_policy(
|
||||
"always_fail", RetryPolicy(max_retries=0)
|
||||
)
|
||||
|
||||
probe = ConcurrencyProbe()
|
||||
|
||||
def failing_handler(data, **kwargs):
|
||||
with probe:
|
||||
raise RuntimeError("permanent failure")
|
||||
|
||||
downstream_calls = {"c": 0}
|
||||
|
||||
def downstream_handler(data, **kwargs):
|
||||
downstream_calls["c"] += 1
|
||||
return {**data, "c": True}
|
||||
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("a", "always_fail", handler=failing_handler, parallel_safe=True)
|
||||
builder.add_step("b", "branch", handler=branch_handler(probe, "b"), parallel_safe=True)
|
||||
builder.add_step(
|
||||
"c", "branch", handler=downstream_handler, dependencies=["a", "b"]
|
||||
)
|
||||
builder.set_parallelism(2)
|
||||
pipeline = builder.build("failure_pipeline")
|
||||
|
||||
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertFalse(result.success)
|
||||
self.assertEqual(downstream_calls["c"], 0)
|
||||
|
||||
failed_step = next(s for s in pipeline.steps if s.name == "a")
|
||||
self.assertEqual(failed_step.status, StepStatus.FAILED)
|
||||
self.assertIsNotNone(failed_step.error)
|
||||
|
||||
def test_parallel_safe_not_passed_to_handler_kwargs(self):
|
||||
received_kwargs = {}
|
||||
|
||||
def capturing_handler(data, **kwargs):
|
||||
received_kwargs.update(kwargs)
|
||||
return data
|
||||
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step(
|
||||
"a", "branch", handler=capturing_handler, parallel_safe=True, batch_size=2
|
||||
)
|
||||
pipeline = builder.build("kwargs_pipeline")
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertIn("batch_size", received_kwargs)
|
||||
self.assertNotIn("parallel_safe", received_kwargs)
|
||||
self.assertNotIn("parallel_safe", pipeline.steps[0].config)
|
||||
|
||||
def test_unchanged_echoed_key_does_not_conflict(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
# "a" echoes the base value of "shared" (unchanged),
|
||||
# "b" legitimately changes it: no false conflict.
|
||||
("a", "branch", branch_handler(probe, "shared", 1), True),
|
||||
("b", "branch", branch_handler(probe, "shared", 2), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
pipeline, data={"text": "hi", "shared": 1}
|
||||
)
|
||||
|
||||
self.assertTrue(result.success, msg=str(result.errors))
|
||||
self.assertEqual(result.output["shared"], 2)
|
||||
|
||||
def test_ambiguous_equality_counts_as_changed(self):
|
||||
sentinel = Uncomparable()
|
||||
|
||||
def echo(data, **kwargs):
|
||||
return {**data, "shared": data["shared"]}
|
||||
|
||||
def change(data, **kwargs):
|
||||
return {**data, "shared": "changed"}
|
||||
|
||||
pipeline = self._build(
|
||||
[("a", "branch", echo, True), ("b", "branch", change, True)],
|
||||
parallelism=2,
|
||||
)
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(
|
||||
pipeline, data={"text": "hi", "shared": sentinel}
|
||||
)
|
||||
|
||||
# Ambiguous equality must not be treated as unchanged, so both
|
||||
# branches count as writes and the merge reports the conflict.
|
||||
self.assertFalse(result.success)
|
||||
error_text = result.errors[0] if result.errors else ""
|
||||
self.assertIn("shared", error_text)
|
||||
|
||||
def test_non_boolean_parallel_safe_stays_serial(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", branch_handler(probe, "b"), True),
|
||||
],
|
||||
parallelism=2,
|
||||
)
|
||||
# Simulate a truthy non-bool value reaching the engine (e.g. set
|
||||
# directly on the step attribute): must not enable concurrency.
|
||||
for step in pipeline.steps:
|
||||
step.parallel_safe = "false"
|
||||
|
||||
result = ExecutionEngine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(probe.max_active, 1)
|
||||
|
||||
def test_non_dict_result_reports_failure_not_completion(self):
|
||||
probe = ConcurrencyProbe()
|
||||
tracking_ids = {}
|
||||
counter = {"n": 0}
|
||||
|
||||
def fake_start(*args, **kwargs):
|
||||
tid = f"tid_{counter['n']}"
|
||||
counter["n"] += 1
|
||||
tracking_ids[kwargs.get("submodule")] = tid
|
||||
return tid
|
||||
|
||||
with patch(
|
||||
"semantica.pipeline.execution_engine.get_progress_tracker"
|
||||
) as mock_get:
|
||||
tracker = MagicMock()
|
||||
tracker.start_tracking.side_effect = fake_start
|
||||
mock_get.return_value = tracker
|
||||
engine = ExecutionEngine()
|
||||
|
||||
def non_dict_handler(data, **kwargs):
|
||||
with probe:
|
||||
return "not-a-dict"
|
||||
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", non_dict_handler, True),
|
||||
],
|
||||
parallelism=2,
|
||||
name="progress_pipeline",
|
||||
)
|
||||
|
||||
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertFalse(result.success)
|
||||
failed_step = next(s for s in pipeline.steps if s.name == "b")
|
||||
self.assertEqual(failed_step.status, StepStatus.FAILED)
|
||||
self.assertIsInstance(failed_step.error, ProcessingError)
|
||||
# Handler ran exactly once: never re-run serially.
|
||||
self.assertEqual(probe.calls, 2)
|
||||
|
||||
b_tid = tracking_ids.get("progress_pipeline:branch:b")
|
||||
self.assertIsNotNone(
|
||||
b_tid, f"expected tracking for step b, got {tracking_ids}"
|
||||
)
|
||||
b_stops = [
|
||||
c
|
||||
for c in tracker.stop_tracking.call_args_list
|
||||
if (c.args[0] if c.args else c.kwargs.get("tracking_id")) == b_tid
|
||||
]
|
||||
self.assertTrue(b_stops)
|
||||
for call in b_stops:
|
||||
status = call.kwargs.get("status")
|
||||
self.assertEqual(status, "failed")
|
||||
self.assertNotEqual(status, "completed")
|
||||
|
||||
def test_same_type_steps_get_distinct_tracking_records(self):
|
||||
probe = ConcurrencyProbe()
|
||||
pipeline = self._build(
|
||||
[
|
||||
("a", "branch", branch_handler(probe, "a"), True),
|
||||
("b", "branch", branch_handler(probe, "b"), True),
|
||||
],
|
||||
parallelism=2,
|
||||
name="tracking_pipeline",
|
||||
)
|
||||
|
||||
with patch(
|
||||
"semantica.pipeline.execution_engine.get_progress_tracker"
|
||||
) as mock_get:
|
||||
tracker = MagicMock()
|
||||
mock_get.return_value = tracker
|
||||
engine = ExecutionEngine()
|
||||
|
||||
result = engine.execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertTrue(result.success, msg=str(result.errors))
|
||||
submodules = [
|
||||
call.kwargs.get("submodule")
|
||||
for call in tracker.start_tracking.call_args_list
|
||||
if call.kwargs.get("module") == "pipeline"
|
||||
]
|
||||
# Concurrent steps of the same step_type get distinct submodules
|
||||
# (and therefore distinct tracking IDs), including pipeline identity.
|
||||
self.assertIn("tracking_pipeline:branch:a", submodules)
|
||||
self.assertIn("tracking_pipeline:branch:b", submodules)
|
||||
|
||||
|
||||
class TestParallelSafeSerialization(unittest.TestCase):
|
||||
"""parallel_safe must survive dict and JSON round-trips."""
|
||||
|
||||
def _build_pipeline(self):
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("extract", "source", parallel_safe=True, batch_size=10)
|
||||
builder.add_step("index", "sink", dependencies=["extract"])
|
||||
builder.set_parallelism(3)
|
||||
return builder.build("parallel-serialization")
|
||||
|
||||
def test_serializer_roundtrip_preserves_parallel_safe(self):
|
||||
pipeline = self._build_pipeline()
|
||||
serializer = PipelineSerializer()
|
||||
|
||||
serialized = serializer.serialize_pipeline(pipeline, format="dict")
|
||||
self.assertTrue(serialized["steps"][0]["parallel_safe"])
|
||||
self.assertFalse(serialized["steps"][1]["parallel_safe"])
|
||||
|
||||
restored = serializer.deserialize_pipeline(serialized)
|
||||
self.assertTrue(restored.steps[0].parallel_safe)
|
||||
self.assertFalse(restored.steps[1].parallel_safe)
|
||||
self.assertEqual(restored.config.get("parallelism"), 3)
|
||||
|
||||
def test_serializer_json_roundtrip_preserves_parallel_safe(self):
|
||||
pipeline = self._build_pipeline()
|
||||
serializer = PipelineSerializer()
|
||||
|
||||
serialized = serializer.serialize_pipeline(pipeline, format="json")
|
||||
restored = serializer.deserialize_pipeline(serialized)
|
||||
|
||||
self.assertTrue(restored.steps[0].parallel_safe)
|
||||
self.assertFalse(restored.steps[1].parallel_safe)
|
||||
|
||||
def test_builder_serialize_outputs_parallel_safe(self):
|
||||
builder = PipelineBuilder()
|
||||
builder.add_step("extract", "source", parallel_safe=True)
|
||||
builder.add_step("index", "sink", dependencies=["extract"])
|
||||
builder.set_parallelism(2)
|
||||
|
||||
serialized = builder.serialize(format="dict")
|
||||
|
||||
self.assertTrue(serialized["steps"][0]["parallel_safe"])
|
||||
self.assertFalse(serialized["steps"][1]["parallel_safe"])
|
||||
self.assertEqual(serialized["config"]["parallelism"], 2)
|
||||
|
||||
|
||||
class Uncomparable:
|
||||
"""Object whose equality comparison always raises TypeError."""
|
||||
|
||||
def __eq__(self, other):
|
||||
raise TypeError("cannot compare")
|
||||
|
||||
|
||||
class TestParallelDependencyValidation(unittest.TestCase):
|
||||
"""Cyclic/unknown dependencies must fail with ValidationError in the
|
||||
parallel path, not RecursionError/KeyError (review finding #1)."""
|
||||
|
||||
def _engine(self):
|
||||
with patch(
|
||||
"semantica.pipeline.execution_engine.get_progress_tracker"
|
||||
) as mock_get:
|
||||
mock_get.return_value = MagicMock()
|
||||
engine = ExecutionEngine()
|
||||
return engine
|
||||
|
||||
def _handler(self):
|
||||
return lambda data, **kwargs: data
|
||||
|
||||
def test_cyclic_dependencies_raise_validation_error(self):
|
||||
# Construct the pipeline directly: the builder already rejects
|
||||
# cycles at build time, so bypassing it exercises the engine-level
|
||||
# grouping guard (which otherwise hits RecursionError).
|
||||
steps = [
|
||||
PipelineStep(
|
||||
name="a",
|
||||
step_type="branch",
|
||||
handler=self._handler(),
|
||||
dependencies=["b"],
|
||||
parallel_safe=True,
|
||||
),
|
||||
PipelineStep(
|
||||
name="b",
|
||||
step_type="branch",
|
||||
handler=self._handler(),
|
||||
dependencies=["a"],
|
||||
parallel_safe=True,
|
||||
),
|
||||
]
|
||||
pipeline = Pipeline(
|
||||
name="cyclic_pipeline",
|
||||
steps=steps,
|
||||
config={"parallelism": 4},
|
||||
)
|
||||
|
||||
result = self._engine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertFalse(result.success)
|
||||
error_text = result.errors[0] if result.errors else ""
|
||||
self.assertIn("Circular dependency", error_text)
|
||||
|
||||
def test_unknown_dependency_raises_validation_error(self):
|
||||
# Construct the pipeline directly so the engine-level unknown
|
||||
# dependency guard is exercised (instead of a builder-time
|
||||
# KeyError from the grouping DFS).
|
||||
steps = [
|
||||
PipelineStep(
|
||||
name="a",
|
||||
step_type="branch",
|
||||
handler=self._handler(),
|
||||
dependencies=["missing"],
|
||||
parallel_safe=True,
|
||||
),
|
||||
]
|
||||
pipeline = Pipeline(
|
||||
name="unknown_dep_pipeline",
|
||||
steps=steps,
|
||||
config={"parallelism": 4},
|
||||
)
|
||||
|
||||
result = self._engine().execute_pipeline(pipeline, data={"text": "hi"})
|
||||
|
||||
self.assertFalse(result.success)
|
||||
error_text = result.errors[0] if result.errors else ""
|
||||
self.assertIn("unknown step 'missing'", error_text)
|
||||
self.assertIn("'a'", error_text)
|
||||
|
||||
|
||||
class TestParallelSafeMustBeBoolean(unittest.TestCase):
|
||||
"""parallel_safe must be an explicit boolean everywhere (review #4)."""
|
||||
|
||||
def test_add_step_rejects_non_boolean_values(self):
|
||||
builder = PipelineBuilder()
|
||||
for invalid in ("false", "true", 1, 0, None, [True]):
|
||||
with self.assertRaises(ValidationError):
|
||||
builder.add_step(
|
||||
"a",
|
||||
"branch",
|
||||
handler=lambda data, **kwargs: data,
|
||||
parallel_safe=invalid,
|
||||
)
|
||||
|
||||
def test_build_pipeline_rejects_non_boolean_values(self):
|
||||
builder = PipelineBuilder()
|
||||
config = {
|
||||
"name": "invalid_parallel_safe",
|
||||
"steps": [
|
||||
{"name": "a", "type": "branch", "parallel_safe": "false"}
|
||||
],
|
||||
}
|
||||
with self.assertRaises(ValidationError):
|
||||
builder.build_pipeline(config)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user