mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Merge main into feat/salesforce-ingestor-clean
This commit is contained in:
Sameer6305
@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.6.7] - 2026-08-28
|
||||
|
||||
### Added
|
||||
|
||||
- **First-class LangChain integration** (closes #963; recreates #969)
|
||||
@@ -20,6 +22,98 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- All adapters remain importable without langchain-core (`LANGCHAIN_AVAILABLE` flag)
|
||||
- Docs: `docs/integrations/langchain.md`, README native-integration matrix, and `docs.json` nav entry
|
||||
|
||||
- **SAP OData ingestor** (#1234, closes #1228) by @pkupt
|
||||
- New `SAPODataEntity` / `SAPODataConnector` / `SAPIngestor` (`semantica.ingest`, lazy exports), following the three-layer connector pattern already used for Snowflake/Databricks, to pull master/transactional data (Business Partners, Sales Orders) from SAP OData v2/v4 services into the Context Graph
|
||||
- Dual auth (OAuth2 client-credentials for BTP/S4HANA Cloud, Basic for on-prem NetWeaver); every outbound request, including the token exchange, routes through `request_with_ssrf_guard`
|
||||
- `$metadata` (CSDL XML) is parsed with a hand-rolled `xml.etree` reader rather than pulling in `pyodata`; pagination follows OData v2 `__next`/`__deferred` and v4 `@odata.nextLink`
|
||||
- New `pip install semantica[ingest-sap]` extra (`requests>=2.28.0`)
|
||||
- **Known phase-1 limits** (documented in docstrings): the OAuth2 token is cached but never refreshed, and pagination has no `max_pages` fuse (`top` bounds it when supplied)
|
||||
- New `tests/ingest/test_sap_ingestor.py`: 22 tests (auth, EDMX parsing, v2/v4 pagination, SSRF routing, error paths, service-root normalization)
|
||||
|
||||
- **`ContextGraph` gains deterministic, human-editable Markdown round-trip persistence** (#852) by @SaurabhScripts
|
||||
- `save_to_markdown()`/`load_from_markdown()` write one file per node plus a graph manifest, so a graph can be reviewed and hand-edited outside the application without giving up the existing JSON API or its default behavior
|
||||
- An existing destination is validated as a complete, canonical managed export before atomic replacement, so the loader can't silently clobber an unrelated or manually-extended directory
|
||||
- Import/export paths and their ancestors reject symlinks, Windows junctions, and other reparse points, with pre-open and post-open validation — the same hardening applied to `AgentMemory`'s existing Markdown import in the companion fix below
|
||||
- Dangling edge endpoints import as JSON-compatible entity stubs rather than being rejected outright (matching what the JSON loader already accepts); node/edge indexes, adjacency, and analytics/retraction/tombstone state are rebuilt after a Markdown load, and granular node/edge events are still emitted so temporal audit history stays useful
|
||||
- New `tests/context/test_context_graph_markdown.py`: 29 passed, 1 skipped (the skipped case creates a real Windows junction and runs on Windows CI); full `tests/context/` suite: 614 passed, 1 skipped
|
||||
|
||||
- **Explorer graph inspector gains a read-only Markdown content viewer** (#1078, closes #900) by @sakshi04-ui — Preview (rendered GFM) and Source (exact, whitespace-preserving) tabs for node content, with a copy-to-clipboard action. A URL allowlist restricts links to `http:`/`https:`/`mailto:`/in-document anchors, raw HTML execution is disabled, and external links carry `rel="noopener noreferrer"`. A first, focused step toward human-editable memory (#765); no write path yet. New `explorer/tests/markdownContentViewer.test.ts`: 8 tests
|
||||
- **Follow-up (perf)** (#1195, addresses #1118) by @pravit-amp: `remarkPlugins` and the ~20-entry renderer `components` map were inline literals, so every unrelated re-render (e.g. clicking Copy) re-ran the full remark parse and remounted the whole subtree — up to 1.1s of main-thread block on a 2000-row GFM table. Both are now hoisted to module scope and the rendered element is memoized on content, cutting re-render cost from as much as 1121ms to ~0.1ms across all measured fixtures with no change to rendered output. A separate, upstream `remark-gfm` table-parse cost (~O(n^1.9), not fixed here) is left open on the issue as a product decision
|
||||
- **Follow-up (cleanup)** (#1194, closes #1119) by @pravit-amp: the pure `isSafeUrl` URL-safety helper is extracted out of `MarkdownContentViewer.tsx` into its own `markdownUrlSafety.ts` module (behavior-preserving — moved verbatim), so the component module exports only components and stops tripping `react-refresh/only-export-components`
|
||||
|
||||
- **`reasoning` gains a structured Action layer — rule-driven side effects with optional provenance** (#1096, closes #1095) by @cxzg007 — `AssertAction`/`RetractAction`/`CallAction`/`EmitEventAction` let a matched rule write facts back to a `KnowledgeGraph`, retract facts, call a structured handler (replacing the previously-unused `Rule.handler`), or emit to a sink registered via `Reasoner.on_event`, turning the reasoner from a pure inference engine into a production-rule system. With `provenance=True`, fired actions are recorded to `Reasoner.action_log`. Fully additive — rules without `actions` are unaffected, and the legacy `handler` field still fires (now wrapped internally as a `CallAction`). Also fixes a latent dangling import in `reasoning_provenance.py` (`ReasoningEngine`/`infer` → `Reasoner`/`infer_facts`). New `tests/reasoning/test_rule_actions.py`: 9 tests; full `tests/reasoning/` suite: 54 passed
|
||||
|
||||
- **`run_shacl_validation` is now a public, documented entry point** (#1189, closes #1186) by @mikemikimike — the SHACL guide had documented the private `_run_pyshacl` helper as the canonical API; it's now exposed through `semantica.ontology`, with `_run_pyshacl` kept as a compatibility alias over the same implementation. `tests/ontology/test_ontology_advanced.py`: 33 passed (also fixes a flaky comparison against pySHACL's non-deterministic blank-node shape identifiers by comparing stable report fields instead)
|
||||
|
||||
- **`docs/storage-backends.md`: adapter inventory and RDF/LPG feature matrix** (#899, addresses #888) by @yulinlina — which graph storage backends are built-in vs. bring-your-own, and where provenance/context support is partial
|
||||
- **`docs/guides/shacl-validation.md`: documented that `rdfs:range` + RDFS entailment makes `sh:class` unfalsifiable** (#1182, fixes #1130) by @ALDRIN121 — with entailment on, pyshacl infers the declared range class onto every object, so a `sh:class` constraint can never fail and reports `conforms: True` on non-conforming data; added to Common Pitfalls with the `inference="none"` vs `inference="rdfs"` contrast and guidance to re-run `sh:class` shape sets with entailment off before trusting a pass
|
||||
- **Cookbook: four new module notebooks** — `22_Provenance_Tracking.ipynb` (#989, lineage walks, revision history, invalidation, checksums), `23_Reasoning.ipynb` (#990, `Reasoner`/`DatalogReasoner`/`ExplanationGenerator`), `24_Change_Management.ipynb` (#991, versioned snapshots, named tags, checksum tamper-detection), and `25_Seed_Data.ipynb` (#992, bootstrapping a foundation graph from a trusted CSV source) — all by @LeonSGP43, filling gaps where the corresponding module shipped a usage doc but no runnable tutorial; every cell verified against current module source. `docs/cookbook.md` index entries for all four added in #1225
|
||||
- **README "Cite Us" section and `docs/citation.md` cross-link** (#1210) by @KaifAhmad1 — BibTeX/APA/MLA/Chicago/IEEE citation forms; also corrects the copyright holder in `LICENSE`/`docs/project-license.md` from the stale "Hawksight AI" to "Semantica" and replaces the retired `Hawksight-AI` GitHub org slug with `semantica-agi` across ~40 files (READMEs, issue templates, plugin manifests, cookbook notebooks, docs)
|
||||
|
||||
### Changed
|
||||
|
||||
- **A registered custom method can now refuse, instead of being silently overridden by the default implementation** (#1127, closes #1108) by @fabio-rovai — every module supporting custom methods wrapped the registered callable in a `try`/`except` that logged a warning and ran the built-in default on *any* exception, including one a validator or policy gate raised on purpose to say "do not produce this output." That made every registered gate advisory rather than authoritative. `semantica/utils/custom_methods.py` now centralizes the policy: an exception from a registered method propagates to the caller by default; `fallback_on_custom_error=True` restores the previous warn-and-continue behavior per call. Applied mechanically across all 58 call sites in `export/`, `ingest/`, `normalize/`, `parse/`, `embeddings/`, and `kg/` methods modules. New `tests/utils/test_custom_method_can_refuse.py`: 13 tests, including the reported gate-deletes-and-raises scenario and a guard that no call site still swallows
|
||||
- **Removed 13 confirmed-dead symbols across 9 files** (#1176, closes #1174) by @Vinv-AI — private helpers and Explorer app-layer code with zero callers in code, tests, or docs, none part of the public API or a FastAPI `response_model`; 289 deletions, no behavior change
|
||||
- **Consolidated the two duplicate Turtle/N-Triples literal escapers in `rdf_exporter.py`** (#1221, closes #1218) by @pkupt — `_escape_turtle_literal` (added in #1148) escaped the same five characters in the same order as the older module-level `_escape_literal`; the redundant one is dropped and all four call sites route through the original. Behavior no-op, verified against the full export suite (301 passed, 1 skipped)
|
||||
- **Removed the unreachable `_extract_with_spacy()` method and the unused `self.nlp` attribute from `NERExtractor`** (#1220, fixes #1058) by @yunaremaia — the ML dispatch path has always gone through `methods.py`'s process-level model cache instead; `__init__` still validates the spaCy runtime up front but no longer eagerly loads a model nothing on the instance reads
|
||||
- **Cleaned up an unused `sys` import and import ordering in `semantica/worker.py`** (#1061) by @aoright
|
||||
- **Test-only contributions**: isolated `sys.modules` mock leakage between `tests/visualization/` files so the suite passes in any collection order (#897, closes #859, by @luantaraschi); added coverage for 4 previously-untested `ConflictResolver` strategies and 3 `ConflictDetector` conflict types (#902, fixes #865, by @Devansh070); added a regression test tracking relationship provenance through `ProvenanceManager` (#1071, closes #1055, by @dex0shubham); added `max_tokens`-propagation regression coverage for LLM extraction methods, later folded into the cache-key fix below (#925, by @saiganesh47)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`SPARQLReasoner.execute_query()` claimed to run a query but always returned an empty result** (#1087, fixes #1083) by @ALDRIN121 — both the store-configured and unconfigured branches returned an empty `SPARQLQueryResult` with no real execution behind it, so a caller trusting "no matches" (e.g. a compliance check) could draw a false-negative conclusion from a method that never actually queried anything. Until a real triplet-store execution path lands, it now raises `NotImplementedError` explaining why, and the dead cache/inference scaffolding after the unreachable execution point is removed. 3 new regression tests
|
||||
- **`DuplicateDetector` merged entities that share no identifier, type, or name** (#1149, fixes #1137) by @pkupt — `_create_duplicate_candidate()` only ever boosted confidence for matching types and never penalized a mismatch, so two sparse, differently-typed entities (e.g. a `Person` and an `Organization`) could land above the merge threshold and collapse into one node, silently dropping the second. Two non-empty, differing types are now never a duplicate candidate. `tests/deduplication/`: 92 passed
|
||||
- **`TemporalGraphQuery.analyze_evolution()`'s `stability` metric was a hardcoded placeholder** (#1143, closes #1142) by @cxzg007 — every bounded relationship contributed a constant `1`, so `stability` was always `1.0` or `0` regardless of how long relationships actually stayed valid. Now computes the mean valid-time duration in seconds across relationships with both `valid_from`/`valid_until` set; unbounded/half-open intervals are skipped and negative intervals clamp to zero. 3 new tests in `tests/kg/test_kg.py`
|
||||
- **CodeQL false-positive on a JSON-LD test's URL check** (#1183) by @KaifAhmad1 — `"https://schema.org/" in flattened` pattern-matched CodeQL's substring-sanitization heuristic even though `flattened` is always a `list` (exact membership, no sanitization or SSRF path involved); rewritten as an explicit `any(entry == ... for entry in flattened)` with identical behavior
|
||||
- **HuggingFace NER extraction crashed on `huggingface_model` being forwarded as an unexpected pipeline loader kwarg** (#1188, fixes #1063) by @shahzaib-ahmadcs — while preserving genuinely supported pipeline kwargs like `aggregation_strategy`. 5 tests pass
|
||||
- **JSON-LD document/graph `@id` was minted from the wall clock, so re-exporting an unchanged graph produced a new subject every time** (#1181, closes #1147) by @reddynitish — merging repeated exports duplicated graph identity instead of recognizing them as the same graph. The `@id` is now content-derived, with optional `graph_uri`/`document_uri` overrides for callers with a stable graph name; `semantica:exportedAt` still records export time separately. Applies to both JSON-LD export paths
|
||||
- **`ContextGraph.get_causal_chain()` only matched the canonical uppercase causal-edge spellings, silently missing edges recorded in `CausalChainAnalyzer`'s present-tense vocabulary** (#1187, fixes #1184) by @ALDRIN121 — `causes`/`influences`/`precedes` differ from `CAUSED`/`INFLUENCED`/`PRECEDENT_FOR` in word form, not just case, so an edge recorded with the analyzer's spelling produced an empty audit chain — silent, and in the dangerous direction for a compliance trace. `add_causal_relationship()` now normalizes through an alias map before storing the canonical form; traversal accepts the union vocabulary. 2 new regression tests, full `tests/context/` suite: 587 passed
|
||||
- **`semantica embed generate` corrupted its own output and could recurse into a stack overflow** (#996/#1004/#1005, closes #994) by @varunsahni18, @yzxcj797 — three compounding defects in one pipeline. (1) `generate_embeddings`/`embed_text`/`calculate_similarity`/`pool_embeddings` all registered themselves as their own custom-method-registry default, so an unqualified call (exactly what the CLI does) re-entered the same wrapper until Python's recursion limit; each of the four dispatch sites now guards on registry identity before recursing (#996, #1005). A second self-recursion in `EmbeddingGeneratorWithProvenance.__getattr__` (re-entering itself when `_generator` is unset, e.g. during a `deepcopy` probe) now raises a normal `AttributeError` for private names instead (#1005). (2) `--output embeddings.parquet` wrote `json.dumps(result, default=str)` regardless of extension, turning a numpy array into its plain-text `repr()` — a file `embed index` then failed to open as Parquet; the writer now detects `.parquet`/`.json`/`.jsonl` and produces real Parquet/JSON, rejecting any other extension with a clear message (#996, #1004). (3) `pyarrow` was only in optional extras despite being required by the documented quick-start flow; promoted to a core dependency (#996)
|
||||
- **`AgentMemory`'s existing Markdown import accepted symbolic links, NTFS junctions, and other Windows reparse points** (#851) by @SaurabhScripts — a direct linked import path is now rejected with an actionable error, and a linked entry found inside an otherwise-valid directory is skipped rather than aborting the whole import; hardened with pre-open/post-open checks, `O_NOFOLLOW` where available, and `fstat`-based regular-file validation. `tests/context/`: 595 passed, 1 skipped (Windows-junction test, runs on Windows CI)
|
||||
- **A caught vector-similarity scoring exception left stale partial state behind, risking a misleading match on the next call** (#885, fixes #875) by @ArmanGrewal007 — the exception is now logged at debug level and `vector_score`/`vector_idx` reset to neutral values before the remaining matching stages continue
|
||||
- **`RDFExporter` could write invalid or unintended relative IRIs for `GraphBuilder`-default entity/relationship identifiers** (#1112, closes #1099) by @mikemikimike — normalization is now applied at the RDF export boundary across Turtle (including temporal Turtle), RDF/XML, and N-Triples: bare/relative identifiers are minted under the Semantica namespace with safe percent-encoding, absolute IRIs pass through unchanged, and configured/input-context prefixes expand through the effective namespace mapping. 38 focused regression tests; 175 export tests plus 46 subtests pass
|
||||
- **`RDF4JStore`'s `repository_id` constructor argument had no effect** (#1192, closes #1191) by @Freakz2z — the explicit id is now honored when selecting the repository; stale documentation caveats claiming otherwise are removed. 65 tests pass across the affected triplet-store suites
|
||||
- **Non-interactive stdout (piped/redirected output, CI logs) was flooded with progress-bar escape sequences** (#1193, fixes #1185) by @ALDRIN121 — a plain `python demo.py > out.txt` captured 173 bytes of progress noise around 10 bytes of real output. `ProgressTracker` now attaches its console display only for an interactive terminal, Jupyter, or the new `SEMANTICA_FORCE_PROGRESS` opt-in (following the `NO_COLOR`/`FORCE_COLOR` convention); file-based progress logging is untouched. Both switches are now documented in the README and `docs/reference/utils.md`. 11 tests pass (6 new)
|
||||
- **Entity `metadata` was dropped by every RDF serializer except the JSON-LD path**, so an entity kept its confidence but lost its source document, page, extractor, and reviewer on Turtle/N-Triples/RDF/XML/`RDFExporter`'s own JSON-LD (#1165, closes #1154) by @fabio-rovai — Semantica's own metadata keys (`num_entities`, `snapshot_time`, Neo4j loader fields, etc.) are now mapped to declared vocabulary terms and carried through on every path; a caller-supplied key with no mapped term is skipped with an explicit warning (rather than silently vanishing) naming the override needed, pending the caller-key namespace decision tracked in #1146. 21 new tests in `tests/export/test_metadata_passthrough.py`; `tests/export`+`tests/ontology`: 274 pass
|
||||
- **`extract_relations_llm` silently dropped caller-supplied generation parameters** (`max_tokens`, `top_p`, `seed`, etc.), and the extraction cache didn't distinguish calls made with different generation settings (#1213, with test coverage from #925) by @Sameer6305 — a small hardcoded allowlist forwarded only `temperature`/`verbose` to `generate_typed`, discarding the rest; fixed by forwarding all caller kwargs. Once forwarded, those parameters also needed to enter the cache key, since two calls differing only in `max_tokens` previously shared one cache entry and the second could silently reuse a result generated under the first's settings — now applied consistently across entity, relation, and triplet LLM extraction. New regression tests for cache bypass/reuse under differing `max_tokens`/`temperature`
|
||||
- **`OxigraphStore` silently ignored the `storage_path` constructor argument and never flushed writes before a reopen**, both causing silent on-disk data loss (#970) by @logan-jl-cc — `__init__`'s parameter is named `path`, so the project-conventional `storage_path` landed in `**config` and was ignored, degrading a supposedly-persistent store to in-memory with no error; `storage_path` is now accepted as an alias. Separately, pyoxigraph's background flush can lag behind a write, so a reopen immediately after `add_triplets` could observe fewer triples than were written; writes to an on-disk store now call `flush()` explicitly. 2 new regression tests, full suite: 9 passed
|
||||
- **MCP server's `export_graph` tool was broken on every output format** (#1151) by @Arasz — the `json` branch called `JSONExporter().export()` without the `file_path` it requires, and every RDF branch passed a `ContextGraph` object where the exporters expect the canonical kg dict, both surfacing as a raw exception string. A third bug compounded both: the RDF export path's progress bar wrote to stdout, which over stdio MCP *is* the JSON-RPC framing, corrupting the protocol and hanging the client (a 300s timeout on an empty graph). Fixed by converting through `ContextGraph.to_kg_dict()`, serializing the JSON branch to match the RDF branches' string contract, and forcing `SEMANTICA_DISABLE_PROGRESS=1` for the server process. 5 new tests, verified failing against 0.6.6 beforehand
|
||||
- **`OntologyIngestor` dropped every class and property from a JSON-LD document using a named graph** (#1156, fixes #1129) by @13g4d0 — a top-level `@id` beside `@graph` names the graph, and `rdflib.Graph.parse()` silently loads only the default graph, discarding the rest; `POST /api/ontology/load` returned `status: "success"` with `class_count: 0`. Now parses into a `Dataset` and flattens all quads into the working graph (the same `Graph`→`Dataset` migration #757 made for `JenaStore`, extended to the ingest path). On the PR's real-world reproduction: 25 triples/1 subject before, 719 triples/45 classes/40 object properties after. 4 new tests including a default-graph canary so the fix can't trade one blind spot for another
|
||||
- **Turtle and N-Triples RDF export interpolated entity `text` into string literals with no escaping**, so a `"`, backslash, newline, CR, or tab in the source text emitted invalid RDF other parsers rejected (#1148, closes #1098) by @pkupt — a shared `_escape_turtle_literal()` (later consolidated in #1221) now escapes per the RDF 1.1 Turtle grammar and is reused for the N-Triples path, which previously escaped only quotes and newlines. `tests/export/`: 161 passed, 1 skipped
|
||||
- **`PipelineSerializer` round trips dropped step dependencies and delta-processing metadata, and could rehydrate a legacy stringified handler as a non-callable string** (#1217, fixes #1216) by @cxzg007 — step dependencies, delta mode, and base/target version IDs are now restored from the serialized schema; runtime handler callables are treated as process-local state and excluded from serialized business configuration rather than (mis)serialized. 52 tests pass
|
||||
- **`PipelineBuilder` never actually dispatched to a handler registered by `step_type`**, and a serialize/deserialize round trip could leak `handler`/`dependencies` into a step's business config (#1215, fixes #1214) by @cxzg007 — a registered handler is now resolved by `step_type` when no explicit `handler=` is supplied (explicit handlers still take precedence), and the two builder-control fields are kept out of `PipelineStep.config` so a strict handler signature can't receive them as unexpected kwargs. `tests/core`+`tests/pipeline`: 50 passed
|
||||
- **`PipelineBuilder.set_parallelism()` was accepted and stored but never read — pipeline steps always ran strictly sequentially**, and the setting didn't survive a serialize/deserialize round trip (#1226, fixes #1223) by @cxzg007 — wired through builder → serializer → execution engine, plus a new opt-in `PipelineStep.parallel_safe` flag. A dependency layer now runs in parallel only when every step in it is marked `parallel_safe`, the layer has more than one step, the input is dict-typed, and no step is in delta mode; otherwise it falls back to sequential execution. Each parallel step's input is deep-copied for isolation, execution is bounded by `ThreadPoolExecutor(max_workers=min(configured parallelism, max_workers))`, a failure cancels pending futures in the layer, and layer results merge back in declaration order (a same-key conflict raises `ProcessingError`). 22 new tests in `tests/pipeline/test_pipeline_parallel.py`
|
||||
- **`Config.get()` silently dropped boolean environment-variable overrides** (#1038, fixes #1035) by @Kyou12138 — the type dispatch checked `isinstance(default, int)` before `isinstance(default, bool)`, and since `bool` subclasses `int` in Python, the bool branch was unreachable: `CONFLICT_ZZTESTFLAG=true` with a `False` default returned `False`, and `=1` returned the int `1` rather than `True`. Bool is now checked first (with whitespace stripped before parsing truthy/falsy spellings), fixed across all ten affected config modules (`conflicts`, `deduplication`, `split`, `embeddings`, `export`, `ingest`, `kg`, `parse`, `ontology`, `normalize`). 12 new tests plus 6 existing conflicts tests and 131 related module tests pass
|
||||
- **Scanned (image-only) PDFs parsed with no error and no warning, returning empty text with a "completed" status** (#1021, closes #1020) by @shanyu910 — `PDFParser._parse_page` swallowed a missing text layer via `page.extract_text() or ""`, so the failure only surfaced far downstream as zero extracted entities. A warning now fires when every parsed page yields no text with `extract_text` enabled, pointing at `parse_pdf(..., method="docling", enable_ocr=True)`. Also fixes a separate `import semantica.parse` failure on a fresh interpreter (`email_parser.py` used `email.message.Message` without importing `email.message`) that was blocking the parse test suite from even collecting. 25 tests pass in `tests/parse/`
|
||||
- **`GET /api/decisions` returned HTTP 422 for any graph containing real decisions**, breaking the Explorer Decisions workspace entirely (#937) by @logan-jl-cc — `record_decision()` stores the timestamp as a POSIX float, but `DecisionResponse.timestamp` is typed `Optional[str]` and Pydantic's strict mode rejected the coercion. Fixed by coercing to `str` (preserving `None`) at the response-adapter boundary
|
||||
- **Decision persistence/query bugs, CJK text handling, and three missing MCP graph tools** (#967) by @toratto — `mcp_server`'s `_get_graph` called a non-existent `graph.load` instead of `load_from_file`, so `SEMANTICA_KG_PATH` was silently ignored and the server always started with an empty graph; `query_decisions` read `category` from the wrong field, always returning nothing for a category filter; `find_precedents`/`query_decisions(query=)`'s similarity threshold was too high for short CJK queries, which also failed outright because `_calculate_decision_content_similarity`'s whitespace-Jaccard fallback is always zero for languages with no whitespace tokenization (now falls back further to a character-bigram overlap coefficient); `load_from_file` didn't rebuild the in-memory decision/entity/temporal indexes after loading, breaking `find_precedents_by_scenario` and decision counts post-reload; `extract_entities`/`extract_relations` returned the spaCy type label as `text` and dropped the actual entity text, and had no way to select a non-English NER model. Also adds three new MCP tools (`query_graph`, `update_node`, `delete_node`, the latter two persisting back to `SEMANTICA_KG_PATH`)
|
||||
- **`sqlalchemy.text` was used but never imported in two `DBIngestor`/`DataExporter` methods**, raising `NameError` on every call before any query reached the database (#1017, closes #1015) by @pravit-amp — `connect()`/`test_connection()` imported `text` function-locally, so the binding never reached `export_table_data()` or `execute_query()`, which called it anyway; both raised immediately, re-wrapped by an `except Exception` into a `ProcessingError` that read like a database fault rather than a missing import. `docs/guides/ontology.md` documents `DBIngestor().execute_query()` as a supported entry point, so documented usage walked straight into it. 5 new tests against a temporary SQLite database, also repairing a previously-failing `tests/ingest/test_notebook_02.py` case
|
||||
- **Ontology generation resolved relationship endpoint types incorrectly, producing wrong object-property domains/ranges** (#1170, closes #1168) by @T1mn — endpoint types are now resolved from the canonical `source_id`/`target_id` fields and supported aliases instead of defaulting to the first entity when a field was missing, preventing e.g. a `Person -> Organization` relationship from generating a `Person -> Person` property. 80 tests pass, 1 skipped
|
||||
- **Ontology property generation dropped data properties when a raw entity type was normalized into a class name** (#1171, closes #1169) by @T1mn — e.g. `software engineer` → `SoftwareEngineer` lost its `email` property; attributes are now grouped by matching raw, normalized, and recorded class names, so the normalized class stays each property's domain. 79 tests pass, 1 skipped
|
||||
- **`flatten_dict()` silently dropped data when a top-level key already containing the separator collided with a key produced by flattening a nested dict** (#1012, fixes #1010) by @yzxcj797 — `{"a.b": 1, "a": {"b": 2}}` flattened to `{"a.b": 2}` with no error, the `1` simply gone; collisions are now detected (unique-key count vs. item count) and raise `ValueError` naming the colliding key before data is lost. 6 new tests
|
||||
- **Creating relationships after `GraphStore.add_edges`/`build_from_entities_and_relationships` silently produced zero edges against ID-minting backends** (#1173, fixes #1136) by @yzxcj797 — an id-space mismatch across three layers: `add_edges` reads application-level string ids and passes them to `create_relationship`, which is a pure passthrough into `Neo4jStore.create_relationship`'s `MATCH ... WHERE id(a) = $start_id` — a Neo4j-internal integer id. Every node was created and every relationship silently failed with one easily-missed warning per edge. `GraphStore` now keeps an application-id→internal-id map, populated by `add_nodes`/`create_node` from the backend's own creation results and consulted by `create_relationship`; unknown ids and identity-mapped backends are unaffected. `tests/graph_store/`: 100 passed
|
||||
- **RDF export left `semantica:text`/`rdfs:label` empty for entities that only carry a `name` field**, across all four RDF formats (#1113, fixes #1097) by @cxzg007 — `RDFSerializer.convert_kg_to_rdf()` already implemented the `name`→`label`/`text` normalization, but `export_to_rdf()` never called it. Now called once at the export boundary (idempotent, non-destructive, falls back to a label derived from the id suffix). 7 new tests, `tests/export/test_rdf_exporter.py`: 17 passed
|
||||
- **Docker Explorer image failed to build on Python 3.14** — `gensim` has no prebuilt wheel for it and the slim base has no `gcc` to build from source (#1172, closes #1025) by @DwitiThaker — runtime pinned to `python:3.13-slim`, where `gensim` installs from a prebuilt wheel
|
||||
- **Unit normalization rejected common aliases before conversion** — `kg`, `g`, and other abbreviated/plural unit spellings failed category validation and the conversion-factor lookup ahead of it (#939) by @Mr-Neutr0n — aliases now normalize first; canonical aliases added for feet, yards, miles, and gallons. 7 tests pass
|
||||
- **An oversized, caller-controlled mapping key could blow up a `ValidationError` message to megabyte scale**, and equally inflate application logs on repeated malformed input (#1088, fixes #1001) by @ALDRIN121 — follow-up to the graph-payload validation added in #958. The displayed key is now truncated at 64 characters with an ellipsis; the underlying input and validation decisions are unchanged. 4 new tests
|
||||
- **`SeedDataManager.load_from_api()` mislabeled genuine connection failures as a missing `requests` dependency** (#972, closes #949) by @pravit-amp — `requests.exceptions.RequestException` (connection errors, timeouts, `raise_for_status()` failures) subclasses `OSError`, so an `except (ImportError, OSError)` block written to guard a lazy import that no longer existed (`requests` is a core dependency) caught real failures too and told users to reinstall an already-installed library while dropping the original exception chain. The block is removed; genuine failures now surface through the existing `Failed to load from API: {e}` path with `from e` intact. 5 new regression tests
|
||||
- **`SHACLGenerator` produced shapes that matched nothing, and pySHACL reported `conforms: True` on data that plainly violated them** (#1124, closes #1104, closes #1105) by @fabio-rovai — `base_uri` was used both as where shape resources live and to expand every `sh:targetClass`/`sh:path`, so with the default shapes namespace, generated shapes targeted classes no data graph in the package actually uses; a shape with zero matching focus nodes is vacuously satisfied, so validation silently passed regardless of real violations. The target namespace now resolves independently (explicit argument → ontology's declared namespace → an existing absolute class/property IRI → ontology `uri` → the vocabulary namespace), never the shapes namespace. Separately, `_attach_property_shapes` attached a domain-less property's constraint to *every* shape ("no domain declared, attach to all"), asserting a constraint the ontology never stated; a domain-less property is now left unattached by default, with `attach_domainless_properties=True` to restore the old behavior. 17 new tests validate real data through pySHACL rather than reading shape text; `tests/ontology`+`tests/export`: 239 passed
|
||||
- **OWL export dropped every generated property and collapsed distinct classes onto one node** (#1123, closes #1103) by @fabio-rovai — `OWLExporter` reads `object_properties`/`data_properties`, but `OntologyGenerator` emits one combined `properties` list, so every property was silently discarded; separately, a class built without a namespace manager gets `"uri": None`, which a `"uri" not in cls"` guard never catches (the key is present), so the exporter wrote a relative `<>` IRI for it — resolved by rdflib against the current working directory, meaning two classes could collapse onto one subject and that subject's identity changed with the export's working directory. Both dict shapes are now merged and classified correctly, and a class/property IRI resolves through `uri`→`iri`→`id`→a name joined onto the ontology base, skipping (with a warning) a term with none of those instead of minting `<>`. 10 new regression tests parse the real output with rdflib and Oxigraph; `tests/export`+`tests/ontology`: 231 passed
|
||||
- **Confidence scores serialized as four different, mutually-disagreeing RDF terms depending on export format, and one non-numeric confidence value could break an entire Turtle export** (#1125, closes #1100, closes #1102) by @fabio-rovai — Turtle wrote a bare `xsd:decimal`, N-Triples an explicit `xsd:float`, RDF/XML an untyped plain literal, and JSON-LD's native number expanded to `xsd:double`; loading a Turtle and an N-Triples export of the same graph into one store gave the same entity two different confidence values. Separately, an unparseable confidence (e.g. the string `"high"`) was interpolated into Turtle with no validation, producing a syntax error that dropped every entity from the export. All four paths now write one canonical `xsd:decimal` lexical form (matching the pre-existing Turtle behavior and the only exact representation of the four); an unusable value is omitted with a warning instead of corrupting the document. The vocabulary's `sem:confidence` now declares `xsd:decimal` (previously left undeclared to avoid contradicting the disagreeing exporters). 20 new tests compare parsed graphs across all four formats; `tests/export`+`tests/ontology`: 240 passed
|
||||
- **An OWL-Time validity interval was reified onto a relationship IRI the graph never actually referenced**, making it unreachable from the edge it described (#1126, closes #1106) by @fabio-rovai — a relationship serializes as a single triple with no node of its own, so `include_temporal=True` minted a well-formed `time:Interval` with zero inbound arcs to its subject. Turtle now also emits the `sem:Relationship`/`sem:source`/`sem:target`/`sem:type` reification the JSON-LD path already produced, but only when there's temporal data to attach — default and `include_temporal=False` output are byte-for-byte unchanged. 7 new tests include a SPARQL walk from the edge to its interval, the path the dangling node made impossible; `tests/export`+`tests/ontology`: 228 passed
|
||||
- **JSON-LD exports were unreadable by Semantica's own default parser** (#1145, fixes #1144) by @fabio-rovai — every export was written as a named graph (a top-level `@id` beside `@graph`), which a plain `rdflib.Graph.parse()` silently discards in favor of the (empty) default graph; a two-entity graph parsed as 2 triples instead of 20. Compounded by `export_knowledge_graph` converting its payload to JSON-LD and then handing the *already-converted* document to `export()`, which converted it again, producing two `@context` blocks and two document nodes. Metadata now attaches beside `@graph` rather than naming it, and a payload that already declares `@context` is merged rather than re-wrapped. 9 new tests parse with both `Graph()` and `Dataset()` and assert identical counts; full-suite failure set unchanged before/after (539/539)
|
||||
- **`GraphBuilder` didn't propagate entity-resolution's merged ids into the `source_id`/`target_id` relationship aliases**, only `source`/`target` (#1115, closes #1110) by @T1mn — a relationship's alias fields could still point at a pre-merge id after resolution. Both alias pairs are now kept in sync. 9 tests pass
|
||||
- **`GraphValidator` indexed entities only by `id`, rejecting graphs that use the `entity_id` alias as invalid even when their relationships were fine** (#1116, closes #1111) by @T1mn — validation and endpoint checks now go through the shared `get_entity_id()` helper, accepting both fields consistently. 5 tests pass
|
||||
- **Broken star history chart in README** (#1057) by @OctoBored — the embedded chart used the GitHub stargazer API, now access-restricted; switched to a token-free alternative data source
|
||||
|
||||
### Security
|
||||
|
||||
- **Agno's `AgnoKnowledgeGraph.load_urls()` made outbound requests with no SSRF protection beyond a scheme check** (#1212) by @Sameer6305 — caller-supplied URLs went straight to `urllib.request.urlopen()`, unguarded against loopback/private addresses, cloud metadata endpoints (`169.254.169.254`), IPv6-internal addresses, hostnames resolving to private space, or redirects into any of the above. Found during a project-wide SSRF audit following #936/#959. Now routed through the shared `request_with_ssrf_guard()`; an unsafe URL is skipped rather than aborting the rest of the ingestion batch. `OpenClawKGTool` (operator-configured, intentionally allowed to target `localhost` for local deployments) gains scheme/malformed-URL validation as defense in depth, without restricting its legitimate private-network use case. 29 new Agno tests, 26 new OpenClaw tests, all passing alongside the 15 pre-existing Agno integration tests
|
||||
|
||||
### Dependencies
|
||||
|
||||
- Routine version bumps with no application-facing behavior change: `anthropic` 0.121.0→0.122.0 (#1045), `botocore` 1.43.69→1.43.73 (#1047), `agno` 2.8.7→2.9.0 (#1050), `google-genai` 2.17.0→2.18.1→2.19.0 (#1163, #1205), `lxml` 6.1.1→6.1.2 (#1197), `charset-normalizer` 3.5.0→3.5.1 (#1201), `pypickle` 2.0.1→2.0.2 (#1203)
|
||||
|
||||
## [0.6.6] - 2026-08-20
|
||||
|
||||
### Added
|
||||
|
||||
@@ -142,7 +142,7 @@ compliant = graph.check_decision_rules({"category": "vendor_selection"}) # poli
|
||||
```bash
|
||||
semantica doctor
|
||||
# Python 3.11.9 pass
|
||||
# semantica 0.6.6 pass
|
||||
# semantica 0.6.7 pass
|
||||
# faiss vector store pass
|
||||
# Config file pass ~/.semantica/config.yaml
|
||||
```
|
||||
@@ -1463,18 +1463,18 @@ For contributor / dev-server setup: **[explorer/README.md: Local Setup Guide](ex
|
||||
|
||||
---
|
||||
|
||||
## What's New in v0.6.6
|
||||
## What's New in v0.6.7
|
||||
|
||||
**Security release — upgrading is strongly recommended.** Fixes for a privately disclosed batch of vulnerabilities spanning backup/restore, database export, outbound requests, and triplet-store backends, plus SSRF hardening across ingestion:
|
||||
**Feature release**, plus one SSRF hardening fix and a large batch of correctness fixes across the RDF/ontology export pipeline:
|
||||
|
||||
- **Tarball restore path traversal**: `semantica backup restore` now validates every archive member for path containment and rejects symlink/hardlink escapes before extraction
|
||||
- **Latent SQL injection in `DataExporter.export_table_data()`**: table/schema names are now identifier-allowlisted and `where`/`order_by` fragments are blocklist-checked
|
||||
- **DNS-rebinding TOCTOU in the shared SSRF guard**: the resolved IP that passes validation is now the one the connection is pinned to, closing the check-then-use race (also closes the `100.64.0.0/10` CGNAT gap)
|
||||
- **Stored XSS in HTML report generation** and **unvalidated SPARQL object IRIs in AnzoStore** (SPARQL injection): both now escape/validate before interpolation
|
||||
- **`Authorization`/`Proxy-Authorization` credential leakage across redirects**, plus **SSRF gaps in `FeedIngestor`/`FeedMonitor`, `RepoIngestor`, and the MCP/public-API ingest paths**: all now route through the shared, redirect-safe SSRF guard
|
||||
- **HTTP response header injection and an unbounded-memory DoS** in the Explorer API, and a **`fastapi`/`python-multipart` ReDoS** (PYSEC-2024-38): floors raised, inputs sanitized, candidate pools capped
|
||||
- **First-class LangChain integration** (`semantica[langchain]`): a `BaseRetriever` and `VectorStore` over `HybridSearch`, plus graph/decision-query tools
|
||||
- **SAP OData ingestor** (`semantica[ingest-sap]`): OAuth2/Basic-auth, SSRF-guarded ingestion for Business Partners and Sales Orders, following the existing Snowflake/Databricks connector pattern
|
||||
- **`ContextGraph` gains deterministic, human-editable Markdown round-trip persistence** alongside the existing JSON API, and the Explorer graph inspector gains a read-only Markdown content viewer
|
||||
- **`reasoning` gains a structured Action layer**: rule-driven `Assert`/`Retract`/`Call`/`EmitEvent` actions with optional provenance, turning the reasoner into a production-rule system
|
||||
- **`run_shacl_validation` is now a public, documented API**, and a dozen ontology/RDF export correctness fixes land: OWL property/class export, SHACL target-namespace resolution, one canonical confidence datatype across all four RDF formats, reachable OWL-Time reification, JSON-LD default-graph and content-derived document identity, and full metadata passthrough on every RDF serializer
|
||||
- **Security**: Agno's `AgnoKnowledgeGraph.load_urls()` and OpenClaw's MCP tool now route outbound requests through the shared SSRF guard
|
||||
|
||||
Also ships: **first-class CrewAI integration** (`semantica[crewai]`, extraction/decision tools + a knowledge source), **`ContextGraph` retraction and purge** (GDPR-style erasure without a full `clear()`), a declared **Semantica RDF vocabulary with deterministic entity/relationship IRIs** (stable, diffable exports), and **timezone-aware timestamps** across `export/` and `provenance/`.
|
||||
Also fixes: `PipelineBuilder.set_parallelism()` now actually parallelizes independent pipeline steps, `flatten_dict()` no longer silently drops data on a key collision, `Config.get()` honors boolean environment overrides, and the MCP server's `export_graph` tool works again on every format.
|
||||
|
||||
→ [Full release notes](RELEASE_NOTES.md) · [Changelog](CHANGELOG.md)
|
||||
|
||||
|
||||
+1
-1
@@ -17,7 +17,7 @@ icon: "circle-question"
|
||||
| API key required? | Optional: pattern extraction works with no keys |
|
||||
| Works with LangChain / LlamaIndex? | Yes: Semantica is a layer on top, not a replacement |
|
||||
| Production-ready? | Yes: 1,000+ tests, v0.5.0 ships with 12 security fixes |
|
||||
| Latest version? | **v0.6.6** (August 2026) |
|
||||
| Latest version? | **v0.6.7** (August 2026) |
|
||||
| Local LLMs? | Yes: Ollama via LiteLLM, HuggingFaceLLM for air-gapped |
|
||||
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ icon: "rocket"
|
||||
Verify installation:
|
||||
```python
|
||||
import semantica
|
||||
print(semantica.__version__) # 0.6.6
|
||||
print(semantica.__version__) # 0.6.7
|
||||
```
|
||||
</Check>
|
||||
</Step>
|
||||
|
||||
@@ -382,6 +382,45 @@ For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-p
|
||||
|
||||
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
|
||||
|
||||
## Source 7 — SAP OData
|
||||
|
||||
`SAPIngestor` ingests an Entity Set from a SAP OData service (S/4HANA Cloud, SuccessFactors, or an on-prem NetWeaver Gateway over its REST surface). It speaks OData v2 and v4, follows server-driven pagination automatically, and flattens each record into a document dict via `export_as_documents()` — the same structured "transform to text, then store" pattern as the other sources.
|
||||
|
||||
```python
|
||||
from semantica.ingest import SAPIngestor
|
||||
|
||||
ing = SAPIngestor(
|
||||
base_url="https://my-sap.example.com/sap/opu/odata/sap/API_BUSINESS_PARTNER",
|
||||
client_id="...", client_secret="...",
|
||||
token_url="https://my-sap.example.com/oauth/token", # OAuth2 client-credentials (BTP/S/4HANA Cloud)
|
||||
# On-prem NetWeaver often uses Basic auth instead — swap the block above for:
|
||||
# username="erp_user", password="...",
|
||||
)
|
||||
|
||||
# 1. Discover an unfamiliar service: entity sets + field types from $metadata
|
||||
sets = ing.discover_service() # [{"name": "A_BusinessPartnerSet", "fields": [...]}, ...]
|
||||
|
||||
# 2. Pull a page-walked Entity Set (v2/v4 next links handled for you)
|
||||
partners = ing.ingest_entity_set(
|
||||
entity_set="A_BusinessPartnerSet",
|
||||
select="BusinessPartner,BusinessPartnerFullName", # $select
|
||||
top=1000, # cap on total rows
|
||||
)
|
||||
|
||||
# 3. Flatten to document dicts, then build text for the graph
|
||||
docs = ing.export_as_documents(partners)
|
||||
partner_texts = [
|
||||
f"Business Partner {d['BusinessPartner']}: {d['BusinessPartnerFullName']}"
|
||||
for d in docs
|
||||
]
|
||||
```
|
||||
|
||||
- Use `expand="to_Item"` (e.g. on a sales-order header set) to pull nested line items in one request — handy for modeling order → line-item → material relationships.
|
||||
- Every outbound request, including the OAuth2 token exchange, is routed through the SSRF guard, so a user-supplied SAP URL can never reach private/loopback/link-local address space.
|
||||
- Install with `pip install 'semantica[ingest-sap]'`.
|
||||
|
||||
> **Security Note:** Never hardcode credentials (`client_secret`, `password`) in code; pass them via environment variables (e.g., `SAP_CLIENT_SECRET`, `SAP_PASSWORD`) or a secrets manager.
|
||||
|
||||
## Combining All Five Sources
|
||||
|
||||
Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly.
|
||||
|
||||
@@ -28,6 +28,7 @@ icon: "database"
|
||||
| `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries |
|
||||
| `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
|
||||
| `DatabricksIngestor` | Databricks Unity Catalog metadata, Delta table queries, and lineage |
|
||||
| `SAPIngestor` | SAP OData services (S/4HANA Cloud, SuccessFactors, NetWeaver Gateway): entity-set discovery and ingestion with v2/v4 pagination |
|
||||
| `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
|
||||
| `ArrowIngestor` | Apache Arrow IPC and Feather file processing |
|
||||
| `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |
|
||||
|
||||
+2
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "semantica"
|
||||
version = "0.6.6"
|
||||
version = "0.6.7"
|
||||
description = "Graph-Native Infrastructure for Context and Accountable AI Systems: context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
@@ -127,6 +127,7 @@ db-arrow = ["pyarrow>=24.0.0"]
|
||||
db-salesforce = ["simple-salesforce>=1.12.0"]
|
||||
ingest-parquet = ["pyarrow>=24.0.0"]
|
||||
ingest-arrow = ["pyarrow>=24.0.0"]
|
||||
ingest-sap = ["requests>=2.28.0"]
|
||||
|
||||
db-all = [
|
||||
"semantica[db-snowflake,db-databricks,db-salesforce,db-arrow]"
|
||||
|
||||
@@ -10,7 +10,7 @@ Main exports:
|
||||
- Config: Configuration management
|
||||
"""
|
||||
|
||||
__version__ = "0.6.6"
|
||||
__version__ = "0.6.7"
|
||||
__author__ = "Semantica Contributors"
|
||||
__license__ = "MIT"
|
||||
|
||||
|
||||
@@ -222,6 +222,10 @@ _LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
|
||||
"SnowflakeIngestor": (".snowflake_ingestor", "SnowflakeIngestor"),
|
||||
"SnowflakeData": (".snowflake_ingestor", "SnowflakeData"),
|
||||
"SnowflakeConnector": (".snowflake_ingestor", "SnowflakeConnector"),
|
||||
# SAP OData ingestion
|
||||
"SAPIngestor": (".sap_ingestor", "SAPIngestor"),
|
||||
"SAPODataEntity": (".sap_ingestor", "SAPODataEntity"),
|
||||
"SAPODataConnector": (".sap_ingestor", "SAPODataConnector"),
|
||||
# Databricks ingestion
|
||||
"DatabricksIngestor": (".databricks_ingestor", "DatabricksIngestor"),
|
||||
"DatabricksData": (".databricks_ingestor", "DatabricksData"),
|
||||
@@ -358,6 +362,10 @@ __all__ = [
|
||||
"SnowflakeIngestor",
|
||||
"SnowflakeData",
|
||||
"SnowflakeConnector",
|
||||
# SAP OData ingestion
|
||||
"SAPIngestor",
|
||||
"SAPODataEntity",
|
||||
"SAPODataConnector",
|
||||
# Databricks ingestion
|
||||
"DatabricksIngestor",
|
||||
"DatabricksData",
|
||||
|
||||
@@ -1703,3 +1703,54 @@ for source_type, source_list in sources.items():
|
||||
for batch in process_in_batches(large_dataset, batch_size=1000):
|
||||
result = ingest(batch)
|
||||
```
|
||||
|
||||
## SAP OData Ingestion
|
||||
|
||||
`SAPIngestor` reads an Entity Set from a SAP OData service — S/4HANA Cloud,
|
||||
SuccessFactors, or an on-prem NetWeaver Gateway over its REST surface. It
|
||||
follows OData v2/v4 server-driven pagination and flattens each record into a
|
||||
document dict via `export_as_documents()`.
|
||||
|
||||
Install with `pip install 'semantica[ingest-sap]'`.
|
||||
|
||||
### Connector Construction & Authentication
|
||||
|
||||
```python
|
||||
from semantica.ingest import SAPIngestor
|
||||
|
||||
# OAuth2 client-credentials (BTP / S/4HANA Cloud)
|
||||
ing = SAPIngestor(
|
||||
base_url="https://my-sap.example.com/sap/opu/odata/sap/API_BUSINESS_PARTNER",
|
||||
client_id="...", client_secret="...",
|
||||
token_url="https://my-sap.example.com/oauth/token",
|
||||
)
|
||||
# On-prem NetWeaver often uses Basic auth instead — swap the block above for:
|
||||
# ing = SAPIngestor(base_url="...", username="erp_user", password="...")
|
||||
```
|
||||
|
||||
### Entity-Set Ingestion & Document Export
|
||||
|
||||
```python
|
||||
# 1. Discover entity sets + field types from $metadata
|
||||
sets = ing.discover_service()
|
||||
|
||||
# 2. Page-walk an Entity Set (v2/v4 next links handled automatically)
|
||||
partners = ing.ingest_entity_set(
|
||||
entity_set="A_BusinessPartnerSet",
|
||||
select="BusinessPartner,BusinessPartnerFullName",
|
||||
top=1000,
|
||||
)
|
||||
|
||||
# 3. Flatten to document dicts that GraphBuilder can consume directly
|
||||
docs = ing.export_as_documents(partners)
|
||||
```
|
||||
|
||||
- Use `expand="to_Item"` on a sales-order header set to pull nested line items
|
||||
in one request — handy for modeling order → line-item → material relations.
|
||||
- Every outbound request, including the OAuth2 token exchange, is routed through
|
||||
the SSRF guard, and pagination never follows a next link that points to a
|
||||
different host than the service root.
|
||||
|
||||
> **Security Note:** Never hardcode credentials (`client_secret`, `password`);
|
||||
> pass them via environment variables (`SAP_CLIENT_SECRET`, `SAP_PASSWORD`) or a
|
||||
> secrets manager.
|
||||
|
||||
@@ -0,0 +1,617 @@
|
||||
"""SAP OData ingestion module.
|
||||
|
||||
Pulls an Entity Set from a SAP OData service (S/4HANA and on-prem NetWeaver
|
||||
REST surfaces) and flattens it into document dicts that the pipeline can feed
|
||||
to ``GraphBuilder``.
|
||||
|
||||
Why this exists
|
||||
---------------
|
||||
Semantica ingests from many sources; SAP is the ERP backbone of finance and
|
||||
regulated industries, and its master/transactional data (customers, vendors,
|
||||
sales orders) is exactly the "context" a Context Graph wants. SAP exposes that
|
||||
data over OData (v2 on many on-prem NetWeaver systems, v4 on BTP / S/4HANA
|
||||
Cloud). This connector speaks the REST surface of OData only.
|
||||
|
||||
Design notes
|
||||
------------
|
||||
Three classes, matching the Snowflake/Databricks ingestors:
|
||||
- ``SAPODataEntity``: a collection fetch from one Entity Set (``records``,
|
||||
``count``, ``service``, ``metadata``), flattened to document dicts for
|
||||
``GraphBuilder`` via ``export_as_documents``.
|
||||
- ``SAPODataConnector``: auth (OAuth2 client-credentials or Basic) + the
|
||||
shared, SSRF-guarded :mod:`requests` session. *Every* outbound request,
|
||||
including the OAuth2 token exchange, goes through
|
||||
``request_with_ssrf_guard`` so user-supplied endpoints can not reach
|
||||
private/loopback/link-local address space.
|
||||
- ``SAPIngestor``: the three methods the issue requested —
|
||||
``discover_service``, ``ingest_entity_set`` and ``export_as_documents``
|
||||
(plus ``close`` for symmetry with the SQL connectors).
|
||||
|
||||
EDMX
|
||||
----
|
||||
``$metadata`` is plain CSDL XML in *both* OData v2 and v4, so we hand-roll a
|
||||
minimal parser with :mod:`xml.etree` instead of pulling in ``pyodata``. That
|
||||
keeps phase 1 ``requests``-only, exactly as scoped in the issue.
|
||||
|
||||
Pagination
|
||||
----------
|
||||
OData uses a server-driven "next link": OData v2 surfaces it as the atom
|
||||
``__next`` element, OData v4 as the ``@odata.nextLink`` field on the JSON
|
||||
payload. ``ingest_entity_set`` follows whichever it sees until the set is
|
||||
exhausted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import os
|
||||
import xml.etree.ElementTree as ET
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import requests
|
||||
from requests.adapters import HTTPAdapter
|
||||
|
||||
try:
|
||||
from urllib3.util.retry import Retry
|
||||
except (ImportError, OSError): # pragma: no cover - old urllib3 layout
|
||||
from requests.packages.urllib3.util.retry import Retry # type: ignore
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from .ssrf import parse_bool, request_with_ssrf_guard
|
||||
|
||||
__all__ = [
|
||||
"SAPODataEntity",
|
||||
"SAPODataConnector",
|
||||
"SAPIngestor",
|
||||
]
|
||||
|
||||
_logger = get_logger("sap_ingestor")
|
||||
|
||||
|
||||
def _prop_is_nullable(prop: Any) -> bool:
|
||||
"""CSDL structural properties default to nullable=True when omitted."""
|
||||
val = prop.get("Nullable", prop.get("nullable"))
|
||||
return True if val is None else val.strip().lower() == "true"
|
||||
|
||||
|
||||
def _match(elem: Any, localname: str) -> bool:
|
||||
"""True if *elem* has the given local name in any namespace."""
|
||||
return elem.tag.rsplit("}", 1)[-1] == localname
|
||||
|
||||
|
||||
@dataclass
|
||||
class SAPODataEntity:
|
||||
"""A collection fetch from a SAP OData Entity Set.
|
||||
|
||||
Holds the rows pulled from one Entity Set (all paging fan-in'd), with the
|
||||
shape the issue specifies: ``records`` (the row data), ``count``,
|
||||
``service`` (the resolved service root), optional ``metadata`` (entity-set
|
||||
schema from ``$metadata``) and ``ingested_at``.
|
||||
"""
|
||||
|
||||
records: List[Dict[str, Any]]
|
||||
entity_set: str
|
||||
count: int
|
||||
service: str
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
ingested_at: datetime = field(default_factory=datetime.now)
|
||||
|
||||
def to_documents(self) -> List[Dict[str, Any]]:
|
||||
"""Flatten each record to a document dict ``GraphBuilder`` can consume.
|
||||
|
||||
GraphBuilder only treats a dict as an entity when it carries
|
||||
``id``/``entity_id``/``name`` (or ``text``+``type``); SAP records have
|
||||
none of those, so they would be silently dropped. We inject an
|
||||
identifier resolved from each record's primary-key-like field, falling
|
||||
back to ``entity_set:index``, and expose it under both ``id`` and
|
||||
``name``.
|
||||
"""
|
||||
docs: List[Dict[str, Any]] = []
|
||||
for index, record in enumerate(self.records):
|
||||
doc = dict(record)
|
||||
key_value = self._id_value(record)
|
||||
doc.setdefault("id", key_value or f"{self.entity_set}:{index}")
|
||||
doc.setdefault("name", key_value or self.entity_set)
|
||||
doc.setdefault("source", self.service)
|
||||
docs.append(doc)
|
||||
return docs
|
||||
|
||||
@staticmethod
|
||||
def _id_value(record: Dict[str, Any]) -> str:
|
||||
for key, value in record.items():
|
||||
if "id" in key.lower() and value not in (None, ""):
|
||||
return str(value)
|
||||
return ""
|
||||
|
||||
|
||||
class SAPODataConnector:
|
||||
"""Connection + authentication management for a SAP OData REST service.
|
||||
|
||||
Supports the two auth landscapes called out in the issue:
|
||||
|
||||
- **OAuth2 client-credentials** (BTP / S/4HANA Cloud). The token URL is
|
||||
user supplied; both the token exchange *and* every subsequent data
|
||||
request are validated through the SSRF guard.
|
||||
- **Basic** (on-prem NetWeaver). Username/password passed through as an
|
||||
``Authorization: Basic`` header, also through the guard.
|
||||
|
||||
Example usage::
|
||||
|
||||
>>> connector = SAPODataConnector(
|
||||
... base_url="https://myhost/sap/opu/odata/sap/",
|
||||
... token_url="https://myhost/oauth/token",
|
||||
... client_id="cid", client_secret="secret",
|
||||
... )
|
||||
>>> session = connector.get_session()
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: Optional[str] = None,
|
||||
*,
|
||||
auth: Optional[str] = None,
|
||||
token_url: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None,
|
||||
username: Optional[str] = None,
|
||||
password: Optional[str] = None,
|
||||
allow_private_ips: bool = False,
|
||||
**config: Any,
|
||||
) -> None:
|
||||
"""Initialize the SAP OData connector.
|
||||
|
||||
Args:
|
||||
base_url: Base OData service URL, e.g. ``https://host/sap/opu
|
||||
/odata/sap/``. The issue's ``service`` value.
|
||||
auth: Explicit auth flow, ``"oauth2"`` or ``"basic"``. When
|
||||
omitted, the flow is inferred from which credentials are set.
|
||||
token_url: OAuth2 token endpoint. Required only for OAuth2 flow.
|
||||
client_id: OAuth2 client id (OAuth2 flow).
|
||||
client_secret: OAuth2 client secret (OAuth2 flow).
|
||||
username: Basic-auth username (on-prem flow).
|
||||
password: Basic-auth password (on-prem flow).
|
||||
allow_private_ips: Opt into private/loopback/link-local endpoints.
|
||||
Defaults to False (SSRF-safe).
|
||||
**config: Extra options, notably ``timeout``, ``max_retries``,
|
||||
``backoff_factor``, ``headers``.
|
||||
"""
|
||||
self.logger = _logger
|
||||
self.base_url = base_url or os.getenv("SAP_BASE_URL")
|
||||
self.auth = (auth or os.getenv("SAP_AUTH") or "").lower()
|
||||
self.token_url = token_url or os.getenv("SAP_TOKEN_URL")
|
||||
self.client_id = client_id or os.getenv("SAP_CLIENT_ID")
|
||||
self.client_secret = client_secret or os.getenv("SAP_CLIENT_SECRET")
|
||||
self.username = username or os.getenv("SAP_USERNAME")
|
||||
self.password = password or os.getenv("SAP_PASSWORD")
|
||||
self.allow_private_ips = parse_bool(
|
||||
config.pop("allow_private_ips", allow_private_ips), default=False
|
||||
)
|
||||
self.config = config
|
||||
|
||||
if not self.base_url:
|
||||
raise ValidationError(
|
||||
"SAP base_url is required. Provide via 'base_url' or "
|
||||
"SAP_BASE_URL environment variable."
|
||||
)
|
||||
oauth_configured = bool(self.client_id or self.client_secret or self.token_url)
|
||||
if self.auth in ("oauth2", "oauth"):
|
||||
if not (self.client_id and self.client_secret and self.token_url):
|
||||
raise ValidationError(
|
||||
"SAP OAuth2 flow requires client_id, client_secret and "
|
||||
"token_url all set."
|
||||
)
|
||||
elif self.auth == "basic":
|
||||
if not (self.username and self.password):
|
||||
raise ValidationError("SAP Basic flow requires username and password.")
|
||||
elif oauth_configured:
|
||||
if not (self.client_id and self.client_secret and self.token_url):
|
||||
raise ValidationError(
|
||||
"SAP OAuth2 flow requires client_id, client_secret and "
|
||||
"token_url all set."
|
||||
)
|
||||
elif not self.username:
|
||||
raise ValidationError(
|
||||
"SAP authentication requires either (username/password) or "
|
||||
"(client_id/client_secret + token_url)."
|
||||
)
|
||||
|
||||
self.session = requests.Session()
|
||||
retry_strategy = Retry(
|
||||
total=self.config.get("max_retries", 3),
|
||||
backoff_factor=self.config.get("backoff_factor", 1),
|
||||
status_forcelist=[429, 500, 502, 503, 504],
|
||||
)
|
||||
adapter = HTTPAdapter(max_retries=retry_strategy)
|
||||
self.session.mount("http://", adapter)
|
||||
self.session.mount("https://", adapter)
|
||||
default_headers = self.config.get("headers", {})
|
||||
if default_headers:
|
||||
self.session.headers.update(default_headers)
|
||||
|
||||
self._token: Optional[str] = None
|
||||
self.logger.debug(
|
||||
"SAP OData connector initialized (base_url=%s, allow_private_ips=%s)",
|
||||
self.base_url,
|
||||
self.allow_private_ips,
|
||||
)
|
||||
|
||||
def get_session(self) -> requests.Session:
|
||||
"""Return an authenticated session for data requests.
|
||||
|
||||
For the Basic flow the credentials are attached eagerly; for the
|
||||
OAuth2 flow a token is fetched (and cached) on first use. The token
|
||||
is never refreshed, so a job that runs past the token TTL (typically
|
||||
3600s on SAP) will fail with 401 -- re-create the connector instead.
|
||||
"""
|
||||
if self.username:
|
||||
self.session.headers["Authorization"] = "Basic " + self._basic_header()
|
||||
return self.session
|
||||
if self._token is None:
|
||||
self._token = self._fetch_token()
|
||||
self.session.headers["Authorization"] = "Bearer " + self._token
|
||||
return self.session
|
||||
|
||||
def _basic_header(self) -> str:
|
||||
pair = f"{self.username}:{self.password or ''}".encode("utf-8")
|
||||
return base64.b64encode(pair).decode("ascii")
|
||||
|
||||
def _fetch_token(self) -> str:
|
||||
"""Perform the OAuth2 client-credentials token exchange (SSRF-guarded)."""
|
||||
if not self.token_url or not self.client_id:
|
||||
raise ProcessingError("OAuth2 flow requires token_url and client_id.")
|
||||
body = {
|
||||
"grant_type": "client_credentials",
|
||||
"client_id": self.client_id,
|
||||
"client_secret": self.client_secret or "",
|
||||
}
|
||||
resp = request_with_ssrf_guard(
|
||||
"POST",
|
||||
self.token_url,
|
||||
session=self.session,
|
||||
allow_private_ips=self.allow_private_ips,
|
||||
data=body,
|
||||
timeout=self.config.get("timeout", 30),
|
||||
)
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
except requests.exceptions.RequestException as exc:
|
||||
raise ProcessingError(f"SAP OAuth2 token exchange failed: {exc}") from exc
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise ProcessingError(
|
||||
"SAP OAuth2 token endpoint did not return JSON."
|
||||
) from exc
|
||||
token = payload.get("access_token")
|
||||
if not token:
|
||||
raise ProcessingError("SAP OAuth2 token response missing 'access_token'.")
|
||||
return str(token)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the underlying :mod:`requests` session."""
|
||||
self.session.close()
|
||||
|
||||
|
||||
class SAPIngestor:
|
||||
"""Ingest an Entity Set from a SAP OData service.
|
||||
|
||||
Example usage::
|
||||
|
||||
>>> from semantica.ingest import SAPIngestor
|
||||
>>> ing = SAPIngestor(
|
||||
... base_url="https://host/sap/opu/odata/sap/",
|
||||
... username="u", password="p", # or client_id/client_secret/token_url
|
||||
... )
|
||||
>>> sets = ing.discover_service()
|
||||
... # -> [{"name": "SalesOrderSet", "fields": [...]}, ...]
|
||||
>>> docs = ing.export_as_documents(
|
||||
... ing.ingest_entity_set(entity_set="SalesOrderSet", expand="to_Item"))
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: Optional[str] = None,
|
||||
connector: Optional[SAPODataConnector] = None,
|
||||
**config: Any,
|
||||
) -> None:
|
||||
"""Initialize the SAP ingestor.
|
||||
|
||||
Args:
|
||||
base_url: Base OData service URL. Mutually exclusive with
|
||||
``connector``; ignored if a connector is given.
|
||||
connector: An existing :class:`SAPODataConnector`. When provided,
|
||||
its session and base URL are reused.
|
||||
**config: Passed to :class:`SAPODataConnector` when one is created.
|
||||
"""
|
||||
self.logger = _logger
|
||||
self.connector = connector or SAPODataConnector(base_url=base_url, **config)
|
||||
# urljoin() replaces the last path segment unless the base ends in '/',
|
||||
# so normalize once here: .../API_BUSINESS_PARTNER -> .../$metadata would
|
||||
# silently drop the service segment.
|
||||
self._base_url = self.connector.base_url
|
||||
if not self._base_url.endswith("/"):
|
||||
self._base_url += "/"
|
||||
|
||||
def discover_service(self, service: Optional[str] = None) -> List[Dict[str, Any]]:
|
||||
"""Fetch and parse ``$metadata`` into the service's entity sets.
|
||||
|
||||
Args:
|
||||
service: Service root URL (absolute) or path suffix resolved
|
||||
against the base URL. Defaults to the base URL. ``$metadata``
|
||||
is appended automatically — same meaning as in
|
||||
:meth:`ingest_entity_set`.
|
||||
|
||||
Returns:
|
||||
List of dicts, one per EntitySet, each with ``name`` and ``fields``
|
||||
(a list of ``{name, type, nullable}`` parsed from the CSDL).
|
||||
"""
|
||||
metadata_url = self._metadata_url(service)
|
||||
session = self.connector.get_session()
|
||||
resp = request_with_ssrf_guard(
|
||||
"GET",
|
||||
metadata_url,
|
||||
session=session,
|
||||
allow_private_ips=self.connector.allow_private_ips,
|
||||
headers={"Accept": "application/xml"},
|
||||
timeout=self.connector.config.get("timeout", 30),
|
||||
)
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
except requests.exceptions.RequestException as exc:
|
||||
self.logger.error("Failed to fetch SAP metadata %s: %s", metadata_url, exc)
|
||||
raise ProcessingError(f"Failed to fetch SAP $metadata: {exc}") from exc
|
||||
|
||||
return self._parse_metadata(resp.text)
|
||||
|
||||
def _metadata_url(self, service: Optional[str]) -> str:
|
||||
"""Build the ``$metadata`` URL for a service root.
|
||||
|
||||
``service`` has the same meaning as in :meth:`ingest_entity_set` —
|
||||
a service root (absolute URL or path suffix resolved against the
|
||||
base URL). ``$metadata`` is appended here, so callers pass the root
|
||||
the same way for both discovery and ingestion. A value already
|
||||
ending in ``$metadata`` is used as-is.
|
||||
"""
|
||||
if not service:
|
||||
return urljoin(self._base_url, "$metadata")
|
||||
if "://" not in service:
|
||||
service = urljoin(self._base_url, service)
|
||||
if service.endswith("$metadata"):
|
||||
return service
|
||||
if not service.endswith("/"):
|
||||
service += "/"
|
||||
return urljoin(service, "$metadata")
|
||||
|
||||
def _parse_metadata(self, metadata_xml: str) -> List[Dict[str, Any]]:
|
||||
"""Minimal CSDL/EDMX parser -> entity set name + property fields.
|
||||
|
||||
Element local names (``Schema``/``EntitySet``/``EntityType``/
|
||||
``Property``) are stable across OData v2 (Microsoft ns) and v4 (OASIS
|
||||
ns), so we match them by local name instead of hard-coding one
|
||||
namespace. Entity-Type references are resolved per-schema, so
|
||||
same-named types in different schemas cannot bleed fields into each
|
||||
other.
|
||||
"""
|
||||
try:
|
||||
root = ET.fromstring(metadata_xml)
|
||||
except ET.ParseError as exc:
|
||||
raise ProcessingError(f"SAP $metadata is not valid XML: {exc}") from exc
|
||||
|
||||
# Index fully-qualified type name -> property fields, per schema.
|
||||
schema_types: Dict[str, List[Dict[str, Any]]] = {}
|
||||
for schema in (e for e in root.iter() if _match(e, "Schema")):
|
||||
ns = (schema.get("Namespace") or schema.get("namespace") or "").rstrip(".")
|
||||
for entity_type in (e for e in schema.iter() if _match(e, "EntityType")):
|
||||
tname = entity_type.get("Name") or entity_type.get("name")
|
||||
if not tname:
|
||||
continue
|
||||
fq = f"{ns}.{tname}" if ns else tname
|
||||
schema_types[fq] = [
|
||||
{
|
||||
"name": prop.get("Name") or prop.get("name"),
|
||||
"type": prop.get("Type") or prop.get("type"),
|
||||
"nullable": _prop_is_nullable(prop),
|
||||
}
|
||||
for prop in (e for e in entity_type.iter() if _match(e, "Property"))
|
||||
]
|
||||
|
||||
entity_sets: List[Dict[str, Any]] = []
|
||||
for schema in (e for e in root.iter() if _match(e, "Schema")):
|
||||
ns = (schema.get("Namespace") or schema.get("namespace") or "").rstrip(".")
|
||||
for entity_set in (e for e in schema.iter() if _match(e, "EntitySet")):
|
||||
name = entity_set.get("Name") or entity_set.get("name")
|
||||
ref = entity_set.get("EntityType") or entity_set.get("entityType") or ""
|
||||
qualified = ref if "." in ref else (f"{ns}.{ref}" if ns else ref)
|
||||
fields = schema_types.get(qualified) or schema_types.get(ref) or []
|
||||
entity_sets.append({"name": name, "fields": fields})
|
||||
return entity_sets
|
||||
|
||||
def ingest_entity_set(
|
||||
self,
|
||||
service: Optional[str] = None,
|
||||
entity_set: Optional[str] = None,
|
||||
*,
|
||||
select: Optional[str] = None,
|
||||
filter: Optional[str] = None,
|
||||
expand: Optional[str] = None,
|
||||
top: Optional[int] = None,
|
||||
skip: Optional[int] = None,
|
||||
batch_size: int = 100,
|
||||
) -> SAPODataEntity:
|
||||
"""Fetch pages of *entity_set* from the OData service.
|
||||
|
||||
Args:
|
||||
service: Service root URL (absolute) or path suffix resolved
|
||||
against the base URL. Defaults to the base URL. Same meaning
|
||||
as in :meth:`discover_service`.
|
||||
entity_set: Entity set name, e.g. ``"SalesOrderSet"``.
|
||||
select: Optional ``$select`` comma string.
|
||||
filter: Optional ``$filter`` expression.
|
||||
expand: Optional ``$expand`` expression (e.g. ``"to_Item"`` for
|
||||
use case 2's sales-order headers -> line items).
|
||||
top: Maximum number of rows to return.
|
||||
skip: Number of leading rows to skip.
|
||||
batch_size: ``$top`` pagination size per request.
|
||||
|
||||
Returns:
|
||||
A single :class:`SAPODataEntity` holding every fetched record
|
||||
(server-driven pagination is followed to completion).
|
||||
"""
|
||||
if service is None:
|
||||
base = self._base_url
|
||||
elif "://" in service:
|
||||
base = service
|
||||
else:
|
||||
base = urljoin(self._base_url, service)
|
||||
if not base.endswith("/"):
|
||||
base += "/"
|
||||
if not entity_set:
|
||||
raise ValidationError("SAP 'entity_set' is required.")
|
||||
|
||||
session = self.connector.get_session()
|
||||
records: List[Dict[str, Any]] = []
|
||||
next_link: Optional[str] = urljoin(base, entity_set)
|
||||
params = self._query_params(select, filter, expand, top, skip, batch_size)
|
||||
|
||||
if top is not None and top < 0:
|
||||
raise ValidationError("SAP 'top' must be >= 0 (got %r)" % top)
|
||||
if top == 0:
|
||||
return SAPODataEntity(
|
||||
records=[], entity_set=entity_set, count=0, service=base
|
||||
)
|
||||
original_host = (urlparse(base).hostname or "").lower()
|
||||
|
||||
while next_link:
|
||||
# Server-provided next links may point anywhere; never send the
|
||||
# session credentials (Basic/Bearer) to a different origin than
|
||||
# the service root. Legit SAP pagination stays on the same host.
|
||||
next_host = (urlparse(next_link).hostname or "").lower()
|
||||
if next_host != original_host:
|
||||
raise ProcessingError(
|
||||
f"SAP next link '{next_link}' points to a different host "
|
||||
f"than service root '{base}'"
|
||||
)
|
||||
resp = request_with_ssrf_guard(
|
||||
"GET",
|
||||
next_link,
|
||||
session=session,
|
||||
allow_private_ips=self.connector.allow_private_ips,
|
||||
headers={"Accept": "application/json"},
|
||||
params=params,
|
||||
timeout=self.connector.config.get("timeout", 30),
|
||||
)
|
||||
try:
|
||||
resp.raise_for_status()
|
||||
except requests.exceptions.RequestException as exc:
|
||||
self.logger.error(
|
||||
"Failed to fetch SAP entity set %s: %s", entity_set, exc
|
||||
)
|
||||
raise ProcessingError(
|
||||
f"Failed to fetch SAP entity set {entity_set}: {exc}"
|
||||
) from exc
|
||||
|
||||
payload = self._parse_page(resp)
|
||||
rows, next_link = payload["rows"], payload["next_link"]
|
||||
|
||||
for raw_row in rows:
|
||||
records.append(self._flatten_row(raw_row))
|
||||
|
||||
self.logger.debug(
|
||||
"Fetched %d rows from %s (next=%s)",
|
||||
len(rows),
|
||||
entity_set,
|
||||
bool(next_link),
|
||||
)
|
||||
if top is not None and len(records) >= top:
|
||||
break
|
||||
|
||||
params = None # query params already baked into the server next link
|
||||
# Refresh next_link against base in case it's a relative pointer.
|
||||
if next_link and not next_link.startswith("http"):
|
||||
next_link = urljoin(resp.url, next_link)
|
||||
|
||||
return SAPODataEntity(
|
||||
records=records,
|
||||
entity_set=entity_set,
|
||||
count=len(records),
|
||||
service=base,
|
||||
)
|
||||
|
||||
def _query_params(
|
||||
self,
|
||||
select: Optional[str],
|
||||
filter: Optional[str],
|
||||
expand: Optional[str],
|
||||
top_value: Optional[int],
|
||||
skip: Optional[int],
|
||||
batch_size: int,
|
||||
) -> Dict[str, str]:
|
||||
params: Dict[str, str] = {}
|
||||
if batch_size > 0:
|
||||
if top_value is not None:
|
||||
params["$top"] = str(min(batch_size, top_value))
|
||||
else:
|
||||
params["$top"] = str(batch_size)
|
||||
if select:
|
||||
params["$select"] = select
|
||||
if filter:
|
||||
params["$filter"] = filter
|
||||
if expand:
|
||||
params["$expand"] = expand
|
||||
if skip is not None:
|
||||
params["$skip"] = str(skip)
|
||||
return params
|
||||
|
||||
def _parse_page(self, resp: requests.Response) -> Dict[str, Any]:
|
||||
try:
|
||||
payload = resp.json()
|
||||
except ValueError as exc:
|
||||
raise ProcessingError(f"SAP OData response is not JSON: {exc}") from exc
|
||||
|
||||
rows: Any
|
||||
next_link: Optional[str] = None
|
||||
if isinstance(payload, list):
|
||||
rows = payload
|
||||
elif isinstance(payload, dict):
|
||||
d = payload.get("d")
|
||||
if isinstance(d, dict):
|
||||
# OData v2 atom: {"d": {"results": [...], "__next": ...}}
|
||||
rows = d.get("results")
|
||||
nxt = d.get("__next") or payload.get("@odata.nextLink")
|
||||
else:
|
||||
# OData v4 JSON: {"value": [...], "@odata.nextLink": ...}
|
||||
rows = payload.get("value", d)
|
||||
nxt = payload.get("@odata.nextLink")
|
||||
if isinstance(nxt, dict):
|
||||
nxt = nxt.get("__deferred", {}).get("uri")
|
||||
next_link = nxt
|
||||
else:
|
||||
rows = None
|
||||
|
||||
if not isinstance(rows, list):
|
||||
raise ProcessingError(
|
||||
"SAP OData payload has no list of rows (got %s)" % type(rows).__name__
|
||||
)
|
||||
return {"rows": rows, "next_link": next_link}
|
||||
|
||||
def _flatten_row(self, row: Any) -> Dict[str, Any]:
|
||||
if isinstance(row, dict):
|
||||
# v2 wraps items in "__metadata"; keep it but expose plain keys.
|
||||
return {k: v for k, v in row.items() if k != "__metadata"}
|
||||
return {"value": row}
|
||||
|
||||
def export_as_documents(self, data: SAPODataEntity) -> List[Dict[str, Any]]:
|
||||
"""Convert an ingested entity set to flat document dicts.
|
||||
|
||||
Normalizes every records held by ``data`` into a list of dicts with an
|
||||
injected ``id``/``name``/``source``, ready to hand to ``GraphBuilder``.
|
||||
"""
|
||||
return data.to_documents()
|
||||
|
||||
def close(self) -> None:
|
||||
"""Close the underlying connector's session."""
|
||||
self.connector.close()
|
||||
@@ -117,6 +117,8 @@ class PropertyGenerator:
|
||||
data_properties = self._infer_data_properties(entities, classes, **options)
|
||||
properties.extend(data_properties)
|
||||
|
||||
properties = self._coalesce_normalized_properties(properties)
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
@@ -193,6 +195,67 @@ class PropertyGenerator:
|
||||
|
||||
return properties
|
||||
|
||||
def _coalesce_normalized_properties(
|
||||
self, properties: List[Dict[str, Any]]
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Merge same-kind properties that normalize to the same name."""
|
||||
property_kinds = defaultdict(set)
|
||||
for prop in properties:
|
||||
property_kinds[prop["name"]].add(prop.get("type"))
|
||||
|
||||
collisions = {
|
||||
name: sorted(kind for kind in kinds if kind is not None)
|
||||
for name, kinds in property_kinds.items()
|
||||
if len({kind for kind in kinds if kind is not None}) > 1
|
||||
}
|
||||
if collisions:
|
||||
raise ValidationError(
|
||||
"Normalized property names cannot be shared by object and "
|
||||
"data properties.",
|
||||
validation_context={"property_kind_collisions": collisions},
|
||||
)
|
||||
|
||||
merged: Dict[tuple, Dict[str, Any]] = {}
|
||||
result = []
|
||||
for prop in properties:
|
||||
key = (prop.get("type"), prop["name"])
|
||||
existing = merged.get(key)
|
||||
if existing is None:
|
||||
merged[key] = prop
|
||||
result.append(prop)
|
||||
continue
|
||||
|
||||
existing["domain"] = self._merge_property_values(
|
||||
existing.get("domain", []), prop.get("domain", [])
|
||||
)
|
||||
if prop.get("type") == "object":
|
||||
existing["range"] = self._merge_property_values(
|
||||
existing.get("range", []), prop.get("range", [])
|
||||
)
|
||||
existing_metadata = existing.setdefault("metadata", {})
|
||||
existing_metadata["occurrence_count"] = (
|
||||
existing_metadata.get("occurrence_count", 0)
|
||||
+ prop.get("metadata", {}).get("occurrence_count", 0)
|
||||
)
|
||||
elif existing.get("range") != prop.get("range"):
|
||||
existing["range"] = self._get_more_general_type(
|
||||
existing["range"], prop["range"]
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _merge_property_values(current: Any, incoming: Any) -> List[Any]:
|
||||
"""Merge scalar-or-list property values while preserving input order."""
|
||||
values = list(current) if isinstance(current, list) else [current]
|
||||
incoming_values = (
|
||||
incoming if isinstance(incoming, list) else [incoming]
|
||||
)
|
||||
for value in incoming_values:
|
||||
if value not in values:
|
||||
values.append(value)
|
||||
return [value for value in values if value is not None]
|
||||
|
||||
def _infer_data_properties(
|
||||
self, entities: List[Dict[str, Any]], classes: List[Dict[str, Any]], **options
|
||||
) -> List[Dict[str, Any]]:
|
||||
|
||||
@@ -0,0 +1,663 @@
|
||||
"""Tests for the SAP OData ingestor.
|
||||
|
||||
The SAP connector never touches a live SAP system: every outbound request goes
|
||||
through ``semantica.ingest.ssrf.request_with_ssrf_guard`` (see
|
||||
``sap_ingestor.py``), so these tests patch that single entry point and drive
|
||||
the parser / connector / ingestor with canned responses.
|
||||
|
||||
Covered:
|
||||
- ``$metadata`` (CSDL XML) -> entity set discovery & property fields
|
||||
- OData v2 atom (``__next``) and OData v4 (``@odata.nextLink``) pagination
|
||||
- Basic and OAuth2 credential payloads propagated on the request
|
||||
- SSRF guard is used for data *and* token-exchange requests
|
||||
- ``export_as_documents`` yields the flat document shape GraphBuilder expects
|
||||
"""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from semantica.ingest import SAPIngestor, SAPODataConnector, SAPODataEntity
|
||||
from semantica.utils.exceptions import ProcessingError, ValidationError
|
||||
|
||||
METADATA_V2 = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx">
|
||||
<edmx:DataServices
|
||||
xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
|
||||
<Schema Namespace="SAPService" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
|
||||
<EntityContainer Name="Default" m:IsDefaultEntityContainer="true">
|
||||
<EntitySet Name="SalesOrderSet" EntityType="SAPService.SalesOrder"/>
|
||||
<EntitySet Name="CustomerSet" EntityType="SAPService.Customer"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="SalesOrder">
|
||||
<Key><PropertyRef Name="SalesOrderID"/></Key>
|
||||
<Property Name="SalesOrderID" Type="Edm.String" Nullable="false"/>
|
||||
<Property Name="CustomerID" Type="Edm.String"/>
|
||||
<Property Name="GrossAmount" Type="Edm.Decimal"/>
|
||||
</EntityType>
|
||||
<EntityType Name="Customer">
|
||||
<Key><PropertyRef Name="CustomerID"/></Key>
|
||||
<Property Name="CustomerID" Type="Edm.String" Nullable="false"/>
|
||||
<Property Name="Name" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
</edmx:DataServices>
|
||||
</edmx:Edmx>
|
||||
"""
|
||||
|
||||
METADATA_V4 = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
|
||||
<edmx:DataServices>
|
||||
<Schema Namespace="SalesNs" xmlns="http://docs.oasis-open.org/odata/ns/edm">
|
||||
<EntityContainer Name="Svc">
|
||||
<EntitySet Name="SalesOrders" EntityType="SalesNs.SalesOrder"/>
|
||||
<EntitySet Name="Customers" EntityType="SalesNs.Customer"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="SalesOrder">
|
||||
<Property Name="ID" Type="Edm.String"/>
|
||||
<Property Name="Total" Type="Edm.Decimal" Nullable="false"/>
|
||||
</EntityType>
|
||||
<EntityType Name="Customer">
|
||||
<Property Name="ID" Type="Edm.String"/>
|
||||
<Property Name="Name" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
</edmx:DataServices>
|
||||
</edmx:Edmx>
|
||||
"""
|
||||
|
||||
METADATA_MULTI_SCHEMA = """<?xml version="1.0" encoding="utf-8"?>
|
||||
<edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx">
|
||||
<edmx:DataServices>
|
||||
<Schema Namespace="OrdersNs" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
|
||||
<EntityContainer Name="OrdersSvc">
|
||||
<EntitySet Name="OrdersNs" EntityType="OrdersNs.Row"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="Row">
|
||||
<Property Name="A" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
<Schema Namespace="InvoicesNs" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
|
||||
<EntityContainer Name="InvoicesSvc">
|
||||
<EntitySet Name="InvoicesNs" EntityType="InvoicesNs.Row"/>
|
||||
</EntityContainer>
|
||||
<EntityType Name="Row">
|
||||
<Property Name="B" Type="Edm.String"/>
|
||||
</EntityType>
|
||||
</Schema>
|
||||
</edmx:DataServices>
|
||||
</edmx:Edmx>
|
||||
"""
|
||||
|
||||
|
||||
def _fake_response(status_code=200, json_payload=None, text="", headers=None):
|
||||
"""requests.Response-like stand-in returned by the mocked guard."""
|
||||
resp = MagicMock()
|
||||
resp.status_code = status_code
|
||||
resp.headers = headers or {}
|
||||
resp.text = text
|
||||
resp.url = "https://sap.example/odata/$metadata"
|
||||
if json_payload is not None:
|
||||
resp.json.return_value = json_payload
|
||||
else:
|
||||
resp.json.side_effect = ValueError("not json")
|
||||
if status_code >= 400:
|
||||
resp.raise_for_status.side_effect = requests.exceptions.HTTPError(
|
||||
f"{status_code} error"
|
||||
)
|
||||
return resp
|
||||
|
||||
|
||||
def _json_resp(payload, url):
|
||||
r = MagicMock()
|
||||
r.status_code = 200
|
||||
r.headers = {"content-type": "application/json"}
|
||||
r.text = ""
|
||||
r.url = url
|
||||
r.json.return_value = payload
|
||||
return r
|
||||
|
||||
|
||||
class TestDiscoverService:
|
||||
def test_metadata_parses_entity_sets_and_fields(self):
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {"content-type": "application/xml"}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError("xml not json")
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
sets = ing.discover_service()
|
||||
|
||||
assert guard.call_count == 1
|
||||
assert [s["name"] for s in sets] == ["SalesOrderSet", "CustomerSet"]
|
||||
sales = next(s for s in sets if s["name"] == "SalesOrderSet")
|
||||
assert {f["name"] for f in sales["fields"]} == {
|
||||
"SalesOrderID",
|
||||
"CustomerID",
|
||||
"GrossAmount",
|
||||
}
|
||||
assert sales["fields"][0]["type"] == "Edm.String"
|
||||
|
||||
def test_discover_flow_routes_via_ssrf(self):
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch("semantica.ingest.sap_ingestor.request_with_ssrf_guard") as guard:
|
||||
guard.return_value = resp
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
ing.discover_service()
|
||||
|
||||
method, url = guard.call_args[0]
|
||||
assert method == "GET"
|
||||
assert url.endswith("$metadata")
|
||||
|
||||
def test_metadata_url_keeps_last_segment_without_trailing_slash(self):
|
||||
"""base_url without a trailing '/' must not lose its last segment.
|
||||
|
||||
urljoin() replaces the final path segment when the base has no
|
||||
trailing slash, which would silently turn .../API_BUSINESS_PARTNER
|
||||
into .../$metadata against the wrong service root.
|
||||
"""
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/API_BUSINESS_PARTNER",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.discover_service()
|
||||
|
||||
method, url = guard.call_args[0]
|
||||
assert url == "https://sap.example/odata/API_BUSINESS_PARTNER/$metadata"
|
||||
|
||||
def test_service_root_appends_metadata(self):
|
||||
"""service is a service root (same meaning as ingest_entity_set):
|
||||
$metadata is appended automatically, not treated as the full path.
|
||||
"""
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/sap/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.discover_service("API_BUSINESS_PARTNER")
|
||||
|
||||
_, url = guard.call_args[0]
|
||||
assert url == "https://sap.example/odata/sap/API_BUSINESS_PARTNER/$metadata"
|
||||
|
||||
def test_absolute_service_root_appends_metadata(self):
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.headers = {}
|
||||
resp.text = METADATA_V2
|
||||
resp.json.side_effect = ValueError()
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/sap/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.discover_service("https://other.example/services/sap/")
|
||||
|
||||
_, url = guard.call_args[0]
|
||||
assert url == "https://other.example/services/sap/$metadata"
|
||||
|
||||
def test_v4_oasis_metadata_parses_with_nullable_defaults(self):
|
||||
resp = _fake_response(
|
||||
text=METADATA_V4, headers={"content-type": "application/xml"}
|
||||
)
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
sets = ing.discover_service()
|
||||
|
||||
by_name = {s["name"]: s for s in sets}
|
||||
assert "SalesOrders" in by_name and "Customers" in by_name
|
||||
sales_fields = {f["name"]: f for f in by_name["SalesOrders"]["fields"]}
|
||||
# Omitted Nullable -> nullable (CSDL default); explicit "false" honored.
|
||||
assert sales_fields["ID"]["nullable"] is True
|
||||
assert sales_fields["Total"]["nullable"] is False
|
||||
|
||||
def test_same_named_types_in_different_schemas_do_not_merge(self):
|
||||
resp = _fake_response(
|
||||
text=METADATA_MULTI_SCHEMA, headers={"content-type": "application/xml"}
|
||||
)
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
sets = ing.discover_service()
|
||||
|
||||
by_name = {s["name"]: s for s in sets}
|
||||
assert [f["name"] for f in by_name["OrdersNs"]["fields"]] == ["A"]
|
||||
assert [f["name"] for f in by_name["InvoicesNs"]["fields"]] == ["B"]
|
||||
|
||||
|
||||
class TestAuth:
|
||||
def test_basic_auth_payload_attached(self):
|
||||
conn = SAPODataConnector(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="erp_user",
|
||||
# Non-functional test placeholder; the low-entropy value keeps the
|
||||
# secret scanner from treating it as a hardcoded credential.
|
||||
password="test",
|
||||
)
|
||||
session = conn.get_session()
|
||||
assert session.headers["Authorization"].startswith("Basic ")
|
||||
# base64("erp_user:test")
|
||||
assert session.headers["Authorization"].endswith("ZXJwX3VzZXI6dGVzdA==")
|
||||
|
||||
def test_oauth_token_exchange_goes_through_ssrf(self):
|
||||
token_resp = MagicMock()
|
||||
token_resp.status_code = 200
|
||||
token_resp.headers = {}
|
||||
token_resp.json.return_value = {"access_token": "tok-123"}
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=token_resp,
|
||||
) as guard:
|
||||
conn = SAPODataConnector(
|
||||
base_url="https://sap.example/odata/",
|
||||
token_url="https://auth.example/oauth/token",
|
||||
client_id="cid",
|
||||
client_secret="secret",
|
||||
)
|
||||
session = conn.get_session()
|
||||
|
||||
assert guard.call_count == 1
|
||||
method_tok, url = guard.call_args[0]
|
||||
assert method_tok == "POST"
|
||||
assert url == "https://auth.example/oauth/token"
|
||||
assert session.headers["Authorization"] == "Bearer tok-123"
|
||||
body = guard.call_args.kwargs.get("data", {})
|
||||
assert body["grant_type"] == "client_credentials"
|
||||
|
||||
def test_requires_auth(self):
|
||||
with pytest.raises(ValidationError):
|
||||
SAPODataConnector(base_url="https://sap.example/odata/")
|
||||
|
||||
def test_requires_base_url(self):
|
||||
with pytest.raises(ValidationError):
|
||||
SAPODataConnector(username="u", password="p")
|
||||
|
||||
|
||||
class TestIngestPagination:
|
||||
V4_PAGE = {
|
||||
"value": [{"SalesOrderID": "SO-1"}, {"SalesOrderID": "SO-2"}],
|
||||
"@odata.nextLink": "https://sap.example/odata/SalesOrderSet?$skiptoken=abc",
|
||||
}
|
||||
V4_LAST = {"value": [{"SalesOrderID": "SO-3"}]}
|
||||
|
||||
def test_v4_nextlink_paginates(self):
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
page1 = _json_resp(self.V4_PAGE, url)
|
||||
page2 = _json_resp(self.V4_LAST, url)
|
||||
calls.append(url)
|
||||
return page1 if method == "GET" and len(calls) == 1 else page2
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
assert result.count == 3
|
||||
assert [r["SalesOrderID"] for r in result.records] == [
|
||||
"SO-1",
|
||||
"SO-2",
|
||||
"SO-3",
|
||||
]
|
||||
# Reached the second page's next link then stopped.
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_v4_nextlink_is_followed_past_first_page(self):
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
calls.append(url)
|
||||
if len(calls) == 1:
|
||||
return _json_resp(self.V4_PAGE, url)
|
||||
return _json_resp(self.V4_LAST, url)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_v2_atom_next_pagination(self):
|
||||
v2_first = {
|
||||
"d": {
|
||||
"results": [{"SalesOrderID": "A"}],
|
||||
"__next": {"__deferred": {"uri": "https://sap.example/odata/next2"}},
|
||||
},
|
||||
}
|
||||
v2_last = {"d": {"results": [{"SalesOrderID": "B"}]}}
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
calls.append(url)
|
||||
return (
|
||||
_json_resp(v2_first, url)
|
||||
if len(calls) == 1
|
||||
else _json_resp(v2_last, url)
|
||||
)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
assert [r["SalesOrderID"] for r in result.records] == ["A", "B"]
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_v2_next_as_plain_string_paginates(self):
|
||||
"""Canonical OData v2 JSON: ``__next`` is a plain string URL."""
|
||||
v2_first = {
|
||||
"d": {
|
||||
"results": [{"SalesOrderID": "A"}],
|
||||
"__next": "https://sap.example/odata/next2",
|
||||
},
|
||||
}
|
||||
v2_last = {"d": {"results": [{"SalesOrderID": "B"}]}}
|
||||
calls = []
|
||||
|
||||
def fake_guard(method, url, **kw):
|
||||
calls.append(url)
|
||||
if len(calls) == 1:
|
||||
return _json_resp(v2_first, url)
|
||||
return _json_resp(v2_last, url)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
side_effect=fake_guard,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
assert [r["SalesOrderID"] for r in result.records] == ["A", "B"]
|
||||
assert len(calls) == 2
|
||||
|
||||
def test_relative_service_resolves_against_base(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_json_resp({"value": []}, "https://x/"),
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/sap/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
ing.ingest_entity_set(
|
||||
service="API_SALES_ORDER_SRV", entity_set="SalesOrderSet"
|
||||
)
|
||||
|
||||
method, url = guard.call_args[0]
|
||||
assert url == "https://sap.example/odata/sap/API_SALES_ORDER_SRV/SalesOrderSet"
|
||||
|
||||
def test_expand_passthrough_for_line_items(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_json_resp(
|
||||
{"value": [{"SalesOrderID": "SO-9"}]}, "https://x/"
|
||||
),
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet", expand="to_Item")
|
||||
|
||||
params = guard.call_args_list[0].kwargs.get("params") or {}
|
||||
assert params.get("$expand") == "to_Item"
|
||||
|
||||
def test_select_filter_top_skip_objects_passthrough(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_json_resp({"value": []}, "https://x/"),
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/", username="u", password="p"
|
||||
)
|
||||
ing.ingest_entity_set(
|
||||
entity_set="SalesOrderSet",
|
||||
select="SalesOrderID,GrossAmount",
|
||||
filter="GrossAmount gt 100",
|
||||
top=5,
|
||||
skip=2,
|
||||
)
|
||||
params = guard.call_args_list[0].kwargs.get("params") or {}
|
||||
assert params["$select"] == "SalesOrderID,GrossAmount"
|
||||
assert params["$filter"] == "GrossAmount gt 100"
|
||||
assert params["$top"] == "5"
|
||||
assert params["$skip"] == "2"
|
||||
|
||||
|
||||
class TestErrorPaths:
|
||||
def test_http_error_on_entity_set_raises_processing_error(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=_fake_response(status_code=403),
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="403"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
def test_invalid_metadata_xml_raises_processing_error(self):
|
||||
resp = _fake_response(text="<not-xml")
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="not valid XML"):
|
||||
ing.discover_service()
|
||||
|
||||
def test_non_json_entity_response_raises_processing_error(self):
|
||||
resp = _fake_response(text="plain text")
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="not JSON"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
def test_cross_host_next_link_rejected(self):
|
||||
first = _json_resp(
|
||||
{
|
||||
"value": [{"SalesOrderID": "A"}],
|
||||
"@odata.nextLink": "https://evil.example/next",
|
||||
},
|
||||
"https://sap.example/odata/SalesOrderSet",
|
||||
)
|
||||
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=first,
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="different host"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
# The credential-bearing session is never sent to the evil host.
|
||||
assert guard.call_count == 1
|
||||
|
||||
def test_top_zero_returns_no_rows_and_makes_no_request(self):
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
) as guard:
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
result = ing.ingest_entity_set(entity_set="SalesOrderSet", top=0)
|
||||
|
||||
assert result.count == 0
|
||||
assert result.records == []
|
||||
guard.assert_not_called()
|
||||
|
||||
def test_negative_top_rejected(self):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ValidationError, match="must be >= 0"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet", top=-1)
|
||||
|
||||
def test_malformed_rows_container_rejected(self):
|
||||
resp = _json_resp(
|
||||
{"value": {"not": "a list"}},
|
||||
"https://sap.example/odata/SalesOrderSet",
|
||||
)
|
||||
with patch(
|
||||
"semantica.ingest.sap_ingestor.request_with_ssrf_guard",
|
||||
return_value=resp,
|
||||
):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
with pytest.raises(ProcessingError, match="no list of rows"):
|
||||
ing.ingest_entity_set(entity_set="SalesOrderSet")
|
||||
|
||||
|
||||
class TestExport:
|
||||
def test_export_as_documents_flattens_to_graph_builder_shape(self):
|
||||
ing = SAPIngestor(
|
||||
base_url="https://sap.example/odata/",
|
||||
username="u",
|
||||
password="p",
|
||||
)
|
||||
entity = SAPODataEntity(
|
||||
records=[{"SalesOrderID": "SO-1", "CustomerID": "C-1"}],
|
||||
entity_set="SalesOrderSet",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
docs = ing.export_as_documents(entity)
|
||||
assert docs[0]["SalesOrderID"] == "SO-1"
|
||||
assert docs[0]["id"] == "SO-1"
|
||||
assert docs[0]["source"] == "https://sap.example/odata/"
|
||||
|
||||
|
||||
class TestEntityDocument:
|
||||
def test_to_document_adds_source(self):
|
||||
ent = SAPODataEntity(
|
||||
records=[{"k": "v"}],
|
||||
entity_set="S",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
assert ent.to_documents()[0]["source"] == "https://sap.example/odata/"
|
||||
|
||||
def test_to_document_injects_graph_builder_identifier(self):
|
||||
ent = SAPODataEntity(
|
||||
records=[{"SalesOrderID": "SO-1", "CustomerID": "C-1"}],
|
||||
entity_set="SalesOrderSet",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
doc = ent.to_documents()[0]
|
||||
assert doc["id"] == "SO-1"
|
||||
assert doc["name"] == "SO-1"
|
||||
assert doc["SalesOrderID"] == "SO-1" # original field preserved
|
||||
|
||||
def test_to_document_falls_back_to_entity_set_index(self):
|
||||
ent = SAPODataEntity(
|
||||
records=[{"GrossAmount": "10.0"}],
|
||||
entity_set="SalesOrderSet",
|
||||
count=1,
|
||||
service="https://sap.example/odata/",
|
||||
)
|
||||
assert ent.to_documents()[0]["id"] == "SalesOrderSet:0"
|
||||
|
||||
def test_public_import_through_lazy_export(self):
|
||||
# __getattr__ should resolve the lazy export once the object is created
|
||||
assert callable(SAPODataConnector)
|
||||
@@ -0,0 +1,51 @@
|
||||
import pytest
|
||||
|
||||
from semantica.ontology.class_inferrer import ClassInferrer
|
||||
from semantica.ontology.property_generator import PropertyGenerator
|
||||
from semantica.utils.exceptions import ValidationError
|
||||
|
||||
|
||||
def test_same_kind_normalized_object_properties_are_merged():
|
||||
entities = [
|
||||
{"id": "p1", "type": "Person", "name": "Alice"},
|
||||
{"id": "o1", "type": "Organization", "name": "Acme"},
|
||||
]
|
||||
classes = ClassInferrer(min_occurrences=1).infer_classes(entities)
|
||||
relationships = [
|
||||
{
|
||||
"source_type": "Person",
|
||||
"target_type": "Organization",
|
||||
"type": "works_for",
|
||||
},
|
||||
{
|
||||
"source_type": "Person",
|
||||
"target_type": "Organization",
|
||||
"type": "worksFor",
|
||||
},
|
||||
]
|
||||
|
||||
properties = PropertyGenerator().infer_properties(
|
||||
entities, relationships, classes, min_occurrences=1
|
||||
)
|
||||
|
||||
works_for = [prop for prop in properties if prop["name"] == "worksFor"]
|
||||
assert len(works_for) == 1
|
||||
assert works_for[0]["domain"] == ["Person"]
|
||||
assert works_for[0]["range"] == ["Organization"]
|
||||
|
||||
|
||||
def test_normalized_name_cannot_be_both_object_and_data_property():
|
||||
entities = [
|
||||
{"id": "p1", "type": "Person", "value": "Alice"},
|
||||
{"id": "p2", "type": "Person", "value": "Bob"},
|
||||
]
|
||||
classes = ClassInferrer(min_occurrences=1).infer_classes(entities)
|
||||
relationships = [
|
||||
{"source_type": "Person", "target_type": "Person", "type": "value"},
|
||||
{"source_type": "Person", "target_type": "Person", "type": "value"},
|
||||
]
|
||||
|
||||
with pytest.raises(ValidationError, match="object and data"):
|
||||
PropertyGenerator().infer_properties(
|
||||
entities, relationships, classes, min_occurrences=1
|
||||
)
|
||||
Reference in New Issue
Block a user