feat(ingest): add SAP OData ingestor (#1228) (#1234)

Adds `SAPODataConnector`, `SAPODataEntity`, and `SAPIngestor` for ingesting master and transactional data from SAP OData services, mainly things like Business Partners and Sales Orders.

Tested around S/4HANA Cloud, SuccessFactors, and on-prem NetWeaver Gateway style OData endpoints.

Main pieces included:

* OAuth2 client credentials and Basic auth support. Both go through the existing `ssrf.py` checks, including the OAuth token request.
* Small EDMX parser used by `discover_service()` so we don't need to pull in `pyodata`.
* Server-side pagination support for both OData versions:

  * v2: `__next`, including plain string and `__deferred` formats
  * v4: `@odata.nextLink`
* Keeps the service path in the base URL correctly whether the URL has a trailing slash or not. This is normalized in `SAPIngestor.__init__`.
* Adds an `ingest-sap` extra with just `requests`, so there is no SAP/proprietary SDK dependency.

This is meant to be a fairly small first version of the connector without adding a lot of SAP-specific dependencies.

Closes #1228
This commit is contained in:
Kevin Zhang
2026-08-28 14:54:02 +05:00
committed by GitHub
parent cce5ea177c
commit 100e95a098
7 changed files with 1380 additions and 0 deletions
+39
View File
@@ -382,6 +382,45 @@ For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-p
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
## Source 7 — SAP OData
`SAPIngestor` ingests an Entity Set from a SAP OData service (S/4HANA Cloud, SuccessFactors, or an on-prem NetWeaver Gateway over its REST surface). It speaks OData v2 and v4, follows server-driven pagination automatically, and flattens each record into a document dict via `export_as_documents()` — the same structured "transform to text, then store" pattern as the other sources.
```python
from semantica.ingest import SAPIngestor
ing = SAPIngestor(
base_url="https://my-sap.example.com/sap/opu/odata/sap/API_BUSINESS_PARTNER",
client_id="...", client_secret="...",
token_url="https://my-sap.example.com/oauth/token", # OAuth2 client-credentials (BTP/S/4HANA Cloud)
# On-prem NetWeaver often uses Basic auth instead — swap the block above for:
# username="erp_user", password="...",
)
# 1. Discover an unfamiliar service: entity sets + field types from $metadata
sets = ing.discover_service() # [{"name": "A_BusinessPartnerSet", "fields": [...]}, ...]
# 2. Pull a page-walked Entity Set (v2/v4 next links handled for you)
partners = ing.ingest_entity_set(
entity_set="A_BusinessPartnerSet",
select="BusinessPartner,BusinessPartnerFullName", # $select
top=1000, # cap on total rows
)
# 3. Flatten to document dicts, then build text for the graph
docs = ing.export_as_documents(partners)
partner_texts = [
f"Business Partner {d['BusinessPartner']}: {d['BusinessPartnerFullName']}"
for d in docs
]
```
- Use `expand="to_Item"` (e.g. on a sales-order header set) to pull nested line items in one request — handy for modeling order → line-item → material relationships.
- Every outbound request, including the OAuth2 token exchange, is routed through the SSRF guard, so a user-supplied SAP URL can never reach private/loopback/link-local address space.
- Install with `pip install 'semantica[ingest-sap]'`.
> **Security Note:** Never hardcode credentials (`client_secret`, `password`) in code; pass them via environment variables (e.g., `SAP_CLIENT_SECRET`, `SAP_PASSWORD`) or a secrets manager.
## Combining All Five Sources
Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly.
+1
View File
@@ -28,6 +28,7 @@ icon: "database"
| `DBIngestor` | SQL databases via SQLAlchemy: tables, views, and custom queries |
| `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
| `DatabricksIngestor` | Databricks Unity Catalog metadata, Delta table queries, and lineage |
| `SAPIngestor` | SAP OData services (S/4HANA Cloud, SuccessFactors, NetWeaver Gateway): entity-set discovery and ingestion with v2/v4 pagination |
| `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
| `ArrowIngestor` | Apache Arrow IPC and Feather file processing |
| `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |