mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
docs(changelog): add PR #602 public API ingestion entries to Unreleased
Documents all added features, hardening fixes, and follow-up patches from PR #602 (PublicAPIIngestor) including contributors Luffy2208 and Sameer6305.
This commit is contained in:
@@ -11,6 +11,52 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- **Public API Ingestion Support** (#602) by @Luffy2208
|
||||
- Added `PublicAPIIngestor` class built on top of `RESTIngestor` for credential-free REST endpoints
|
||||
- Added `PublicAPIExample` and `PublicAPIExamples` catalog with 6 pre-configured no-auth examples:
|
||||
- `jsonplaceholder_posts`, `jsonplaceholder_users`, `jsonplaceholder_todos` — fake REST resources for testing
|
||||
- `rest_countries_all` — country reference data
|
||||
- `data_gov_datasets` — Data.gov CKAN catalog search
|
||||
- `open_meteo_forecast` — weather forecast (Berlin sample)
|
||||
- Added `PublicAPIDetection` dataclass for endpoint-level public/no-auth detection
|
||||
- Endpoint-level public API detection via `detect_public_api()` (informational, never raises)
|
||||
- No-auth validation: rejects `Authorization`, `X-Api-Key`, and all common auth headers before sending the request
|
||||
- Auth credential detection in URL query strings (`api_key=`, `token=`, `access_token=`, etc.)
|
||||
- Polite rate limiting with per-request and per-ingestor `rate_limit_delay` controls
|
||||
- Response parsing for JSON, CSV, and XML with `response_format="auto"` content-type detection
|
||||
- HTML response guard — `text/html` responses are never misclassified as XML
|
||||
- Nested `record_path` dot-notation extraction (e.g. `"result.results"` for Data.gov envelope)
|
||||
- `_to_records` normalization with automatic envelope unwrapping for `items`, `data`, `results`, `records` keys
|
||||
- `batch_public_apis()` for multi-endpoint ingestion with optional `fail_fast`
|
||||
- `ingest_examples()` for bulk example ingestion
|
||||
- `sample_response()` fixtures on `PublicAPIExamples` for mocked unit tests without live network calls
|
||||
- `ingest_public_api()` convenience function and `ingest(..., source_type="public_api")` unified dispatch
|
||||
- `source_type="api"` alias supported in `ingest()`
|
||||
- Registry integration: `public_api` and `api` task namespaces with `endpoint`, `example`, `detect`, `batch`, `examples` methods
|
||||
- Lazy-import exports of `RESTIngestor`, `APIData`, `PublicAPIIngestor`, `PublicAPIExample`, `PublicAPIExamples`, `PublicAPIDetection` from `semantica.ingest`
|
||||
- Documentation: updated `docs/reference/ingest.md`, `docs/modules.md`, and `semantica/ingest/ingest_usage.md` with full usage examples
|
||||
- 18 mocked tests covering JSON/CSV/XML parsing, nested record extraction, auth rejection, detection, string boolean config, batch dispatch, and unified `ingest()` routing
|
||||
- 3 optional-import tests covering `defusedxml` fallback path and import isolation without web-scraping backends
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Public API XML parsing hardened against malicious payloads** (#602) by @Luffy2208
|
||||
- Replaced stdlib `xml.etree.ElementTree` with `defusedxml.ElementTree` (XXE/entity-expansion safe); falls back to a hardened `lxml` parser (`resolve_entities=False`, `no_network=True`, `load_dtd=False`, `huge_tree=False`) when `defusedxml` is not installed
|
||||
- Added regression test asserting XXE entity payloads raise `ProcessingError`
|
||||
|
||||
- **`validate_no_auth` config value not honoured when passed as a string** (#602) by @Luffy2208
|
||||
- `bool("false")` evaluated to `True`, making `validate_no_auth=False` impossible via config files or environment variables; replaced with explicit `_coerce_bool()` that maps `"false"`, `"0"`, `"no"`, `"off"` → `False` and rejects unrecognised strings with `ValidationError`
|
||||
|
||||
- **Auth credential detection extended to URL query strings** (#602) by @Sameer6305
|
||||
- `detect_public_api()` and `ingest_public_api()` now scan the endpoint URL itself for auth parameters (`api_key`, `token`, `access_token`, etc.) via `urllib.parse.parse_qs`, not only request headers and explicit `params=` dicts
|
||||
- Added regression tests for URL auth rejection (3 parametrized cases)
|
||||
|
||||
- **`ingest_examples` and `batch_public_apis` mutable options mutation** (#602)
|
||||
- Shared `**options` dict was passed by reference across loop iterations; mutable values such as `params` dicts were silently mutated after the first call, causing subsequent calls to receive a different (partially modified) options set; fixed by deep-copying options on each iteration
|
||||
|
||||
- **`rate_limit_delay` forwarded twice in `ingest_public_api` method dispatcher** (#602)
|
||||
- `rate_limit_delay` was consumed by the `PublicAPIIngestor` constructor via `config` but also leaked into `request_kwargs` forwarded to the ingestor method; added to the `config_only_key` strip list so it is consumed once at construction time only
|
||||
|
||||
- **XML File Ingestion Support** (#560) by @Luffy2208
|
||||
- Added `XMLIngestor` class with `lxml` backend for parsing local XML files
|
||||
- Nested element hierarchy and flat element list extraction
|
||||
|
||||
Reference in New Issue
Block a user