Bump version, cut CHANGELOG's Unreleased section into 0.6.7, backfill
changelog entries for merged PRs missing from it, and refresh
version-dependent references in README/docs.
Adds `SAPODataConnector`, `SAPODataEntity`, and `SAPIngestor` for ingesting master and transactional data from SAP OData services, mainly things like Business Partners and Sales Orders.
Tested around S/4HANA Cloud, SuccessFactors, and on-prem NetWeaver Gateway style OData endpoints.
Main pieces included:
* OAuth2 client credentials and Basic auth support. Both go through the existing `ssrf.py` checks, including the OAuth token request.
* Small EDMX parser used by `discover_service()` so we don't need to pull in `pyodata`.
* Server-side pagination support for both OData versions:
* v2: `__next`, including plain string and `__deferred` formats
* v4: `@odata.nextLink`
* Keeps the service path in the base URL correctly whether the URL has a trailing slash or not. This is normalized in `SAPIngestor.__init__`.
* Adds an `ingest-sap` extra with just `requests`, so there is no SAP/proprietary SDK dependency.
This is meant to be a fairly small first version of the connector without adding a lot of SAP-specific dependencies.
Closes#1228
Bump version, cut CHANGELOG's Unreleased section into 0.6.6, backfill
changelog entries for merged PRs missing from it, and refresh
version-dependent references in README/docs.
Closes#1107, closes#1101.
Every RDF export mints terms in https://semantica.dev/ns#, and nothing declared
what those terms meant. The namespace returns 404 and no vocabulary shipped with
the package, so a consumer receiving an export could not tell semantica:text
from a typo of it: in the open world an undeclared IRI is unknown rather than
wrong, and every RDF tool treats the two alike. Closed-world checking is what
separates them, and it needs a document to check against.
semantica/ontology/vocabulary/semantica-ns.ttl declares the fourteen terms the
exporters actually emit, drawn from the emitting call sites rather than from
what a vocabulary ought to contain. It ships inside the package so it loads
without a network round trip, and is the same document intended to be served at
the namespace IRI once hosting and content negotiation are sorted.
tests/ontology/test_vocabulary.py ties the document to the code: every term the
serializers can write must be declared, so adding a term to an exporter without
declaring it fails the build rather than shipping an undeclared IRI.
The vocabulary alone would not have made those IRIs resolve, because the
fallback path minted them from Python's builtin hash(). That is randomised per
process, so the same entity received a different IRI on every run and exports
could not be diffed, deduplicated against an earlier load, or joined to a
provenance record written by an earlier process. Minting now uses SHA-256 and
writes a full IRI in the declared namespace rather than semantica:entity_N,
which inside angle brackets is an IRI in the scheme "semantica" rather than the
prefix expansion, and so never joined with anything written through the prefix.
The same applies to the default entity and relationship types in the Turtle
path.
134 export tests and 91 ontology tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(crewai): add first-class CrewAI integration (#962)
Add native CrewAI support so Crew agents can share a ContextGraph and
AgentContext via BaseTool subclasses and a BaseKnowledgeSource, matching
the existing agno integration pattern.
- SemanticaKGTool: 5 KG actions (extract_entities, extract_relations,
add_to_graph, query_graph, find_related) with sync run()/async arun()
- SemanticaDecisionTool: 5 decision-intelligence actions
(record_decision, find_precedents, trace_causal_chain,
analyze_impact, check_policy) over AgentContext
- SemanticaKnowledgeSource: serializes a ContextGraph into crew
knowledge storage; bridges legacy load_content() and current
validate_content()/aadd() contracts for crewai>=0.80.0
- All classes degrade gracefully when crewai is absent
- New pip extra crewai=... included in the all bundle
- 70 new tests (stub-based present-case + subprocess degradation path)
- Docs: integrations/crewai.md, docs.json nav, README matrix updates
* fix(crewai): harden tools against real Semantica dataclass shapes (#962)
Bugs found during live testing with crewai 1.15.16:
- SemanticaKGTool.add_to_graph crashed on real Entity/Relation dataclasses
('str' object has no attribute 'end_char'): string names were passed to
extract_relations(entities=...), which requires Entity objects, and the
tool read .name/.source/.target instead of Entity's .text/.label and
Relation's .subject/.object. Add shape-agnostic field helpers.
- SemanticaDecisionTool() created an AgentContext without a knowledge_graph,
so _decision_backend was never set and record_decision raised 'Decision
tracking is not enabled'. Wire in a ContextGraph.
- record_decision hard-failed when the agent omitted optional fields; fall
back to category='general', reasoning='agent decision',
outcome='recorded'.
Add tests covering real Entity/Relation dataclass shapes and the live
auto-created AgentContext path (now 77 crewai tests, 212 total).
* fix(crewai): make find_related traverse edges undirected (#962)
ContextGraph.get_neighbors only follows outgoing edges, so a node whose
only edge is incoming (A -> B) reported no related concepts. Rebuild a
bidirectional adjacency from find_edges() in SemanticaKGTool._find_related
so 'related' honors both directions.
* fix(crewai): harden tools for checkpoint serialization and correct action semantics (#962)
- Exclude live graph/context/extractor state from JSON serialization
(model_dump(mode="json")) so CrewAI checkpointing no longer raises
PydanticSerializationError; model_post_init self-heals defaults on restore
- query_graph now searches node content via graph.query() plus id/type
- trace_causal_chain returns an explicit error when causal tracing is
unavailable instead of substituting similarity precedents; call
trace_decision_causality(..., max_depth=...) with the correct kwarg name
- find_precedents propagates max_precedents/limit to the backend instead of
being silently capped at 10
- Serialize add_to_graph batches under a module lock to prevent concurrent
double-counting; skip nameless entities instead of creating repr()-junk nodes
- aadd() runs CPU-bound serialization in a thread executor
- Mirror crewai args_schema serialize/restore in the conftest stub and add
serialization regression tests (crewai: 92 tests)
* fix(crewai): correct check_policy coercion, guard causal tracing, and harden concurrency (#962)
- _eval_rule now coerces rule values type-aware: bool("false") was truthy, so
'enabled == false' reported a violation for enabled=false, and string datums
like "0.90" were compared lexicographically instead of numerically
- _trace_causal_chain no longer raises AttributeError (which escaped _run) when
the decision context lacks knowledge_graph; returns honest error JSON
- SemanticaKnowledgeSource storage failures log an actionable ERROR; without a
configured crew embedder agents previously retrieved nothing silently
- add_to_graph uses a per-graph re-entrant lock (WeakKeyDictionary) instead of a
process-global one: independent graphs no longer serialize each other and
re-entrant extractor callbacks cannot deadlock
- entity/relation confidence=None normalizes to 1.0 instead of failing the
whole extraction with float(None)
- add subprocess integration test against real crewai covering Crew-level
serialization round-trip and checkpoint restore (stub tests cannot see it)
- docs: embedder requirement for SemanticaKnowledgeSource; resume contract note
* fix(crewai): surface knowledge-source save failures at ERROR when storage is wired (#962)
Re-verification against real crewai showed the embedder-missing failure raises
ValueError even though storage IS wired, so the old except-ValueError branch
mislabeled it as 'storage not wired' and logged DEBUG — hiding the failure.
Distinguish by storage presence instead of exception type: storage is None ->
DEBUG keep-in-memory (legitimate standalone use); storage wired but save()
raises -> actionable ERROR. Add regression test mirroring real crewai's
ValueError-on-missing-embedder behavior.
* fix(crewai): expose run()/arun() entry points in degraded mode (#962)
The public crewai contract is run()/arun(); without crewai installed they were
missing (only the private _run existed), so the documented 'usable without
crewai' path raised AttributeError at the entry point. Define them in degraded
mode only, leaving crewai's BaseTool implementations untouched when present.
Extend the degradation subprocess test to exercise run() and arun().
* fix(crewai): standardize query shape, field-name rules, and restore-state flag
- _query_graph: id/type matches now return the same schema as content
matches (id/type/label/content/score) instead of a bare list
- _eval_rule: non-greedy field capture so hyphen/dot/space JSON keys
(e.g. "risk-score >= 0.9") are addressable in policy rules
- add had_live_state/reconstructed_state so checkpoint-restored tools
and knowledge sources signal that their live graph/context was lost
and an empty one reconstructed; knowledge source no longer hides the
loss by eagerly rebuilding its graph inside __init__ (pydantic calls
__init__ during model_validate)
* fix(crewai): address Qodo review — confidence errors, string trim, holistic availability
- record_decision: stop calling float() in _run, so malformed confidence
values surface as JSON errors (via _record_decision's handling) instead
of crashing the tool
- _coerce_value: return the stripped string for non-numeric literals so
whitespace-padded decision_data fields match policy rules
- centralize crewai availability in _availability.py so the exported
CREWAI_AVAILABLE flag is holistic across tools and knowledge source
(previously each module probed crewai independently and the package
flag came from decision_tool only)
* ci: regenerate requirements-ci.txt for the crewai extra
The crewai extra in pyproject.toml brings in crewai, crewai-tools and
transitive deps (chromadb, lancedb, ...). Recompile with
uv==0.12.1 per CONTRIBUTING.md so the CI staleness check passes.
* ci: keep crewai out of the locked CI dependency set
crewai (all versions) hard-requires chromadb~=1.1.0, which carries a
pre-authentication code-injection advisory (CVE-2026-45829 / GHSA-f4j7-r4q5-qw2c)
with NO fixed release — even the latest 1.5.9 is affected. Keeping crewai in
the 'all' extra failed pip-audit and the safety check on requirements-ci.txt.
- drop crewai from the 'all' aggregate (standalone semantica[crewai] extra is
unchanged and still installs crewai)
- stop listing crewai-tools in the extra: the integration only uses crewai core
(BaseTool, BaseKnowledgeSource) and crewai-tools pulled extra transitive deps
- regenerate requirements-ci.txt: OSV/pip-audit 0 vulnerabilities, safety 0
vulnerabilities, staleness check matches
* docs(crewai): document crewai extra scope and chromadb CVE-2026-45829
- CHANGELOG: extra is crewai>=0.80.0 only (no crewai-tools) and is not
part of the 'all' bundle, with the chromadb CVE-2026-45829 reason
- integrations/crewai/README.md: add a security warning that installing
the extra pulls chromadb~=1.1.0, which is affected by the unpatched
pre-auth code-injection CVE-2026-45829
---------
Fixes#994:
1. Prevent self-recursion in methods.py: generate_embeddings, embed_text,
calculate_similarity, and pool_embeddings all registered themselves as
custom methods, causing infinite self-calls when dispatch invoked them
without explicitly passing method parameter.
Fix: check custom_method is not the function itself before recursing.
2. Fix embed generate --output corrupt output: the CLI wrote
json.dumps(result, default=str) which produced plaintext repr of numpy
arrays (e.g. '[1.49e-01 4.85e-02 ...]') instead of proper Parquet.
Fix: detect .parquet extension (case-insensitive), convert numpy array
to pandas DataFrame with dim_* columns and id index, use to_parquet().
Non-parquet extensions fall back to JSON with clear ImportError message.
3. Add pyarrow>=14.0.0 to core dependencies (previously only in
ingest-parquet/ingest-arrow optional extras). The documented quick-start
flow of embed generate --output ... requires pyarrow out of the box.
(Note: pandas>=1.3.0 is already a core dependency; pyarrow is the
missing piece.)
Note: .github/workflows/* files are excluded from this PR as they require
a token with workflow scope. Upstream workflows are unchanged.
* ci: pin Python dependencies in requirements-ci.txt for reproducible CI
Adds a committed lockfile pinning all transitive dependencies at exact
versions (uv pip compile, Python 3.11, all extras — 1581 lines), the
Python equivalent of explorer/package-lock.json + npm ci.
- CI installs from requirements-ci.txt before building the wheel
- CI verifies the lockfile is byte-identical to a fresh compile (fails
on staleness after pyproject.toml changes)
- CONTRIBUTING documents the regeneration command
Closes#938
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: address Qodo review — security scans use pinned deps, exclude gpu extras
- security-scan.yml installs from requirements-ci.txt instead of
"./[llm-litellm]" so Safety scans the exact CI/release dependency tree
- security.yml runs pip-audit -r requirements-ci.txt for the same parity
- lockfile regenerated with --extra all (the cross-platform set) instead
of --all-extras, which pulled faiss-gpu/cupy from the Linux-only gpu
extra and co-installed faiss-cpu + faiss-gpu in CI
- uv pinned to 0.12.1 (the version that generated the lockfile) in CI and
CONTRIBUTING so regeneration is deterministic
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: make lockfile staleness check immune to upstream releases
The previous check re-resolved pyproject.toml without constraints, so any
upstream package release (e.g. boto3 1.43.69 -> 1.43.70) failed CI even
when nothing in the repo changed — exactly the time-dependent drift Qodo
flagged. The check now re-resolves with requirements-ci.txt as a
constraint and compares only version lines, so it detects intentional
pyproject.toml changes but ignores upstream releases. CONTRIBUTING
updated to match.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* ci: fix security workflows — install pip-audit; order tooling after pinned deps
Security workflow: the pip-audit install step was lost in the rebase
conflict merge — pip-audit was invoked but never installed (exit 127).
Security-scan workflow: installing safety first let the pinned
requirements-ci.txt overwrite its transitive deps (rich), breaking the
safety CLI at runtime (RuntimeError: Type not yet supported). Tooling is
now installed AFTER the pinned set.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): address review — hashes, build isolation, release builds, docs (4/4)
ZohaibHassan16's review flagged 4 supply-chain gaps; all addressed:
1. **Release builds now use the lockfile**: release.yml installs
requirements-ci.txt and runs `python -m build --no-isolation` so the
sdist/wheel is built against the exact tested dependency set.
2. **Build isolation pinned**: [build-system].requires is now
setuptools==84.0.0 + wheel==0.48.0 (exact pins, no ranges).
3. **Hashes**: requirements-ci.txt regenerated with --generate-hashes
(5,708 sha256 hashes, verified against PyPI). Staleness check updated
to strip the `\` line continuations hashes introduce.
4. **CONTRIBUTING.md documents the separate environment**: hashes,
never-install-into-dev note, build-system pins, --no-isolation release
builds.
Validated: stale-check diff clean, hash spot-check matches PyPI.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): apply --no-isolation to CI build + align benchmark to Python 3.11
Follow-up to ZohaibHassan16's second review round:
1. ci.yml was still running `python -m build` with build isolation
(unpinned setuptools/wheel from PyPI) — now `python -m build
--no-isolation` against the pinned deps, matching release.yml.
2. benchmark.yml was on Python 3.12 while the lockfile is compiled for
3.11 — aligned to 3.11 so every workflow runs the same environment.
Signed-off-by: Yunare Maia <yunare@gmail.com>
* fix(ci): install pinned wheel before --no-isolation build
python -m build --no-isolation failed with 'Missing dependencies:
wheel==0.48.0' because wheel is build-time only — uv's lockfile
excludes it, so installing requirements-ci.txt alone left the build
env without it. Both ci.yml and release.yml now install wheel==0.48.0
(the same pin [build-system] declares) before building. Validated
locally: wheel builds clean with --no-isolation.
Signed-off-by: Yunare Maia <yunare@gmail.com>
---------
Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* fix(ingest): harden RepoIngestor against GitPython URL and option injection
Bump GitPython to >=3.1.58, allowlist clone kwargs, and validate repo URLs
before clone_from to close env-var exfiltration and option-injection paths.
* fix(ingest): accept scp-like SSH remotes in RepoIngestor URL validation
* fix(ingest): resolve repo hostnames to block SSRF via private IPs
* fix(ingest): map malformed repo URL parse errors to ValidationError
* fix(ingest): bound and prune repo host resolve cache
Cap the repository host DNS cache, prune expired entries on access, and evict the oldest entries so long-running processes cannot accumulate unbounded host lookups from user-supplied repo URLs.
* fix(ingest): cap host resolve cache and tighten env-var token checks
Bound the repo host DNS cache with pruning and oldest-entry eviction, and narrow URL env-var blocking to actual $VAR/${VAR} tokens so literal dollar signs are not rejected.
* fix(ingest): preserve repo path compatibility and NAT64 support
* docs(changelog): document RepoIngestor GitPython hardening (#905, closes#868)
Records the clone-surface hardening (GitPython floor, clone-option
allowlist, URL/SSRF validation), the two fixes made during review
(NAT64 false-positive, local-path regression), and a known residual
gap: the SSRF host check doesn't classify RFC 6598 CGNAT space
(100.64.0.0/10) as blocked since ipaddress.is_private doesn't cover it.
---------
Co-authored-by: Pravit Ampapathini <pravitampapathini@wifi-10-43-175-99.wifi.berkeley.edu>
Co-authored-by: Pravit Ampapathini <pravitampapathini@Pravits-MacBook-Air-3.local>
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* security(deps): bump fastapi floor to >=0.109.1 (PYSEC-2024-38)
The [explorer] extra declared fastapi>=0.100.0, which allows the
vulnerable 0.109.0 (PYSEC-2024-38, HTTP response splitting). Raise the
floor to 0.109.1, the patched release. One-line change, no functional
impact -- the 0.109.x API is stable and backward-compatible.
Fixes#869
* fix(deps): bump fastapi to >=0.109.2 and python-multipart to >=0.0.7 for PYSEC-2024-38
PYSEC-2024-38 (CVE-2024-24762 / GHSA-2jv5-9r88-3w3p) is a ReDoS in
python-multipart < 0.0.7: an attacker sends a crafted Content-Type header
that causes catastrophic backtracking in the multipart regex, stalling the
event loop and causing a DoS on any endpoint that parses form data.
The original PR bumped fastapi to >=0.109.1, but that version pins
starlette<0.36.0,>=0.35.0 and cannot install starlette 0.36.2+ (which
contains the fix via python-multipart>=0.0.7). FastAPI 0.109.2 is the
first version that pins starlette>=0.36.3 (verified against PyPI metadata).
Two changes are necessary:
1. fastapi>=0.109.1 -> fastapi>=0.109.2: ensures starlette>=0.36.3 is
installed as a transitive dependency, which in turn pulls the fixed
python-multipart>=0.0.7.
2. python-multipart>=0.0.6 -> python-multipart>=0.0.7: closes the direct
dependency path. python-multipart is listed explicitly in the explorer
extra, so without this floor a resolver could still install 0.0.6 and
leave the vulnerability present even with the fastapi bump.
The fix targets only the 'explorer' optional dependency group, which is
the only code surface where FastAPI and form-data parsing are used.
No functional API changes between 0.109.1 and 0.109.2; 239 Explorer tests
pass without modification.
* ci(security): gate pip-audit on explorer-extra dependency PRs, add changelog entry for PYSEC-2024-38
The Security workflow's pip-audit job ran weekly against a bare Python
env with none of Semantica's optional extras installed, and always
continue-on-error'd -- it would never have flagged the vulnerable
fastapi/python-multipart floors this PR fixes, or the first attempt at
the fix that left python-multipart>=0.0.6 in place. security-scan.yml's
Safety check has the same blind spot (only installs [llm-litellm]).
pip-audit now also runs on pull_request when pyproject.toml changes,
installs semantica[all] so it can actually see extras like [explorer],
and fails the build on findings for that trigger. Scheduled/dispatch
runs stay non-blocking pending a full pass over the [all] tree.
Also documents the fix (#871, closes#869) in CHANGELOG.md, including
the correction made during review after the original fastapi-only bump
turned out not to close the vulnerability.
* fix(deps): raise setuptools floor to >=83.0.0 (CVE-2026-59890), harden audit env
The new pull_request pip-audit gate (previous commit) caught this on its
first run: pip install -e ".[all]" resolved setuptools==79.0.1, vulnerable
to CVE-2026-59890 / GHSA-h35f-9h28-mq5c / PYSEC-2026-3447 (Unicode
normalization lets a MANIFEST.in exclude/prune pattern be bypassed on
macOS APFS/HFS+, leaking excluded files into a built sdist). Fixed in
setuptools 83.0.0.
[build-system] requires had the same too-permissive floor this whole PR
is about (setuptools>=61.0). Raised to >=83.0.0. Also upgrade pip/
setuptools explicitly in the Security workflow before running pip-audit,
since [build-system] requires only governs isolated build environments,
not the ambient one actions/setup-python provisions and pip-audit scans.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* security: SHA-pin all Actions, harden release pipeline, add pin verification
Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a
compromised third-party Action with a mutable tag stealing a long-lived
publishing token) and closes several related gaps found in an audit of the
actual repository state.
- Pin every third-party GitHub Action across all workflows to a full commit
SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI
check that confirms via the GitHub API that each pin still matches its
tag, on every workflow change, push to main, and weekly.
- Scope release.yml permissions to the job level (workflow defaults to
contents: read); add a concurrency group so simultaneous tag pushes can't
race the publish job.
- Add SLSA build provenance attestation (actions/attest-build-provenance)
for every released wheel.
- Fix a latent bug in security-scan.yml: the PR-comment step was missing
pull-requests: write and silently failing; add bounded artifact retention
for uploaded scan reports.
- Group github-actions Dependabot updates to cut review noise.
- Document the resulting posture in SECURITY.md for auditors/regulated
adopters, including what's enforced and what a fork needs to reconfigure
for itself (environment/branch protection, Trusted Publishing trust).
Also (via GitHub API, not in this diff): created a protected `pypi`
environment with a required reviewer restricted to v* tags, and enabled
branch protection on main (required review, required status checks, no
force-push/deletion).
* fix: harden verify-action-pins per PR #824 bot review
Addresses real findings from the automated review on #824:
- The script previously only matched uses: lines that already contained a
40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1)
would never be scanned at all and the check would pass silently. It now
matches every uses: line and hard-fails on any ref that isn't a full
commit SHA.
- A tag that fails to resolve via the GitHub API (rate limit, deleted tag)
previously only logged a warning and continued; that's now a hard
failure too, since an unverifiable pin is exactly the failure mode this
check exists to catch.
- verify-action-pins.yml only triggered on .github/workflows/** changes,
so an edit to the verifier script itself wouldn't run the check that
verifies it. Added the script path to both trigger filters.
The reviewer's claim that slash-containing tag comments (release/v1) break
the API lookup did not reproduce - tested directly against
pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves
multi-segment refs natively - so no change was needed there.
Verified with a synthetic test workflow containing a mutable-tag action,
a correctly-pinned SHA, and a deliberately mismatched SHA: the updated
script now catches the first and third cases and passes the second. Also
re-ran against the real workflow tree (40/40 pins still verify clean).
* fix: repair broken Safety scan and PR comment formatting
The "Comment PR with Security Results" step was producing garbled output
(literal \n characters instead of newlines, "undefined:" labels) because:
- Every line in the JS comment builder used \n (escaped backslash-n)
inside template literals, which JS renders as the literal two-character
string \n, not a newline.
- The Semgrep section read issue.rule_id, but Semgrep's JSON field is
check_id - hence "undefined: <path>" for every entry.
Rewrote the comment builder to construct each section as an array of
lines joined with a real '\n', with correct field names, and collapsed
long finding lists into a <details> block instead of a flat list.
Verified by extracting the exact script and running it under node against
synthetic fixtures matching each tool's real JSON schema (found/clean/
missing-report paths all render correctly).
While tracing the "undefined" and always-empty Safety section, found the
Safety step itself was silently broken:
- `safety check --json --output safety-report.json` is invalid in
Safety 3.x: --output now selects a console format (json/text/screen),
not a file path. The command errored on every run (swallowed by
`|| true`), so safety-report.json was never created and the PR comment
always fell back to a generic "scan completed" message. Switched to
`--save-json`, which is the correct flag for writing a JSON report to
disk, and confirmed against the real safety 3.8.1 CLI locally.
- Even with a report, the code read vuln.package - the real field is
package_name.
- The job never installed Semantica's own dependencies before scanning,
so `safety check` (which defaults to scanning the environment) was
auditing the scanner tools' own dependencies, not Semantica's. Added
`pip install -e ".[llm-litellm]"` so the project's actual dependency
tree - including the LiteLLM extra this whole hardening effort is
about - is what gets scanned.
Also updated the corresponding SECURITY.md bullet to describe what Safety
actually covers now.
* fix: remove unused pypdf2 dependency (CVE-2023-36464)
Now that the Safety scan step actually runs (see previous commit), it
correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a
real, pre-existing vulnerability that was invisible until the scan was
fixed.
PyPDF2 is not a patchable dependency here: the project is discontinued
(merged into `pypdf`), 3.0.1 is its final release, and there is no fixed
version to upgrade to. Grepping the repo for `import PyPDF2` / `from
PyPDF2` turns up nothing - it was never actually imported anywhere. Its
only presence outside pyproject.toml was in docstrings describing a
"PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented
in code; pdfplumber is the library actually used. Removed the dependency
and corrected the stale docstrings in parse/__init__.py, parse/methods.py,
parse/pdf_parser.py, and ingest/email_ingestor.py accordingly.
* fix: suppress Bandit B324 false positives on non-cryptographic MD5 use
Same pattern as the previous pypdf2 commit: fixing the Safety scan
surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10
pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for
security").
Checked each of the 10 call sites: every one uses hashlib.md5() to build
a short deterministic cache key, entity ID, or IRI suffix from already-
non-secret input (query text, entity text/type, class/property names) -
none are used for passwords, tokens, or integrity verification of
untrusted data. This is exactly the case Bandit's own message points at
("Consider usedforsecurity=False").
Did not use usedforsecurity=False itself: that keyword argument was
added to hashlib in Python 3.9, and pyproject.toml declares
`requires-python = ">=3.8"` - adding it unconditionally risks a TypeError
on 3.8. Used a targeted `# nosec B324` comment with a one-line
justification instead, which suppresses only this specific check and
carries no runtime behavior change on any supported Python version.
Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity
findings (was 10).
* docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening
Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml
hardening (job-scoped permissions, concurrency, SLSA provenance), the
pypi environment/branch protection GitHub-side config, the
security-scan.yml Safety/comment-formatting fixes, and the two
vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal,
Bandit B324 suppression).
* fix: close two remaining gaps missed by upstream bot-review fixes
verify-action-pins.sh:
- Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched
by the existing regex, so a SHA-pinned action written with quotes would
silently skip verification. Updated the main ERE to accept an optional
leading/trailing single or double quote around the owner/action@ref
value, and excluded quote chars from the inner character classes so the
ref is still extracted cleanly.
- The grep input glob only covered *.yml. GitHub also treats *.yaml as a
valid workflow extension. Added *.yaml to the glob and a 2>/dev/null
guard so the command doesn't fail when no *.yaml files exist.
security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a):
- Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all
five scanner-failure modes (file missing, empty file, malformed JSON,
valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still
silently produce VULNS=0 or VULNS=null and pass the merge-blocker check.
- Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety
crashed before writing a report (covers missing and empty-file cases).
- Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~
^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON,
missing key, and null cases). Both guards emit ::error:: annotations.
- Verified with a 7-case simulation: all 5 failure modes now exit 1;
genuine zero-vuln and real-vuln cases still behave correctly.
* fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI
The regex for matching uses: lines was embedded directly inline in a
[[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional-
expression parser interprets these as shell syntax rather than regex
literals, producing:
syntax error in conditional expression: unexpected token ')'
at line 27 on every CI run.
Fix: move the regex into a USES_PATTERN variable using safe single-quote
shell-string concatenation so the [[ =~ ]] parser receives an unquoted
variable reference ($USES_PATTERN) rather than a literal pattern containing
bash-special characters. The regex semantics are identical: optional
leading/trailing quote around owner/action@ref, quote chars excluded from
capture groups.
Verified in real bash 5.2.21 (Git for Windows):
No syntax error on the real 40-pin workflow tree (Checked 40)
Unquoted SHA pin: MATCH, correct repo+ref extracted
Double-quoted SHA pin: MATCH, correct repo+ref extracted
Single-quoted SHA pin: MATCH, correct repo+ref extracted
.yaml extension file: MATCH, correct repo+ref extracted
./local-action: NO MATCH (correct)
docker://: NO MATCH (correct)
* docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md
The checklist covered Trusted Publishing trust, protected environment,
branch protection, and Dependabot github-actions entry. Three non-forking
controls described elsewhere in SECURITY.md were omitted:
- GitHub secret scanning and push protection (repo settings, not copied
on fork)
- GitGuardian (GitHub App installation scoped to this specific repo,
requires separate install on any fork)
- CodeQL Default Setup vs Advanced Setup state (repo setting that affects
whether the upload-sarif step in codeql.yml does anything)
Added as items 5, 6, 7 matching the existing numbered bullet style.
* fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check)
verify-action-pins caught that github/codeql-action@v4 tag was re-pointed
upstream:
old: f205ea1c3313d32999d8d6a48b4f6530d4437b38
new: d1ba80a13dd99fba24a470575428917156a28b43
Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze,
upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment
# v4 unchanged — the tag itself hasn't changed, only what commit it points to.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
Adds an explicit httpx<0.28.0 constraint to [project.dependencies] so it
applies globally across all environments, not just dev. Without this,
different environments could resolve an incompatible transitive httpx
version and hit the same Starlette TestClient breakage independently.
Verified via git stash comparison: the unmodified baseline fails to even
collect the test suite (TestClient TypeError during collection), so this
pin doesn't just fix tests, it's what allows the full suite to run at all.
Closes#788
Promotes the Unreleased changelog section (Databricks connector, SQLite
vector store, SPARQL CONSTRUCT templates, JenaStore named-graph support)
to 0.6.0 and syncs version references across pyproject.toml, __init__.py,
and docs.
Add graph-apache-age extra (psycopg2-binary) which was previously
undeclared despite age_store.py depending on it, wire it into
graph-all, and document install commands for FalkorDB/AGE/Neptune
alongside Neo4j. Note that RDF triple stores need no extra since they
talk SPARQL over HTTP via the core `requests` dependency. Also align
README's "Triplet Stores" table label to "Triple Stores (RDF)" to
match the standard term used elsewhere in the docs, while keeping the
TripletStore interface name in backticks.
Adds DatabricksIngestor to semantica/ingest/, mirroring SnowflakeIngestor's
structure and public API shape: table/query ingestion via
databricks-sql-connector, Unity Catalog metadata and lineage via
databricks-sdk, and export-as-documents for KG construction.
Closes#747
* Add shacl extra to pyproject.toml (fixes#736)
* docs(changelog): add entry for shacl extra fix (#736)
by @Sameer6305
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
* feat: implement Apache Arrow and Feather file ingestion support (#235)
* fix(arrow): eliminate double full-scan and clean up reader wrapper
- Replace _read_batches with _read_batches_with_info which collects
batch metadata (total_rows, record_batches) during the same pass as
the data read, so ingest_file no longer calls _file_metadata before
_read_batches. For a limit=1 read on a large file this previously
scanned every batch twice; now it stops after the first batch.
- _file_metadata is now only invoked for include_data=False (where a
full scan is unavoidable to report accurate row counts).
- Remove the dead num_record_batches property from _ArrowReaderWrapper;
it was never called by production code and its is_table branch
materialised all batches just to count them.
- Fix _open_file exception chain: raise ... from file_err instead of
from feather_err so the most diagnostic IPC error appears in the
Python traceback chain, not the least informative fallback error.
* docs(changelog): add [Unreleased] entries for Arrow ingestion (#705)
---------
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
KaifAhmad1