* security(deps): bump fastapi floor to >=0.109.1 (PYSEC-2024-38)
The [explorer] extra declared fastapi>=0.100.0, which allows the
vulnerable 0.109.0 (PYSEC-2024-38, HTTP response splitting). Raise the
floor to 0.109.1, the patched release. One-line change, no functional
impact -- the 0.109.x API is stable and backward-compatible.
Fixes#869
* fix(deps): bump fastapi to >=0.109.2 and python-multipart to >=0.0.7 for PYSEC-2024-38
PYSEC-2024-38 (CVE-2024-24762 / GHSA-2jv5-9r88-3w3p) is a ReDoS in
python-multipart < 0.0.7: an attacker sends a crafted Content-Type header
that causes catastrophic backtracking in the multipart regex, stalling the
event loop and causing a DoS on any endpoint that parses form data.
The original PR bumped fastapi to >=0.109.1, but that version pins
starlette<0.36.0,>=0.35.0 and cannot install starlette 0.36.2+ (which
contains the fix via python-multipart>=0.0.7). FastAPI 0.109.2 is the
first version that pins starlette>=0.36.3 (verified against PyPI metadata).
Two changes are necessary:
1. fastapi>=0.109.1 -> fastapi>=0.109.2: ensures starlette>=0.36.3 is
installed as a transitive dependency, which in turn pulls the fixed
python-multipart>=0.0.7.
2. python-multipart>=0.0.6 -> python-multipart>=0.0.7: closes the direct
dependency path. python-multipart is listed explicitly in the explorer
extra, so without this floor a resolver could still install 0.0.6 and
leave the vulnerability present even with the fastapi bump.
The fix targets only the 'explorer' optional dependency group, which is
the only code surface where FastAPI and form-data parsing are used.
No functional API changes between 0.109.1 and 0.109.2; 239 Explorer tests
pass without modification.
* ci(security): gate pip-audit on explorer-extra dependency PRs, add changelog entry for PYSEC-2024-38
The Security workflow's pip-audit job ran weekly against a bare Python
env with none of Semantica's optional extras installed, and always
continue-on-error'd -- it would never have flagged the vulnerable
fastapi/python-multipart floors this PR fixes, or the first attempt at
the fix that left python-multipart>=0.0.6 in place. security-scan.yml's
Safety check has the same blind spot (only installs [llm-litellm]).
pip-audit now also runs on pull_request when pyproject.toml changes,
installs semantica[all] so it can actually see extras like [explorer],
and fails the build on findings for that trigger. Scheduled/dispatch
runs stay non-blocking pending a full pass over the [all] tree.
Also documents the fix (#871, closes#869) in CHANGELOG.md, including
the correction made during review after the original fastapi-only bump
turned out not to close the vulnerability.
* fix(deps): raise setuptools floor to >=83.0.0 (CVE-2026-59890), harden audit env
The new pull_request pip-audit gate (previous commit) caught this on its
first run: pip install -e ".[all]" resolved setuptools==79.0.1, vulnerable
to CVE-2026-59890 / GHSA-h35f-9h28-mq5c / PYSEC-2026-3447 (Unicode
normalization lets a MANIFEST.in exclude/prune pattern be bypassed on
macOS APFS/HFS+, leaking excluded files into a built sdist). Fixed in
setuptools 83.0.0.
[build-system] requires had the same too-permissive floor this whole PR
is about (setuptools>=61.0). Raised to >=83.0.0. Also upgrade pip/
setuptools explicitly in the Security workflow before running pip-audit,
since [build-system] requires only governs isolated build environments,
not the ambient one actions/setup-python provisions and pip-audit scans.
---------
Co-authored-by: Sameer Kadam <sskadam6305@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Upstream moved the v4 tag to 5595ccaf912efad79be6eef63a5619ff05969be3
(v4.37.6), which the repo's own verify-action-pins.sh now (correctly)
flags as a mismatch against the previously-pinned commit. Pre-existing
drift unrelated to #830/#836, but it was failing this PR's required
"verify" check, so fixing it here.
- Wire the Explorer frontend's node --test suites (test:graph-store,
test:graph-workspace, and the new test:plugin-registry regression
test) into CI. Previously only `npm run build` ran, so none of the
frontend tests -- including this fix's own regression coverage --
executed anywhere except a contributor's local machine.
- Broaden the diagnostics dedup's structureLayer comparison to also
cover disabledReason/curveCount/bridgeCurveCount/backboneCurveCount,
not just cacheKey/lastDrawAt/enabled, so a disabledReason-only
transition doesn't leave the dev diagnostics panel stale.
* security: SHA-pin all Actions, harden release pipeline, add pin verification
Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a
compromised third-party Action with a mutable tag stealing a long-lived
publishing token) and closes several related gaps found in an audit of the
actual repository state.
- Pin every third-party GitHub Action across all workflows to a full commit
SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI
check that confirms via the GitHub API that each pin still matches its
tag, on every workflow change, push to main, and weekly.
- Scope release.yml permissions to the job level (workflow defaults to
contents: read); add a concurrency group so simultaneous tag pushes can't
race the publish job.
- Add SLSA build provenance attestation (actions/attest-build-provenance)
for every released wheel.
- Fix a latent bug in security-scan.yml: the PR-comment step was missing
pull-requests: write and silently failing; add bounded artifact retention
for uploaded scan reports.
- Group github-actions Dependabot updates to cut review noise.
- Document the resulting posture in SECURITY.md for auditors/regulated
adopters, including what's enforced and what a fork needs to reconfigure
for itself (environment/branch protection, Trusted Publishing trust).
Also (via GitHub API, not in this diff): created a protected `pypi`
environment with a required reviewer restricted to v* tags, and enabled
branch protection on main (required review, required status checks, no
force-push/deletion).
* fix: harden verify-action-pins per PR #824 bot review
Addresses real findings from the automated review on #824:
- The script previously only matched uses: lines that already contained a
40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1)
would never be scanned at all and the check would pass silently. It now
matches every uses: line and hard-fails on any ref that isn't a full
commit SHA.
- A tag that fails to resolve via the GitHub API (rate limit, deleted tag)
previously only logged a warning and continued; that's now a hard
failure too, since an unverifiable pin is exactly the failure mode this
check exists to catch.
- verify-action-pins.yml only triggered on .github/workflows/** changes,
so an edit to the verifier script itself wouldn't run the check that
verifies it. Added the script path to both trigger filters.
The reviewer's claim that slash-containing tag comments (release/v1) break
the API lookup did not reproduce - tested directly against
pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves
multi-segment refs natively - so no change was needed there.
Verified with a synthetic test workflow containing a mutable-tag action,
a correctly-pinned SHA, and a deliberately mismatched SHA: the updated
script now catches the first and third cases and passes the second. Also
re-ran against the real workflow tree (40/40 pins still verify clean).
* fix: repair broken Safety scan and PR comment formatting
The "Comment PR with Security Results" step was producing garbled output
(literal \n characters instead of newlines, "undefined:" labels) because:
- Every line in the JS comment builder used \n (escaped backslash-n)
inside template literals, which JS renders as the literal two-character
string \n, not a newline.
- The Semgrep section read issue.rule_id, but Semgrep's JSON field is
check_id - hence "undefined: <path>" for every entry.
Rewrote the comment builder to construct each section as an array of
lines joined with a real '\n', with correct field names, and collapsed
long finding lists into a <details> block instead of a flat list.
Verified by extracting the exact script and running it under node against
synthetic fixtures matching each tool's real JSON schema (found/clean/
missing-report paths all render correctly).
While tracing the "undefined" and always-empty Safety section, found the
Safety step itself was silently broken:
- `safety check --json --output safety-report.json` is invalid in
Safety 3.x: --output now selects a console format (json/text/screen),
not a file path. The command errored on every run (swallowed by
`|| true`), so safety-report.json was never created and the PR comment
always fell back to a generic "scan completed" message. Switched to
`--save-json`, which is the correct flag for writing a JSON report to
disk, and confirmed against the real safety 3.8.1 CLI locally.
- Even with a report, the code read vuln.package - the real field is
package_name.
- The job never installed Semantica's own dependencies before scanning,
so `safety check` (which defaults to scanning the environment) was
auditing the scanner tools' own dependencies, not Semantica's. Added
`pip install -e ".[llm-litellm]"` so the project's actual dependency
tree - including the LiteLLM extra this whole hardening effort is
about - is what gets scanned.
Also updated the corresponding SECURITY.md bullet to describe what Safety
actually covers now.
* fix: remove unused pypdf2 dependency (CVE-2023-36464)
Now that the Safety scan step actually runs (see previous commit), it
correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a
real, pre-existing vulnerability that was invisible until the scan was
fixed.
PyPDF2 is not a patchable dependency here: the project is discontinued
(merged into `pypdf`), 3.0.1 is its final release, and there is no fixed
version to upgrade to. Grepping the repo for `import PyPDF2` / `from
PyPDF2` turns up nothing - it was never actually imported anywhere. Its
only presence outside pyproject.toml was in docstrings describing a
"PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented
in code; pdfplumber is the library actually used. Removed the dependency
and corrected the stale docstrings in parse/__init__.py, parse/methods.py,
parse/pdf_parser.py, and ingest/email_ingestor.py accordingly.
* fix: suppress Bandit B324 false positives on non-cryptographic MD5 use
Same pattern as the previous pypdf2 commit: fixing the Safety scan
surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10
pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for
security").
Checked each of the 10 call sites: every one uses hashlib.md5() to build
a short deterministic cache key, entity ID, or IRI suffix from already-
non-secret input (query text, entity text/type, class/property names) -
none are used for passwords, tokens, or integrity verification of
untrusted data. This is exactly the case Bandit's own message points at
("Consider usedforsecurity=False").
Did not use usedforsecurity=False itself: that keyword argument was
added to hashlib in Python 3.9, and pyproject.toml declares
`requires-python = ">=3.8"` - adding it unconditionally risks a TypeError
on 3.8. Used a targeted `# nosec B324` comment with a one-line
justification instead, which suppresses only this specific check and
carries no runtime behavior change on any supported Python version.
Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity
findings (was 10).
* docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening
Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml
hardening (job-scoped permissions, concurrency, SLSA provenance), the
pypi environment/branch protection GitHub-side config, the
security-scan.yml Safety/comment-formatting fixes, and the two
vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal,
Bandit B324 suppression).
* fix: close two remaining gaps missed by upstream bot-review fixes
verify-action-pins.sh:
- Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched
by the existing regex, so a SHA-pinned action written with quotes would
silently skip verification. Updated the main ERE to accept an optional
leading/trailing single or double quote around the owner/action@ref
value, and excluded quote chars from the inner character classes so the
ref is still extracted cleanly.
- The grep input glob only covered *.yml. GitHub also treats *.yaml as a
valid workflow extension. Added *.yaml to the glob and a 2>/dev/null
guard so the command doesn't fail when no *.yaml files exist.
security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a):
- Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all
five scanner-failure modes (file missing, empty file, malformed JSON,
valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still
silently produce VULNS=0 or VULNS=null and pass the merge-blocker check.
- Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety
crashed before writing a report (covers missing and empty-file cases).
- Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~
^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON,
missing key, and null cases). Both guards emit ::error:: annotations.
- Verified with a 7-case simulation: all 5 failure modes now exit 1;
genuine zero-vuln and real-vuln cases still behave correctly.
* fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI
The regex for matching uses: lines was embedded directly inline in a
[[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional-
expression parser interprets these as shell syntax rather than regex
literals, producing:
syntax error in conditional expression: unexpected token ')'
at line 27 on every CI run.
Fix: move the regex into a USES_PATTERN variable using safe single-quote
shell-string concatenation so the [[ =~ ]] parser receives an unquoted
variable reference ($USES_PATTERN) rather than a literal pattern containing
bash-special characters. The regex semantics are identical: optional
leading/trailing quote around owner/action@ref, quote chars excluded from
capture groups.
Verified in real bash 5.2.21 (Git for Windows):
No syntax error on the real 40-pin workflow tree (Checked 40)
Unquoted SHA pin: MATCH, correct repo+ref extracted
Double-quoted SHA pin: MATCH, correct repo+ref extracted
Single-quoted SHA pin: MATCH, correct repo+ref extracted
.yaml extension file: MATCH, correct repo+ref extracted
./local-action: NO MATCH (correct)
docker://: NO MATCH (correct)
* docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md
The checklist covered Trusted Publishing trust, protected environment,
branch protection, and Dependabot github-actions entry. Three non-forking
controls described elsewhere in SECURITY.md were omitted:
- GitHub secret scanning and push protection (repo settings, not copied
on fork)
- GitGuardian (GitHub App installation scoped to this specific repo,
requires separate install on any fork)
- CodeQL Default Setup vs Advanced Setup state (repo setting that affects
whether the upload-sarif step in codeql.yml does anything)
Added as items 5, 6, 7 matching the existing numbered bullet style.
* fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check)
verify-action-pins caught that github/codeql-action@v4 tag was re-pointed
upstream:
old: f205ea1c3313d32999d8d6a48b4f6530d4437b38
new: d1ba80a13dd99fba24a470575428917156a28b43
Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze,
upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment
# v4 unchanged — the tag itself hasn't changed, only what commit it points to.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
The CodeQL Analyze Python job failed on the #757 merge commit with
ECONNRESET while streaming the CodeQL bundle download in
codeql-action/init's "Setup CodeQL tools" step. This is unrelated to
the merged code — it's a known, currently-unaddressed gap in
codeql-action: the download error is retryable but the action doesn't
retry it internally (confirmed via codeql-action's issue tracker and
changelog).
Since a `uses:` step can't be wrapped by a shell-level retry action,
Initialize CodeQL now runs up to 3 times, cascading to the next
attempt only if the previous one failed, so the common case (success
on attempt 1) costs nothing extra.
Root cause of 6 consecutive CI failures:
MSDO 0.215.0's guardian.cmd wrapper breaks the build whenever checkov exits
with code 1. Checkov exits 1 on ANY violation, including MEDIUM/LOW findings
that are all 'below minimum severity'. This makes Active results = 0 and
'Found no breaking results', yet Guardian still raises BreakException because
it treats the tool's exit code as a first-class breaking signal. The
.checkov.yaml soft-fail setting was never read because the MSDO runner
bypasses repository config files.
Fix:
- Remove checkov from the MSDO tools list (stops the guardian.cmd crash)
- Add a dedicated 'checkov' job on ubuntu-latest using the official
bridgecrewio/checkov-action@v12, which runs a current checkov release,
runs on Linux, and correctly reads .checkov.yaml and respects soft_fail
- Set soft_fail: true in the action so low/medium findings appear in the
Security tab without ever blocking the build
- MSDO continues to run eslint, templateanalyzer (Bicep/ARM), and terrascan;
these tools all have well-behaved exit codes and produce no active results
after the security fixes applied earlier in this PR
.checkov.yaml:
- Replace soft-fail: true (was a failed workaround for MSDO) with
skip-check: [CKV_K8S_28, CKV_K8S_30] — correct suppression for the
Knative false-positives (Cloud Run enforces seccomp + AppArmor at
platform level without requiring K8s annotations)
Adds an explicit top-level `permissions` block to the GitHub Actions CI workflow.
This change sets the `GITHUB_TOKEN` permission scope to the minimum required level (`contents: read`), following the principle of least privilege and addressing the CodeQL alert `actions/missing-workflow-permissions`.
The workflow only requires read access to repository contents for checkout and CI tasks, so no additional permissions are needed.
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(explorer): resolve blank dashboard UI and ship frontend bundle in wheel
Fixes#631 — the Explorer server started successfully but the browser showed
a blank page because semantica/static/ was gitignored and never present after
a fresh install or clone.
Changes:
- ci.yml / release.yml: add Node 20 setup + npm ci && npm run build before
python -m build so every wheel contains a CI-built frontend bundle
- pyproject.toml: add package-data patterns (static/*, static/assets/*) so
setuptools includes the bundle in the wheel; add MANIFEST.in for sdist coverage
- app.py: replace silent empty-HTML fallback with a 200 page that clearly
explains the missing bundle and links to /docs; fix CORS allow_credentials
to default false, gated behind EXPLORER_CORS_CREDENTIALS env var to prevent
credentialed cross-origin requests on unauthenticated endpoints
- __init__.py: warn at startup when --host is non-loopback (unauthenticated
network exposure)
- explorer/README.md: full rewrite covering pip-install mode (primary path,
no Node required) and dev-server mode (contributors), CLI flags, env vars,
workspace table, troubleshooting for the blank-page symptom
- README.md: update Knowledge Explorer section with correct command and link
to the new setup guide
* fix(explorer): set build.target esnext to fix esbuild CI failure
esbuild >=0.28 (forced via npm overrides) conflicts with Vite 6 defaults on
Linux CI — it tries to lower destructuring syntax for the implicit browser
target list but errors out. Explicit target: 'esnext' tells esbuild to emit
native syntax unchanged, bypassing the transpilation error entirely. Safe for
a developer tool that runs in modern browsers.
* test(explorer): verify packaged frontend bundle
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* security: fix 9 Dependabot/CodeQL alerts — DOMPurify, vite, uuid, workflow permissions
- Add explicit permissions block to defender-for-devops.yml (CodeQL #25)
- Upgrade vite 5.4.x → 6.4.3; bundled esbuild 0.21.5 → 0.25.12 (Dependabot #2, #7)
- Force dompurify ^3.4.0 via npm overrides; resolves 6 DOMPurify XSS alerts (#4–#6, #8–#11)
- Force uuid ^13.0.1 via npm overrides; fixes buffer bounds check (Dependabot #12)
* fix(ci): exclude bandit from MSDO scan on windows-latest
bandit_runner.exe builds a per-file command line; on a large Python repo
the total command string exceeds the Windows CreateProcess limit and the
process fails to start (Win32 ERROR_FILENAME_EXCED_RANGE 206).
Exclude bandit via the tools param and retain checkov, eslint,
templateanalyzer, terrascan, and binskim.
* fix(ci): drop binskim (no binaries), enable Neptune audit logging
- Remove binskim from MSDO tools: repo has no compiled binaries so
BinSkim raises AnalyzeArgumentNoValuesException and breaks the run
- Add EnableCloudwatchLogsExports: [audit] to NeptuneCluster to fix
Checkov CKV_AWS_101 (the one error-level result breaking the build)
- Validate docs structure with docs_check.py (Python)
- Validate Mintlify build with mint validate (Node 20 LTS)
- Export static site with mint export, deploy to GitHub Pages
- Deploy job skipped on PRs (validate-only for branches)
- docs.yml: replace mkdocs build/deploy with python docs_check.py;
Mintlify deployment is handled by its own GitHub App
- ci.yml: remove dead paths-ignore refs to deleted mkdocs.yml and
requirements-docs.txt
fix(agent_memory): implement MemoryItem.to_dict() / from_dict() for safe JSON
persistence — timestamps serialised via isoformat(), embeddings dropped (not
JSON-safe, regenerated on demand); save() and load() now round-trip correctly
without TypeError or AttributeError (Bug #1)
fix(sparql): add asyncio.Semaphore(_SPARQL_MAX_CONCURRENT=4) around graph.query
so timed-out threads cannot exhaust the default ThreadPoolExecutor; add
`truncated: bool` field to SparqlResponse so callers know when the 5 000-row
cap was hit (Bug #2)
fix(export_import): trim _ALLOWED_IMPORT_EXTENSIONS to {.json, .csv} — the only
formats the handler actually parses; removes .graphml/.gexf/.ttl/.rdf that
passed the allowlist check but hit a hard 422 inside the handler (Bug #3)
fix(codeql): remove blanket rule-ID auto-dismiss job; replace with a commented
template for pinning specific alert numbers — prevents future real alerts of
the same rule being silently suppressed (Bug #4)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GITHUB_TOKEN cannot change Default Setup (requires admin rights — HTTP 403).
Removed the disable-default-setup job entirely.
New approach:
- analyze job: runs CodeQL with upload:false then uploads SARIF via
upload-sarif with continue-on-error:true so the workflow does not fail
if Default Setup is still active
- dismiss-fixed-alerts job: runs on push to main, fetches all open alerts
matching the 3 fixed rule IDs and dismisses them via PATCH API which
only requires security-events:write (no admin needed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous fix used || true in a single-step which masked API failures
and had no propagation delay — Default Setup remained active when the
SARIF upload ran, causing the same conflict error.
Changes:
- New job `disable-default-setup` runs first: calls the API, waits 30s,
then polls to confirm state=not-configured before exiting
- `analyze` job depends on `disable-default-setup` via `needs:` so CodeQL
only runs after the state change is confirmed propagated
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Advanced Setup and Default Setup cannot run simultaneously — SARIF upload
fails with "cannot be processed when the default setup is enabled".
Added a pre-analysis step that calls the GitHub code-scanning API to switch
Default Setup to not-configured before CodeQL runs, eliminating the conflict.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds explicit CodeQL analysis workflow triggered on push/PR to main and
weekly schedule. Without this, GitHub Default Setup only runs on a
schedule — alerts do not re-scan after a PR merge, leaving fixed
vulnerabilities still shown as open.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Register 'integration' pytest mark in pyproject.toml to eliminate
PytestUnknownMarkWarning across the test suite
- Add -m "not integration" and --ignore for external-service tests,
notebook tests, comprehensive real-world tests, and API-key-dependent
tests (Groq, Novita, Snowflake, Neptune, HF deepdive)
- Keeps fast unit tests: context, kg, semantic_extract, reasoning,
pipeline, export, deduplication, parse, normalize, utils, provenance
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace '*.md' with '**/*.md' in paths-ignore across ci.yml,
benchmark.yml, and security-scan.yml — '*.md' only matches root-level
markdown; '**/*.md' covers all subdirectories (cookbook/, docs/, etc.)
- Add cache: 'pip' to setup-python in ci.yml to avoid re-downloading
heavy packages (torch, spacy, faiss) on every run
- Update security-scan PR comment text to accurately reflect that it
skips doc/markdown-only PRs, not "every PR"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
