* security: SHA-pin all Actions, harden release pipeline, add pin verification
Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a
compromised third-party Action with a mutable tag stealing a long-lived
publishing token) and closes several related gaps found in an audit of the
actual repository state.
- Pin every third-party GitHub Action across all workflows to a full commit
SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI
check that confirms via the GitHub API that each pin still matches its
tag, on every workflow change, push to main, and weekly.
- Scope release.yml permissions to the job level (workflow defaults to
contents: read); add a concurrency group so simultaneous tag pushes can't
race the publish job.
- Add SLSA build provenance attestation (actions/attest-build-provenance)
for every released wheel.
- Fix a latent bug in security-scan.yml: the PR-comment step was missing
pull-requests: write and silently failing; add bounded artifact retention
for uploaded scan reports.
- Group github-actions Dependabot updates to cut review noise.
- Document the resulting posture in SECURITY.md for auditors/regulated
adopters, including what's enforced and what a fork needs to reconfigure
for itself (environment/branch protection, Trusted Publishing trust).
Also (via GitHub API, not in this diff): created a protected `pypi`
environment with a required reviewer restricted to v* tags, and enabled
branch protection on main (required review, required status checks, no
force-push/deletion).
* fix: harden verify-action-pins per PR #824 bot review
Addresses real findings from the automated review on #824:
- The script previously only matched uses: lines that already contained a
40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1)
would never be scanned at all and the check would pass silently. It now
matches every uses: line and hard-fails on any ref that isn't a full
commit SHA.
- A tag that fails to resolve via the GitHub API (rate limit, deleted tag)
previously only logged a warning and continued; that's now a hard
failure too, since an unverifiable pin is exactly the failure mode this
check exists to catch.
- verify-action-pins.yml only triggered on .github/workflows/** changes,
so an edit to the verifier script itself wouldn't run the check that
verifies it. Added the script path to both trigger filters.
The reviewer's claim that slash-containing tag comments (release/v1) break
the API lookup did not reproduce - tested directly against
pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves
multi-segment refs natively - so no change was needed there.
Verified with a synthetic test workflow containing a mutable-tag action,
a correctly-pinned SHA, and a deliberately mismatched SHA: the updated
script now catches the first and third cases and passes the second. Also
re-ran against the real workflow tree (40/40 pins still verify clean).
* fix: repair broken Safety scan and PR comment formatting
The "Comment PR with Security Results" step was producing garbled output
(literal \n characters instead of newlines, "undefined:" labels) because:
- Every line in the JS comment builder used \n (escaped backslash-n)
inside template literals, which JS renders as the literal two-character
string \n, not a newline.
- The Semgrep section read issue.rule_id, but Semgrep's JSON field is
check_id - hence "undefined: <path>" for every entry.
Rewrote the comment builder to construct each section as an array of
lines joined with a real '\n', with correct field names, and collapsed
long finding lists into a <details> block instead of a flat list.
Verified by extracting the exact script and running it under node against
synthetic fixtures matching each tool's real JSON schema (found/clean/
missing-report paths all render correctly).
While tracing the "undefined" and always-empty Safety section, found the
Safety step itself was silently broken:
- `safety check --json --output safety-report.json` is invalid in
Safety 3.x: --output now selects a console format (json/text/screen),
not a file path. The command errored on every run (swallowed by
`|| true`), so safety-report.json was never created and the PR comment
always fell back to a generic "scan completed" message. Switched to
`--save-json`, which is the correct flag for writing a JSON report to
disk, and confirmed against the real safety 3.8.1 CLI locally.
- Even with a report, the code read vuln.package - the real field is
package_name.
- The job never installed Semantica's own dependencies before scanning,
so `safety check` (which defaults to scanning the environment) was
auditing the scanner tools' own dependencies, not Semantica's. Added
`pip install -e ".[llm-litellm]"` so the project's actual dependency
tree - including the LiteLLM extra this whole hardening effort is
about - is what gets scanned.
Also updated the corresponding SECURITY.md bullet to describe what Safety
actually covers now.
* fix: remove unused pypdf2 dependency (CVE-2023-36464)
Now that the Safety scan step actually runs (see previous commit), it
correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a
real, pre-existing vulnerability that was invisible until the scan was
fixed.
PyPDF2 is not a patchable dependency here: the project is discontinued
(merged into `pypdf`), 3.0.1 is its final release, and there is no fixed
version to upgrade to. Grepping the repo for `import PyPDF2` / `from
PyPDF2` turns up nothing - it was never actually imported anywhere. Its
only presence outside pyproject.toml was in docstrings describing a
"PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented
in code; pdfplumber is the library actually used. Removed the dependency
and corrected the stale docstrings in parse/__init__.py, parse/methods.py,
parse/pdf_parser.py, and ingest/email_ingestor.py accordingly.
* fix: suppress Bandit B324 false positives on non-cryptographic MD5 use
Same pattern as the previous pypdf2 commit: fixing the Safety scan
surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10
pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for
security").
Checked each of the 10 call sites: every one uses hashlib.md5() to build
a short deterministic cache key, entity ID, or IRI suffix from already-
non-secret input (query text, entity text/type, class/property names) -
none are used for passwords, tokens, or integrity verification of
untrusted data. This is exactly the case Bandit's own message points at
("Consider usedforsecurity=False").
Did not use usedforsecurity=False itself: that keyword argument was
added to hashlib in Python 3.9, and pyproject.toml declares
`requires-python = ">=3.8"` - adding it unconditionally risks a TypeError
on 3.8. Used a targeted `# nosec B324` comment with a one-line
justification instead, which suppresses only this specific check and
carries no runtime behavior change on any supported Python version.
Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity
findings (was 10).
* docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening
Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml
hardening (job-scoped permissions, concurrency, SLSA provenance), the
pypi environment/branch protection GitHub-side config, the
security-scan.yml Safety/comment-formatting fixes, and the two
vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal,
Bandit B324 suppression).
* fix: close two remaining gaps missed by upstream bot-review fixes
verify-action-pins.sh:
- Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched
by the existing regex, so a SHA-pinned action written with quotes would
silently skip verification. Updated the main ERE to accept an optional
leading/trailing single or double quote around the owner/action@ref
value, and excluded quote chars from the inner character classes so the
ref is still extracted cleanly.
- The grep input glob only covered *.yml. GitHub also treats *.yaml as a
valid workflow extension. Added *.yaml to the glob and a 2>/dev/null
guard so the command doesn't fail when no *.yaml files exist.
security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a):
- Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all
five scanner-failure modes (file missing, empty file, malformed JSON,
valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still
silently produce VULNS=0 or VULNS=null and pass the merge-blocker check.
- Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety
crashed before writing a report (covers missing and empty-file cases).
- Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~
^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON,
missing key, and null cases). Both guards emit ::error:: annotations.
- Verified with a 7-case simulation: all 5 failure modes now exit 1;
genuine zero-vuln and real-vuln cases still behave correctly.
* fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI
The regex for matching uses: lines was embedded directly inline in a
[[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional-
expression parser interprets these as shell syntax rather than regex
literals, producing:
syntax error in conditional expression: unexpected token ')'
at line 27 on every CI run.
Fix: move the regex into a USES_PATTERN variable using safe single-quote
shell-string concatenation so the [[ =~ ]] parser receives an unquoted
variable reference ($USES_PATTERN) rather than a literal pattern containing
bash-special characters. The regex semantics are identical: optional
leading/trailing quote around owner/action@ref, quote chars excluded from
capture groups.
Verified in real bash 5.2.21 (Git for Windows):
No syntax error on the real 40-pin workflow tree (Checked 40)
Unquoted SHA pin: MATCH, correct repo+ref extracted
Double-quoted SHA pin: MATCH, correct repo+ref extracted
Single-quoted SHA pin: MATCH, correct repo+ref extracted
.yaml extension file: MATCH, correct repo+ref extracted
./local-action: NO MATCH (correct)
docker://: NO MATCH (correct)
* docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md
The checklist covered Trusted Publishing trust, protected environment,
branch protection, and Dependabot github-actions entry. Three non-forking
controls described elsewhere in SECURITY.md were omitted:
- GitHub secret scanning and push protection (repo settings, not copied
on fork)
- GitGuardian (GitHub App installation scoped to this specific repo,
requires separate install on any fork)
- CodeQL Default Setup vs Advanced Setup state (repo setting that affects
whether the upload-sarif step in codeql.yml does anything)
Added as items 5, 6, 7 matching the existing numbered bullet style.
* fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check)
verify-action-pins caught that github/codeql-action@v4 tag was re-pointed
upstream:
old: f205ea1c3313d32999d8d6a48b4f6530d4437b38
new: d1ba80a13dd99fba24a470575428917156a28b43
Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze,
upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment
# v4 unchanged — the tag itself hasn't changed, only what commit it points to.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
The CodeQL Analyze Python job failed on the #757 merge commit with
ECONNRESET while streaming the CodeQL bundle download in
codeql-action/init's "Setup CodeQL tools" step. This is unrelated to
the merged code — it's a known, currently-unaddressed gap in
codeql-action: the download error is retryable but the action doesn't
retry it internally (confirmed via codeql-action's issue tracker and
changelog).
Since a `uses:` step can't be wrapped by a shell-level retry action,
Initialize CodeQL now runs up to 3 times, cascading to the next
attempt only if the previous one failed, so the common case (success
on attempt 1) costs nothing extra.
Root cause of 6 consecutive CI failures:
MSDO 0.215.0's guardian.cmd wrapper breaks the build whenever checkov exits
with code 1. Checkov exits 1 on ANY violation, including MEDIUM/LOW findings
that are all 'below minimum severity'. This makes Active results = 0 and
'Found no breaking results', yet Guardian still raises BreakException because
it treats the tool's exit code as a first-class breaking signal. The
.checkov.yaml soft-fail setting was never read because the MSDO runner
bypasses repository config files.
Fix:
- Remove checkov from the MSDO tools list (stops the guardian.cmd crash)
- Add a dedicated 'checkov' job on ubuntu-latest using the official
bridgecrewio/checkov-action@v12, which runs a current checkov release,
runs on Linux, and correctly reads .checkov.yaml and respects soft_fail
- Set soft_fail: true in the action so low/medium findings appear in the
Security tab without ever blocking the build
- MSDO continues to run eslint, templateanalyzer (Bicep/ARM), and terrascan;
these tools all have well-behaved exit codes and produce no active results
after the security fixes applied earlier in this PR
.checkov.yaml:
- Replace soft-fail: true (was a failed workaround for MSDO) with
skip-check: [CKV_K8S_28, CKV_K8S_30] — correct suppression for the
Knative false-positives (Cloud Run enforces seccomp + AppArmor at
platform level without requiring K8s annotations)
Adds an explicit top-level `permissions` block to the GitHub Actions CI workflow.
This change sets the `GITHUB_TOKEN` permission scope to the minimum required level (`contents: read`), following the principle of least privilege and addressing the CodeQL alert `actions/missing-workflow-permissions`.
The workflow only requires read access to repository contents for checkout and CI tasks, so no additional permissions are needed.
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* fix(explorer): resolve blank dashboard UI and ship frontend bundle in wheel
Fixes#631 — the Explorer server started successfully but the browser showed
a blank page because semantica/static/ was gitignored and never present after
a fresh install or clone.
Changes:
- ci.yml / release.yml: add Node 20 setup + npm ci && npm run build before
python -m build so every wheel contains a CI-built frontend bundle
- pyproject.toml: add package-data patterns (static/*, static/assets/*) so
setuptools includes the bundle in the wheel; add MANIFEST.in for sdist coverage
- app.py: replace silent empty-HTML fallback with a 200 page that clearly
explains the missing bundle and links to /docs; fix CORS allow_credentials
to default false, gated behind EXPLORER_CORS_CREDENTIALS env var to prevent
credentialed cross-origin requests on unauthenticated endpoints
- __init__.py: warn at startup when --host is non-loopback (unauthenticated
network exposure)
- explorer/README.md: full rewrite covering pip-install mode (primary path,
no Node required) and dev-server mode (contributors), CLI flags, env vars,
workspace table, troubleshooting for the blank-page symptom
- README.md: update Knowledge Explorer section with correct command and link
to the new setup guide
* fix(explorer): set build.target esnext to fix esbuild CI failure
esbuild >=0.28 (forced via npm overrides) conflicts with Vite 6 defaults on
Linux CI — it tries to lower destructuring syntax for the implicit browser
target list but errors out. Explicit target: 'esnext' tells esbuild to emit
native syntax unchanged, bypassing the transpilation error entirely. Safe for
a developer tool that runs in modern browsers.
* test(explorer): verify packaged frontend bundle
---------
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
* security: fix 9 Dependabot/CodeQL alerts — DOMPurify, vite, uuid, workflow permissions
- Add explicit permissions block to defender-for-devops.yml (CodeQL #25)
- Upgrade vite 5.4.x → 6.4.3; bundled esbuild 0.21.5 → 0.25.12 (Dependabot #2, #7)
- Force dompurify ^3.4.0 via npm overrides; resolves 6 DOMPurify XSS alerts (#4–#6, #8–#11)
- Force uuid ^13.0.1 via npm overrides; fixes buffer bounds check (Dependabot #12)
* fix(ci): exclude bandit from MSDO scan on windows-latest
bandit_runner.exe builds a per-file command line; on a large Python repo
the total command string exceeds the Windows CreateProcess limit and the
process fails to start (Win32 ERROR_FILENAME_EXCED_RANGE 206).
Exclude bandit via the tools param and retain checkov, eslint,
templateanalyzer, terrascan, and binskim.
* fix(ci): drop binskim (no binaries), enable Neptune audit logging
- Remove binskim from MSDO tools: repo has no compiled binaries so
BinSkim raises AnalyzeArgumentNoValuesException and breaks the run
- Add EnableCloudwatchLogsExports: [audit] to NeptuneCluster to fix
Checkov CKV_AWS_101 (the one error-level result breaking the build)
- Validate docs structure with docs_check.py (Python)
- Validate Mintlify build with mint validate (Node 20 LTS)
- Export static site with mint export, deploy to GitHub Pages
- Deploy job skipped on PRs (validate-only for branches)
- docs.yml: replace mkdocs build/deploy with python docs_check.py;
Mintlify deployment is handled by its own GitHub App
- ci.yml: remove dead paths-ignore refs to deleted mkdocs.yml and
requirements-docs.txt
fix(agent_memory): implement MemoryItem.to_dict() / from_dict() for safe JSON
persistence — timestamps serialised via isoformat(), embeddings dropped (not
JSON-safe, regenerated on demand); save() and load() now round-trip correctly
without TypeError or AttributeError (Bug #1)
fix(sparql): add asyncio.Semaphore(_SPARQL_MAX_CONCURRENT=4) around graph.query
so timed-out threads cannot exhaust the default ThreadPoolExecutor; add
`truncated: bool` field to SparqlResponse so callers know when the 5 000-row
cap was hit (Bug #2)
fix(export_import): trim _ALLOWED_IMPORT_EXTENSIONS to {.json, .csv} — the only
formats the handler actually parses; removes .graphml/.gexf/.ttl/.rdf that
passed the allowlist check but hit a hard 422 inside the handler (Bug #3)
fix(codeql): remove blanket rule-ID auto-dismiss job; replace with a commented
template for pinning specific alert numbers — prevents future real alerts of
the same rule being silently suppressed (Bug #4)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GITHUB_TOKEN cannot change Default Setup (requires admin rights — HTTP 403).
Removed the disable-default-setup job entirely.
New approach:
- analyze job: runs CodeQL with upload:false then uploads SARIF via
upload-sarif with continue-on-error:true so the workflow does not fail
if Default Setup is still active
- dismiss-fixed-alerts job: runs on push to main, fetches all open alerts
matching the 3 fixed rule IDs and dismisses them via PATCH API which
only requires security-events:write (no admin needed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous fix used || true in a single-step which masked API failures
and had no propagation delay — Default Setup remained active when the
SARIF upload ran, causing the same conflict error.
Changes:
- New job `disable-default-setup` runs first: calls the API, waits 30s,
then polls to confirm state=not-configured before exiting
- `analyze` job depends on `disable-default-setup` via `needs:` so CodeQL
only runs after the state change is confirmed propagated
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Advanced Setup and Default Setup cannot run simultaneously — SARIF upload
fails with "cannot be processed when the default setup is enabled".
Added a pre-analysis step that calls the GitHub code-scanning API to switch
Default Setup to not-configured before CodeQL runs, eliminating the conflict.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds explicit CodeQL analysis workflow triggered on push/PR to main and
weekly schedule. Without this, GitHub Default Setup only runs on a
schedule — alerts do not re-scan after a PR merge, leaving fixed
vulnerabilities still shown as open.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Register 'integration' pytest mark in pyproject.toml to eliminate
PytestUnknownMarkWarning across the test suite
- Add -m "not integration" and --ignore for external-service tests,
notebook tests, comprehensive real-world tests, and API-key-dependent
tests (Groq, Novita, Snowflake, Neptune, HF deepdive)
- Keeps fast unit tests: context, kg, semantic_extract, reasoning,
pipeline, export, deduplication, parse, normalize, utils, provenance
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace '*.md' with '**/*.md' in paths-ignore across ci.yml,
benchmark.yml, and security-scan.yml — '*.md' only matches root-level
markdown; '**/*.md' covers all subdirectories (cookbook/, docs/, etc.)
- Add cache: 'pip' to setup-python in ci.yml to avoid re-downloading
heavy packages (torch, spacy, faiss) on every run
- Update security-scan PR comment text to accurately reflect that it
skips doc/markdown-only PRs, not "every PR"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
