From b59211ea7f8b8cd6172f04afc222d4b4f3f03e24 Mon Sep 17 00:00:00 2001 From: Mohd Kaif <98801504+KaifAhmad1@users.noreply.github.com> Date: Mon, 3 Aug 2026 19:17:10 +0530 Subject: [PATCH] security: SHA-pin all Actions, harden release pipeline, add pin verification (#824) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * security: SHA-pin all Actions, harden release pipeline, add pin verification Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a compromised third-party Action with a mutable tag stealing a long-lived publishing token) and closes several related gaps found in an audit of the actual repository state. - Pin every third-party GitHub Action across all workflows to a full commit SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI check that confirms via the GitHub API that each pin still matches its tag, on every workflow change, push to main, and weekly. - Scope release.yml permissions to the job level (workflow defaults to contents: read); add a concurrency group so simultaneous tag pushes can't race the publish job. - Add SLSA build provenance attestation (actions/attest-build-provenance) for every released wheel. - Fix a latent bug in security-scan.yml: the PR-comment step was missing pull-requests: write and silently failing; add bounded artifact retention for uploaded scan reports. - Group github-actions Dependabot updates to cut review noise. - Document the resulting posture in SECURITY.md for auditors/regulated adopters, including what's enforced and what a fork needs to reconfigure for itself (environment/branch protection, Trusted Publishing trust). Also (via GitHub API, not in this diff): created a protected `pypi` environment with a required reviewer restricted to v* tags, and enabled branch protection on main (required review, required status checks, no force-push/deletion). * fix: harden verify-action-pins per PR #824 bot review Addresses real findings from the automated review on #824: - The script previously only matched uses: lines that already contained a 40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1) would never be scanned at all and the check would pass silently. It now matches every uses: line and hard-fails on any ref that isn't a full commit SHA. - A tag that fails to resolve via the GitHub API (rate limit, deleted tag) previously only logged a warning and continued; that's now a hard failure too, since an unverifiable pin is exactly the failure mode this check exists to catch. - verify-action-pins.yml only triggered on .github/workflows/** changes, so an edit to the verifier script itself wouldn't run the check that verifies it. Added the script path to both trigger filters. The reviewer's claim that slash-containing tag comments (release/v1) break the API lookup did not reproduce - tested directly against pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves multi-segment refs natively - so no change was needed there. Verified with a synthetic test workflow containing a mutable-tag action, a correctly-pinned SHA, and a deliberately mismatched SHA: the updated script now catches the first and third cases and passes the second. Also re-ran against the real workflow tree (40/40 pins still verify clean). * fix: repair broken Safety scan and PR comment formatting The "Comment PR with Security Results" step was producing garbled output (literal \n characters instead of newlines, "undefined:" labels) because: - Every line in the JS comment builder used \n (escaped backslash-n) inside template literals, which JS renders as the literal two-character string \n, not a newline. - The Semgrep section read issue.rule_id, but Semgrep's JSON field is check_id - hence "undefined: " for every entry. Rewrote the comment builder to construct each section as an array of lines joined with a real '\n', with correct field names, and collapsed long finding lists into a
block instead of a flat list. Verified by extracting the exact script and running it under node against synthetic fixtures matching each tool's real JSON schema (found/clean/ missing-report paths all render correctly). While tracing the "undefined" and always-empty Safety section, found the Safety step itself was silently broken: - `safety check --json --output safety-report.json` is invalid in Safety 3.x: --output now selects a console format (json/text/screen), not a file path. The command errored on every run (swallowed by `|| true`), so safety-report.json was never created and the PR comment always fell back to a generic "scan completed" message. Switched to `--save-json`, which is the correct flag for writing a JSON report to disk, and confirmed against the real safety 3.8.1 CLI locally. - Even with a report, the code read vuln.package - the real field is package_name. - The job never installed Semantica's own dependencies before scanning, so `safety check` (which defaults to scanning the environment) was auditing the scanner tools' own dependencies, not Semantica's. Added `pip install -e ".[llm-litellm]"` so the project's actual dependency tree - including the LiteLLM extra this whole hardening effort is about - is what gets scanned. Also updated the corresponding SECURITY.md bullet to describe what Safety actually covers now. * fix: remove unused pypdf2 dependency (CVE-2023-36464) Now that the Safety scan step actually runs (see previous commit), it correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a real, pre-existing vulnerability that was invisible until the scan was fixed. PyPDF2 is not a patchable dependency here: the project is discontinued (merged into `pypdf`), 3.0.1 is its final release, and there is no fixed version to upgrade to. Grepping the repo for `import PyPDF2` / `from PyPDF2` turns up nothing - it was never actually imported anywhere. Its only presence outside pyproject.toml was in docstrings describing a "PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented in code; pdfplumber is the library actually used. Removed the dependency and corrected the stale docstrings in parse/__init__.py, parse/methods.py, parse/pdf_parser.py, and ingest/email_ingestor.py accordingly. * fix: suppress Bandit B324 false positives on non-cryptographic MD5 use Same pattern as the previous pypdf2 commit: fixing the Safety scan surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10 pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for security"). Checked each of the 10 call sites: every one uses hashlib.md5() to build a short deterministic cache key, entity ID, or IRI suffix from already- non-secret input (query text, entity text/type, class/property names) - none are used for passwords, tokens, or integrity verification of untrusted data. This is exactly the case Bandit's own message points at ("Consider usedforsecurity=False"). Did not use usedforsecurity=False itself: that keyword argument was added to hashlib in Python 3.9, and pyproject.toml declares `requires-python = ">=3.8"` - adding it unconditionally risks a TypeError on 3.8. Used a targeted `# nosec B324` comment with a one-line justification instead, which suppresses only this specific check and carries no runtime behavior change on any supported Python version. Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity findings (was 10). * docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml hardening (job-scoped permissions, concurrency, SLSA provenance), the pypi environment/branch protection GitHub-side config, the security-scan.yml Safety/comment-formatting fixes, and the two vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal, Bandit B324 suppression). * fix: close two remaining gaps missed by upstream bot-review fixes verify-action-pins.sh: - Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched by the existing regex, so a SHA-pinned action written with quotes would silently skip verification. Updated the main ERE to accept an optional leading/trailing single or double quote around the owner/action@ref value, and excluded quote chars from the inner character classes so the ref is still extracted cleanly. - The grep input glob only covered *.yml. GitHub also treats *.yaml as a valid workflow extension. Added *.yaml to the glob and a 2>/dev/null guard so the command doesn't fail when no *.yaml files exist. security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a): - Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all five scanner-failure modes (file missing, empty file, malformed JSON, valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still silently produce VULNS=0 or VULNS=null and pass the merge-blocker check. - Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety crashed before writing a report (covers missing and empty-file cases). - Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~ ^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON, missing key, and null cases). Both guards emit ::error:: annotations. - Verified with a 7-case simulation: all 5 failure modes now exit 1; genuine zero-vuln and real-vuln cases still behave correctly. * fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI The regex for matching uses: lines was embedded directly inline in a [[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional- expression parser interprets these as shell syntax rather than regex literals, producing: syntax error in conditional expression: unexpected token ')' at line 27 on every CI run. Fix: move the regex into a USES_PATTERN variable using safe single-quote shell-string concatenation so the [[ =~ ]] parser receives an unquoted variable reference ($USES_PATTERN) rather than a literal pattern containing bash-special characters. The regex semantics are identical: optional leading/trailing quote around owner/action@ref, quote chars excluded from capture groups. Verified in real bash 5.2.21 (Git for Windows): No syntax error on the real 40-pin workflow tree (Checked 40) Unquoted SHA pin: MATCH, correct repo+ref extracted Double-quoted SHA pin: MATCH, correct repo+ref extracted Single-quoted SHA pin: MATCH, correct repo+ref extracted .yaml extension file: MATCH, correct repo+ref extracted ./local-action: NO MATCH (correct) docker://: NO MATCH (correct) * docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md The checklist covered Trusted Publishing trust, protected environment, branch protection, and Dependabot github-actions entry. Three non-forking controls described elsewhere in SECURITY.md were omitted: - GitHub secret scanning and push protection (repo settings, not copied on fork) - GitGuardian (GitHub App installation scoped to this specific repo, requires separate install on any fork) - CodeQL Default Setup vs Advanced Setup state (repo setting that affects whether the upload-sarif step in codeql.yml does anything) Added as items 5, 6, 7 matching the existing numbered bullet style. * fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check) verify-action-pins caught that github/codeql-action@v4 tag was re-pointed upstream: old: f205ea1c3313d32999d8d6a48b4f6530d4437b38 new: d1ba80a13dd99fba24a470575428917156a28b43 Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze, upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment # v4 unchanged — the tag itself hasn't changed, only what commit it points to. --------- Co-authored-by: Sameer6305 --- .github/dependabot.yml | 7 + .github/scripts/verify-action-pins.sh | 70 ++++++++ .github/workflows/benchmark.yml | 6 +- .github/workflows/ci.yml | 6 +- .github/workflows/codeql.yml | 14 +- .github/workflows/defender-for-devops.yml | 12 +- .github/workflows/docs.yml | 16 +- .github/workflows/release.yml | 27 +++- .github/workflows/security-scan.yml | 188 ++++++++++++++-------- .github/workflows/security.yml | 4 +- .github/workflows/verify-action-pins.yml | 28 ++++ CHANGELOG.md | 23 +++ SECURITY.md | 95 +++++++++++ pyproject.toml | 1 - semantica/context/agent_memory.py | 2 +- semantica/context/context_graph.py | 2 +- semantica/context/entity_linker.py | 4 +- semantica/graph_store/graph_store.py | 4 +- semantica/ingest/email_ingestor.py | 6 +- semantica/ontology/namespace_manager.py | 6 +- semantica/parse/__init__.py | 4 +- semantica/parse/methods.py | 4 +- semantica/parse/pdf_parser.py | 2 +- semantica/triplet_store/query_engine.py | 2 +- 24 files changed, 408 insertions(+), 125 deletions(-) create mode 100644 .github/scripts/verify-action-pins.sh create mode 100644 .github/workflows/verify-action-pins.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml index d1da3ddf..42a46136 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -70,6 +70,13 @@ updates: - "dependencies" - "github-actions" - "ci" + # All our actions are SHA-pinned with a "# vX" comment; Dependabot + # resolves the new tag's SHA and updates both the pin and the comment + # together, so this stays the source of truth (no separate script needed). + groups: + github-actions: + patterns: + - "*" # Optional dependencies (separate schedule for stability) - package-ecosystem: "pip" diff --git a/.github/scripts/verify-action-pins.sh b/.github/scripts/verify-action-pins.sh new file mode 100644 index 00000000..5022297e --- /dev/null +++ b/.github/scripts/verify-action-pins.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# Verifies that every third-party GitHub Action referenced in +# .github/workflows/*.yml and .github/workflows/*.yaml is pinned to a full +# commit SHA (not a mutable tag +# or branch), and that any pin's trailing "# vX" comment still matches what +# that tag resolves to today. +# +# Fails closed on purpose: +# - a `uses:` line pinned to anything other than a 40-hex-char SHA is a +# hard failure, not a skip - this is what stops a newly-added mutable +# tag (e.g. `uses: some/action@v1`) from slipping past unnoticed. +# - a tag that can't be resolved via the GitHub API (rate limit, deleted +# tag, typo) is also a hard failure rather than a warning - an +# unverifiable pin is exactly the failure mode this check exists to +# catch, so it must not pass silently. +set -uo pipefail + +fail=0 +checked=0 + +# Pattern for a third-party uses: line — stored in a variable so bash's +# [[ =~ ]] parser never sees literal \" or \' escapes, which cause a +# "syntax error in conditional expression: unexpected token )" at runtime. +# Semantics: optional leading quote, owner/repo, optional subpath, @ref, +# optional trailing quote; quote chars excluded from the ref capture group. +USES_PATTERN='uses:[[:space:]]+["'"'"']?([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)(/[^[:space:]@"'"'"']+)?@([^[:space:]"'"'"']+)["'"'"']?' + +while IFS=: read -r file lineno content; do + # Local composite actions (./x) and Docker image refs (docker://...) use a + # different pinning mechanism and aren't in scope here. + [[ "$content" =~ uses:\ +\./ ]] && continue + [[ "$content" =~ uses:\ +docker:// ]] && continue + + if [[ "$content" =~ $USES_PATTERN ]]; then + repo="${BASH_REMATCH[1]}" + ref="${BASH_REMATCH[3]}" + checked=$((checked + 1)) + + if [[ ! "$ref" =~ ^[0-9a-fA-F]{40}$ ]]; then + echo "::error file=$file,line=$lineno::$repo is pinned to '$ref', not a full commit SHA. Mutable tags/branches can be silently re-pointed (see the LiteLLM/Trivy 2026 incident) - pin to a commit SHA instead." + fail=1 + continue + fi + sha="$ref" + + if [[ "$content" =~ \#[[:space:]]*([^[:space:]]+)[[:space:]]*$ ]]; then + tag="${BASH_REMATCH[1]}" + else + echo "::warning file=$file,line=$lineno::$repo@$sha has no trailing '# vX' comment recording which tag it corresponds to - add one for auditability." + continue + fi + + resolved=$(gh api "repos/$repo/commits/$tag" --jq '.sha' 2>/dev/null) + if [[ -z "$resolved" ]]; then + echo "::error file=$file,line=$lineno::Could not resolve '$repo@$tag' via the GitHub API (rate limit, deleted tag, or typo). Treating as unverifiable = failure." + fail=1 + continue + fi + + if [[ "$resolved" != "$sha" ]]; then + echo "::error file=$file,line=$lineno::$repo is pinned to $sha but tag '$tag' now resolves to $resolved. Update the pin or the comment." + fail=1 + else + echo "OK $repo@$tag -> $sha ($file:$lineno)" + fi + fi +done < <(grep -rHn "uses:" .github/workflows/*.yml .github/workflows/*.yaml 2>/dev/null) + +echo "Checked $checked action reference(s)." +exit $fail diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 3c79c367..f75c199a 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -13,12 +13,12 @@ jobs: steps: - name: Checkout Code - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: fetch-depth: 0 - name: Set up Python 3.12 - uses: actions/setup-python@v7 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: "3.12" cache: 'pip' @@ -43,7 +43,7 @@ jobs: # pytest-benchmark --storage file://benchmarks/results --benchmark-compare - name: Upload Benchmark Results - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 if: always() with: name: benchmark-report-${{ github.run_id }} diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4f51f1b7..18a246fb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -21,11 +21,11 @@ jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: '3.11' - - uses: actions/setup-node@v7 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: node-version: '20' cache: 'npm' diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index a78fbf42..c32d6955 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -20,7 +20,7 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v7 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 # The CodeQL bundle download (github/codeql-action/init's "Setup CodeQL # tools" step) streams a ~1GB tarball from GitHub's release CDN and @@ -32,7 +32,7 @@ jobs: # meaningful state carried over from a failed attempt. - name: Initialize CodeQL (attempt 1) id: codeql-init-1 - uses: github/codeql-action/init@v4 + uses: github/codeql-action/init@d1ba80a13dd99fba24a470575428917156a28b43 # v4 continue-on-error: true with: languages: python @@ -42,7 +42,7 @@ jobs: - name: Initialize CodeQL (attempt 2) id: codeql-init-2 if: steps.codeql-init-1.outcome == 'failure' - uses: github/codeql-action/init@v4 + uses: github/codeql-action/init@d1ba80a13dd99fba24a470575428917156a28b43 # v4 continue-on-error: true with: languages: python @@ -52,17 +52,17 @@ jobs: - name: Initialize CodeQL (attempt 3) id: codeql-init-3 if: steps.codeql-init-2.outcome == 'failure' - uses: github/codeql-action/init@v4 + uses: github/codeql-action/init@d1ba80a13dd99fba24a470575428917156a28b43 # v4 with: languages: python queries: security-and-quality config-file: .github/codeql/codeql-config.yml - name: Autobuild - uses: github/codeql-action/autobuild@v4 + uses: github/codeql-action/autobuild@d1ba80a13dd99fba24a470575428917156a28b43 # v4 - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v4 + uses: github/codeql-action/analyze@d1ba80a13dd99fba24a470575428917156a28b43 # v4 with: category: "/language:python" upload: false @@ -72,7 +72,7 @@ jobs: # Uploads results only when Default Setup is not active. # If Default Setup is still enabled, this step skips gracefully # instead of failing the workflow with HTTP 409. - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@d1ba80a13dd99fba24a470575428917156a28b43 # v4 with: sarif_file: ${{ steps.codeql.outputs.sarif-output }} category: "/language:python" diff --git a/.github/workflows/defender-for-devops.yml b/.github/workflows/defender-for-devops.yml index 0fd59ada..aa71bd69 100644 --- a/.github/workflows/defender-for-devops.yml +++ b/.github/workflows/defender-for-devops.yml @@ -36,14 +36,14 @@ jobs: runs-on: windows-latest steps: - - uses: actions/checkout@v7 - - uses: actions/setup-dotnet@v6 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6 with: dotnet-version: | 5.0.x 6.0.x - name: Run Microsoft Security DevOps - uses: microsoft/security-devops-action@v1.12.0 + uses: microsoft/security-devops-action@08976cb623803b1b36d7112d4ff9f59eae704de0 # v1.12.0 id: msdo with: # checkov is intentionally excluded from this MSDO step. @@ -57,11 +57,11 @@ jobs: # avoiding the guardian.cmd/checkov exit-code bug in the MSDO wrapper. tools: eslint,templateanalyzer,terrascan - name: Upload results to Security tab - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@d1ba80a13dd99fba24a470575428917156a28b43 # v4 with: sarif_file: ${{ steps.msdo.outputs.sarifFile }} - - uses: actions/setup-python@v7 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: "3.12" @@ -82,7 +82,7 @@ jobs: } - name: Upload Checkov results to Security tab - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@d1ba80a13dd99fba24a470575428917156a28b43 # v4 if: always() with: sarif_file: reports/checkov.sarif diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index cc417a98..ca149d7e 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -29,11 +29,11 @@ jobs: name: Validate Documentation runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: '3.11' - - uses: actions/setup-node@v7 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: node-version: '20' - run: python docs_check.py @@ -44,9 +44,9 @@ jobs: runs-on: ubuntu-latest needs: validate steps: - - uses: actions/checkout@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - - uses: actions/setup-node@v7 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: node-version: '20' @@ -57,12 +57,12 @@ jobs: cd .. unzip -q export.zip -d site - - uses: actions/configure-pages@v6 + - uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6 - - uses: actions/upload-pages-artifact@v5 + - uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5 with: path: ./site - name: Deploy to GitHub Pages id: deployment - uses: actions/deploy-pages@v5 + uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6aaea2eb..df704ea3 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -5,19 +5,28 @@ on: tags: ['v*'] permissions: - contents: write - id-token: write + contents: read jobs: release: runs-on: ubuntu-latest environment: pypi + concurrency: + group: release-${{ github.ref }} + cancel-in-progress: false + permissions: + contents: write # for the GitHub Release + id-token: write # for PyPI Trusted Publishing (OIDC) and attestation signing + attestations: write # for SLSA build provenance + # If you add another job to this workflow, give it its own explicit + # `permissions:` block rather than relying on the workflow-level default + # above (contents: read) - do not widen the workflow-level default. steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: '3.11' - - uses: actions/setup-node@v7 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: node-version: '20' cache: 'npm' @@ -46,7 +55,11 @@ jobs: print("Explorer frontend is packaged") PY - - uses: softprops/action-gh-release@v3 + - name: Attest build provenance + uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373 # v4 + with: + subject-path: 'dist/*' + - uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3 with: files: dist/* - - uses: pypa/gh-action-pypi-publish@release/v1 + - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1 diff --git a/.github/workflows/security-scan.yml b/.github/workflows/security-scan.yml index d2ddd94e..ab742e76 100644 --- a/.github/workflows/security-scan.yml +++ b/.github/workflows/security-scan.yml @@ -28,13 +28,17 @@ jobs: contents: read security-events: write actions: read - + # Needed for the "Comment PR with Security Results" step below. Safe on + # pull_request (not pull_request_target): GitHub always forces a + # read-only token for PRs from forks regardless of this permission. + pull-requests: write + steps: - name: Checkout repository - uses: actions/checkout@v7 - + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: Set up Python - uses: actions/setup-python@v7 + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: '3.11' @@ -42,21 +46,50 @@ jobs: run: | python -m pip install --upgrade pip pip install safety bandit semgrep jq - + # Install the project itself (core deps + the LiteLLM provider extra) + # so Safety scans Semantica's actual dependency tree, not just the + # scanner tools' own dependencies. + pip install -e ".[llm-litellm]" + - name: Run Safety Check (Package Vulnerabilities) run: | - safety check --json --output safety-report.json || true + # NOTE: Safety 3.x repurposed --output to select a console format + # (json/text/screen/...), not a file path. Writing JSON to a file + # now requires --save-json; the previous `--output safety-report.json` + # usage was silently invalid and never produced a report. + safety check --save-json safety-report.json || true + + # Guard 1: fail loudly if Safety exited before writing a report at all + # (network error, API auth failure, tool crash). Without this check a + # missing or empty file causes jq to fall back to "0", making a broken + # scanner indistinguishable from a clean scan. + if [ ! -s safety-report.json ]; then + echo "::error::Safety scan produced no report (safety-report.json is missing or empty). Treating as failure — check for network errors, API auth failures, or Safety crashes in the logs above." + exit 1 + fi + echo "Checking for package vulnerabilities..." - - # Count vulnerabilities safely - VULNS=$(safety check --json --output /dev/stdout 2>/dev/null | jq '.vulnerabilities | length' 2>/dev/null || echo "0") - + + # No || echo "0" fallback: if jq fails (malformed JSON, missing key, + # vulnerabilities:null) VULNS will be empty or "null" so guard 2 below + # catches it rather than silently treating the broken report as zero. + VULNS=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null) + + # Guard 2: ensure VULNS is a non-negative integer before the -gt + # comparison. "null" (missing/null key) or "" (jq parse failure) would + # cause bash's -gt to throw an arithmetic error and fall through to the + # success branch — the same silent-pass bug as a missing file. + if ! [[ "$VULNS" =~ ^[0-9]+$ ]]; then + echo "::error::Safety report exists but 'vulnerabilities' is missing or non-numeric (got: '${VULNS}'). The report may be malformed or Safety may have written an error-only JSON. Treating as failure." + exit 1 + fi + if [ "$VULNS" -gt 0 ]; then echo "❌ Security vulnerabilities found: $VULNS" echo "CI will fail to prevent merging of vulnerable dependencies" echo "" echo "Vulnerability details:" - safety check || true + jq -r '.vulnerabilities[] | "- \(.package_name)==\(.analyzed_version): \(.vulnerability_id) (\(.CVE // "no CVE assigned"))"' safety-report.json || true exit 1 else echo "✅ No security vulnerabilities found" @@ -99,9 +132,10 @@ jobs: fi - name: Upload Security Reports - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: security-reports + retention-days: 14 path: | safety-report.json bandit-report.json @@ -109,77 +143,91 @@ jobs: - name: Comment PR with Security Results if: github.event_name == 'pull_request' - uses: actions/github-script@v9 + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 with: script: | const fs = require('fs'); - - // Read safety report - let safetyResults = ''; - try { - const safetyData = JSON.parse(fs.readFileSync('safety-report.json', 'utf8')); - if (safetyData.vulnerabilities && safetyData.vulnerabilities.length > 0) { - safetyResults = `## Safety Vulnerabilities Found\\n`; - safetyData.vulnerabilities.forEach(vuln => { - safetyResults += `- **${vuln.package}**: ${vuln.advisory}\\n`; - }); - } else { - safetyResults = '## No Safety Vulnerabilities Found\\n'; + + // Renders one tool's findings as a section. `items` is already + // the list of pre-formatted "- `thing` in `where`" strings; this + // just handles the found/not-found/report-missing framing and + // collapses long lists into a
block so the comment + // doesn't turn into a wall of text. + function renderSection(title, reportPath, parse) { + let data; + try { + data = JSON.parse(fs.readFileSync(reportPath, 'utf8')); + } catch (e) { + return [ + `### ${title}`, + `⚠️ No report found at \`${reportPath}\` — the scan may have failed before producing output. Check the job logs.`, + ].join('\n'); } - } catch (e) { - safetyResults = '## Safety scan completed\\n'; - } - - // Read bandit report - let banditResults = ''; - try { - const banditData = JSON.parse(fs.readFileSync('bandit-report.json', 'utf8')); - if (banditData.results && banditData.results.length > 0) { - const highIssues = banditData.results.filter(issue => issue.issue_severity === 'HIGH'); - if (highIssues.length > 0) { - banditResults = `## High Severity Security Issues Found\\n`; - highIssues.forEach(issue => { - banditResults += `- **${issue.test_name}**: ${issue.filename}:${issue.line_number}\\n`; - }); - } else { - banditResults = '## No High Severity Security Issues Found\\n'; - } - } else { - banditResults = '## No Bandit Issues Found\\n'; + + const items = parse(data); + if (items.length === 0) { + return [`### ${title}`, `✅ No findings.`].join('\n'); } - } catch (e) { - banditResults = '## Bandit scan completed\\n'; - } - - // Read semgrep report - let semgrepResults = ''; - try { - const semgrepData = JSON.parse(fs.readFileSync('semgrep-report.json', 'utf8')); - if (semgrepData.results && semgrepData.results.length > 0) { - semgrepResults = `## Security Patterns Found\\n`; - semgrepData.results.slice(0, 10).forEach(issue => { - semgrepResults += `- **${issue.rule_id}**: ${issue.path}\\n`; - }); - if (semgrepData.results.length > 10) { - semgrepResults += `- ... and ${semgrepData.results.length - 10} more\\n`; - } + + const lines = [`### ${title}`, `Found **${items.length}**.`, '']; + const shown = items.slice(0, 15); + if (items.length > 15) { + lines.push('
', 'Show all findings', ''); + lines.push(...items); + lines.push('', '
'); } else { - semgrepResults = '## No Security Patterns Found\\n'; + lines.push(...shown); } - } catch (e) { - semgrepResults = '## Semgrep scan completed\\n'; + return lines.join('\n'); } - - // Create summary comment - const comment = `# 🔒 Security Scan Results\\n\\n${safetyResults}\\n\\n${banditResults}\\n\\n${semgrepResults}\\n\\n---\\n\\n*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*\\n\\n📊 **Security Policy**: CI fails on vulnerabilities and HIGH severity issues.`; - - // Post comment with error handling + + const safetySection = renderSection( + 'Safety — dependency vulnerabilities', + 'safety-report.json', + (data) => (data.vulnerabilities || []).map( + (v) => `- \`${v.package_name}==${v.analyzed_version}\`: ${v.vulnerability_id}` + + (v.CVE ? ` (${v.CVE})` : '') + ` — ${v.advisory || 'no advisory text'}` + ) + ); + + const banditSection = renderSection( + 'Bandit — HIGH-severity code issues', + 'bandit-report.json', + (data) => (data.results || []) + .filter((issue) => issue.issue_severity === 'HIGH') + .map((issue) => `- \`${issue.test_name}\` in \`${issue.filename}:${issue.line_number}\``) + ); + + const semgrepSection = renderSection( + 'Semgrep — static analysis patterns', + 'semgrep-report.json', + (data) => (data.results || []).map( + (issue) => `- \`${issue.check_id}\` in \`${issue.path}:${issue.start?.line ?? '?'}\`` + ) + ); + + const comment = [ + '# 🔒 Security Scan Results', + '', + safetySection, + '', + banditSection, + '', + semgrepSection, + '', + '---', + '', + '*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*', + '', + '📊 **Security Policy**: CI fails on Safety vulnerabilities and Bandit HIGH-severity findings. Semgrep findings above are informational and do not block merge.', + ].join('\n'); + try { await github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, - body: comment + body: comment, }); console.log('✅ Security comment posted successfully'); } catch (error) { diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml index d45bae3d..9f093bc7 100644 --- a/.github/workflows/security.yml +++ b/.github/workflows/security.yml @@ -12,8 +12,8 @@ jobs: audit: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7 - - uses: actions/setup-python@v7 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7 with: python-version: '3.11' - run: pip install pip-audit diff --git a/.github/workflows/verify-action-pins.yml b/.github/workflows/verify-action-pins.yml new file mode 100644 index 00000000..6e7dada9 --- /dev/null +++ b/.github/workflows/verify-action-pins.yml @@ -0,0 +1,28 @@ +name: Verify Action Pins + +on: + pull_request: + paths: + - '.github/workflows/**' + - '.github/scripts/verify-action-pins.sh' + push: + branches: [main] + paths: + - '.github/workflows/**' + - '.github/scripts/verify-action-pins.sh' + schedule: + - cron: '0 3 * * 1' # weekly, in case an upstream tag is deliberately moved + workflow_dispatch: + +permissions: + contents: read + +jobs: + verify: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + - name: Verify pinned action SHAs match their tag comments + env: + GH_TOKEN: ${{ github.token }} + run: bash .github/scripts/verify-action-pins.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index badd93d8..389779fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -168,6 +168,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `GET /api/analytics` sets `response.status_code = 207` (Multi-Status) when some, but not all, of the requested metrics fail, and raises `HTTPException(500)` when every requested metric fails — a plain 2xx (including 207) reads as success to callers that only check `response.ok`, so an all-failed request now surfaces as a hard error rather than a body full of `{"error": ...}` - Added regression tests covering all three failure paths (`test_patterns_failure_returns_500`, `test_analytics_partial_failure_returns_207`, `test_analytics_total_failure_returns_500`, and two `TestOntologyCreateFailures` cases) +### Security + +- **CI/CD supply-chain hardening against mutable-tag Action compromise (LiteLLM/Trivy-class attack)** (#824) by @KaifAhmad1 + - Every third-party GitHub Action across all 8 workflows is now pinned to a full commit SHA instead of a mutable tag (`@v7` → `@3d3c42e... # v7`), closing the exact vector used against LiteLLM in March 2026 (a compromised Trivy Action tag stole a long-lived publishing token) + - Added `verify-action-pins.yml` + `.github/scripts/verify-action-pins.sh`: a CI check that fails closed on any `uses:` reference that isn't a full SHA (catching a newly introduced mutable tag, not just auditing existing pins) and re-verifies every pin against the GitHub API on each workflow change, on push to `main`, and weekly; an unresolvable API lookup is treated as a failure rather than a silent skip + - `release.yml`: scoped `permissions` to the job level (workflow default is now `contents: read`), added a `concurrency` group so simultaneous tag pushes can't race the publish job, and added SLSA build provenance attestation (`actions/attest-build-provenance`) for every released wheel + - Created a protected `pypi` GitHub Environment (required reviewer, restricted to `v*` tag deployments) and enabled branch protection on `main` (required PR review with stale-approval dismissal, required status checks, no force-push/deletion, required conversation resolution) — PyPI publishing already used Trusted Publishing (OIDC) with no long-lived token + - Grouped Dependabot's `github-actions` updates into a single PR + +- **`security-scan.yml`'s Safety dependency-vulnerability check was silently non-functional** (#824) by @KaifAhmad1 + - `safety check --json --output safety-report.json` is invalid in Safety 3.x (`--output` now selects a console format, not a file path); the command errored on every run, swallowed by `|| true`, so no report was ever produced and the job always fell back to a generic "scan completed" message with the vulnerability count hardcoded to 0 + - Switched to `--save-json`, the correct flag for writing a JSON report to disk; also fixed `vuln.package` → `vuln.package_name` and Semgrep's `issue.rule_id` → `issue.check_id` (both produced `undefined` in the PR comment) + - The job never installed Semantica's own dependencies before scanning, so Safety was auditing the scanner tools' own transitive deps, not the project's; added `pip install -e ".[llm-litellm]"` so the actual dependency tree — including the LiteLLM extra — is what gets scanned + - Rewrote the PR-comment builder: every line previously used `\\n` inside JS template literals, which renders as the literal text `\n` rather than a newline, producing an unreadable wall of text; now builds real line arrays and collapses long finding lists into a `
` block + - Added the `pull-requests: write` permission the comment-posting step was missing (silently failing via its own try/catch on every prior run) + +- **`pypdf2==3.0.1` removed (CVE-2023-36464)** (#824) by @KaifAhmad1 + - Surfaced by the Safety fix above: PyPDF2 is a discontinued project (merged into `pypdf`) permanently frozen at the vulnerable 3.0.1 with no patched release possible. `grep -rn "import PyPDF2"` found zero real usages anywhere in the codebase — it was only referenced in docstrings describing a `PyPDF2.PdfReader()` fallback for PDF parsing that was never actually implemented (`pdfplumber` does the real work). Removed the dependency and corrected the stale docstrings in `parse/__init__.py`, `parse/methods.py`, `parse/pdf_parser.py`, and `ingest/email_ingestor.py` + +- **10 Bandit B324 false positives suppressed (non-cryptographic MD5 use)** (#824) by @KaifAhmad1 + - Surfaced by the same Safety fix restoring a working CI gate: Bandit's HIGH-severity check was blocking on 10 pre-existing `hashlib.md5()` calls, all generating short deterministic cache keys, entity IDs, or IRI suffixes from non-secret input — none used for passwords, tokens, or verifying untrusted data + - Bandit's own message suggests `usedforsecurity=False`, but that keyword argument needs Python 3.9+ and `pyproject.toml` declares `requires-python = ">=3.8"`; used a targeted `# nosec B324` with a one-line justification instead, which suppresses only this check with no runtime behavior change on any supported Python version + ## [0.6.0] - 2026-07-21 ### Added diff --git a/SECURITY.md b/SECURITY.md index 6945cde3..6b395811 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -112,6 +112,101 @@ We regularly update dependencies to address security vulnerabilities. However, y - Be cautious with external API calls - Implement proper authentication and authorization +## CI/CD Supply-Chain Security + +Semantica's build and release pipeline is explicitly hardened against +CI/CD supply-chain attacks — the class of attack behind the March 2026 +LiteLLM/Trivy incident, where a compromised third-party Action with a +**mutable tag** was used to steal a long-lived publishing token, after which +malicious packages were pushed straight to PyPI without ever touching the +source repository. Every control below maps directly to closing one step of +that attack chain. + +### Immutable build inputs + +- **Risk**: a tag (`@v4`, `@release/v1`) is re-pointed by a compromised upstream maintainer or account, silently changing what every consumer's CI runs. + **Control**: every third-party GitHub Action in every workflow is pinned to a full 40-character commit SHA, with the human-readable tag kept only as a trailing comment (e.g. `actions/checkout@3d3c42e... # v7`). +- **Risk**: a SHA pin drifts out of sync with its own comment over time, or is mistyped. + **Control**: `verify-action-pins.yml` fails closed on any `uses:` reference that isn't a full commit SHA (catching a newly added mutable tag, not just auditing existing pins), resolves every pinned tag via the GitHub API on each workflow change, on every push to `main`, and weekly, and fails if the SHA no longer matches the tag it claims to be — an API lookup that can't be resolved is treated as a failure, not a silent skip. +- **Risk**: manually re-pinning ~15 actions across 8 workflow files on every upstream release is error-prone. + **Control**: Dependabot (`github-actions` ecosystem) opens a grouped PR that bumps the SHA *and* the tag comment together whenever an action releases — pins never require hand-editing. + +### Publishing pipeline (highest-privilege path) + +- **Risk**: a long-lived `PYPI_TOKEN` sitting in repo/org secrets is exfiltrated by any compromised step. + **Control**: PyPI publishing uses Trusted Publishing (OIDC) (`id-token: write`) — there is no long-lived PyPI credential anywhere in this repository to steal. +- **Risk**: a compromised CI run publishes to PyPI with no human in the loop. + **Control**: the publish job runs only inside a protected `pypi` GitHub Environment with a required human reviewer — every release needs manual approval in the Actions UI before it runs. +- **Risk**: the release job could be triggered from an arbitrary branch/ref. + **Control**: the `pypi` environment's deployment-branch policy is restricted to `v*` tags only. +- **Risk**: a scanner or unrelated job inherits publish-level credentials. + **Control**: `release.yml` sets `permissions: contents: read` at the workflow level; `contents: write` / `id-token: write` / `attestations: write` are granted only to the release job, never workflow-wide. +- **Risk**: two tag pushes race through the publish pipeline simultaneously. + **Control**: `concurrency: group: release-${{ github.ref }}` serializes releases per tag. +- **Risk**: a consumer can't verify a wheel on PyPI actually came from this repo's CI. + **Control**: SLSA build provenance is attested for every release via `actions/attest-build-provenance`, producing a signed, verifiable record of the exact commit and workflow run that produced the artifact (checkable with `gh attestation verify`). + +### Repository controls + +- **Risk**: unreviewed or force-pushed changes land on `main`. + **Control**: `main` requires 1 approving PR review (stale approvals dismissed on new pushes), resolved conversations, and blocks force-pushes and branch deletion. +- **Risk**: a PR merges without its security/CI checks passing. + **Control**: merges require the `build`, `Analyze Python` (CodeQL), and `security-scan` checks to pass, in strict mode (checks must be re-run against the latest `main`). +- **Risk**: a compromised scanner job reaches secrets or write access. + **Control**: scanning jobs (`CodeQL`, `security-scan.yml`, `security.yml`, `defender-for-devops.yml`) run with read-only, least-privilege permissions (typically `contents: read` + `security-events: write` only) and never share a job, environment, or secret scope with the publish job. +- **Risk**: secrets are committed accidentally. + **Control**: GitHub secret scanning and push protection are both enabled at the repository level, rejecting pushes that contain recognizable credential patterns before they land in history. + +## Automated Security Scanning + +Every scan below runs continuously in CI, not just at release time: + +- **CodeQL** (`security-and-quality` query pack) — Python source: injection, unsafe deserialization, and other code-level vulnerability classes. Runs in `codeql.yml` on every push/PR to `main` and weekly. +- **Bandit** — Python-specific security anti-patterns (hardcoded secrets, unsafe `eval`/`pickle`, weak crypto, etc.); CI fails on any HIGH-severity finding. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly. +- **Semgrep** (`p/security` ruleset) — cross-language static-analysis security patterns. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly. +- **Safety** — known CVEs in Semantica's own installed dependencies, including optional LLM-provider extras such as LiteLLM; CI fails on any match. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly. +- **pip-audit** — independent, PyPA-maintained vulnerability database cross-check against installed dependencies (Safety and pip-audit use different advisory sources, so both run). Runs in `security.yml` weekly. +- **Microsoft Defender for DevOps** (`eslint`, `templateanalyzer`, `terrascan`) — JavaScript/TypeScript lint-security rules and infrastructure-as-code misconfigurations. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly. +- **Checkov** — Kubernetes, Helm, Dockerfile, GitHub Actions, and secrets-pattern IaC scanning; results upload to the same Security tab as CodeQL. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly. +- **GitGuardian** — secret-detection check on every pull request, installed as a GitHub App integration (not a repo-local workflow). Runs on every PR. +- **GitHub secret scanning + push protection** — blocks known credential patterns before they're pushed, and continuously scans existing history. Platform-level, continuous. +- **Dependabot** — version/security PRs for Python, Docker, and GitHub Actions dependencies, grouped where relevant to reduce review noise. Configured in `.github/dependabot.yml`, runs weekly for security-relevant packages and monthly for docs dependencies. +- **`verify-action-pins.yml`** — enforces that every Action reference is a full commit SHA (failing on a newly introduced mutable tag) and confirms each SHA still matches the tag it claims to be. Runs on every workflow change, every push to `main`, and weekly. + +All SARIF-producing scanners (CodeQL, Checkov, Microsoft Defender) publish +findings to the repository's **Security → Code scanning alerts** tab, giving +a single audit trail across tools rather than scattered per-tool reports. + +### Adopting this posture in a fork or downstream deployment + +Teams standing up their own instance of Semantica, or forking it for an +internal/regulated deployment, can reuse this posture directly: + +1. Keep Dependabot's `github-actions` ecosystem entry — it is what keeps + SHA pins current without manual maintenance. +2. Re-run `verify-action-pins.yml` after re-pointing the repository's Actions + at your own mirrors, if you do so. +3. If you publish your own PyPI package from a fork, configure your own + Trusted Publishing trust relationship on PyPI (Trusted Publishing is + scoped to a specific `owner/repo` + workflow filename) and your own + protected environment with your own required reviewers — these are not + transferable from this repository. +4. Branch protection, environment protection, and repository secret + scanning are repository *settings*, not workflow files — cloning or + forking the repo does **not** copy them. They must be re-applied via + the GitHub UI or API on the new repository. +5. GitHub secret scanning and push protection are repository settings that + don't carry over to a fork either — re-enable both under the new + repository's Security settings, not just Dependabot. +6. GitGuardian runs as a GitHub App installation scoped to this specific + repository, not a workflow file — a fork gets no secret-detection + coverage from it until the app is installed separately on the new repo. +7. CodeQL's `upload-sarif` step in `codeql.yml` only runs meaningfully if + Default Setup is *not* already enabled for the repository (it's designed + to skip gracefully otherwise) — check whether Default Setup or Advanced + Setup is active on the new repository and adjust expectations for where + CodeQL findings show up accordingly. + ## Dependency Security Policy ### Regular Updates diff --git a/pyproject.toml b/pyproject.toml index d729e9b2..92b89a78 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -66,7 +66,6 @@ dependencies = [ "grpcio>=1.81.1", "beautifulsoup4>=4.15.0", "lxml>=6.1.1", - "pypdf2>=2.10.0", "python-docx>=1.2.0", "openpyxl>=3.1.5", "pillow>=12.2.0", diff --git a/semantica/context/agent_memory.py b/semantica/context/agent_memory.py index 53e5816c..2df42647 100644 --- a/semantica/context/agent_memory.py +++ b/semantica/context/agent_memory.py @@ -727,7 +727,7 @@ class AgentMemory: timestamp = str(time.time()) random_str = str(hash(str(self.memory_items)) % 10000) - memory_hash = hashlib.md5(f"{timestamp}_{random_str}".encode()).hexdigest()[:12] + memory_hash = hashlib.md5(f"{timestamp}_{random_str}".encode()).hexdigest()[:12] # nosec B324 - short unique ID, not security-sensitive return f"mem_{memory_hash}" diff --git a/semantica/context/context_graph.py b/semantica/context/context_graph.py index ebfc0464..06419759 100644 --- a/semantica/context/context_graph.py +++ b/semantica/context/context_graph.py @@ -1691,7 +1691,7 @@ class ContextGraph: # Fallback ID generation import hashlib - entity_hash = hashlib.md5( + entity_hash = hashlib.md5( # nosec B324 - deterministic entity ID, not security-sensitive f"{entity_text}_{entity_type}".encode() ).hexdigest()[:12] entity_id = f"{entity_type.lower()}_{entity_hash}" diff --git a/semantica/context/entity_linker.py b/semantica/context/entity_linker.py index 1cab04cc..eb35edd9 100644 --- a/semantica/context/entity_linker.py +++ b/semantica/context/entity_linker.py @@ -170,7 +170,7 @@ class EntityLinker: uri = f"{self.base_uri}{uri_safe}" else: # Use hash of entity_id - entity_hash = hashlib.md5(entity_id.encode()).hexdigest()[:8] + entity_hash = hashlib.md5(entity_id.encode()).hexdigest()[:8] # nosec B324 - deterministic URI suffix, not security-sensitive uri = f"{self.base_uri}{entity_hash}" # Add type if available @@ -500,7 +500,7 @@ class EntityLinker: def _generate_entity_id(self, text: str, entity_type: str) -> str: """Generate entity ID from text and type.""" - entity_hash = hashlib.md5(f"{text}_{entity_type}".encode()).hexdigest()[:12] + entity_hash = hashlib.md5(f"{text}_{entity_type}".encode()).hexdigest()[:12] # nosec B324 - deterministic entity ID, not security-sensitive return f"{entity_type.lower()}_{entity_hash}" def build_entity_web(self) -> Dict[str, Any]: diff --git a/semantica/graph_store/graph_store.py b/semantica/graph_store/graph_store.py index 2d185c19..5b7e2637 100644 --- a/semantica/graph_store/graph_store.py +++ b/semantica/graph_store/graph_store.py @@ -294,7 +294,7 @@ class QueryEngine: import hashlib key_str = f"{query}:{str(parameters)}" - return hashlib.md5(key_str.encode()).hexdigest() + return hashlib.md5(key_str.encode()).hexdigest() # nosec B324 - cache key, not security-sensitive def clear_cache(self) -> None: """Clear query cache.""" @@ -1056,7 +1056,7 @@ class GraphStore: if not entity_id and entity_text: import hashlib - entity_hash = hashlib.md5( + entity_hash = hashlib.md5( # nosec B324 - deterministic entity ID, not security-sensitive f"{entity_text}_{entity_type}".encode() ).hexdigest()[:12] entity_id = f"{entity_type.lower()}_{entity_hash}" diff --git a/semantica/ingest/email_ingestor.py b/semantica/ingest/email_ingestor.py index 0b453def..81c2af20 100644 --- a/semantica/ingest/email_ingestor.py +++ b/semantica/ingest/email_ingestor.py @@ -180,7 +180,7 @@ class AttachmentProcessor: This method attempts to extract text content from various document types. Currently supports plain text files. PDF and Word document - extraction would require additional libraries (PyPDF2/pdfplumber for + extraction would require additional libraries (pdfplumber for PDF, python-docx for Word). Args: @@ -193,9 +193,9 @@ class AttachmentProcessor: """ try: if file_type == "application/pdf": - # PDF text extraction would require PyPDF2 or pdfplumber + # PDF text extraction would require pdfplumber self.logger.debug( - "PDF text extraction not implemented (requires PyPDF2/pdfplumber)" + "PDF text extraction not implemented (requires pdfplumber)" ) return None elif file_type in [ diff --git a/semantica/ontology/namespace_manager.py b/semantica/ontology/namespace_manager.py index f7ed0047..69f26c66 100644 --- a/semantica/ontology/namespace_manager.py +++ b/semantica/ontology/namespace_manager.py @@ -120,7 +120,7 @@ class NamespaceManager: # Use hash-based IRI import hashlib - hash_id = hashlib.md5(class_name.encode()).hexdigest()[:8] + hash_id = hashlib.md5(class_name.encode()).hexdigest()[:8] # nosec B324 - deterministic IRI suffix, not security-sensitive iri = urljoin(self.get_base_uri(), f"class/{hash_id}") return iri @@ -146,7 +146,7 @@ class NamespaceManager: # Use hash-based IRI import hashlib - hash_id = hashlib.md5(property_name.encode()).hexdigest()[:8] + hash_id = hashlib.md5(property_name.encode()).hexdigest()[:8] # nosec B324 - deterministic IRI suffix, not security-sensitive iri = urljoin(self.get_base_uri(), f"property/{hash_id}") return iri @@ -170,7 +170,7 @@ class NamespaceManager: else: import hashlib - hash_id = hashlib.md5(individual_name.encode()).hexdigest()[:8] + hash_id = hashlib.md5(individual_name.encode()).hexdigest()[:8] # nosec B324 - deterministic IRI suffix, not security-sensitive iri = urljoin(self.get_base_uri(), f"individual/{hash_id}") return iri diff --git a/semantica/parse/__init__.py b/semantica/parse/__init__.py index 5091f562..7890b3ae 100644 --- a/semantica/parse/__init__.py +++ b/semantica/parse/__init__.py @@ -8,7 +8,7 @@ emails, code files, and media files. Algorithms Used: Document Parsing: - - PDF Parsing: pdfplumber integration (pdfplumber.PDF()) for text extraction, PyPDF2.PdfReader() fallback, table extraction (pdfplumber.extract_tables()), image extraction, metadata extraction (title, author, dates via pdf.metadata), page-level processing (page iteration) + - PDF Parsing: pdfplumber integration (pdfplumber.PDF()) for text extraction, table extraction (pdfplumber.extract_tables()), image extraction, metadata extraction (title, author, dates via pdf.metadata), page-level processing (page iteration) - DOCX Parsing: python-docx integration (docx.Document()), paragraph extraction (document.paragraphs), table extraction (docx.table.Table), section/heading detection (paragraph.style), metadata extraction (core_properties), formatting extraction - Docling Parsing: Docling integration (DocumentConverter.convert()) for enhanced table extraction and document structure understanding, supports PDF, DOCX, PPTX, XLSX, HTML, images, markdown/HTML/JSON export formats, OCR support (optional dependency) - PPTX Parsing: python-pptx integration (pptx.Presentation()), slide extraction (presentation.slides), shape extraction, notes extraction, metadata extraction @@ -45,7 +45,7 @@ Media Parsing: - Audio/Video Parsing: Metadata extraction (format, duration, codec), file information extraction (future support) Format-Specific Parsers: - - PDFParser: pdfplumber.PDF() for text/tables, PyPDF2.PdfReader() fallback, page iteration (pdf.pages), metadata extraction + - PDFParser: pdfplumber.PDF() for text/tables, page iteration (pdf.pages), metadata extraction - DOCXParser: docx.Document() for document loading, paragraph iteration, table extraction, core_properties access - DoclingParser: DocumentConverter.convert() for multi-format parsing with enhanced table extraction, supports PDF/DOCX/PPTX/XLSX/HTML/images, markdown/HTML/JSON export (optional dependency) - PPTXParser: pptx.Presentation() for presentation loading, slide iteration, shape extraction diff --git a/semantica/parse/methods.py b/semantica/parse/methods.py index 0fddd607..e83cf6ff 100644 --- a/semantica/parse/methods.py +++ b/semantica/parse/methods.py @@ -49,7 +49,7 @@ Media Parsing: Algorithms Used: Document Parsing: - - PDF Parsing: pdfplumber integration (pdfplumber.PDF()) for text extraction, PyPDF2.PdfReader() fallback, table extraction (pdfplumber.extract_tables()), image extraction, metadata extraction (title, author, dates via pdf.metadata), page-level processing (page iteration) + - PDF Parsing: pdfplumber integration (pdfplumber.PDF()) for text extraction, table extraction (pdfplumber.extract_tables()), image extraction, metadata extraction (title, author, dates via pdf.metadata), page-level processing (page iteration) - DOCX Parsing: python-docx integration (docx.Document()), paragraph extraction (document.paragraphs), table extraction (docx.table.Table), section/heading detection (paragraph.style), metadata extraction (core_properties), formatting extraction - HTML Parsing: BeautifulSoup integration (BeautifulSoup(html, 'html.parser')), text extraction (soup.get_text()), link extraction (find_all('a')), metadata extraction (meta tags), structure analysis - Text Parsing: Plain text file reading (open().read()), encoding detection, line-by-line processing @@ -83,7 +83,7 @@ Media Parsing: - Audio/Video Parsing: Metadata extraction (format, duration, codec), file information extraction (future support) Format-Specific Parsers: - - PDFParser: pdfplumber.PDF() for text/tables, PyPDF2.PdfReader() fallback, page iteration (pdf.pages), metadata extraction + - PDFParser: pdfplumber.PDF() for text/tables, page iteration (pdf.pages), metadata extraction - DOCXParser: docx.Document() for document loading, paragraph iteration, table extraction, core_properties access - JSONParser: json.load()/json.loads(), recursive structure traversal, path extraction - CSVParser: csv.DictReader() for row-by-row processing, delimiter detection, header handling diff --git a/semantica/parse/pdf_parser.py b/semantica/parse/pdf_parser.py index 7ad31ced..639f60a9 100644 --- a/semantica/parse/pdf_parser.py +++ b/semantica/parse/pdf_parser.py @@ -1,7 +1,7 @@ """ PDF Document Parser Module -This module handles PDF document parsing using PyPDF2 and pdfplumber for text, +This module handles PDF document parsing using pdfplumber for text, table, and image extraction, including metadata and page-level processing. Key Features: diff --git a/semantica/triplet_store/query_engine.py b/semantica/triplet_store/query_engine.py index f0e908c2..5ddfe087 100644 --- a/semantica/triplet_store/query_engine.py +++ b/semantica/triplet_store/query_engine.py @@ -527,7 +527,7 @@ class QueryEngine: import hashlib normalized = " ".join(query.split()) - return hashlib.md5(normalized.encode()).hexdigest() + return hashlib.md5(normalized.encode()).hexdigest() # nosec B324 - cache key, not security-sensitive def _cache_result(self, query: str, result: QueryResult) -> None: """Cache query result."""