ci: pin Python dependencies in requirements-ci.txt for reproducible CI (#945)

* ci: pin Python dependencies in requirements-ci.txt for reproducible CI

Adds a committed lockfile pinning all transitive dependencies at exact
versions (uv pip compile, Python 3.11, all extras — 1581 lines), the
Python equivalent of explorer/package-lock.json + npm ci.

- CI installs from requirements-ci.txt before building the wheel
- CI verifies the lockfile is byte-identical to a fresh compile (fails
  on staleness after pyproject.toml changes)
- CONTRIBUTING documents the regeneration command

Closes #938

Signed-off-by: Yunare Maia <yunare@gmail.com>

* ci: address Qodo review — security scans use pinned deps, exclude gpu extras

- security-scan.yml installs from requirements-ci.txt instead of
  "./[llm-litellm]" so Safety scans the exact CI/release dependency tree
- security.yml runs pip-audit -r requirements-ci.txt for the same parity
- lockfile regenerated with --extra all (the cross-platform set) instead
  of --all-extras, which pulled faiss-gpu/cupy from the Linux-only gpu
  extra and co-installed faiss-cpu + faiss-gpu in CI
- uv pinned to 0.12.1 (the version that generated the lockfile) in CI and
  CONTRIBUTING so regeneration is deterministic

Signed-off-by: Yunare Maia <yunare@gmail.com>

* ci: make lockfile staleness check immune to upstream releases

The previous check re-resolved pyproject.toml without constraints, so any
upstream package release (e.g. boto3 1.43.69 -> 1.43.70) failed CI even
when nothing in the repo changed — exactly the time-dependent drift Qodo
flagged. The check now re-resolves with requirements-ci.txt as a
constraint and compares only version lines, so it detects intentional
pyproject.toml changes but ignores upstream releases. CONTRIBUTING
updated to match.

Signed-off-by: Yunare Maia <yunare@gmail.com>

* ci: fix security workflows — install pip-audit; order tooling after pinned deps

Security workflow: the pip-audit install step was lost in the rebase
conflict merge — pip-audit was invoked but never installed (exit 127).

Security-scan workflow: installing safety first let the pinned
requirements-ci.txt overwrite its transitive deps (rich), breaking the
safety CLI at runtime (RuntimeError: Type not yet supported). Tooling is
now installed AFTER the pinned set.

Signed-off-by: Yunare Maia <yunare@gmail.com>

* fix(ci): address review — hashes, build isolation, release builds, docs (4/4)

ZohaibHassan16's review flagged 4 supply-chain gaps; all addressed:

1. **Release builds now use the lockfile**: release.yml installs
   requirements-ci.txt and runs `python -m build --no-isolation` so the
   sdist/wheel is built against the exact tested dependency set.
2. **Build isolation pinned**: [build-system].requires is now
   setuptools==84.0.0 + wheel==0.48.0 (exact pins, no ranges).
3. **Hashes**: requirements-ci.txt regenerated with --generate-hashes
   (5,708 sha256 hashes, verified against PyPI). Staleness check updated
   to strip the `\` line continuations hashes introduce.
4. **CONTRIBUTING.md documents the separate environment**: hashes,
   never-install-into-dev note, build-system pins, --no-isolation release
   builds.

Validated: stale-check diff clean, hash spot-check matches PyPI.
Signed-off-by: Yunare Maia <yunare@gmail.com>

* fix(ci): apply --no-isolation to CI build + align benchmark to Python 3.11

Follow-up to ZohaibHassan16's second review round:

1. ci.yml was still running `python -m build` with build isolation
   (unpinned setuptools/wheel from PyPI) — now `python -m build
   --no-isolation` against the pinned deps, matching release.yml.
2. benchmark.yml was on Python 3.12 while the lockfile is compiled for
   3.11 — aligned to 3.11 so every workflow runs the same environment.

Signed-off-by: Yunare Maia <yunare@gmail.com>

* fix(ci): install pinned wheel before --no-isolation build

python -m build --no-isolation failed with 'Missing dependencies:
wheel==0.48.0' because wheel is build-time only — uv's lockfile
excludes it, so installing requirements-ci.txt alone left the build
env without it. Both ci.yml and release.yml now install wheel==0.48.0
(the same pin [build-system] declares) before building. Validated
locally: wheel builds clean with --no-isolation.

Signed-off-by: Yunare Maia <yunare@gmail.com>

---------

Signed-off-by: Yunare Maia <yunare@gmail.com>
Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>

.github/workflows/benchmark.yml

@@ -17,10 +17,10 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: Set up Python 3.12
+      - name: Set up Python 3.11
         uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
         with:
-          python-version: "3.12"
+          python-version: "3.11"
           cache: 'pip'
 
       - name: Install Dependencies

.github/workflows/ci.yml

@@ -42,8 +42,28 @@ jobs:
       - name: Build Explorer frontend
         working-directory: explorer
         run: npm run build
+      - name: Install pinned Python dependencies
+        run: |
+          pip install -r requirements-ci.txt
+      - name: Verify requirements-ci.txt is up to date
+        run: |
+          pip install uv==0.12.1
+          # Re-resolve with the committed file as a constraint: upstream package
+          # releases must NOT fail CI (deps only change when pyproject.toml
+          # changes intentionally). Compare only version lines (pkg==ver),
+          # ignoring the -c constraint comments and the `\` line continuations
+          # that --generate-hashes emits.
+          uv pip compile pyproject.toml --python-version 3.11 --extra all \
+            --constraint requirements-ci.txt -o /tmp/requirements-ci-check.txt
+          diff \
+            <(grep -E '^[a-zA-Z0-9._-]+==' requirements-ci.txt | sed 's/ \\$//') \
+            <(grep -E '^[a-zA-Z0-9._-]+==' /tmp/requirements-ci-check.txt)
       - run: pip install build
-      - run: python -m build
+      # wheel is build-time only (not in requirements-ci.txt) — install the
+      # same pinned version [build-system] declares so --no-isolation works.
+      - run: pip install wheel==0.48.0
+      - name: Build package (no isolation — pinned deps)
+        run: python -m build --no-isolation
       - name: Verify Explorer frontend is packaged
         run: |
           python - <<'PY'

.github/workflows/release.yml

@@ -36,8 +36,16 @@ jobs:
         run: |
           npm ci
           npm run build
+      # Install the pinned dependency set (with hashes) so the sdist/wheel
+      # build runs against the same versions CI tests against.
+      - name: Install pinned build dependencies
+        run: pip install -r requirements-ci.txt
       - run: pip install build
-      - run: python -m build
+      # wheel is build-time only (not in requirements-ci.txt) — install the
+      # same pinned version [build-system] declares so --no-isolation works.
+      - run: pip install wheel==0.48.0
+      - name: Build package (no isolation — pinned deps)
+        run: python -m build --no-isolation
       - name: Verify Explorer frontend is packaged
         run: |
           python - <<'PY'

.github/workflows/security-scan.yml

@@ -45,11 +45,14 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
+          # Install the pinned dependency set FIRST so Safety scans Semantica's
+          # exact CI/release dependency tree (requirements-ci.txt is generated
+          # from pyproject.toml extras, so this covers the project's real deps).
+          pip install -r requirements-ci.txt
+          # Tooling AFTER the pinned set: installing safety/bandit/semgrep/jq
+          # first lets the pinned requirements overwrite their transitive deps
+          # (e.g. rich), which breaks the safety CLI at runtime.
           pip install safety bandit semgrep jq
-          # Install the project itself (core deps + the LiteLLM provider extra)
-          # so Safety scans Semantica's actual dependency tree, not just the
-          # scanner tools' own dependencies.
-          pip install -e ".[llm-litellm]"
 
       - name: Run Safety Check (Package Vulnerabilities)
         run: |

.github/workflows/security.yml

@@ -8,6 +8,7 @@ on:
     branches: [main]
     paths:
       - 'pyproject.toml'
+      - 'requirements-ci.txt'
       - '.github/workflows/security.yml'
 
 permissions:
@@ -23,21 +24,19 @@ jobs:
           python-version: '3.11'
       # Upgrade first: actions/setup-python's baked-in setuptools has been
       # behind known-vulnerable floors before (e.g. PYSEC-2026-3447 /
-      # CVE-2026-59890, fixed in 83.0.0) regardless of what this project's
-      # own [build-system] requires -- that only governs isolated build
-      # environments, not the ambient one pip-audit scans here.
+      # setuptools 75.1.0), so don't trust the preinstalled one.
       - run: python -m pip install --upgrade pip setuptools
-      - run: pip install pip-audit
-      # Install the [all] extra so pip-audit sees every optional dependency
-      # group (fastapi, python-multipart, etc.), not just pip-audit's own
-      # deps. PYSEC-2024-38 (#869) shipped in the first place because
-      # neither this job (bare env, no extras) nor security-scan.yml's
-      # Safety check (installs only [llm-litellm]) ever had fastapi or
+      # Audit the pinned dependency set (requirements-ci.txt is compiled from
+      # pyproject.toml with --extra all — the same coverage as the [all]
+      # extra, minus the Linux-only gpu set — so this keeps scan parity with
+      # CI/release builds without a time-dependent resolution). This is the
+      # fix for PYSEC-2024-38 (#869): the bare-env job never had fastapi or
       # python-multipart installed to look at.
-      - run: pip install -e ".[all]"
+      - run: pip install -r requirements-ci.txt
       # PR runs gate on findings, since they're scoped to actual
       # pyproject.toml changes under review. The schedule/workflow_dispatch
       # runs stay non-blocking until a full pass over pre-existing findings
       # across the whole [all] tree has been done.
-      - run: pip-audit
+      - run: pip install pip-audit
+      - run: pip-audit -r requirements-ci.txt
         continue-on-error: ${{ github.event_name != 'pull_request' }}

CONTRIBUTING.md

@@ -181,6 +181,39 @@ pip install -e ".[dev]"
 pre-commit install
 ```
 
+### Pinned CI dependencies
+
+`requirements-ci.txt` pins every transitive dependency at exact versions so CI,
+security scans, and release builds install the same packages every run (the
+Python equivalent of `explorer/package-lock.json` + `npm ci`). It is a
+**separate build environment**: every package carries a SHA-256 hash
+(`--generate-hashes`), so installs are reproducible and supply-chain safe —
+never install into your local dev environment from it.
+
+Regenerate it after changing `pyproject.toml` dependencies:
+
+```bash
+pip install uv==0.12.1
+uv pip compile pyproject.toml --python-version 3.11 --extra all --generate-hashes -o requirements-ci.txt
+```
+
+The `all` extra is the repo's cross-platform dependency set (GPU extras like
+`faiss-gpu`/`cupy` are excluded and installed separately on Linux — see
+`pyproject.toml`). Keep the pinned `uv` version in sync with CI so regeneration
+is deterministic.
+
+CI's staleness check re-resolves with the committed lockfile as a constraint
+and compares version lines only: upstream package releases never fail CI —
+the lockfile changes only when `pyproject.toml` changes intentionally.
+
+CI fails if `requirements-ci.txt` is stale relative to `pyproject.toml`
+(the version-line comparison detects new/removed/changed dependencies).
+
+Build-system pins: `[build-system].requires` is pinned to exact versions
+(`setuptools==84.0.0`, `wheel==0.48.0`) and release builds run
+`python -m build --no-isolation` against the lockfile — no unpinned
+build-time isolation anywhere.
+
 ### 3. Create Branch
 
 ```bash

pyproject.toml

@@ -1,5 +1,5 @@
 [build-system]
-requires = ["setuptools>=83.0.0", "wheel"]
+requires = ["setuptools==84.0.0", "wheel==0.48.0"]
 build-backend = "setuptools.build_meta"
 
 [project]