mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
* security: SHA-pin all Actions, harden release pipeline, add pin verification
Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a
compromised third-party Action with a mutable tag stealing a long-lived
publishing token) and closes several related gaps found in an audit of the
actual repository state.
- Pin every third-party GitHub Action across all workflows to a full commit
SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI
check that confirms via the GitHub API that each pin still matches its
tag, on every workflow change, push to main, and weekly.
- Scope release.yml permissions to the job level (workflow defaults to
contents: read); add a concurrency group so simultaneous tag pushes can't
race the publish job.
- Add SLSA build provenance attestation (actions/attest-build-provenance)
for every released wheel.
- Fix a latent bug in security-scan.yml: the PR-comment step was missing
pull-requests: write and silently failing; add bounded artifact retention
for uploaded scan reports.
- Group github-actions Dependabot updates to cut review noise.
- Document the resulting posture in SECURITY.md for auditors/regulated
adopters, including what's enforced and what a fork needs to reconfigure
for itself (environment/branch protection, Trusted Publishing trust).
Also (via GitHub API, not in this diff): created a protected `pypi`
environment with a required reviewer restricted to v* tags, and enabled
branch protection on main (required review, required status checks, no
force-push/deletion).
* fix: harden verify-action-pins per PR #824 bot review
Addresses real findings from the automated review on #824:
- The script previously only matched uses: lines that already contained a
40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1)
would never be scanned at all and the check would pass silently. It now
matches every uses: line and hard-fails on any ref that isn't a full
commit SHA.
- A tag that fails to resolve via the GitHub API (rate limit, deleted tag)
previously only logged a warning and continued; that's now a hard
failure too, since an unverifiable pin is exactly the failure mode this
check exists to catch.
- verify-action-pins.yml only triggered on .github/workflows/** changes,
so an edit to the verifier script itself wouldn't run the check that
verifies it. Added the script path to both trigger filters.
The reviewer's claim that slash-containing tag comments (release/v1) break
the API lookup did not reproduce - tested directly against
pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves
multi-segment refs natively - so no change was needed there.
Verified with a synthetic test workflow containing a mutable-tag action,
a correctly-pinned SHA, and a deliberately mismatched SHA: the updated
script now catches the first and third cases and passes the second. Also
re-ran against the real workflow tree (40/40 pins still verify clean).
* fix: repair broken Safety scan and PR comment formatting
The "Comment PR with Security Results" step was producing garbled output
(literal \n characters instead of newlines, "undefined:" labels) because:
- Every line in the JS comment builder used \n (escaped backslash-n)
inside template literals, which JS renders as the literal two-character
string \n, not a newline.
- The Semgrep section read issue.rule_id, but Semgrep's JSON field is
check_id - hence "undefined: <path>" for every entry.
Rewrote the comment builder to construct each section as an array of
lines joined with a real '\n', with correct field names, and collapsed
long finding lists into a <details> block instead of a flat list.
Verified by extracting the exact script and running it under node against
synthetic fixtures matching each tool's real JSON schema (found/clean/
missing-report paths all render correctly).
While tracing the "undefined" and always-empty Safety section, found the
Safety step itself was silently broken:
- `safety check --json --output safety-report.json` is invalid in
Safety 3.x: --output now selects a console format (json/text/screen),
not a file path. The command errored on every run (swallowed by
`|| true`), so safety-report.json was never created and the PR comment
always fell back to a generic "scan completed" message. Switched to
`--save-json`, which is the correct flag for writing a JSON report to
disk, and confirmed against the real safety 3.8.1 CLI locally.
- Even with a report, the code read vuln.package - the real field is
package_name.
- The job never installed Semantica's own dependencies before scanning,
so `safety check` (which defaults to scanning the environment) was
auditing the scanner tools' own dependencies, not Semantica's. Added
`pip install -e ".[llm-litellm]"` so the project's actual dependency
tree - including the LiteLLM extra this whole hardening effort is
about - is what gets scanned.
Also updated the corresponding SECURITY.md bullet to describe what Safety
actually covers now.
* fix: remove unused pypdf2 dependency (CVE-2023-36464)
Now that the Safety scan step actually runs (see previous commit), it
correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a
real, pre-existing vulnerability that was invisible until the scan was
fixed.
PyPDF2 is not a patchable dependency here: the project is discontinued
(merged into `pypdf`), 3.0.1 is its final release, and there is no fixed
version to upgrade to. Grepping the repo for `import PyPDF2` / `from
PyPDF2` turns up nothing - it was never actually imported anywhere. Its
only presence outside pyproject.toml was in docstrings describing a
"PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented
in code; pdfplumber is the library actually used. Removed the dependency
and corrected the stale docstrings in parse/__init__.py, parse/methods.py,
parse/pdf_parser.py, and ingest/email_ingestor.py accordingly.
* fix: suppress Bandit B324 false positives on non-cryptographic MD5 use
Same pattern as the previous pypdf2 commit: fixing the Safety scan
surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10
pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for
security").
Checked each of the 10 call sites: every one uses hashlib.md5() to build
a short deterministic cache key, entity ID, or IRI suffix from already-
non-secret input (query text, entity text/type, class/property names) -
none are used for passwords, tokens, or integrity verification of
untrusted data. This is exactly the case Bandit's own message points at
("Consider usedforsecurity=False").
Did not use usedforsecurity=False itself: that keyword argument was
added to hashlib in Python 3.9, and pyproject.toml declares
`requires-python = ">=3.8"` - adding it unconditionally risks a TypeError
on 3.8. Used a targeted `# nosec B324` comment with a one-line
justification instead, which suppresses only this specific check and
carries no runtime behavior change on any supported Python version.
Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity
findings (was 10).
* docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening
Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml
hardening (job-scoped permissions, concurrency, SLSA provenance), the
pypi environment/branch protection GitHub-side config, the
security-scan.yml Safety/comment-formatting fixes, and the two
vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal,
Bandit B324 suppression).
* fix: close two remaining gaps missed by upstream bot-review fixes
verify-action-pins.sh:
- Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched
by the existing regex, so a SHA-pinned action written with quotes would
silently skip verification. Updated the main ERE to accept an optional
leading/trailing single or double quote around the owner/action@ref
value, and excluded quote chars from the inner character classes so the
ref is still extracted cleanly.
- The grep input glob only covered *.yml. GitHub also treats *.yaml as a
valid workflow extension. Added *.yaml to the glob and a 2>/dev/null
guard so the command doesn't fail when no *.yaml files exist.
security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a):
- Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all
five scanner-failure modes (file missing, empty file, malformed JSON,
valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still
silently produce VULNS=0 or VULNS=null and pass the merge-blocker check.
- Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety
crashed before writing a report (covers missing and empty-file cases).
- Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~
^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON,
missing key, and null cases). Both guards emit ::error:: annotations.
- Verified with a 7-case simulation: all 5 failure modes now exit 1;
genuine zero-vuln and real-vuln cases still behave correctly.
* fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI
The regex for matching uses: lines was embedded directly inline in a
[[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional-
expression parser interprets these as shell syntax rather than regex
literals, producing:
syntax error in conditional expression: unexpected token ')'
at line 27 on every CI run.
Fix: move the regex into a USES_PATTERN variable using safe single-quote
shell-string concatenation so the [[ =~ ]] parser receives an unquoted
variable reference ($USES_PATTERN) rather than a literal pattern containing
bash-special characters. The regex semantics are identical: optional
leading/trailing quote around owner/action@ref, quote chars excluded from
capture groups.
Verified in real bash 5.2.21 (Git for Windows):
No syntax error on the real 40-pin workflow tree (Checked 40)
Unquoted SHA pin: MATCH, correct repo+ref extracted
Double-quoted SHA pin: MATCH, correct repo+ref extracted
Single-quoted SHA pin: MATCH, correct repo+ref extracted
.yaml extension file: MATCH, correct repo+ref extracted
./local-action: NO MATCH (correct)
docker://: NO MATCH (correct)
* docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md
The checklist covered Trusted Publishing trust, protected environment,
branch protection, and Dependabot github-actions entry. Three non-forking
controls described elsewhere in SECURITY.md were omitted:
- GitHub secret scanning and push protection (repo settings, not copied
on fork)
- GitGuardian (GitHub App installation scoped to this specific repo,
requires separate install on any fork)
- CodeQL Default Setup vs Advanced Setup state (repo setting that affects
whether the upload-sarif step in codeql.yml does anything)
Added as items 5, 6, 7 matching the existing numbered bullet style.
* fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check)
verify-action-pins caught that github/codeql-action@v4 tag was re-pointed
upstream:
old: f205ea1c3313d32999d8d6a48b4f6530d4437b38
new: d1ba80a13dd99fba24a470575428917156a28b43
Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze,
upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment
# v4 unchanged — the tag itself hasn't changed, only what commit it points to.
---------
Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
275 lines
7.7 KiB
TOML
275 lines
7.7 KiB
TOML
[build-system]
|
|
requires = ["setuptools>=61.0", "wheel"]
|
|
build-backend = "setuptools.build_meta"
|
|
|
|
[project]
|
|
name = "semantica"
|
|
version = "0.6.0"
|
|
description = "Accountability and context layer for AI agents. Context graphs, decision intelligence, full provenance tracking, and explainable reasoning engines — every AI decision traceable, every output auditable."
|
|
readme = "README.md"
|
|
license = { text = "MIT" }
|
|
|
|
authors = [{ name = "Semantica", email = "kaif@getsemantica.ai" }]
|
|
maintainers = [{ name = "Semantica", email = "kaif@getsemantica.ai" }]
|
|
|
|
requires-python = ">=3.8"
|
|
|
|
classifiers = [
|
|
"Development Status :: 5 - Production/Stable",
|
|
"Intended Audience :: Developers",
|
|
"Intended Audience :: Science/Research",
|
|
"Intended Audience :: Information Technology",
|
|
"License :: OSI Approved :: MIT License",
|
|
"Operating System :: OS Independent",
|
|
"Programming Language :: Python :: 3",
|
|
"Programming Language :: Python :: 3.8",
|
|
"Programming Language :: Python :: 3.9",
|
|
"Programming Language :: Python :: 3.10",
|
|
"Programming Language :: Python :: 3.11",
|
|
"Programming Language :: Python :: 3.12",
|
|
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
|
"Topic :: Scientific/Engineering :: Information Analysis",
|
|
"Topic :: Text Processing :: Linguistic",
|
|
"Topic :: Database :: Database Engines/Servers",
|
|
"Topic :: Software Development :: Libraries :: Python Modules"
|
|
]
|
|
|
|
keywords = [
|
|
"knowledge-graph", "context-graph", "ai-agents", "llm", "decision-intelligence",
|
|
"provenance", "explainability", "reasoning-engine", "entity-extraction",
|
|
"relation-extraction", "graph-rag", "knowledge-intelligence", "semantic-layer",
|
|
"nlp", "embeddings", "ontology", "rdf", "triplet-extraction", "agentic-ai",
|
|
"knowledge-base", "entity-resolution", "w3c-prov", "audit-trail"
|
|
]
|
|
|
|
# ---------------- CORE DEPENDENCIES (SAFE DEFAULT) ----------------
|
|
dependencies = [
|
|
"numpy>=2.0.2",
|
|
"pandas>=1.3.0",
|
|
"scipy>=1.13.1",
|
|
"scikit-learn>=1.7.2",
|
|
"umap-learn>=0.5.12",
|
|
"spacy>=3.4.0",
|
|
"transformers>=4.20.0",
|
|
"torch>=1.13.1",
|
|
"sentence-transformers>=2.2.0",
|
|
"rdflib>=6.2.0",
|
|
"networkx>=2.8.0",
|
|
"matplotlib>=3.9.4",
|
|
"seaborn>=0.13.2",
|
|
"plotly>=6.8.0",
|
|
"ipywidgets>=8.0.0",
|
|
"requests>=2.34.2",
|
|
"GitPython>=3.1.50",
|
|
"chardet>=7.4.3",
|
|
"protobuf>=5.29.1,<8.0",
|
|
"grpcio>=1.81.1",
|
|
"beautifulsoup4>=4.15.0",
|
|
"lxml>=6.1.1",
|
|
"python-docx>=1.2.0",
|
|
"openpyxl>=3.1.5",
|
|
"pillow>=12.2.0",
|
|
"librosa>=0.9.0",
|
|
"opencv-python>=4.13.0.92",
|
|
"faiss-cpu>=1.7.0",
|
|
"fastembed>=0.2.0",
|
|
"onnxruntime>=1.20.1",
|
|
"tokenizers>=0.15.0",
|
|
"pydantic>=2.13.4",
|
|
"click>=8.4.2",
|
|
"rich>=12.5.0",
|
|
"tqdm>=4.68.3",
|
|
"pyyaml>=6.0",
|
|
"toml>=0.10.0",
|
|
"python-dotenv>=1.2.1",
|
|
"loguru>=0.7.3",
|
|
"structlog>=22.1.0",
|
|
"gensim>=4.4.0",
|
|
"httpx<0.29.0"
|
|
]
|
|
|
|
[project.urls]
|
|
Homepage = "https://getsemantica.ai"
|
|
Documentation = "https://docs.getsemantica.ai"
|
|
Repository = "https://github.com/semantica-agi/semantica"
|
|
Changelog = "https://github.com/semantica-agi/semantica/blob/main/CHANGELOG.md"
|
|
"Bug Tracker" = "https://github.com/semantica-agi/semantica/issues"
|
|
Discord = "https://discord.gg/sV34vps5hH"
|
|
|
|
# ---------------- OPTIONAL DEPENDENCIES ----------------
|
|
[project.optional-dependencies]
|
|
|
|
# ---- LLM Providers ----
|
|
llm-openai = ["openai>=1.0.0"]
|
|
llm-groq = ["groq>=0.4.0"]
|
|
llm-gemini = ["google-genai>=0.1.0"]
|
|
llm-anthropic = ["anthropic>=0.18.0"]
|
|
llm-ollama = ["ollama>=0.1.0"]
|
|
llm-deepseek = ["openai>=1.0.0"]
|
|
llm-litellm = ["litellm>=1.83.9"]
|
|
llm-instructor = ["instructor>=1.15.3"]
|
|
|
|
llm-all = [
|
|
"semantica[llm-openai,llm-groq,llm-gemini,llm-anthropic,llm-ollama,llm-deepseek,llm-litellm,llm-instructor]"
|
|
]
|
|
|
|
# ---- Document Parsing ----
|
|
parse-docling = ["docling>=2.107.0"]
|
|
|
|
# ---- SHACL Validation ----
|
|
shacl = ["pyshacl>=0.25.0"]
|
|
|
|
# ---- Database Connectors ----
|
|
db-snowflake = ["snowflake-connector-python>=4.6.0", "cryptography>=49.0.0"]
|
|
db-databricks = ["databricks-sdk>=0.60.0", "databricks-sql-connector>=4.0.0"]
|
|
db-arrow = ["pyarrow>=24.0.0"]
|
|
ingest-parquet = ["pyarrow>=24.0.0"]
|
|
ingest-arrow = ["pyarrow>=24.0.0"]
|
|
|
|
db-all = [
|
|
"semantica[db-snowflake,db-databricks,db-arrow]"
|
|
]
|
|
|
|
# ---- Embedding / Models ----
|
|
models-huggingface = [
|
|
"transformers>=4.20.0",
|
|
"torch>=1.13.1"
|
|
]
|
|
|
|
# ---- Graph Backends ----
|
|
graph-neo4j = ["neo4j>=5.0.0"]
|
|
graph-falkordb = ["falkordb>=1.0.0", "redis>=4.3.0"]
|
|
graph-amazon-neptune = ["boto3>=1.24.0", "neo4j>=5.0.0"]
|
|
graph-apache-age = ["psycopg2-binary>=2.9.0"]
|
|
|
|
graph-all = [
|
|
"semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune,graph-apache-age]"
|
|
]
|
|
|
|
# ---- Vector Store Backends ----
|
|
vectorstore-qdrant = ["qdrant-client>=1.0.0"]
|
|
vectorstore-weaviate = ["weaviate-client>=4.0.0"]
|
|
vectorstore-pinecone = ["pinecone-client>=3.0.0"]
|
|
vectorstore-milvus = ["pymilvus>=2.0.0"]
|
|
vectorstore-pgvector = ["psycopg[binary,pool]>=3.0.0", "pgvector>=0.2.0"]
|
|
vectorstore-sqlite = ["sqlite-vec>=0.1.1"]
|
|
|
|
vectorstore-all = [
|
|
"semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus,vectorstore-pgvector,vectorstore-sqlite]"
|
|
]
|
|
|
|
# ---- Infra / Queues / Workers ----
|
|
infra = [
|
|
"redis>=4.3.0",
|
|
"celery>=5.2.0",
|
|
"kafka-python>=3.0.2",
|
|
"pulsar-client>=3.0.0",
|
|
"pika>=1.3.0"
|
|
]
|
|
|
|
# ---- Cloud Providers ----
|
|
cloud = [
|
|
"boto3>=1.24.0",
|
|
"azure-storage-blob>=12.30.0",
|
|
"google-cloud-storage>=2.5.0"
|
|
]
|
|
|
|
# ---- Monitoring (FIXED) ----
|
|
monitoring = [
|
|
"prometheus-client>=0.14.0",
|
|
"opentelemetry-api>=1.30.0,<2.0.0",
|
|
"opentelemetry-sdk>=1.30.0,<2.0.0",
|
|
"opentelemetry-semantic-conventions>=0.58b0,<0.65",
|
|
"opentelemetry-instrumentation>=0.62b1,<0.65"
|
|
]
|
|
|
|
# ---- Visualization ----
|
|
viz = [
|
|
"pyvis>=0.3.0",
|
|
"graphviz>=0.21",
|
|
"d3blocks>=1.0.0"
|
|
]
|
|
|
|
# ---- GPU ----
|
|
gpu = [
|
|
"faiss-gpu>=1.7.0",
|
|
"cupy>=10.0.0"
|
|
]
|
|
|
|
# ---- Agentic Framework Integrations ----
|
|
agno = ["agno>=1.0.0"]
|
|
|
|
# ---- File Watching ----
|
|
watch = ["watchdog>=6.0.0"]
|
|
|
|
# ---- Splitting / Chunking ----
|
|
split-tiktoken = ["tiktoken>=0.5.0"]
|
|
split-community = ["python-louvain>=0.16"]
|
|
split-topic = ["bertopic>=0.15.0", "gensim>=4.4.0"]
|
|
|
|
split-all = [
|
|
"semantica[split-tiktoken,split-community,split-topic]"
|
|
]
|
|
|
|
# ---- Dev ----
|
|
dev = [
|
|
"pytest>=7.1.0",
|
|
"pytest-cov>=7.1.0",
|
|
"pytest-asyncio>=0.19.0",
|
|
"black>=22.6.0",
|
|
"isort>=6.1.0",
|
|
"flake8>=4.0.0",
|
|
"mypy>=0.971",
|
|
"pre-commit>=4.6.0",
|
|
"jupyter>=1.0.0",
|
|
"ipykernel>=6.15.0"
|
|
]
|
|
|
|
# Explorer Dashboard
|
|
explorer = [
|
|
"fastapi>=0.100.0",
|
|
"uvicorn[standard]>=0.22.0",
|
|
"websockets>=15.0.1",
|
|
"python-multipart>=0.0.6"
|
|
]
|
|
explorer-lite = [
|
|
"streamlit>=1.25.0",
|
|
"streamlit-agraph>=0.0.45"
|
|
]
|
|
|
|
# Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
|
|
all = [
|
|
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
|
|
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
|
|
]
|
|
|
|
# ---------------- ENTRYPOINTS ----------------
|
|
[project.scripts]
|
|
semantica = "semantica.cli:main"
|
|
semantica-server = "semantica.server:main"
|
|
semantica-worker = "semantica.worker:main"
|
|
semantica-explorer = "semantica.explorer:main"
|
|
semantica-mcp = "semantica.mcp_server:main"
|
|
|
|
# ---------------- TOOLING ----------------
|
|
[tool.setuptools.packages.find]
|
|
where = ["."]
|
|
include = ["semantica*", "integrations*"]
|
|
|
|
[tool.setuptools.package-data]
|
|
# Explicit patterns are more reliable than **/* across setuptools versions.
|
|
# static/* covers index.html / favicon; static/assets/* covers all JS/CSS chunks.
|
|
"semantica" = ["static/*", "static/assets/*"]
|
|
|
|
[tool.black]
|
|
line-length = 88
|
|
|
|
[tool.isort]
|
|
profile = "black"
|
|
|
|
[tool.pytest.ini_options]
|
|
testpaths = ["tests"]
|
|
markers = [
|
|
"integration: marks tests that require external services or API keys (deselect with '-m not integration')",
|
|
]
|