Compare commits

...
Author SHA1 Message Date
Zohaib Hassnain 9a293bb67c fix(security): implement DNS pinning for ontology and fix SPARQL regex 2026-08-11 15:02:19 +05:00
Sunil 44f585ffce test(security): add regression tests for all security fixes 2026-08-11 15:17:23 +05:30
Sunil f5332589d5 fix(security): harden SPARQL read-only check against comment/prefix bypass 2026-08-11 15:17:21 +05:30
Sunil 9a21ca9834 fix(security): prevent Cypher injection via graph_name and dollar-delimiter breakout 2026-08-11 15:17:19 +05:30
Sunil a169cf3fb9 feat(security): wire API key auth middleware into Explorer app 2026-08-11 15:17:17 +05:30
Sunil 656baa7aee feat(security): add opt-in API key auth middleware for Explorer API 2026-08-11 15:17:15 +05:30
KaifAhmad1 e1725fd763 fix(ontology): close the final (non-redirect) response in _fetch_url_sync
The previous rework of the redirect loop closed the response on each
redirect hop but dropped the try/finally around the success path, so the
terminal response (the one actually read and returned) was left
unclosed, leaking the connection back to the pool unclosed under load.
2026-08-11 15:11:07 +05:30
Mohd Kaif 7ed1d49625 Merge branch 'main' into security/fix-critical-vulnerabilities 2026-08-11 15:05:45 +05:30
Sunil c94be3f9a6 fix(ontology): resolve relative redirects with urljoin, close resp on redirect 2026-08-11 15:01:34 +05:30
Sunil 142707db93 fix(sparql): wrap graph cap ValueError in SparqlResponse instead of 500 2026-08-11 15:01:31 +05:30
Sunil 3357c14ee3 fix(vector_store): use v.tolist() for numpy array serialization 2026-08-11 15:01:29 +05:30
Mohd KaifandZohaib Hassnain 3496d62335 security: require API-key auth on all Explorer API routes (GHSA-j4mq) (#909)
* security: require API-key auth on all Explorer API routes (GHSA-j4mq-hprp-987v)

Every Explorer route (bulk import/export, delete, LLM-backed ontology
generation, SPARQL, etc.) was mounted with no authentication, and both
server entrypoints bind 0.0.0.0 by default. Anyone reaching the port got
full read/write/delete on the graph.

- Add require_auth dependency (explorer/dependencies.py): checks
  X-API-Key against SEMANTICA_API_KEY, fails closed with 503 if
  unconfigured (not silently anonymous), 401 on wrong/missing key.
  SEMANTICA_ALLOW_ANONYMOUS=true opts out explicitly for local dev.
- Wire dependencies=[Depends(require_auth)] into all 11 API routers in
  both explorer/app.py and server.py. /health, /api/info, static assets,
  and the SPA catch-all stay public.
- /ws/graph-updates handshake now checks the same key via header or
  ?api_key= query param (browsers can't set custom WS headers) before
  accepting the connection.
- Default bind changed from 0.0.0.0 to 127.0.0.1 in server.py's main()
  and cli.py's `server start`; the CLI warns if a non-loopback host is
  passed explicitly without a key configured.
- Startup logging reports the resolved auth mode in both app factories.
- Document/generate SEMANTICA_API_KEY in the deploy recipes that expose
  a public endpoint by default: docker-compose, Railway, Fly, Render.

Added tests/explorer/test_explorer_auth.py covering fail-closed default,
wrong/missing/correct key, anonymous opt-in, public-route exemptions, and
the WS handshake. Added tests/explorer/conftest.py defaulting the
pre-existing ~200 explorer tests to SEMANTICA_ALLOW_ANONYMOUS=true so
they keep exercising route logic without needing a key.

* fix CORS

---------

Co-authored-by: Zohaib Hassnain <109234410+ZohaibHassan16@users.noreply.github.com>
2026-08-11 14:05:00 +05:00
pravit-ampandPravit Ampapathini 64f6c5cba2 test(split): cover untested chunker classes (#864) (#904)
* test(split): add coverage for untested chunker classes

* test(split): address Qodo gaps for chunker coverage

Cover exported KG/structural/sliding-window helpers, assert heading
boundaries, and use importorskip instead of mocking optional deps.

* fix(split): normalize sliding-window stride when omitted

* fix(split): pass entities to relation extraction and harden graph-based tests

---------

Co-authored-by: Pravit Ampapathini
2026-08-11 14:00:40 +05:00
pravit-ampandPravit Ampapathini ab5c12f9af fix(ingest): SSRF protection for Web and API ingestors (#867) (#906)
* fix(ingest): add SSRF protection for WebIngestor and RESTIngestor

Block non-http(s) schemes and private/loopback/link-local targets before
outbound requests, with allow_private_ips opt-in for trusted deployments.

* fix(ingest): fail closed on SSRF DNS resolution errors

* fix(ingest): validate SSRF targets on every HTTP redirect hop

* fix(ingest): avoid blocking on SSRF DNS executor shutdown

* fix(ingest): parse allow_private_ips without truthy-string pitfalls

* docs(ingest): clarify robots.txt SSRF/validation comment

---------

Co-authored-by: Pravit Ampapathini
2026-08-11 13:52:15 +05:00
Mohd Kaif fc9af2ebf8 Merge branch 'main' into security/fix-critical-vulnerabilities 2026-08-11 14:04:08 +05:30
KaifAhmad1 3e9ba1b7fb fix: address Qodo review findings on security PR (numpy/JSON, relative redirects, SPARQL 500)
- vector_store.save(): use v.tolist() instead of list(v) so numpy float32
  vectors round-trip through JSON instead of raising TypeError.
- ontology._fetch_url_sync(): resolve relative Location headers via urljoin
  before re-validating (previously any relative redirect was rejected
  outright), and close every response instead of leaking the connection
  across redirect hops.
- sparql.execute_sparql(): move _build_rdflib_graph inside the handler's
  error handling so the graph-size cap returns a clean SparqlResponse
  error instead of an unhandled 500.
- add regression tests for all three.
2026-08-11 14:01:07 +05:30
pravit-ampandPravit Ampapathini 51cf97765d test(deduplication): cover ClusterBuilder, MergeStrategyManager, and batch paths (#907)
* test(deduplication): cover ClusterBuilder, MergeStrategyManager, and batch paths
Add focused coverage for union-find clustering, property merge rules,
merge_duplicates, embedding similarity, and incremental detection (#866).

* test(deduplication): assert unrelated clusters without conditional skip
Make cluster-separation coverage fail closed by using mocked pairs and
unconditional assertions for distinct Apple vs Microsoft cluster IDs.

* test(deduplication): tighten update_clusters attachment assertions
Require the incremental path to place the new near-duplicate in the
same rebuilt cluster instead of accepting a vacuous cluster-count check.

* test(deduplication): strengthen incremental detect_duplicates wrapper checks
Assert real DuplicateCandidate matches, score threshold, and new×existing
routing instead of only checking that the wrapper returns a list.

* test(deduplication): verify metadata provenance behavior

Assert that preserve_provenance writes metadata.provenance fields and add a disabled-path test so regressions do not pass through merge_entities metadata alone.

---------

Co-authored-by: Pravit Ampapathini <pravitampapathini@users.noreply.github.com>
2026-08-11 12:52:15 +05:00
Sunil 8fa2037619 fix: re-push vector_store.py with correct UTF-8 encoding 2026-08-10 22:28:48 +05:30
Sunil 5573ab7a9f fix: re-push ontology.py with correct UTF-8 encoding 2026-08-10 22:28:27 +05:30
Sunil 26f236923c fix: re-push pyproject.toml with correct UTF-8 encoding 2026-08-10 22:28:06 +05:30
Sunil c35899711d fix: re-push sparql.py with correct UTF-8 encoding 2026-08-10 22:27:45 +05:30
Sunil 0a113b9702 fix: make defusedxml required, fail closed if missing (reviewer feedback) 2026-08-10 22:26:49 +05:30
Sunil 2de6ff898a Merge branch 'main' into security/fix-critical-vulnerabilities 2026-08-10 22:01:52 +05:30
Sunil 22ea189d0b security: replace unsafe pickle with JSON in vector store (CWE-502) 2026-08-10 21:53:11 +05:30
Sunil 30d5fef180 security: fix SSRF via redirect bypass in ontology URL fetcher (CWE-918) 2026-08-10 21:52:47 +05:30
Sunil c85df419ae security: add defusedxml to explorer dependencies for XXE protection 2026-08-10 21:52:03 +05:30
Sunil 55f3ee6f84 security: fix SPARQL DoS via unbounded graph materialization (CWE-770) 2026-08-10 21:51:56 +05:30
Sunil 924765b042 security: fix XXE vulnerability in RDF/XML parser (CWE-611) 2026-08-10 21:51:27 +05:30
Mohd Kaif 6f310d1d7a docs: link CONTRIBUTING.md issue workflow from PR template (#896)
Surfaces the comment-before-you-PR workflow from CONTRIBUTING.md
directly on the PR creation page to reduce duplicate PRs on the
same issue.
2026-08-10 21:18:01 +05:30
Sameer Kadam bd35d6031b docs: clarify contributor issue workflow (#895) 2026-08-10 19:45:54 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 53caefaf58 ci(deps): bump actions/attest-build-provenance (#880)
Bumps the github-actions group with 1 update: [actions/attest-build-provenance](https://github.com/actions/attest-build-provenance).


Updates `actions/attest-build-provenance` from 4.1.1 to 4.2.2
- [Release notes](https://github.com/actions/attest-build-provenance/releases)
- [Changelog](https://github.com/actions/attest-build-provenance/blob/main/RELEASE.md)
- [Commits](https://github.com/actions/attest-build-provenance/compare/0f67c3f4856b2e3261c31976d6725780e5e4c373...4d101475d8b20a2381f78447822ac1eab6504dd8)

---
updated-dependencies:
- dependency-name: actions/attest-build-provenance
  dependency-version: 4.2.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-10 16:18:08 +05:30
Mohd Kaif fc2083aa17 Merge pull request #854 from Sameer6305/fix/848-decision-context-persistent-backends
fix(vector_store): stop bypassing backend abstraction in build_decision_context/explain_decision/_filter_by_metadata (closes #848)
2026-08-10 11:32:25 +05:30
Mohd Kaif 1258edfe7f Merge branch 'main' into fix/848-decision-context-persistent-backends 2026-08-10 11:14:15 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 5048665d35 chore(deps): bump dompurify from 3.4.12 to 3.4.13 in /explorer (#872)
Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.4.12 to 3.4.13.
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.12...3.4.13)

---
updated-dependencies:
- dependency-name: dompurify
  dependency-version: 3.4.13
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-09 21:31:11 +05:30
Mohd Kaif 7dce9f1b69 Merge pull request #853 from Sameer6305/fix/845-standardize-search-vectors-output-schema
fix(vector-store): standardize search_vectors() output schema across backend implementations (#845)
2026-08-09 17:37:22 +05:30
Mohd Kaif 1b09f1ca5b Merge branch 'main' into fix/845-standardize-search-vectors-output-schema 2026-08-09 17:29:06 +05:30
KaifAhmad1 03ed4b94e9 fix(vector-store): preserve ranking for unbounded scores in Pinecone/Qdrant
The 1.0 / (1.0 + max(0.0, 1.0 - score)) normalization added in the last
commit clamped every raw score >= 1.0 to an identical 1.0, collapsing
result ranking for dot-product-metric indexes (unbounded), which cosine
(bounded to [-1, 1]) never exercised. Replaced with x/(1+|x|) rescaled
to (0, 1), which is strictly monotonic for any real score.

Also adds regression tests for scores >= 1 and a CHANGELOG entry.
2026-08-09 17:28:05 +05:30
SaurabhandKaifAhmad1 9059a44731 fix(vector-store): reconstruct FAISS vectors (#850)
* fix(vector-store): reconstruct FAISS vectors

* fix(vector-store): surface FAISS reconstruction failures

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-08-09 16:42:24 +05:30
Mohd Kaif e90bd048e1 Add Trendshift badge to README
Added Trendshift badge to README for repository tracking.
2026-08-08 21:37:59 +05:30
Sameer6305 8e0419c864 fixed qodo review
Adds similarity_unavailable marker and warning logs to build_decision_context and explain_decision when a persistent backend (like FAISS) fails to reconstruct a vector. Updates docstrings to explicitly state this degraded-path behavior and guarantees schema stability. Adds regression tests to test vector retrieval failure behavior via caplog and context assertions.
2026-08-08 13:16:50 +05:30
Sameer6305 0d51608547 fix(vector_store): stop bypassing backend abstraction in build_decision_context/explain_decision/_filter_by_metadata (closes #848)
- build_decision_context() and explain_decision(include_paths=True) both
  accessed self.vectors directly, which is only initialized for the
  inmemory backend, crashing with AttributeError on any persistent
  backend (FAISS, Qdrant, Pinecone, etc.). Replaced with self.get_vector()
  (#843's backend-agnostic accessor) + an is-not-None check — a verified
  1:1 behavioral equivalent for the old 'decision_id in self.vectors'
  guard on the inmemory path.
- Found a third, undocumented instance of the same bug during
  verification: _filter_by_metadata() also accessed self.metadata/
  self.vectors directly. Initial fix silently returned [] for persistent
  backends, which was itself a new silent-failure bug (indistinguishable
  from a genuine zero-match result). Reconciled to raise
  NotImplementedError instead, matching the established precedent from
  get_vector()/get_metadata() (#843) for 'backend exists but doesn't
  support this operation' — confirmed via full grep of all 7 backend
  wrapper classes that none currently implement filter_by_metadata,
  so this path was previously dead-code-masked-as-working.

Tests: 14 new tests across two rounds — inmemory behavioral equivalence,
real (non-mocked) FAISS backend regression tests for all three methods,
and explicit coverage proving the NotImplementedError fires with a clear
message rather than the old silent-[] behavior. Full suite: 53 passed,
0 failed, 0 regressions across the 39 pre-existing tests.
2026-08-08 12:57:42 +05:30
Mohd Kaif aa7b7fe525 Merge pull request #847 from Sameer6305/fix/843-vectorstore-persistent-backend-accessors
fix(vector-store): fix get_vector/get_metadata crash on persistent backends (#843)
2026-08-08 11:57:04 +05:30
KaifAhmad1 916d3974e3 fix(vector-store): guard save()/load() indexer access for persistent backends
self.indexer is only set for backend="inmemory", so save()/load() still
raised AttributeError for persistent backends (faiss, qdrant, etc.) even
after this PR's getattr() guards on self.vectors/self.metadata, since the
unguarded `self.indexer` access happened first. Guard it the same way and
delegate to the backend store's native save_index/load_index (currently
only FAISSStore implements these) so persistent-backend saves actually
persist instead of silently no-oping.
2026-08-08 11:50:09 +05:30
Sameer6305 f75469f472 Merge remote-tracking branch 'semantica-agi/main' into fix/843-vectorstore-persistent-backend-accessors 2026-08-07 22:23:39 +05:30
Sameer6305 40b81d0582 fixed qodo reviews: standardize search results schema, score metric, and ID types
- Removed total=False from SearchResult TypedDict so all fields are strictly required

- Ensured distance: None is returned from backends that don't natively expose distance (Qdrant, Pinecone, SQLite, pgvector, in-memory)

- Standardized search result score to a consistent 0.0 - 1.0 similarity metric scale across all backend adapters

- Relaxed SearchResult id type to Union[str, int] to accommodate native integer IDs from Milvus and Qdrant without casting

- Updated schema verification tests
2026-08-07 21:34:43 +05:30
Sameer6305 5db0adc18a fix(vector-store): standardize search_vectors output schema 2026-08-07 19:39:14 +05:30
Mohd Kaif 50758f6f25 Merge pull request #846 from SaurabhScripts/codex/fix-markdown-import-path-errors
fix(context): preserve Markdown import path errors
2026-08-07 16:49:24 +05:30
Saurabh 2756916573 Merge branch 'main' into codex/fix-markdown-import-path-errors 2026-08-07 16:17:17 +05:30
Mohd Kaif f47c730f7e Merge pull request #842 from Sameer6305/fix/839-decisionembeddingpipeline-backend-support
Fix #839: Support persistent backends in DecisionEmbeddingPipeline
2026-08-07 16:05:25 +05:30
KaifAhmad1 721a2f0e9c Fix candidate-embeddings loop dropping matches when pool exhausted
_get_candidate_embeddings()'s expand-and-retry loop widens the search
pool (up to limit*10) when post-filtering leaves too few candidates.
If the backend keeps returning a full page and filtered matches never
reach `limit`, the loop exited via the while condition instead of the
break branch, so the pre-loop empty embeddings/metadata/scores lists
were returned instead of the matches actually found in the final
iteration. This silently returned [] for filtered queries against
large persistent-backend stores even when matches existed - exactly
the scenario this PR adds support for.

Falls back to the last collected batch instead of discarding it.
Also documents this PR and #839 in the changelog.
2026-08-07 15:53:55 +05:30
Sameer6305 dd42b7fa95 fix(milvus): add backward compatibility alias and sanitize query
- Added insert_vectors alias to add_vectors for backward compatibility.
- Sanitized vector_id in get_vector and get_metadata to prevent query injection.
2026-08-07 12:11:48 +05:30
Sameer6305 248d028b09 fixed qodo reviews
- FAISSStore: get_metadata now correctly retrieves from self.metadata instead of raising NotImplementedError.
- MilvusStore:
  - Changed schema to support String IDs (VARCHAR) instead of auto-generated INT64, preventing loss of IDs during insert.
  - Added metadata storage using JSON.
  - Replaced insert_vectors with add_vectors accepting ids and metadata (added insert_vectors alias for backward compatibility).
  - Implemented get_vector and get_metadata with safe parameterized querying to prevent query injection.
- PgVectorStore & SQLiteVecStore:
  - Fixed get_vector and get_metadata to call self.get([vector_id]) instead of the non-existent get_vectors([vector_id]), fixing the silent None return bug.
2026-08-07 12:04:47 +05:30
Saurabh Meena 77a2ab7b18 fix(context): retain path inspection diagnostics 2026-08-07 11:27:52 +05:30
Sameer6305 c8b59b47f5 fix(vector-store): fix get_vector/get_metadata crash on persistent backends (#843)
- VectorStore.get_vector() and get_metadata() were hardcoded to access
  self.vectors and self.metadata dicts, which are only initialized for
  the inmemory backend, causing AttributeError on all persistent backends
  (FAISS, Qdrant, Pinecone, Milvus, Weaviate, PgVector, SQLiteVec).

Changes:
- Refactor VectorStore.get_vector() and get_metadata() to branch on
  self.backend == 'inmemory' (zero behavior change) and delegate to
  self._backend_store otherwise.
- Harden save() to use getattr(self, 'vectors', {}) / getattr(self,
  'metadata', {}) to prevent crash when saving a persistent backend store.
- Add get_vector() and get_metadata() to all 7 backend wrappers:
  - FAISSStore: get_vector uses index.reconstruct(); get_metadata raises
    NotImplementedError (FAISS has no metadata storage natively).
  - QdrantStore: uses client.retrieve() with with_vectors/with_payload.
  - PineconeStore: wraps existing fetch_vectors() call.
  - MilvusStore: raises NotImplementedError (auto_id=True schema discards
    string IDs at insert time, making by-ID lookup impossible in this
    wrapper's current schema).
  - WeaviateStore: uses collection.query.fetch_object_by_id().
  - PgVectorStore: wraps existing get_vectors() SQL method.
  - SQLiteVecStore: wraps existing get_vectors() SQL method.
- Add TestVectorStoreRetrieval regression tests covering inmemory and
  FAISS backends with real (non-mocked) assertions.

All 28 tests pass.
2026-08-07 11:21:42 +05:30
Saurabh Meena c0b6a80480 fix(context): preserve Markdown path errors 2026-08-07 01:15:55 +05:30
Sameer Kadam 36071819b5 Merge branch 'main' into fix/839-decisionembeddingpipeline-backend-support 2026-08-06 20:15:05 +05:30
Sameer6305 a4dac2342b fixed qodo reviews 2026-08-06 19:44:23 +05:30
Sameer6305 7d272f40e8 Fix #839: Support persistent backends in DecisionEmbeddingPipeline
- Replace direct .vectors and .metadata access with VectorStore.search_vectors().
- Add a fallback in HybridSimilarityCalculator (via ind_similar_decisions) to use the search score when backend vector databases do not natively return the raw vector array.
- Fix get_decision_statistics to gracefully fall back when .metadata is not fully supported by the underlying DB.
- Add regression tests utilizing the real FAISS and inmemory backends directly without mocking.
2026-08-06 19:11:35 +05:30
Mohd Kaif 3e5d2672ad Merge pull request #841 from divyankshah/fix/gh-840-qdrant-metadata-key
fix(vector_store): normalize QdrantStore.search_vectors() to return "metadata"
2026-08-06 19:10:42 +05:30
KaifAhmad1 b4b10a4928 docs(changelog): document Qdrant metadata key normalization
Adds an Unreleased/Fixed entry for #841 (closes #840) — QdrantStore
search results were keyed "payload" instead of "metadata", breaking
HybridSearch.filter_by_metadata() for Qdrant results.
2026-08-06 18:53:40 +05:30
Mohd Kaif 48c58a0753 Merge branch 'main' into fix/gh-840-qdrant-metadata-key 2026-08-06 17:26:02 +05:30
Mohd Kaif 6b143ef401 Merge pull request #838 from Linxiushen/feat/embedded-triplet-store
feat(triplet-store): add embedded Oxigraph backend
2026-08-06 16:33:13 +05:30
Mohd Kaif 49f458e927 Merge branch 'main' into feat/embedded-triplet-store 2026-08-06 16:22:56 +05:30
KaifAhmad1 c77184bd77 docs(changelog): document embedded Oxigraph backend and ImportError fix
Adds an Unreleased/Added entry for #838 (closes #834), including the
follow-up fix that preserves ImportError for a missing pyoxigraph
install instead of masking it as a generic ProcessingError.
2026-08-06 16:15:47 +05:30
Mohd Kaif fa77f5cc47 Merge pull request #836 from Sameer6305/fix/830-temporal-panel-render-loop
fix(explorer): resolve infinite render loop preventing Temporal panel from rendering (#830)
2026-08-06 13:24:17 +05:30
KaifAhmad1 7bddee0111 ci: update stale github/codeql-action v4 pin
Upstream moved the v4 tag to 5595ccaf912efad79be6eef63a5619ff05969be3
(v4.37.6), which the repo's own verify-action-pins.sh now (correctly)
flags as a mismatch against the previously-pinned commit. Pre-existing
drift unrelated to #830/#836, but it was failing this PR's required
"verify" check, so fixing it here.
2026-08-06 13:10:13 +05:30
KaifAhmad1 5cd4407e57 fix(explorer): review follow-ups for #830 render-loop fix
- Wire the Explorer frontend's node --test suites (test:graph-store,
  test:graph-workspace, and the new test:plugin-registry regression
  test) into CI. Previously only `npm run build` ran, so none of the
  frontend tests -- including this fix's own regression coverage --
  executed anywhere except a contributor's local machine.
- Broaden the diagnostics dedup's structureLayer comparison to also
  cover disabledReason/curveCount/bridgeCurveCount/backboneCurveCount,
  not just cacheKey/lastDrawAt/enabled, so a disabledReason-only
  transition doesn't leave the dev diagnostics panel stale.
2026-08-06 13:03:56 +05:30
KaifAhmad1 1850cdd617 Merge remote-tracking branch 'origin/main' into fix-830-followup
# Conflicts:
#	CHANGELOG.md
2026-08-06 13:03:28 +05:30
Mohd Kaif 5f00c00be3 Merge pull request #837 from semantica-agi/fix/833-hybridsearch-attributeerror-non-inmemory-backends
fix: HybridSearch.search() crashes with AttributeError on non-inmemor…
2026-08-06 12:06:18 +05:30
Mohd Kaif dee55112ef Merge branch 'main' into fix/833-hybridsearch-attributeerror-non-inmemory-backends 2026-08-06 11:59:30 +05:30
shah b7ac05b6f2 fix(vector_store): normalize QdrantStore.search_vectors() to return "metadata"
QdrantStore.search_vectors() returned results keyed by "payload" while
HybridSearch and PineconeStore both expect/return "metadata". This silently
dropped metadata from Qdrant results and caused HybridSearch.filter_by_metadata
to reject every candidate when a filter was applied (empty result sets).

Fixes #840
2026-08-06 02:33:50 +02:00
Mohd Kaif d9118410bc Merge pull request #829 from Sameer6305/feat/793-temporal-diff-ui
feat(explorer): add temporal diff comparison UI to the Temporal panel (#793)
2026-08-05 21:30:30 +05:30
Mohd Kaif d16db085d8 Merge branch 'main' into feat/793-temporal-diff-ui 2026-08-05 21:23:59 +05:30
Sameer6305 cb716cec61 Merge semantica-agi/main into fix/833-hybridsearch-attributeerror-non-inmemory-backends 2026-08-05 21:12:11 +05:30
Sameer6305 712a6e6d4c test(hybrid_search): add backend delegation regression coverage 2026-08-05 20:31:59 +05:30
林SO b52cdd5bbe fix(triplet-store): preserve missing backend dependency errors 2026-08-05 22:36:19 +08:00
Mohd KaifandSameer6305 d0e018a1c9 fix(vector_store): stop dropping metadata for add_vectors-only backends (#835)
* fix(vector_store): stop dropping metadata for add_vectors-only backends

VectorStore.store_vectors() previously discarded the metadata argument
whenever the backend only exposed add_vectors() (e.g. FAISSStore), even
though add_vectors() supports it. Now metadata is forwarded, and is only
passed when the backend's add_vectors() signature actually accepts it
(checked via inspect.signature), avoiding a TypeError for stricter
backend signatures.

Fixes #832

* fix(vector_store): guard signature introspection in store_vectors

inspect.signature() can raise ValueError/TypeError for some callables
(e.g. certain C-implemented or dynamically built methods). Wrap the
add_vectors() signature probe in try/except, consistent with the same
pattern already used in ProvenanceManager.trace_lineage(), defaulting
to attempting to pass metadata when introspection fails.

* docs(changelog): document VectorStore metadata-drop fix (#832, #835)

* test(vector_store): add regression coverage for metadata forwarding

---------

Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
2026-08-05 19:59:53 +05:30
Sameer6305 bd8d6c5913 docs: add #830 Explorer Temporal panel fix to CHANGELOG.md 2026-08-05 18:06:45 +05:30
Sameer6305 667e69a0c1 refactor: tighten comments across #830 changes for clarity
- pluginRegistryPredicates.ts: consolidate 8-line JSDoc to 5 lines,
  removing redundant detail that restated implementation mechanics
  already obvious from the code.

- GraphWorkspace.tsx: shorten the lastScrubberMsRef comment from 5 lines
  to 2; trim the handleDiagnosticsChange block comment by removing the
  'rather than bailing out' implementation-alternative sentence; tighten
  the distanceVisual inline comment.

- pluginRegistry.temporal.test.mjs: replace 17-line file-level JSDoc
  with 9 lines focused on the invariant rather than the root-cause
  narrative (already covered in pluginRegistryPredicates.ts); remove
  two tsx loader implementation-detail comments; tighten two test-level
  inline comments.

No logic, types, or test assertions changed. All 42 tests pass.
2026-08-05 17:52:21 +05:30
KaifAhmad1 9c7dd16126 docs: add CHANGELOG entry for HybridSearch AttributeError fix (#833, #837) 2026-08-05 17:45:13 +05:30
KaifAhmad1 94adcf7ad3 fix: address code review findings on backend-delegated search path
- Legacy top_k kwarg was read but not removed from options, so it got
  forwarded via **options into VectorStore.search_vectors(), colliding
  with backends that call search(..., top_k=k, **options) (e.g. sqlite,
  pgvector) and raising "got multiple values for keyword argument
  'top_k'". Now popped instead of just read.
- VectorStore.search_vectors()'s dispatch only recognized backend
  methods named search/search_similar, so HybridSearch's delegation
  still hit NotImplementedError for qdrant/milvus/pinecone, which name
  their method search_vectors() with a differently-named count
  parameter (limit vs k). Added a third dispatch branch that binds the
  count positionally so it works regardless of the backend's parameter
  name.
- Backend-delegated results defaulted a missing "distance" to the raw
  score, silently reusing the local path's cosine-similarity convention
  (distance = 1 - score) even for backends using unrelated metrics
  (L2, inner product). A missing distance is now left as None instead
  of a fabricated, metric-inconsistent value.
2026-08-05 17:41:31 +05:30
Sameer6305 80de3652cf fixed qodo findings
Two issues addressed:

1. Plugin-loading useEffect unnecessarily depended on temporalState.
   After the #830 fix, no shouldLoad predicate reads temporalState, but
   the effect's dep array still included it, causing extra re-runs on
   every scrubber update. Removed temporalState from the dep array and
   the shouldLoad call site. Made temporalState optional in the
   LazyPluginRegistryEntry shouldLoad context type to match.

2. Regression test imported a local copy of shouldLoad instead of the
   production predicate. Extracted all three shouldLoad predicates into
   pluginRegistryPredicates.ts (pure module, no React/DOM dependencies),
   wired GraphWorkspace.tsx to use the imported functions, and updated
   the test to import and exercise the real production code via tsx.
   Verified: introducing the old broken condition causes the test to fail;
   the correct implementation passes all 7 assertions.
2026-08-05 17:37:33 +05:30
林SO 0e1b88a593 feat(triplet-store): add embedded Oxigraph backend 2026-08-05 20:06:20 +08:00
KaifAhmad1 b4f820568a fix: HybridSearch.search() crashes with AttributeError on non-inmemory backends
HybridSearch.search() directly accessed self.vector_store.vectors, a dict
that VectorStore only creates for backend="inmemory". Every other backend
(faiss, weaviate, qdrant, milvus, pinecone, pgvector, sqlite) raised
AttributeError. It now delegates to VectorStore.search_vectors() for
non-inmemory backends, applying metadata_filter as a post-filter and
normalizing results to a consistent {id, score, distance, metadata} shape.

Also fixes two related bugs surfaced while testing the backend-delegated
path end to end:
- vector_ids could remain None when explicit vectors/metadata were passed
  without vector_ids, crashing downstream indexing.
- query_vector passed as a plain list crashed backend stores (e.g.
  FAISSStore.search_similar) that call .ndim on it; now normalized to a
  numpy array up front.

And in vector_store.py: VectorStore.store_vectors() silently dropped
metadata for FAISS (and any add_vectors-only backend) because it called
add_vectors(vectors, **options) without forwarding metadata, even though
FAISSStore.add_vectors() accepts it. This blocked HybridSearch's metadata
filtering from working at all against FAISS.

Fixes #833
2026-08-05 17:16:56 +05:30
Sameer6305 8d52281cdf chore(explorer): clean up #830 branch — remove #793 file, add regression test
temporalDiffState.ts belongs to feat/793-temporal-diff-ui and should not
appear in the #830 diff. Remove it from this branch's tracked files.

Add the pluginRegistry.temporal.test.mjs regression test that covers the
shouldLoad fix committed in the main #830 commit (it was never committed).

Add test:plugin-registry script to package.json so the regression test
can be run via npm run test:plugin-registry.
2026-08-05 16:26:24 +05:30
Sameer6305 6a0eecbe02 fix(explorer): resolve #830 — Maximum update depth exceeded on Temporal panel open
Two independent render loops were causing the Temporal panel to remain
stuck on 'Loading temporal...' in npm run dev:

Loop 1 — diagnostics state churn (GraphWorkspace.tsx):
  handleDiagnosticsChange unconditionally called setGraphDiagnosticsState
  with a new object on every invocation. buildEffectAvailability (called
  inside GraphCanvas's diagnostics useEffect) always returns a new object,
  so setGraphDiagnosticsState was called on every effect run, creating a
  cycle: setGraphDiagnosticsState  graphDiagnosticsState new
  diagnosticsSnapshot new  pluginContext new  handleInteractionStateChange
  new  GraphCanvas re-renders  diagnostics effect fires again.

  Fix: before calling setGraphDiagnosticsState, compare the incoming
  diagnostics field-by-field against the last accepted snapshot via a ref
  (lastDiagnosticsRef). All effectAvailability entries, edgeClasses.updatedAt,
  structureLayer.cacheKey/lastDrawAt/enabled, and distanceVisual identity
  must differ for a state update to proceed. The ref approach avoids
  scheduling a re-render at all, rather than bailing out inside a functional
  updater after the render has already been committed.

Loop 2 — scrubberTime churn (GraphWorkspace.tsx + GraphWorkspaceShell.tsx):
  TimelinePanel.tsx calls onTimeChange(defaultTime) whenever its useEffect
  re-runs. React 18 concurrent mode re-runs effects with structurally-new
  Date objects for the same timestamp when speculative renders discard
  useMemo caches, causing setScrubberTime to be called repeatedly with a
  new Date that has the same millisecond value — triggering temporalState
  churn, the diagnostics effect, and eventually the same loop.

  Fix: wrap setScrubberTime in an onTimeChange useCallback that compares the
  incoming time's millisecond value against the last sent value (via
  lastScrubberMsRef). Redundant calls with the same timestamp are dropped
  before reaching setScrubberTime. Stable useCallback identity also prevents
  TimelinePanel's useEffect from re-firing solely due to prop identity churn.

Both fixes applied to GraphWorkspace.tsx and identically to
GraphWorkspaceShell.tsx which has the same pattern.

Verified:
- npm run dev: 0 'Maximum update depth exceeded' errors
- Temporal panel renders with real data in dev mode
- Effects and Neighbors panels unaffected
- npm run build + preview: identical behavior, 0 errors
- All 42 frontend tests pass (34 graph-workspace, 1 graph-store, 7 plugin-registry)
2026-08-05 16:01:18 +05:30
Sameer6305 aa85535d47 fixed copilot review 2026-08-04 16:45:33 +05:30
Sameer6305 7d936d0f7c fix(explorer): write diff highlights to displayGraph as well as store graph
fixed qodo review

applyDiffHighlight/clearDiffHighlight were writing baseColor only to
graphStore.graph (the store singleton), but Sigma is constructed with
displayGraphRef.current and the nodeReducer reads attributes from that
instance. When the display graph is a derived copy (aggregated,
focused, or grouped view), the store write has no effect on the
currently-rendered frame -- sigma.scheduleRefresh() flushes the
reducer over the display graph, which did not receive the mutation.

Fix: introduce writeBaseColor(context, nodeId, color) which writes to
BOTH the store graph (so the color propagates into the next display
graph rebuild via aggregateDisplayGraph's shallow attribute copy) AND
context.displayGraph (the live Graph instance currently bound to
Sigma, so the change is visible in the current frame immediately).

The dg !== graph guard skips the display-graph write when they happen
to be the same object (non-aggregated full view), avoiding a redundant
double-write in that case.

Original baseColor is still captured from the store graph (the
authoritative source, since aggregateDisplayGraph copies from there),
so restore remains correct across all view modes.
2026-08-04 16:32:04 +05:30
Sameer6305 47531d8365 feat(explorer): add temporal diff comparison to the Temporal panel
Adds a Compare section to the existing Temporal Context panel
(temporalOverlayPlugin.tsx) that lets a user pick two ISO timestamps
and diff the graph's node set between them via the existing, previously
UI-less GET /api/temporal/diff backend route.

- New temporalDiffState.ts: typed fetch wrapper (fetchTemporalDiff)
  matching the route's added_nodes/removed_nodes response shape.
- Diff results recolor affected nodes via baseColor (not
  ringColor/haloColor -- traced and confirmed those are only read by
  the sigma reducer for hovered/selected/path-state nodes and are
  silently discarded for default-state nodes).
- Validates both timestamps are present, parseable, and from < to
  before firing a request.
- Distinct idle/loading/error/empty/success states -- an empty diff
  (no changes) is rendered as its own state, not as an error.
- Cancels any in-flight request via AbortController on re-submission
  and on unmount; restores each highlighted node's original baseColor
  (captured before overwrite, not cleared to a fallback default) on
  both paths.
- Reuses existing theme tokens (GRAPH_THEME.palette.semantic[2],
  ui.control.dangerText) and existing button/input/loading/error
  visual patterns already established in this same plugins directory
  and in GraphInspectorPanel.tsx, rather than introducing new styling.
2026-08-04 15:57:53 +05:30
Mohd Kaif 86f115d200 docs: surface pip install command at the top of README and docs (#828)
Makes the install command the first actionable thing visible on both
the README and docs landing page, ahead of the fold.
2026-08-04 12:55:51 +05:30
Mohd Kaif 9c5c3c4ce0 Merge pull request #826 from Sameer6305/fix/785-provenance-storage-failure-tests
test(provenance): expand storage failure regression coverage (#785)
2026-08-04 12:13:41 +05:30
Mohd Kaif 26a5c4a1fb Merge branch 'main' into fix/785-provenance-storage-failure-tests 2026-08-04 12:08:23 +05:30
Mohd Kaif 2adc67e25e Merge pull request #827 from semantica-agi/feat/825-provenance-prov-o-compliance
Provenance: close PROV-O compliance gaps and high-stakes trust blockers
2026-08-04 11:33:42 +05:30
Sameer6305 e9e05fedbd fix(provenance): reset in-memory chain state on clear 2026-08-04 00:01:50 +05:30
KaifAhmad1 0a8330cbb0 fix(provenance): address code review findings on PR #827
- SQLiteStorage now migrates an existing (pre-#825) provenance.db in place
  via ALTER TABLE ADD COLUMN for any columns introduced since, instead of
  only ever running CREATE TABLE IF NOT EXISTS. Without this, opening an
  older database with the new code would break on the first insert/select
  since the row width and _row_to_entry's fixed indices grew past the old
  schema. Added test_migrates_pre_existing_old_schema_database.

- verify_chain() now also checks that sequence_id is exactly the
  predecessor's plus one (no gap, no duplicate), in addition to the existing
  previous_checksum comparison. Hardens against the narrow case where
  compute_checksum()'s deliberate exclusion of entity_id could let two
  distinct rows coincidentally share a checksum, which alone would let a
  checksum-only comparison miss a gap. Added
  test_verify_chain_detects_tampered_sequence_gap.

- Explorer provenance route: edge ids now include direction
  (f"{src}-{eid}-{direction}") to match the seen_edges dedupe key, which
  already included it. The same (src, target) pair can legitimately appear
  in both the upstream and downstream chains (cycles/overlap), and without
  this the two edges collided on the same id. Added
  test_add_chain_edges_ids_distinguish_direction.

- Removed an unused `Any` import in parse_provenance.py.
2026-08-03 22:52:34 +05:30
KaifAhmad1 db4361ad46 feat(provenance): close PROV-O compliance gaps and high-stakes trust blockers (closes #825)
Part A - high-stakes trust blockers:
- Invalidation tombstones via ProvenanceManager.invalidate() (archive-then-append,
  never mutates or deletes) instead of hard delete
- Hash-chained integrity: sequence_id/previous_checksum chain every entry to its
  predecessor; new verify_chain() detects wholesale row deletion that a lone
  per-row checksum cannot
- Typed Agent (AgentRecord: agent_type/is_automated) and Activity (ActivityRecord:
  start/end timing), wired through all 18 *_provenance.py wrappers
- Split parent_entity_id into previous_version_id (correction) vs derived_from_id
  (cross-source derivation), additive alongside the legacy combined field
- Downstream/descendant lineage traversal (get_descendants/trace_descendants,
  reverse BFS) closing the dead direction="downstream" code path in the
  Explorer's provenance route
- Qualified Association+hadRole and Invalidation in export_prov()
- New CLI: provenance invalidate|verify-chain|descendants

Part B - general PROV-O spec completeness:
- Qualified Generation/Usage/Derivation in export_prov()
- wasAssociatedWith, actedOnBehalfOf, wasInformedBy relations
- Bitemporal fields (valid_from/valid_until/revision_type/supersedes) plus
  revision_history()/query_recorded_between(), closing the deprecated
  kg.ProvenanceTracker's "no direct equivalent yet" migration gaps
- prov:Bundle/hadMember membership via bundle_id
- Configurable base_uri (--base-uri CLI flag), shared by RDFExporter's
  NamespaceManager and OWLExporter's default ontology_uri so KG/OWL/PROV
  exports co-resolve under one namespace instead of three hardcoded ones

Bugs fixed along the way:
- agent_id was a dead field: no track_* method read it from kwargs
- track_entities_batch silently absorbed typed kwargs into the metadata blob
- compute_checksum() had to exclude entity_id itself: hashing it made
  track_entity's versioning-archive relabel permanently orphan any entry
  already chained from the pre-relabel checksum, a false-positive "broken
  chain" for a legitimate rename
- InMemoryStorage.get_chain_head() ignored the committed head whenever the
  current transaction had staged entries, corrupting the next chain link
- several new ProvenanceEntry fields were wired into the dataclass and
  export_prov() but not into SQLiteStorage's DDL/INSERT/row-mapping;
  InMemoryStorage masked the gap. Added a permanent round-trip regression
  test to catch this class of bug for future field additions

Flagged, not fixed (separate pre-existing issues, out of scope for #825):
- pipeline/pipeline_provenance.py imports a nonexistent module and wraps a
  Pipeline dataclass with no run() method
- most *_provenance.py wrappers' backing classes are themselves missing or
  incomplete (context_manager, deduplicator, normalizer, etc.)
- kg_provenance.py passes entity_type inside metadata={} instead of as a
  top-level track_entity() kwarg across most of its call sites
2026-08-03 22:28:48 +05:30
Sameer6305 74c093facb fixed issues from qodo and copilot 2026-08-03 21:37:26 +05:30
Sameer6305 aae4c946ea test(provenance): expand storage failure regression coverage (#785) 2026-08-03 21:05:26 +05:30
Mohd KaifandSameer6305 b59211ea7f security: SHA-pin all Actions, harden release pipeline, add pin verification (#824)
* security: SHA-pin all Actions, harden release pipeline, add pin verification

Hardens the CI/CD supply chain against the LiteLLM/Trivy-style attack (a
compromised third-party Action with a mutable tag stealing a long-lived
publishing token) and closes several related gaps found in an audit of the
actual repository state.

- Pin every third-party GitHub Action across all workflows to a full commit
  SHA (tag kept as a trailing comment); add verify-action-pins.yml, a CI
  check that confirms via the GitHub API that each pin still matches its
  tag, on every workflow change, push to main, and weekly.
- Scope release.yml permissions to the job level (workflow defaults to
  contents: read); add a concurrency group so simultaneous tag pushes can't
  race the publish job.
- Add SLSA build provenance attestation (actions/attest-build-provenance)
  for every released wheel.
- Fix a latent bug in security-scan.yml: the PR-comment step was missing
  pull-requests: write and silently failing; add bounded artifact retention
  for uploaded scan reports.
- Group github-actions Dependabot updates to cut review noise.
- Document the resulting posture in SECURITY.md for auditors/regulated
  adopters, including what's enforced and what a fork needs to reconfigure
  for itself (environment/branch protection, Trusted Publishing trust).

Also (via GitHub API, not in this diff): created a protected `pypi`
environment with a required reviewer restricted to v* tags, and enabled
branch protection on main (required review, required status checks, no
force-push/deletion).

* fix: harden verify-action-pins per PR #824 bot review

Addresses real findings from the automated review on #824:

- The script previously only matched uses: lines that already contained a
  40-hex SHA, so a newly added mutable-tag action (e.g. some/action@v1)
  would never be scanned at all and the check would pass silently. It now
  matches every uses: line and hard-fails on any ref that isn't a full
  commit SHA.
- A tag that fails to resolve via the GitHub API (rate limit, deleted tag)
  previously only logged a warning and continued; that's now a hard
  failure too, since an unverifiable pin is exactly the failure mode this
  check exists to catch.
- verify-action-pins.yml only triggered on .github/workflows/** changes,
  so an edit to the verifier script itself wouldn't run the check that
  verifies it. Added the script path to both trigger filters.

The reviewer's claim that slash-containing tag comments (release/v1) break
the API lookup did not reproduce - tested directly against
pypa/gh-action-pypi-publish@release/v1 and GitHub's commits API resolves
multi-segment refs natively - so no change was needed there.

Verified with a synthetic test workflow containing a mutable-tag action,
a correctly-pinned SHA, and a deliberately mismatched SHA: the updated
script now catches the first and third cases and passes the second. Also
re-ran against the real workflow tree (40/40 pins still verify clean).

* fix: repair broken Safety scan and PR comment formatting

The "Comment PR with Security Results" step was producing garbled output
(literal \n characters instead of newlines, "undefined:" labels) because:

- Every line in the JS comment builder used \n (escaped backslash-n)
  inside template literals, which JS renders as the literal two-character
  string \n, not a newline.
- The Semgrep section read issue.rule_id, but Semgrep's JSON field is
  check_id - hence "undefined: <path>" for every entry.

Rewrote the comment builder to construct each section as an array of
lines joined with a real '\n', with correct field names, and collapsed
long finding lists into a <details> block instead of a flat list.
Verified by extracting the exact script and running it under node against
synthetic fixtures matching each tool's real JSON schema (found/clean/
missing-report paths all render correctly).

While tracing the "undefined" and always-empty Safety section, found the
Safety step itself was silently broken:

- `safety check --json --output safety-report.json` is invalid in
  Safety 3.x: --output now selects a console format (json/text/screen),
  not a file path. The command errored on every run (swallowed by
  `|| true`), so safety-report.json was never created and the PR comment
  always fell back to a generic "scan completed" message. Switched to
  `--save-json`, which is the correct flag for writing a JSON report to
  disk, and confirmed against the real safety 3.8.1 CLI locally.
- Even with a report, the code read vuln.package - the real field is
  package_name.
- The job never installed Semantica's own dependencies before scanning,
  so `safety check` (which defaults to scanning the environment) was
  auditing the scanner tools' own dependencies, not Semantica's. Added
  `pip install -e ".[llm-litellm]"` so the project's actual dependency
  tree - including the LiteLLM extra this whole hardening effort is
  about - is what gets scanned.

Also updated the corresponding SECURITY.md bullet to describe what Safety
actually covers now.

* fix: remove unused pypdf2 dependency (CVE-2023-36464)

Now that the Safety scan step actually runs (see previous commit), it
correctly failed this PR's checks on CVE-2023-36464 in pypdf2==3.0.1 - a
real, pre-existing vulnerability that was invisible until the scan was
fixed.

PyPDF2 is not a patchable dependency here: the project is discontinued
(merged into `pypdf`), 3.0.1 is its final release, and there is no fixed
version to upgrade to. Grepping the repo for `import PyPDF2` / `from
PyPDF2` turns up nothing - it was never actually imported anywhere. Its
only presence outside pyproject.toml was in docstrings describing a
"PyPDF2.PdfReader() fallback" for PDF parsing that was never implemented
in code; pdfplumber is the library actually used. Removed the dependency
and corrected the stale docstrings in parse/__init__.py, parse/methods.py,
parse/pdf_parser.py, and ingest/email_ingestor.py accordingly.

* fix: suppress Bandit B324 false positives on non-cryptographic MD5 use

Same pattern as the previous pypdf2 commit: fixing the Safety scan
surfaced this PR's own Bandit HIGH-severity gate actually blocking on 10
pre-existing findings, all Bandit B324 ("Use of weak MD5 hash for
security").

Checked each of the 10 call sites: every one uses hashlib.md5() to build
a short deterministic cache key, entity ID, or IRI suffix from already-
non-secret input (query text, entity text/type, class/property names) -
none are used for passwords, tokens, or integrity verification of
untrusted data. This is exactly the case Bandit's own message points at
("Consider usedforsecurity=False").

Did not use usedforsecurity=False itself: that keyword argument was
added to hashlib in Python 3.9, and pyproject.toml declares
`requires-python = ">=3.8"` - adding it unconditionally risks a TypeError
on 3.8. Used a targeted `# nosec B324` comment with a one-line
justification instead, which suppresses only this specific check and
carries no runtime behavior change on any supported Python version.

Verified locally: bandit -r semantica/ -ll now reports 0 HIGH-severity
findings (was 10).

* docs: add CHANGELOG entry for #824 CI/CD supply-chain hardening

Covers the SHA-pinning + verify-action-pins.yml enforcement, release.yml
hardening (job-scoped permissions, concurrency, SLSA provenance), the
pypi environment/branch protection GitHub-side config, the
security-scan.yml Safety/comment-formatting fixes, and the two
vulnerabilities those fixes surfaced (pypdf2 CVE-2023-36464 removal,
Bandit B324 suppression).

* fix: close two remaining gaps missed by upstream bot-review fixes

verify-action-pins.sh:
- Quoted uses: lines (e.g. uses: owner/action@SHA) were not matched
  by the existing regex, so a SHA-pinned action written with quotes would
  silently skip verification. Updated the main ERE to accept an optional
  leading/trailing single or double quote around the owner/action@ref
  value, and excluded quote chars from the inner character classes so the
  ref is still extracted cleanly.
- The grep input glob only covered *.yml. GitHub also treats *.yaml as a
  valid workflow extension. Added *.yaml to the glob and a 2>/dev/null
  guard so the command doesn't fail when no *.yaml files exist.

security-scan.yml (on top of Kaif's --save-json fix in 67c7ec2a):
- Kaif's fix kept the '|| echo 0' fallback on the VULNS= line, so all
  five scanner-failure modes (file missing, empty file, malformed JSON,
  valid JSON with no 'vulnerabilities' key, vulnerabilities: null) still
  silently produce VULNS=0 or VULNS=null and pass the merge-blocker check.
- Added guard 1: '[ ! -s safety-report.json ]' fails loudly if Safety
  crashed before writing a report (covers missing and empty-file cases).
- Dropped the '|| echo 0' fallback and added guard 2: '[[ ! VULNS =~
  ^[0-9]+$ ]]' fails loudly on non-integer VULNS (covers malformed JSON,
  missing key, and null cases). Both guards emit ::error:: annotations.
- Verified with a 7-case simulation: all 5 failure modes now exit 1;
  genuine zero-vuln and real-vuln cases still behave correctly.

* fix: correct bash [[ =~ ]] quoting that broke verify-action-pins.sh in CI

The regex for matching uses: lines was embedded directly inline in a
[[ =~ ]] test with literal \" and \' escape sequences. Bash's conditional-
expression parser interprets these as shell syntax rather than regex
literals, producing:

  syntax error in conditional expression: unexpected token ')'

at line 27 on every CI run.

Fix: move the regex into a USES_PATTERN variable using safe single-quote
shell-string concatenation so the [[ =~ ]] parser receives an unquoted
variable reference ($USES_PATTERN) rather than a literal pattern containing
bash-special characters. The regex semantics are identical: optional
leading/trailing quote around owner/action@ref, quote chars excluded from
capture groups.

Verified in real bash 5.2.21 (Git for Windows):
  No syntax error on the real 40-pin workflow tree (Checked 40)
  Unquoted SHA pin:      MATCH, correct repo+ref extracted
  Double-quoted SHA pin: MATCH, correct repo+ref extracted
  Single-quoted SHA pin: MATCH, correct repo+ref extracted
  .yaml extension file:  MATCH, correct repo+ref extracted
  ./local-action:        NO MATCH (correct)
  docker://:             NO MATCH (correct)

* docs: add 3 missing items to fork-reconfiguration checklist in SECURITY.md

The checklist covered Trusted Publishing trust, protected environment,
branch protection, and Dependabot github-actions entry. Three non-forking
controls described elsewhere in SECURITY.md were omitted:

- GitHub secret scanning and push protection (repo settings, not copied
  on fork)
- GitGuardian (GitHub App installation scoped to this specific repo,
  requires separate install on any fork)
- CodeQL Default Setup vs Advanced Setup state (repo setting that affects
  whether the upload-sarif step in codeql.yml does anything)

Added as items 5, 6, 7 matching the existing numbered bullet style.

* fix: update github/codeql-action pins to v4 tip (SHA drift caught by verify check)

verify-action-pins caught that github/codeql-action@v4 tag was re-pointed
upstream:

  old: f205ea1c3313d32999d8d6a48b4f6530d4437b38
  new: d1ba80a13dd99fba24a470575428917156a28b43

Updated all 8 occurrences across codeql.yml (init x3, autobuild, analyze,
upload-sarif) and defender-for-devops.yml (upload-sarif x2). Tag comment
# v4 unchanged — the tag itself hasn't changed, only what commit it points to.

---------

Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
2026-08-03 19:17:10 +05:30
Mohd Kaif c7c7250d88 Merge pull request #823 from Sameer6305/fix/775-ontology-atomic-writes
fix(explorer): complete atomic ontology refresh writes - #775
2026-08-03 13:40:08 +05:30
Mohd Kaif 21365cb0e6 Merge branch 'main' into fix/775-ontology-atomic-writes 2026-08-03 13:13:07 +05:30
Sameer KadamandKaifAhmad1 76edaeb1c0 fix(agno): make _eval_rule raise instead of silently returning compliant=True on unevaluable rules (closes #778) (#822)
* fix(agno): surface unevaluable policy rules (#778)

* fix(agno): fixed qodo reviews

check_policy previously let unevaluable policy rules silently return
compliant=True with no signal (issue #778): a rule referencing a field
missing from decision_data, or a rule string not matching the expected
<field> <op> <value> format, both fell through _eval_rule's `return
True` and were treated as passed.

Both now raise ValueError, which routes through check_policy's existing
exception handler and surfaces as a `warnings` entry instead. compliant/
violations semantics are unchanged for every case that previously worked
correctly; an unevaluable rule is not counted as a violation since it's
genuinely unknown whether it would have passed.

Follow-up fixes from code review:
- policy_rules decoded via json.loads without checking it was a list;
  a JSON-encoded bare string decoded to a Python str, so iterating it
  evaluated one "rule" per character, amplifying a single input-shape
  mistake into a wall of per-character warnings. A decoded string is
  now treated as a single rule; any other non-list shape or non-string
  list element produces exactly one warning instead.
- _eval_rule used `data.get(field) is None` to detect a missing field,
  which can't distinguish an absent key from a key present with JSON
  null - both produced the same "undefined field" warning. Field
  presence is now checked with `field not in data` first, and a
  present-but-null value gets its own distinct message.

Added regression tests for all of the above in
tests/integrations/agno/test_decision_kit.py (38 tests in the file,
128 passing across tests/integrations/agno/).

* fix(agno): reject non-object decision_data in check_policy

check_policy only validated that decision_data was well-formed JSON,
not that it decoded to an object. When it decoded to a list, `field
not in data` in _eval_rule silently became list-membership testing
of values instead of a dict key check - e.g. "confidence" not in
["confidence", 0.95] evaluates to False - so a matching rule fell
through to data["confidence"], raising a raw internal TypeError
("list indices must be integers or slices, not str") instead of any
meaningful diagnostic. Numbers, strings, and bools produced similarly
opaque TypeErrors deep inside _eval_rule.

check_policy now checks isinstance(data, dict) right after decoding
and rejects any other shape with a single clear violations entry,
the same way it already rejects malformed JSON.

Added 5 regression tests in tests/integrations/agno/test_decision_kit.py
covering list/number/string/bool/null decision_data shapes (43 tests
in the file, 133 passing across tests/integrations/agno/).

Addresses Copilot PR review comment on the #778 fix branch.

* docs(changelog): reference PR #822 in the check_policy changelog entry

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-08-03 12:23:49 +05:30
Mohd Kaif 1ad00075a3 Merge pull request #821 from Sameer6305/fix/779-record-decision-logging
fix(agno): log shared context decision tracking failures - #779
2026-08-02 13:19:12 +05:30
KaifAhmad1 46dcbbe731 Merge remote-tracking branch 'origin/main' into fix/779-record-decision-logging
# Conflicts:
#	CHANGELOG.md
2026-08-02 13:10:20 +05:30
Mohd KaifandKaifAhmad1 0d447560bc fix(provenance): log tracking failures and return None on storage error (closes #783) (#820)
* fix(provenance): log tracking failures and return None on storage error (closes #783)

* docs(provenance): document Optional return types and failure behavior (#783)

* fixed qodo reviews

- split.ProvenanceTracker: Check _unified_manager.track_chunk() return value and fall back to legacy storage when None is returned (storage failure)

- split.ProvenanceTracker: Initialize legacy stores (_provenance_store and _chunk_registry) unconditionally in __init__ so legacy fallback storage works safely even when initialized with use_unified=True

- split.ProvenanceTracker: Update get_provenance() to check legacy storage when no record is found in unified storage, ensuring fallback-tracked chunks remain retrievable

- SourceTracker: Change track_sources_batch() condition from if entry is not None: to if ok: to respect the boolean return contract (-> bool) of track_entity_source(), track_property_source(), and track_relationship_source()

- Tests: Add regression test test_unified_tracking_returns_none_triggers_fallback and update test_track_sources_batch_failure_not_counted to test boolean failure return_value=False

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-08-01 20:52:55 +05:30
KaifAhmad1 5094235ce1 Merge branch 'main' into fix/783-tracking-methods-honest-failures
Resolves CHANGELOG.md conflict with #819's SKOS cycle-detection entry
by keeping both entries.
2026-08-01 20:43:53 +05:30
Sameer6305 6f6c825f3d fixed qodo reviews
- Removed unused `validate_skos_hierarchy` import from
  test_ontology_subissue3.py (flake8 F401); the test uses a `wraps=` spy
  on the real add_nodes_and_edges instead of calling the helper directly.
- refresh_ontology tests now percent-encode the ontology URI with
  urllib.parse.quote before interpolating it into the {ontology_uri:path}
  request path, matching the already-encoded unknown-uri refresh test in
  the same file instead of embedding a raw http://... URI with slashes.
- Reworded the cyclic-SKOS refresh test's comment and section header:
  GraphSession.add_nodes_and_edges() documents pre-write validation and
  lock-based mutual exclusion, not transactional rollback, so "atomic"
  was replaced with "single combined add_nodes_and_edges() call" to avoid
  implying rollback guarantees that don't exist.

Verified: tests/explorer/test_ontology_subissue3.py (34 passed) and
tests/explorer/ (204 passed), no regressions.
2026-08-01 19:36:16 +05:30
Sameer6305 d293ca6009 fix(explorer): complete atomic ontology refresh writes (#775) 2026-08-01 19:12:00 +05:30
Mohd Kaif 352db64b33 Merge pull request #819 from mikemikimike/agent/validate-skos-cycles
Reject cyclic SKOS hierarchies at write time
2026-08-01 12:02:46 +05:30
KaifAhmad1andmikemikimike bc75768afe Fix two review findings in SKOS cycle validation
- validate_skos_hierarchy() re-walked every existing hierarchy edge in
  the graph on each write, so one pre-existing cycle anywhere would
  block all unrelated future SKOS writes. It now only traverses
  concepts touched by the edges actually being written, while still
  checking those against existing edges for cross-boundary cycles.
- In /api/ontology/load, `except HTTPException: raise` sat after a
  broader `except Exception` clause that already matched HTTPException,
  so a 422 raised after a successful OntologyIngestor parse was
  silently swallowed and retried via the fallback RDF parser instead of
  reaching the caller. Reordered the except clauses.

Co-authored-by: mikemikimike <13286568797@163.com>
2026-08-01 11:46:13 +05:30
mikemikimike f992504227 Make SKOS hierarchy imports atomic 2026-07-31 23:23:10 +05:30
mikemikimike d41530930d Centralize SKOS cycle validation 2026-07-31 23:07:58 +05:30
mikemikimike 692260cc76 Reject cyclic SKOS hierarchies 2026-07-31 23:07:58 +05:30
Sameer6305 aa58b46d4d Merge semantica-agi/main into fix/779-record-decision-logging 2026-07-31 18:28:20 +05:30
Sameer6305 633d485045 fixed qodo review
- Added exc_info=True to both store failed and record_decision failed warning logs in _AgentScopedStore.upsert_memory() to preserve full traceback context for debugging

- Updated CHANGELOG.md entry to document traceback preservation
2026-07-31 18:19:50 +05:30
Mohd Kaif 424b63a27d Merge pull request #818 from Sameer6305/fix/780-agno-tool-registration-validation
fix(agno): fail fast on toolkit registration failures - #780
2026-07-31 18:17:40 +05:30
KaifAhmad1 6e44d98d46 Merge remote-tracking branch 'origin/main' into pr818-fix
# Conflicts:
#	CHANGELOG.md
2026-07-31 17:51:53 +05:30
Sameer6305 d21e5e9944 fix(agno): log decision tracking failures in shared context - #779 2026-07-31 17:50:40 +05:30
KaifAhmad1 67aed43997 docs: add changelog entry for Agno toolkit fail-fast fix (#780, #818) 2026-07-31 17:44:14 +05:30
Sameer6305 ac64943965 Merge remote-tracking branch 'semantica-agi/main' into fix/783-tracking-methods-honest-failures
# Conflicts:
#	CHANGELOG.md
2026-07-31 16:11:32 +05:30
Sameer6305 4dea295f0d fixed qodo reviews
- split.ProvenanceTracker: Check _unified_manager.track_chunk() return value and fall back to legacy storage when None is returned (storage failure)

- split.ProvenanceTracker: Initialize legacy stores (_provenance_store and _chunk_registry) unconditionally in __init__ so legacy fallback storage works safely even when initialized with use_unified=True

- split.ProvenanceTracker: Update get_provenance() to check legacy storage when no record is found in unified storage, ensuring fallback-tracked chunks remain retrievable

- SourceTracker: Change track_sources_batch() condition from if entry is not None: to if ok: to respect the boolean return contract (-> bool) of track_entity_source(), track_property_source(), and track_relationship_source()

- Tests: Add regression test test_unified_tracking_returns_none_triggers_fallback and update test_track_sources_batch_failure_not_counted to test boolean failure return_value=False
2026-07-31 16:05:37 +05:30
Mohd Kaif 6c6cb3f3b5 Merge pull request #817 from Sameer6305/fix/781-causal-chain-error-signaling
fix(mcp): improve causal chain fallback error handling - #781
2026-07-31 15:42:07 +05:30
Sameer6305 1ae1e6d57a docs(provenance): document Optional return types and failure behavior (#783) 2026-07-31 15:01:33 +05:30
Sameer6305 495e29d543 fix(provenance): log tracking failures and return None on storage error (closes #783) 2026-07-31 15:00:23 +05:30
KaifAhmad1 04d2a726b9 Merge remote-tracking branch 'origin/main' into pr817-conflict-fix
# Conflicts:
#	CHANGELOG.md
2026-07-31 13:17:10 +05:30
KaifAhmad1 62a027d6fd fix(mcp): call backend get_causal_chain only once on internal TypeError
Signature introspection and the resulting call were sharing one
try/except, so a genuine bug inside a backend's get_causal_chain
(raising an unrelated TypeError) was misread as a signature mismatch,
causing an identical retry call before the real error surfaced.
Split introspection from the call so a successfully-introspected call
happens exactly once; the trial-and-error cascade now only runs when
inspect.signature itself fails. Also adds the CHANGELOG entry for
#781/#817, which was missing.
2026-07-31 13:14:29 +05:30
Mohd Kaif 7cab35bbc0 Merge pull request #816 from Sameer6305/fix/782-track-entity-atomic-write
fix(provenance): make track_entity's two-step write atomic (closes #782)
2026-07-31 12:14:12 +05:30
KaifAhmad1 938f846dde docs: cite PR number alongside issue in CHANGELOG for track_entity fix
Follow-up to the #782 entry — other entries in this section cite both
the issue and PR number, this one was missing the PR reference.
2026-07-31 12:08:52 +05:30
Sameer6305 66be1630fa fix(agno): fail fast on toolkit registration failures - #780 2026-07-30 20:43:26 +05:30
Sameer6305 a1f835c9b2 refactor(mcp): harden handle_get_causal_chain inputs and signature introspection (#781)
- Add safe input validation and bounds clamping on max_depth (1..100) to prevent DoS/memory exhaustion

- Use inspect.signature for accurate keyword dispatch with precise TypeError fallback

- Prevent masking of genuine internal TypeError exceptions inside graph backends

- Add security and input hardening regression tests
2026-07-30 19:09:22 +05:30
Sameer6305 12172d03b4 fix(mcp): fixed qodo reviews (#781)
- Support legacy (depth kwarg) and positional-only get_causal_chain backend signatures in fallback path

- Add regression tests for signature compatibility
2026-07-30 19:04:00 +05:30
Sameer6305 60f362817f fix(mcp): return error when causal chain analysis unsupported (#781)
- Return explicit error dictionary when graph lacks get_causal_chain instead of silent empty list

- Forward direction and max_depth in fallback graph.get_causal_chain call

- Add regression tests for error signaling and parameter forwarding
2026-07-30 18:11:04 +05:30
Sameer6305 4d1e5cf37c fixed qodo reviews 2026-07-30 16:58:19 +05:30
Sameer6305 16893c28a4 docs(provenance): document atomic rollback behavior (#782) 2026-07-30 16:28:40 +05:30
Sameer6305 577967a549 fix(provenance): make track_entity writes atomic (closes #782) 2026-07-30 16:28:40 +05:30
Mohd KaifandSameer6305 7fb94b6528 feat(triplet_store): add Altair Anzo triplet store backend (#814)
* feat(triplet_store): add Altair Anzo triplet store backend

Adds AnzoStore as a fourth peer to BlazegraphStore/RDF4JStore/JenaStore,
speaking plain SPARQL 1.1 over HTTP (no new dependency needed). The one
structural difference from the existing backends is that Anzo addresses
data by a dataset/graphmart URI rather than a short namespace/repository
name, so the endpoint path percent-encodes it. Reuses the shared
sparql_escaping.py helpers and wires "anzo" into TripletStore's backend
dispatch and config env vars.

Closes #813

* fix(triplet_store): correct AnzoStore SPARQL syntax and validate IRIs

Addresses review findings from Qodo and Codex on PR #814:

- get_triplets(): constraints are now expressed via FILTER(...) instead of
  bare equality expressions appended inside the WHERE group graph pattern
  (e.g. "?s ?p ?o ?s = <...>"), which is not valid SPARQL and was rejected
  by standards-compliant endpoints.
- bulk_load(): named-graph inserts now nest the GRAPH block inside the
  INSERT DATA braces (INSERT DATA { GRAPH <g> { ... } }) per the SPARQL 1.1
  Update grammar, instead of "INSERT DATA GRAPH <g> { ... }".
- bulk_load()/_build_insert_data()/delete_triplet()/get_triplets() now
  validate subject/predicate/graph URIs via sparql_escaping.validate_uri
  before interpolating them into SPARQL Update/Query strings, closing an
  injection path where a value containing ">" or "}" could break out of
  the intended <...> token.
- Corrected the store_type docstring/usage example: Anzo's linked-data-set
  store type is "lds", not "dataset".

Extended tests/triplet_store/test_anzo_store.py with coverage for the
corrected query shapes and the new validation/injection-rejection paths
(38 tests total, up from 32). Full tests/triplet_store/ suite: 299/299
passing.

* test(triplet_store): expand AnzoStore regression coverage

---------

Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
2026-07-30 11:22:52 +05:30
Sameer KadamandKaifAhmad1 e197977172 refactor(provenance): centralize duplicated store-and-swallow logic into _save_entry() (#784) (#815)
* refactor(provenance): centralize duplicated store-and-swallow logic into _save_entry() (closes #784)

- Added ProvenanceManager._save_entry(entry) as the single shared
  checksum-compute + storage.store() + graceful-failure-swallow
  pipeline, previously duplicated identically across track_entity,
  track_relationship, track_chunk, and track_property_source.
- track_entities_batch/track_chunks_batch already delegate to
  track_entity/track_chunk in a loop, so they inherit the fix for
  free — left untouched, confirmed no direct duplication there.
- Byte-for-byte preserves today's swallow-and-continue behavior and
  comment text; this is an architecture-only refactor. The silent-
  failure behavior itself is unchanged and out of scope here — a fix
  to it now only needs to happen in one place instead of four.
- Added 4 new regression tests (previously 0 of the 4 single-item
  methods had failure-path coverage) proving storage.store() raising
  is still caught and each method still returns its ProvenanceEntry.

Tests: tests/provenance/ 228 passed (+4 new), tests/explorer/test_provenance_manager_wiring.py 8 passed. 236/236, 0 failed.

* fix(provenance): drop out-of-transaction store attempt in track_entity fallback

The _save_entry refactor changed track_entity's pre-build exception
fallback (entry is None branch) to call _save_entry(), which makes a
real self.storage.store(entry) call. The original code only computed
a checksum here and never attempted storage again, since this branch
fires when something already failed before the entry was built inside
the atomic transaction. Storing outside that transaction bypasses the
BEGIN IMMEDIATE serialization #807 added, risking the same race it
fixed. Restored checksum-only behavior and added a regression test
asserting storage.store is not called on this path.

Also removed an untested hasattr(_store_with_conn) defensive branch
added during the refactor that wasn't in the original code, and added
a changelog entry.

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-07-29 22:18:05 +05:30
Mohd Kaif ecd0a26a8d Merge pull request #812 from Sameer6305/fix/807-sqlite-storage-performance
perf(provenance): optimize SQLite transaction lifecycle, concurrency, and lineage traversal
2026-07-29 13:04:36 +05:30
KaifAhmad1 f99241ca88 fix(provenance): stop reads from taking the writer lock, fix batch count inflation
Two review findings on #807/#812:

- retrieve() and trace_lineage() were routed through transaction()'s
  BEGIN IMMEDIATE, so plain reads took SQLite's writer lock and
  serialized behind every other read/write, defeating the WAL
  concurrency this PR was meant to add. They now use a dedicated
  _read_connection() (configured, no explicit BEGIN).

- track_entity()/track_chunk() swallowed all internal storage
  exceptions unconditionally, so a single item's failure inside
  track_entities_batch()/track_chunks_batch()'s shared transaction
  never reached the batch loop's per-item except, inflating
  tracked_count for entries that were never persisted. Both now
  re-raise when called with a shared _conn (batch context) while
  still degrading gracefully on standalone calls.

Added regression tests for both, corrected the CHANGELOG entry and
docs that described the prior (overly broad) behavior.
2026-07-29 12:54:43 +05:30
Sameer6305 dabeb0e833 docs: add changelog entry for #807 2026-07-28 23:44:14 +05:30
Sameer6305 9458cf5b2b docs: update provenance documentation for SQLiteStorage WAL and batch tracking (#807) 2026-07-28 23:42:14 +05:30
Sameer6305 3db35a6871 fixed qodo reviews 2026-07-28 23:37:46 +05:30
Sameer6305 b1daf238ca perf(provenance): optimize SQLite transaction lifecycle and lineage traversal 2026-07-28 23:06:23 +05:30
Mohd Kaif 0205ecd711 Merge pull request #811 from semantica-agi/ai-findings-autofix/SECURITY.md
Potential fixes for 3 code quality findings
2026-07-28 21:03:31 +05:30
Mohd Kaif 9677f25d27 Merge pull request #810 from semantica-agi/ai-findings-autofix/semantica-triplet_store-query_engine.py
Potential fixes for 2 code quality findings
2026-07-28 21:03:00 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> 4a221554de Apply suggested fix to SECURITY.md from Copilot Autofix
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-07-28 18:49:06 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> 7e513a12a3 Apply suggested fix to SECURITY.md from Copilot Autofix
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-07-28 18:49:06 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> 39bebe7da9 Apply suggested fix to SECURITY.md from Copilot Autofix
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-07-28 18:49:05 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> 9537bf17b3 Apply suggested fix to semantica/triplet_store/query_engine.py from Copilot Autofix
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-07-28 18:47:35 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> e8cb337946 Apply suggested fix to semantica/triplet_store/query_engine.py from Copilot Autofix
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-07-28 18:47:35 +05:30
Mohd Kaif 840629762f Merge pull request #809 from Sameer6305/fix/792-provenance-manager-wiring
fix(explorer): wire ProvenanceManager into provenance routes (closes #792)
2026-07-28 18:17:54 +05:30
KaifAhmad1 df6c30653c docs: add changelog entry for ProvenanceManager Explorer wiring (#792, #809) 2026-07-28 18:11:17 +05:30
Sameer6305 8d3c99ba30 perf(provenance): optimize lineage integrity checks and clean up top-level imports (#792)
- Reuse lineage['integrity_verified'] in _build_provenance in O(1) time when available, eliminating redundant SHA-256 verification loops across lineage chains.

- Improve compute_checksum dictionary handling in semantica/provenance/integrity.py so None values fall back cleanly to ProvenanceEntry defaults.

- Move json and verify_checksum imports to module top-level in semantica/provenance/manager.py to avoid function-local import overhead during get_lineage calls.
2026-07-28 17:25:01 +05:30
Sameer6305 c2079d8e92 fix(explorer): preserve audit evidence fields in provenance nodes and reports (#792)
- Extend ProvenanceNode in semantica/explorer/schemas.py with audit evidence fields: source_document, source_location, source_quote, confidence, and checksum.

- Update _transform_audit_lineage in semantica/explorer/routes/provenance.py to populate these evidence fields for each lineage node from ProvenanceEntry records, while keeping default None values for orphan nodes.

- Include source_document, confidence, and checksum in markdown report rendering (_render_markdown) so exported markdown reports surface attribution and integrity evidence.

- Add unit test test_provenance_audit_evidence_fields_preserved in test_provenance_manager_wiring.py verifying that evidence fields are present across /api/provenance JSON responses and exported JSON/markdown reports.
2026-07-28 17:15:49 +05:30
Sameer6305 cc864362aa fix(provenance): verify checksum integrity of lineage entries before labeling source as audit (#792)
- Update verify_checksum and compute_checksum in semantica/provenance/integrity.py to support both ProvenanceEntry objects and serialized dictionary entries.

- Add integrity_verified flag computed via verify_checksum to the dictionary returned by ProvenanceManager.get_lineage().

- Update _build_provenance in semantica/explorer/routes/provenance.py to verify every returned lineage entry before labeling the result as source=audit. If verification fails due to missing checksums or corrupted records, log a warning and fall back cleanly to graph traversal.

- Add unit test test_provenance_manager_wiring_checksum_failure_falls_back in test_provenance_manager_wiring.py verifying that tampered lineage entries trigger fallback to source=graph_traversal.
2026-07-28 17:08:12 +05:30
Sameer6305 d30aea79a7 fix(explorer): classify multi-hop audit lineage as upstream and enforce provenance storage configuration (#792)
- Fix _transform_audit_lineage to classify all non-downstream ancestor derivation edges as 'upstream' instead of 'lateral', correcting multi-hop lineage direction in JSON and markdown reports.

- Add GraphSession.set_provenance_storage_path() to explicitly reject conflicting preconfigured storage paths or path mutations after provenance_manager initialization.

- Update create_app() to call active_session.set_provenance_storage_path(prov_path), preventing silent retention of conflicting paths or un-redirectable cached managers.

- Remove unused logging import in app.py.

- Add comprehensive unit tests in test_provenance_manager_wiring.py for upstream edge classification, markdown report grouping, conflicting path rejection, and manager initialization lockouts.
2026-07-28 16:57:38 +05:30
Sameer6305 2ee4da0b47 fix(explorer): wire ProvenanceManager into provenance routes (closes #792)
- Explorer's /api/provenance now queries the audit-grade ProvenanceManager
  (SQLite-backed, checksummed) first, falling back to the naive 2-hop
  graph traversal when no audit records exist for a node.
- Fixed a process-global mutable-state risk in the initial approach:
  provenance storage path is threaded per-session via GraphSession,
  not via ProvenanceManager's global set_default_storage_path classmethod.
- Added source: 'audit' | 'graph_traversal' to the response so callers
  can distinguish which path served the data.
- Documented a known limitation: ProvenanceManager currently only
  traces upstream/ancestor lineage, not descendants — the naive
  fallback remains the only source for downstream relationships until
  ProvenanceManager gains a reverse lookup (tracked separately).
- Warns (rather than silently no-ops) if a provided session's
  provenance_manager was already constructed before create_app()
  applied a provenance_storage_path.
- Never lets a provenance-manager failure crash the route; degrades
  to the naive path with a logged warning instead.

Tests: 5 new tests in test_provenance_manager_wiring.py covering the
audit path, empty-record fallback, storage-failure degradation, app
startup wiring, and cross-session storage isolation. Full
tests/explorer/ + tests/provenance/ suite passing, order-invariant.
2026-07-28 16:23:02 +05:30
Mohd Kaif 016661463e Merge pull request #808 from semantica-agi/docs-enterprise-data-platforms-databricks-snowflake
docs: highlight Databricks/Snowflake enterprise data ingestion, fix ingest doc bugs
2026-07-28 15:53:14 +05:30
Sameer6305 ce914b396a docs: clarify Snowflake OAuth auth, add ArrowIngestor and non-re-exported ingestors (PR #808) 2026-07-28 15:00:02 +05:30
KaifAhmad1 b105b8ea97 fix: address review feedback on Databricks/Snowflake docs (PR #808)
- README: get_table_lineage() takes table_name first, then catalog/schema
  keyword args — the example had them in the wrong order, which would have
  queried lineage for the wrong fully-qualified table when copy-pasted.
- modules.md: the ingest example used DatabricksIngestor without importing
  it, causing a NameError if copy-pasted as-is.
- guides/ingest.md: corrected the claim that Databricks/Snowflake ingestors
  return "the same shape as DBIngestor" — DBIngestor.execute_query() returns
  a raw List[Dict] with no wrapper, unlike DatabricksData/SnowflakeData.
2026-07-28 12:10:37 +05:30
KaifAhmad1 6ed5aea993 docs: highlight Databricks/Snowflake enterprise data ingestion, fix ingest doc bugs
Makes enterprise lakehouse/warehouse ingestion (Databricks Unity Catalog +
Delta Lake, Snowflake) a first-class, prominently documented capability
across the README and guides, and adds matching runnable examples to
docs/guides/ingest.md. Also fixes several pre-existing inaccuracies caught
while auditing the ingest module docs against the actual source:
WebIngestor has no ingest_urls() (only singular ingest_url()), XMLIngestor's
XSD option is schema_path (not validate_xsd) and belongs on ingest() not the
constructor, and the "Available ingestors" list was missing DatabricksIngestor
while listing several classes not actually exported from semantica.ingest.
2026-07-28 11:58:09 +05:30
Mohd Kaif 80bce453c3 Merge pull request #805 from Sameer6305/fix/773-sparql-test-coverage
test(explorer): add coverage for SPARQL route (#773)
2026-07-27 19:33:07 +05:30
KaifAhmad1 d102584af6 fix(explorer): dedupe SPARQL row-cap logic and cover CONSTRUCT/DESCRIBE truncation
Extracts the row-cap-and-truncate loop (duplicated between the
CONSTRUCT/DESCRIBE and SELECT branches) into a shared _cap_rows()
helper, and adds a test for the previously-uncovered CONSTRUCT/DESCRIBE
truncation path. Addresses review nits on PR #805.
2026-07-27 19:27:35 +05:30
Mohd Kaif 4f27d3dcae Merge pull request #804 from Sameer6305/fix/772-live-shacl-validation-v2
fix(ontology): wire live SHACL validation into /shacl/validate and /health (closes #772) #803
2026-07-27 19:05:32 +05:30
KaifAhmad1 35f8c0527c Merge remote-tracking branch 'origin/main' into fix/772-live-shacl-validation-v2
# Conflicts:
#	CHANGELOG.md
2026-07-27 18:58:16 +05:30
KaifAhmad1 28fe304f76 fix(ontology): address review follow-ups on live SHACL validation (#804)
- Revert create_ontology silently falling back to a near-empty ontology on
  generation failure; restores the HTTPException(500) behavior from #770/#787
  that this PR had accidentally undone (and re-enables TestOntologyCreateFailures)
- Fold sh:Warning/sh:Info severity pySHACL results into the /shacl/validate
  response's violations array instead of silently dropping them, so a
  non-conforming report is never returned with an empty violations list
- Share a single nodes/edges fetch between _generated_shacl_for_uri and
  _data_graph_turtle_for_uri via new _fetch_analysis_graph(), so /health
  no longer re-queries and re-truncation-checks the same ontology twice
2026-07-27 18:28:11 +05:30
Mohd KaifandSameer6305 9eea49a070 fix(security): restrict Neptune cookbook SG, add VPC flow logs, harden IaC scan suppressions (#806)
* fix(security): restrict Neptune cookbook SG, add VPC flow logs, harden IaC scan suppressions

Addresses open GHAS code scanning alerts:
- Neptune cookbook stack (neptune-setup.yaml) no longer opens the Bolt/OpenCypher
  port to 0.0.0.0/0; a required ClientCidr parameter must be supplied instead.
  Updated 21_Amazon_Neptune_Store.ipynb deploy instructions to match.
- Added VPC Flow Logs (CloudWatch Logs + IAM role) to the same stack.
- Documented why an account-wide IAM password policy resource does not belong
  in a disposable per-learner CFN stack, with a justified ts:skip.
- Added inline `checkov:skip` / `ts:skip` comments to the knowledge-explorer
  Helm templates (deployment/service/configmap) as a second suppression path
  for the CKV_K8S_21/AC_K8S_0086/AC_K8S_0080 false positives, since the prior
  annotation-only suppression was not being honored by the scanner.

* docs(changelog): document the Neptune and Helm chart security scan fixes

* fix(security): correct flow-log IAM scope and ClientCidr regex from review

- FlowLogRole granted logs:CreateLogStream/PutLogEvents on the bare log
  group ARN, but those actions apply to log streams, not the group itself;
  scoped them to "${FlowLogGroup.Arn}:log-stream:*" instead and moved the
  Describe* actions (which don't support group/stream-level resource
  restriction) to Resource: "*", matching AWS's documented flow-log IAM
  policy shape. Without this, flow log delivery could silently fail.
- ClientCidr's AllowedPattern only checked digit count (1-3 digits per
  octet), so malformed values like 999.999.999.999/32 passed parameter
  validation and would only fail later when CloudFormation tried to
  create the security group rule. Tightened the regex to enforce valid
  IPv4 octet ranges (0-255) and prefix lengths (0-32).

* fix(security): harden IAM policy in neptune-setup and standardize Helm chart scan suppressions

- neptune-setup.yaml: split FlowLogRole policy into account-level statement (CreateLogGroup, DescribeLogGroups, DescribeLogStreams with Resource: '*') and log-group-scoped statement (CreateLogStream, PutLogEvents with !GetAtt FlowLogGroup.Arn) per AWS VPC Flow Logs least-privilege documentation.
- deployment.yaml: remove unreliable file-header skip comments (# checkov:skip / # ts:skip) and replace with resource-level metadata.annotations (checkov.io/skip and runterrascan.io/skip). Update seccomp rule ID from CKV_K8S_28 to checkov's actual seccomp rule CKV_K8S_31 on both Deployment and pod-template metadata.
- configmap.yaml / service.yaml: remove stale # ts:skip=AC_K8S_0086 file-header comments and add runterrascan.io/skip resource-level metadata annotations for consistency across all chart templates.
- .checkov.yaml: update documentation to explain resource-level metadata.annotations and reference CKV_K8S_31.

---------

Co-authored-by: Sameer6305 <sskadam6305@gmail.com>
2026-07-27 17:54:38 +05:30
Sameer6305 db95cedf34 fixed qodo reviews and hardened implementation 2026-07-27 15:42:49 +05:30
Sameer6305 3a9c7c082f Merge branch 'main' into fix/773-sparql-test-coverage 2026-07-27 15:08:14 +05:30
Mohd Kaif 4a3cf37679 Merge pull request #802 from Sameer6305/feat/provenance-shared-storage-wiring
feat(provenance): implement global default storage pattern and fix CLI lineage integration
2026-07-27 15:06:29 +05:30
Sameer6305 dd7b090aec test(explorer): add coverage for SPARQL route (#773)
sparql.py handles direct SPARQL query execution against the live graph with no test coverage anywhere in the repo. Adds coverage for the read-only allowlist (the actual security boundary here), row/timeout limits, error handling, and RDF projection fidelity.
2026-07-27 15:00:47 +05:30
KaifAhmad1 eaa0f823a8 Merge remote-tracking branch 'origin/main' into pr-802
# Conflicts:
#	CHANGELOG.md
2026-07-27 14:24:46 +05:30
KaifAhmad1 7045d7b94e fix(provenance): address review nits and add CHANGELOG entry
- track_entity() no longer aliases a caller-supplied used_entities list
  (it stored the reference directly and later mutated it via .append())
- Remove dead fallback branches in orchestrator.py/manager.py that
  duplicated what Config.get()'s dotted-path resolution already does
- Add local --dry-run to `provenance audit` for parity with
  `provenance export`
- `provenance check --strict` now warns instead of printing a success
  checkmark before raising on a failed check
2026-07-27 14:19:51 +05:30
Sameer6305 8430d4a56e fix(ontology): address qodo review findings for SHACL validation
- DoS guardrails: enforce byte size, triple count, concurrency, and timeout limits on /shacl/validate

- Slash namespace resolution: preserve trailing slash in _resolve_uri so local terms match SHACL shapes

- Truncation safety: raise GraphTruncationError and report unavailable/413/critical when graphs exceed analysis limits

- JSON-LD dict lists: unwrap uri/id/@id in _as_uri_list and _data_graph_turtle_for_uri property loops

- Observability & efficiency: add warning logs on truncation and avoid duplicate UTF-8 encoding in size check
2026-07-27 14:01:45 +05:30
Mohd Kaif ea71ea1823 Merge pull request #786 from SaurabhScripts/codex/agent-memory-markdown-round-trip
Add Markdown round-trip support to AgentMemory
2026-07-27 13:19:56 +05:30
Sameer6305 3a1ab1a8a6 fix(ontology): wire live SHACL validation into /shacl/validate and /health
Closes #772
2026-07-27 13:02:04 +05:30
KaifAhmad1 87714ec1ad Merge remote-tracking branch 'origin/main' into codex/agent-memory-markdown-round-trip
# Conflicts:
#	CHANGELOG.md
2026-07-27 12:51:12 +05:30
KaifAhmad1 1c590c622b docs(changelog): document AgentMemory Markdown round-trip support
Add an Unreleased/Added entry for #786 covering the new export/import
Markdown format, idempotency and rollback guarantees, and the
Explorer/ContextGraph scoping decision from #765.
2026-07-27 12:48:10 +05:30
Sameer6305 40d6fa05d0 fix(context): normalize timestamps in _markdown_record_matches for idempotency 2026-07-26 18:49:30 +05:30
Sameer6305 ac69c27b86 fixes reviews from qodo free for open source 2026-07-26 16:56:06 +05:30
Sameer6305 a16bd9600b fixes reviews from qodo free for open source 2026-07-26 16:46:50 +05:30
Sameer6305 fd84be8b66 fixed qodo reviews 2026-07-26 16:26:47 +05:30
Sameer6305 6cc2e67b92 fix(provenance): populate entries alias in get_lineage and read lineage_chain in lineage()
- Add entries alias in get_lineage() return dictionary so CLI and programmatic callers can access lineage entries via either key

- Update lineage() wrapper method to fallback to lineage_chain when entries is missing

- Add assertions in test_cli_lineage confirming lineage and entries lists are non-empty
2026-07-26 16:10:26 +05:30
Sameer6305 2dd06aadcd feat(provenance): implement Global Default Storage pattern, CLI methods, and orchestrator config wiring
- Add _default_storage_path, set_default_storage_path(), and test-isolation context manager default_storage_path() in ProvenanceManager

- Accept config kwarg in ProvenanceManager.__init__ to fix CLI initialization bug

- Implement audit_log(), lineage(), export_prov(), and check() on ProvenanceManager matching cli.py expectations

- Wire provenance.storage_path in Semantica.__init__ before pipeline stages execute

- Add comprehensive unit tests in tests/provenance/test_manager.py for CLI methods and test isolation
2026-07-26 16:01:50 +05:30
Mohd Kaif 86db4f923d Merge pull request #796 from Sameer6305/fix/769-lint-effect-setstate
Fix #769: Eradicate react-hooks/set-state-in-effect cascading renders project-wide
2026-07-26 13:50:16 +05:30
KaifAhmad1 dcd936a9ab fix: restore error surfacing dropped by inlined mount-effect fetches
The set-state-in-effect refactor inlined each initial-fetch effect as a
standalone `fetchInitial`, duplicating the logic of the existing
reload/fetchOverview/fetchRegistry/loadVersions callbacks instead of
reusing them (required, since eslint-plugin-react-hooks v7 flags calling
an outside setState-touching function directly from an effect body, even
through an async gap - verified via a local lint probe). The duplicates
dropped the setError/flashMsg calls the originals had, so a failed
initial page load in AlignmentsTab, KGOverviewTab, OntologyManager, and
VersionsTab now failed silently instead of showing an error - a
regression of the exact bug #767/#790 fixed for these same files.

Also fixes LineageDiagram only clearing nodes/edges when the new
activeId was falsy, leaving the previous lineage view's stale diagram
on screen while switching directly between two ids.
2026-07-26 13:40:16 +05:30
KaifAhmad1 b663c6bbbf Merge branch 'main' into fix/769-lint-effect-setstate 2026-07-26 13:22:45 +05:30
Saurabh Meena 5ab21c089e Address AgentMemory Markdown review feedback 2026-07-26 09:42:23 +05:30
Mohd Kaif 84775fdea0 Merge pull request #801 from semantica-agi/fix/779-checkov-default-namespacet
fix: suppress CKV_K8S_21 default-namespace false positive on knowledge-explorer Helm chart
2026-07-25 18:24:40 +05:30
Sameer6305 2a0bc7051a fix(security): switch to metadata.annotations for CKV_K8S_21 suppressions 2026-07-25 17:45:53 +05:30
KaifAhmad1 8beca57238 fix: wrap checkov:skip comment to respect yamllint's 120-char line-length limit
The single-line checkov:skip=CKV_K8S_21 comment added in ed44260 was 286
characters, exceeding the repo's yamllint line-length rule (max 120,
.pre-commit-config.yaml). Split into three short comment lines: the skip
directive itself, then the rationale, in service.yaml, deployment.yaml,
and configmap.yaml.
2026-07-25 16:59:26 +05:30
KaifAhmad1 ed44260ec3 fix: suppress CKV_K8S_21 false positive on knowledge-explorer Helm chart
Checkov's helm framework renders the chart without a namespace override,
so metadata.namespace (set to .Release.Namespace, bound only at install
time) always resolves to "default" and trips CKV_K8S_21 on service.yaml,
deployment.yaml, and configmap.yaml even though the chart is
namespace-agnostic by design.

Suppressed via per-file checkov:skip comments, following the same
convention already used for the Cloud Run false positives in
deploy/gcp/cloudrun-service.yaml.
2026-07-25 16:49:25 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 8f09d4f57b chore(deps): bump dompurify from 3.4.11 to 3.4.12 in /explorer (#800)
Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.4.11 to 3.4.12.
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.11...3.4.12)

---
updated-dependencies:
- dependency-name: dompurify
  dependency-version: 3.4.12
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-25 16:36:25 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 7e05f196b5 chore(deps): bump postcss from 8.5.10 to 8.5.23 in /explorer (#799)
Bumps [postcss](https://github.com/postcss/postcss) from 8.5.10 to 8.5.23.
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.5.10...8.5.23)

---
updated-dependencies:
- dependency-name: postcss
  dependency-version: 8.5.23
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-25 16:33:19 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 47c7f4adbe chore(deps): bump brace-expansion and eslint in /explorer (#797)
Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) to 5.0.8 and updates ancestor dependency [eslint](https://github.com/eslint/eslint). These dependencies need to be updated together.


Updates `brace-expansion` from 5.0.6 to 5.0.8
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v5.0.6...v5.0.8)

Updates `eslint` from 9.39.4 to 10.8.0
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/compare/v9.39.4...v10.8.0)

---
updated-dependencies:
- dependency-name: brace-expansion
  dependency-version: 5.0.8
  dependency-type: indirect
- dependency-name: eslint
  dependency-version: 10.8.0
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-25 16:31:43 +05:30
Mohd KaifandKaifAhmad1 0698ba7656 fix(#768): Prevent application crashes by wrapping workspaces in Error Boundaries (#794)
* fix(#768): add ErrorBoundary to workspace Suspense blocks

* fix(#768): ensure ErrorBoundary retryCount only resets on recovery transition

* fix(#768): remove componentDidUpdate auto-reset to avoid premature reset on Suspense fallback

* fix(#768): reset ErrorBoundary retryCount only after a retry settles

Previously retryCount never reset on success (removed in adb4613 to
avoid resetting mid-Suspense-fallback), so unrelated transient errors
across a session could permanently exhaust the 3-retry budget even
though each prior retry had actually recovered. Now the counter resets
via a short settle timer after a retry stays error-free, avoiding both
the premature-reset and never-reset failure modes.

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-07-25 16:15:21 +05:30
KaifAhmad1 33d8f806c2 Merge remote-tracking branch 'origin/main' into fix/768-error-boundaries-review
# Conflicts:
#	CHANGELOG.md
2026-07-25 16:11:15 +05:30
KaifAhmad1 530e297d17 fix(#768): reset ErrorBoundary retryCount only after a retry settles
Previously retryCount never reset on success (removed in adb4613 to
avoid resetting mid-Suspense-fallback), so unrelated transient errors
across a session could permanently exhaust the 3-retry budget even
though each prior retry had actually recovered. Now the counter resets
via a short settle timer after a retry stays error-free, avoiding both
the premature-reset and never-reset failure modes.
2026-07-25 16:09:07 +05:30
Sameer6305 be2f8f8cd8 Implement global default storage pattern for ProvenanceManager 2026-07-24 23:27:58 +05:30
Sameer6305 a2f10dfbdd trigger CI re-run for failed runners 2026-07-24 23:16:10 +05:30
Sameer6305 495c2a2fbd fixes qodo reviews 2026-07-24 21:39:57 +05:30
Sameer6305 eb8156ddb3 Fix GraphWorkspace infinite render loop by tracking stringified open panel IDs 2026-07-24 21:19:54 +05:30
Sameer6305 1c3d2b949f Merge main to fix conflicts 2026-07-24 21:09:34 +05:30
Sameer6305 343d2bc418 Fix #769: Resolve all react-hooks/set-state-in-effect lint errors project-wide 2026-07-24 20:56:11 +05:30
Mohd Kaif 297d959f63 Merge pull request #790 from Sameer6305/fix/767-frontend-silent-failures
Fix #767: Harden workspaces against silent failures and handle 207 Partial Success
2026-07-24 16:39:26 +05:30
KaifAhmad1 161d47f4d9 Merge remote-tracking branch 'origin/main' into fix/767-frontend-silent-failures
# Conflicts:
#	CHANGELOG.md
2026-07-24 16:29:56 +05:30
KaifAhmad1 99b0a517fd Fix remaining silent-failure gaps flagged in review of #790
KGOverviewTab dropped the nodes-fetch 207 warning whenever stats also
returned 207; HealthTab and AlignmentsTab still had the exact
silent-swallow pattern this PR set out to fix elsewhere in the same
folder. Also documents all of #790's fixes in the changelog.
2026-07-24 16:25:21 +05:30
Sameer6305 adb46134c4 fix(#768): remove componentDidUpdate auto-reset to avoid premature reset on Suspense fallback 2026-07-24 16:15:34 +05:30
Sameer6305 d2d38a0509 fix(#768): ensure ErrorBoundary retryCount only resets on recovery transition 2026-07-24 16:11:14 +05:30
Sameer6305 9a21e523f0 fix(#768): add ErrorBoundary to workspace Suspense blocks 2026-07-24 15:53:33 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> ca11a48fef deps(deps): update httpx requirement from <0.28.0 to <0.29.0 (#791)
Updates the requirements on [httpx](https://github.com/encode/httpx) to permit the latest version.
- [Release notes](https://github.com/encode/httpx/releases)
- [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/encode/httpx/compare/0.0.1...0.28.1)

---
updated-dependencies:
- dependency-name: httpx
  dependency-version: 0.28.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-24 15:41:00 +05:30
Mohd KaifandKaifAhmad1 aa171c16f1 Fix #788: pin httpx<0.28.0 globally to fix Explorer test suite TestClient breakage (#789)
* Fix #788: pin httpx<0.28.0 globally to fix TestClient breakage

Adds an explicit httpx<0.28.0 constraint to [project.dependencies] so it
applies globally across all environments, not just dev. Without this,
different environments could resolve an incompatible transitive httpx
version and hit the same Starlette TestClient breakage independently.

Verified via git stash comparison: the unmodified baseline fails to even
collect the test suite (TestClient TypeError during collection), so this
pin doesn't just fix tests, it's what allows the full suite to run at all.

Closes #788

* Add CHANGELOG entry for #788 httpx pin fix

Documents the httpx<0.28.0 global pin from this PR under Unreleased/Fixed.

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-07-24 13:18:36 +05:30
KaifAhmad1 cb5e93ebc7 Merge remote-tracking branch 'origin/main' into fix/788-httpx-pin
# Conflicts:
#	CHANGELOG.md
2026-07-24 13:13:41 +05:30
KaifAhmad1 a256a77277 Add CHANGELOG entry for #788 httpx pin fix
Documents the httpx<0.28.0 global pin from this PR under Unreleased/Fixed.
2026-07-24 13:10:59 +05:30
Sameer6305 e7696f462a fixing qodo findings 2026-07-24 00:38:49 +05:30
Sameer6305 d6c7154fa9 Fix #767: Harden workspaces against silent error swallowing and 207 statuses
Ensures network errors, API failures, and 207 Partial Success statuses are surfaced correctly to the user instead of failing silently in the frontend UI.
2026-07-24 00:16:46 +05:30
Mohd Kaif b473e0bd8a Merge pull request #787 from Sameer6305/fix/770-explorer-200-on-failure
Fix #770: Explorer backend routes return proper error status codes instead of 200 OK on failure
2026-07-23 18:26:13 +05:30
KaifAhmad1 b1deed5857 Address review: harden analytics status codes, add failure-path tests
207 alone is indistinguishable from 200 to callers that only check
response.ok, so /api/analytics now raises 500 when every requested
metric fails and reserves 207 for genuine partial failure. Adds
regression tests for the temporal, analytics, and ontology-create
failure paths introduced in this PR, and logs the fix in the
changelog's Unreleased section.
2026-07-23 18:13:26 +05:30
Sameer6305 637bfe7314 Fix #788: pin httpx<0.28.0 globally to fix TestClient breakage
Adds an explicit httpx<0.28.0 constraint to [project.dependencies] so it
applies globally across all environments, not just dev. Without this,
different environments could resolve an incompatible transitive httpx
version and hit the same Starlette TestClient breakage independently.

Verified via git stash comparison: the unmodified baseline fails to even
collect the test suite (TestClient TypeError during collection), so this
pin doesn't just fix tests, it's what allows the full suite to run at all.

Closes #788
2026-07-23 16:36:58 +05:30
Sameer6305 9b7a33031c solving qodo review 2026-07-23 15:34:53 +05:30
Sameer6305 443a9b78d7 Fix #770: Explorer backend routes return proper error status codes instead of 200 OK on failure
- routes/temporal.py: temporal_patterns raises HTTPException(500) instead of
  silently returning an empty-but-valid TemporalPatternResponse on exception
- routes/analytics.py: preserves existing partial-success body shape
  (frontend already parses this), but sets response.status_code = 207 when
  any individual metric computation fails, so callers get a real signal
  instead of an indistinguishable 200
- routes/ontology.py: POST /create now raises HTTPException(500) on
  generation failure instead of silently falling back to a partial/minimal
  ontology and returning 200 with a misleading nodes_added count

Verified via git stash comparison that pre-existing test suite failures
(58 errors, Starlette TestClient/httpx version mismatch) are unrelated to
this change - identical failure count on modified and unmodified code.

Closes #770
2026-07-23 15:09:44 +05:30
Saurabh Meena 36856cc92a Add Markdown round-trip support to AgentMemory 2026-07-22 22:56:16 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 6e4ff0c7c5 ci(deps): bump actions/setup-node from 6 to 7 (#760)
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 6 to 7.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 14:30:44 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 095d7d3e52 ci(deps): bump actions/setup-dotnet from 5 to 6 (#759)
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 5 to 6.
- [Release notes](https://github.com/actions/setup-dotnet/releases)
- [Commits](https://github.com/actions/setup-dotnet/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-dotnet
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 14:19:16 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 1b706e539f ci(deps): bump actions/setup-python from 4 to 7 (#758)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 7.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v4...v7)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 14:11:25 +05:30
184 changed files with 18145 additions and 1804 deletions
+14
View File
@@ -2,4 +2,18 @@
# Cloud Run false-positives (CKV_K8S_21/28/30) are suppressed via per-file
# inline checkov:skip comments in deploy/gcp/cloudrun-service.yaml rather than
# globally here, so future real Kubernetes manifests are not silently exempted.
#
# The knowledge-explorer Helm chart's unconditional templates (service.yaml,
# deployment.yaml, configmap.yaml) set metadata.namespace to .Release.Namespace,
# which is only bound at `helm install`/`helm template` time. Checkov's helm
# framework renders the chart without a namespace override, so it always
# resolves to "default" and trips CKV_K8S_21 even though the chart is
# namespace-agnostic by design. Suppressed via metadata annotations
# (checkov.io/skip1 / runterrascan.io/skip) on each resource's metadata.annotations,
# as both Checkov and Terrascan require K8s/Helm resource-level annotations
# rather than file-header comments.
# deployment.yaml additionally suppresses AC_K8S_0080 and CKV_K8S_31 (seccomp) via
# metadata.annotations on both the Deployment resource and the pod template:
# the seccomp profile is set correctly in values.yaml and only resolves once
# Helm actually renders `toYaml`, which static template scanning does not do.
skip-check: []
+7
View File
@@ -70,6 +70,13 @@ updates:
- "dependencies"
- "github-actions"
- "ci"
# All our actions are SHA-pinned with a "# vX" comment; Dependabot
# resolves the new tag's SHA and updates both the pin and the comment
# together, so this stays the source of truth (no separate script needed).
groups:
github-actions:
patterns:
- "*"
# Optional dependencies (separate schedule for stability)
- package-ecosystem: "pip"
+2
View File
@@ -1,3 +1,5 @@
> **Before you submit:** make sure you followed the [issue workflow in CONTRIBUTING.md](https://github.com/semantica-agi/semantica/blob/main/CONTRIBUTING.md#-working-on-an-existing-issue) — comment on the issue and wait for assignment before opening a PR, to avoid duplicate work.
## Description
<!-- Provide a clear description of your changes -->
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Verifies that every third-party GitHub Action referenced in
# .github/workflows/*.yml and .github/workflows/*.yaml is pinned to a full
# commit SHA (not a mutable tag
# or branch), and that any pin's trailing "# vX" comment still matches what
# that tag resolves to today.
#
# Fails closed on purpose:
# - a `uses:` line pinned to anything other than a 40-hex-char SHA is a
# hard failure, not a skip - this is what stops a newly-added mutable
# tag (e.g. `uses: some/action@v1`) from slipping past unnoticed.
# - a tag that can't be resolved via the GitHub API (rate limit, deleted
# tag, typo) is also a hard failure rather than a warning - an
# unverifiable pin is exactly the failure mode this check exists to
# catch, so it must not pass silently.
set -uo pipefail
fail=0
checked=0
# Pattern for a third-party uses: line — stored in a variable so bash's
# [[ =~ ]] parser never sees literal \" or \' escapes, which cause a
# "syntax error in conditional expression: unexpected token )" at runtime.
# Semantics: optional leading quote, owner/repo, optional subpath, @ref,
# optional trailing quote; quote chars excluded from the ref capture group.
USES_PATTERN='uses:[[:space:]]+["'"'"']?([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)(/[^[:space:]@"'"'"']+)?@([^[:space:]"'"'"']+)["'"'"']?'
while IFS=: read -r file lineno content; do
# Local composite actions (./x) and Docker image refs (docker://...) use a
# different pinning mechanism and aren't in scope here.
[[ "$content" =~ uses:\ +\./ ]] && continue
[[ "$content" =~ uses:\ +docker:// ]] && continue
if [[ "$content" =~ $USES_PATTERN ]]; then
repo="${BASH_REMATCH[1]}"
ref="${BASH_REMATCH[3]}"
checked=$((checked + 1))
if [[ ! "$ref" =~ ^[0-9a-fA-F]{40}$ ]]; then
echo "::error file=$file,line=$lineno::$repo is pinned to '$ref', not a full commit SHA. Mutable tags/branches can be silently re-pointed (see the LiteLLM/Trivy 2026 incident) - pin to a commit SHA instead."
fail=1
continue
fi
sha="$ref"
if [[ "$content" =~ \#[[:space:]]*([^[:space:]]+)[[:space:]]*$ ]]; then
tag="${BASH_REMATCH[1]}"
else
echo "::warning file=$file,line=$lineno::$repo@$sha has no trailing '# vX' comment recording which tag it corresponds to - add one for auditability."
continue
fi
resolved=$(gh api "repos/$repo/commits/$tag" --jq '.sha' 2>/dev/null)
if [[ -z "$resolved" ]]; then
echo "::error file=$file,line=$lineno::Could not resolve '$repo@$tag' via the GitHub API (rate limit, deleted tag, or typo). Treating as unverifiable = failure."
fail=1
continue
fi
if [[ "$resolved" != "$sha" ]]; then
echo "::error file=$file,line=$lineno::$repo is pinned to $sha but tag '$tag' now resolves to $resolved. Update the pin or the comment."
fail=1
else
echo "OK $repo@$tag -> $sha ($file:$lineno)"
fi
fi
done < <(grep -rHn "uses:" .github/workflows/*.yml .github/workflows/*.yaml 2>/dev/null)
echo "Checked $checked action reference(s)."
exit $fail
+3 -3
View File
@@ -13,12 +13,12 @@ jobs:
steps:
- name: Checkout Code
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
fetch-depth: 0
- name: Set up Python 3.12
uses: actions/setup-python@v5
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.12"
cache: 'pip'
@@ -43,7 +43,7 @@ jobs:
# pytest-benchmark --storage file://benchmarks/results --benchmark-compare
- name: Upload Benchmark Results
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: always()
with:
name: benchmark-report-${{ github.run_id }}
+13 -6
View File
@@ -21,20 +21,27 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v5
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: '3.11'
- uses: actions/setup-node@v6
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: explorer/package-lock.json
- name: Build Explorer frontend
- name: Install Explorer frontend dependencies
working-directory: explorer
run: npm ci
- name: Test Explorer frontend
working-directory: explorer
run: |
npm ci
npm run build
npm run test:graph-store
npm run test:graph-workspace
npm run test:plugin-registry
- name: Build Explorer frontend
working-directory: explorer
run: npm run build
- run: pip install build
- run: python -m build
- name: Verify Explorer frontend is packaged
+7 -7
View File
@@ -20,7 +20,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
# The CodeQL bundle download (github/codeql-action/init's "Setup CodeQL
# tools" step) streams a ~1GB tarball from GitHub's release CDN and
@@ -32,7 +32,7 @@ jobs:
# meaningful state carried over from a failed attempt.
- name: Initialize CodeQL (attempt 1)
id: codeql-init-1
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
continue-on-error: true
with:
languages: python
@@ -42,7 +42,7 @@ jobs:
- name: Initialize CodeQL (attempt 2)
id: codeql-init-2
if: steps.codeql-init-1.outcome == 'failure'
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
continue-on-error: true
with:
languages: python
@@ -52,17 +52,17 @@ jobs:
- name: Initialize CodeQL (attempt 3)
id: codeql-init-3
if: steps.codeql-init-2.outcome == 'failure'
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
with:
languages: python
queries: security-and-quality
config-file: .github/codeql/codeql-config.yml
- name: Autobuild
uses: github/codeql-action/autobuild@v4
uses: github/codeql-action/autobuild@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
uses: github/codeql-action/analyze@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
with:
category: "/language:python"
upload: false
@@ -72,7 +72,7 @@ jobs:
# Uploads results only when Default Setup is not active.
# If Default Setup is still enabled, this step skips gracefully
# instead of failing the workflow with HTTP 409.
uses: github/codeql-action/upload-sarif@v4
uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
with:
sarif_file: ${{ steps.codeql.outputs.sarif-output }}
category: "/language:python"
+6 -6
View File
@@ -36,14 +36,14 @@ jobs:
runs-on: windows-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-dotnet@v5
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6
with:
dotnet-version: |
5.0.x
6.0.x
- name: Run Microsoft Security DevOps
uses: microsoft/security-devops-action@v1.12.0
uses: microsoft/security-devops-action@08976cb623803b1b36d7112d4ff9f59eae704de0 # v1.12.0
id: msdo
with:
# checkov is intentionally excluded from this MSDO step.
@@ -57,11 +57,11 @@ jobs:
# avoiding the guardian.cmd/checkov exit-code bug in the MSDO wrapper.
tools: eslint,templateanalyzer,terrascan
- name: Upload results to Security tab
uses: github/codeql-action/upload-sarif@v4
uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
with:
sarif_file: ${{ steps.msdo.outputs.sarifFile }}
- uses: actions/setup-python@v5
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: "3.12"
@@ -82,7 +82,7 @@ jobs:
}
- name: Upload Checkov results to Security tab
uses: github/codeql-action/upload-sarif@v4
uses: github/codeql-action/upload-sarif@5595ccaf912efad79be6eef63a5619ff05969be3 # v4
if: always()
with:
sarif_file: reports/checkov.sarif
+8 -8
View File
@@ -29,11 +29,11 @@ jobs:
name: Validate Documentation
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v5
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: '3.11'
- uses: actions/setup-node@v6
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: '20'
- run: python docs_check.py
@@ -44,9 +44,9 @@ jobs:
runs-on: ubuntu-latest
needs: validate
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@v6
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: '20'
@@ -57,12 +57,12 @@ jobs:
cd ..
unzip -q export.zip -d site
- uses: actions/configure-pages@v6
- uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6
- uses: actions/upload-pages-artifact@v5
- uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5
with:
path: ./site
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v5
uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5
+20 -7
View File
@@ -5,19 +5,28 @@ on:
tags: ['v*']
permissions:
contents: write
id-token: write
contents: read
jobs:
release:
runs-on: ubuntu-latest
environment: pypi
concurrency:
group: release-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: write # for the GitHub Release
id-token: write # for PyPI Trusted Publishing (OIDC) and attestation signing
attestations: write # for SLSA build provenance
# If you add another job to this workflow, give it its own explicit
# `permissions:` block rather than relying on the workflow-level default
# above (contents: read) - do not widen the workflow-level default.
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v5
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: '3.11'
- uses: actions/setup-node@v6
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: '20'
cache: 'npm'
@@ -46,7 +55,11 @@ jobs:
print("Explorer frontend is packaged")
PY
- uses: softprops/action-gh-release@v3
- name: Attest build provenance
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4
with:
subject-path: 'dist/*'
- uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3
with:
files: dist/*
- uses: pypa/gh-action-pypi-publish@release/v1
- uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # release/v1
+118 -70
View File
@@ -28,13 +28,17 @@ jobs:
contents: read
security-events: write
actions: read
# Needed for the "Comment PR with Security Results" step below. Safe on
# pull_request (not pull_request_target): GitHub always forces a
# read-only token for PRs from forks regardless of this permission.
pull-requests: write
steps:
- name: Checkout repository
uses: actions/checkout@v7
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: '3.11'
@@ -42,21 +46,50 @@ jobs:
run: |
python -m pip install --upgrade pip
pip install safety bandit semgrep jq
# Install the project itself (core deps + the LiteLLM provider extra)
# so Safety scans Semantica's actual dependency tree, not just the
# scanner tools' own dependencies.
pip install -e ".[llm-litellm]"
- name: Run Safety Check (Package Vulnerabilities)
run: |
safety check --json --output safety-report.json || true
# NOTE: Safety 3.x repurposed --output to select a console format
# (json/text/screen/...), not a file path. Writing JSON to a file
# now requires --save-json; the previous `--output safety-report.json`
# usage was silently invalid and never produced a report.
safety check --save-json safety-report.json || true
# Guard 1: fail loudly if Safety exited before writing a report at all
# (network error, API auth failure, tool crash). Without this check a
# missing or empty file causes jq to fall back to "0", making a broken
# scanner indistinguishable from a clean scan.
if [ ! -s safety-report.json ]; then
echo "::error::Safety scan produced no report (safety-report.json is missing or empty). Treating as failure — check for network errors, API auth failures, or Safety crashes in the logs above."
exit 1
fi
echo "Checking for package vulnerabilities..."
# Count vulnerabilities safely
VULNS=$(safety check --json --output /dev/stdout 2>/dev/null | jq '.vulnerabilities | length' 2>/dev/null || echo "0")
# No || echo "0" fallback: if jq fails (malformed JSON, missing key,
# vulnerabilities:null) VULNS will be empty or "null" so guard 2 below
# catches it rather than silently treating the broken report as zero.
VULNS=$(jq '.vulnerabilities | length' safety-report.json 2>/dev/null)
# Guard 2: ensure VULNS is a non-negative integer before the -gt
# comparison. "null" (missing/null key) or "" (jq parse failure) would
# cause bash's -gt to throw an arithmetic error and fall through to the
# success branch — the same silent-pass bug as a missing file.
if ! [[ "$VULNS" =~ ^[0-9]+$ ]]; then
echo "::error::Safety report exists but 'vulnerabilities' is missing or non-numeric (got: '${VULNS}'). The report may be malformed or Safety may have written an error-only JSON. Treating as failure."
exit 1
fi
if [ "$VULNS" -gt 0 ]; then
echo "❌ Security vulnerabilities found: $VULNS"
echo "CI will fail to prevent merging of vulnerable dependencies"
echo ""
echo "Vulnerability details:"
safety check || true
jq -r '.vulnerabilities[] | "- \(.package_name)==\(.analyzed_version): \(.vulnerability_id) (\(.CVE // "no CVE assigned"))"' safety-report.json || true
exit 1
else
echo "✅ No security vulnerabilities found"
@@ -99,9 +132,10 @@ jobs:
fi
- name: Upload Security Reports
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: security-reports
retention-days: 14
path: |
safety-report.json
bandit-report.json
@@ -109,77 +143,91 @@ jobs:
- name: Comment PR with Security Results
if: github.event_name == 'pull_request'
uses: actions/github-script@v9
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9
with:
script: |
const fs = require('fs');
// Read safety report
let safetyResults = '';
try {
const safetyData = JSON.parse(fs.readFileSync('safety-report.json', 'utf8'));
if (safetyData.vulnerabilities && safetyData.vulnerabilities.length > 0) {
safetyResults = `## Safety Vulnerabilities Found\\n`;
safetyData.vulnerabilities.forEach(vuln => {
safetyResults += `- **${vuln.package}**: ${vuln.advisory}\\n`;
});
} else {
safetyResults = '## No Safety Vulnerabilities Found\\n';
// Renders one tool's findings as a section. `items` is already
// the list of pre-formatted "- `thing` in `where`" strings; this
// just handles the found/not-found/report-missing framing and
// collapses long lists into a <details> block so the comment
// doesn't turn into a wall of text.
function renderSection(title, reportPath, parse) {
let data;
try {
data = JSON.parse(fs.readFileSync(reportPath, 'utf8'));
} catch (e) {
return [
`### ${title}`,
`⚠️ No report found at \`${reportPath}\` — the scan may have failed before producing output. Check the job logs.`,
].join('\n');
}
} catch (e) {
safetyResults = '## Safety scan completed\\n';
}
// Read bandit report
let banditResults = '';
try {
const banditData = JSON.parse(fs.readFileSync('bandit-report.json', 'utf8'));
if (banditData.results && banditData.results.length > 0) {
const highIssues = banditData.results.filter(issue => issue.issue_severity === 'HIGH');
if (highIssues.length > 0) {
banditResults = `## High Severity Security Issues Found\\n`;
highIssues.forEach(issue => {
banditResults += `- **${issue.test_name}**: ${issue.filename}:${issue.line_number}\\n`;
});
} else {
banditResults = '## No High Severity Security Issues Found\\n';
}
} else {
banditResults = '## No Bandit Issues Found\\n';
const items = parse(data);
if (items.length === 0) {
return [`### ${title}`, `✅ No findings.`].join('\n');
}
} catch (e) {
banditResults = '## Bandit scan completed\\n';
}
// Read semgrep report
let semgrepResults = '';
try {
const semgrepData = JSON.parse(fs.readFileSync('semgrep-report.json', 'utf8'));
if (semgrepData.results && semgrepData.results.length > 0) {
semgrepResults = `## Security Patterns Found\\n`;
semgrepData.results.slice(0, 10).forEach(issue => {
semgrepResults += `- **${issue.rule_id}**: ${issue.path}\\n`;
});
if (semgrepData.results.length > 10) {
semgrepResults += `- ... and ${semgrepData.results.length - 10} more\\n`;
}
const lines = [`### ${title}`, `Found **${items.length}**.`, ''];
const shown = items.slice(0, 15);
if (items.length > 15) {
lines.push('<details>', '<summary>Show all findings</summary>', '');
lines.push(...items);
lines.push('', '</details>');
} else {
semgrepResults = '## No Security Patterns Found\\n';
lines.push(...shown);
}
} catch (e) {
semgrepResults = '## Semgrep scan completed\\n';
return lines.join('\n');
}
// Create summary comment
const comment = `# 🔒 Security Scan Results\\n\\n${safetyResults}\\n\\n${banditResults}\\n\\n${semgrepResults}\\n\\n---\\n\\n*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*\\n\\n📊 **Security Policy**: CI fails on vulnerabilities and HIGH severity issues.`;
// Post comment with error handling
const safetySection = renderSection(
'Safety — dependency vulnerabilities',
'safety-report.json',
(data) => (data.vulnerabilities || []).map(
(v) => `- \`${v.package_name}==${v.analyzed_version}\`: ${v.vulnerability_id}` +
(v.CVE ? ` (${v.CVE})` : '') + ` — ${v.advisory || 'no advisory text'}`
)
);
const banditSection = renderSection(
'Bandit — HIGH-severity code issues',
'bandit-report.json',
(data) => (data.results || [])
.filter((issue) => issue.issue_severity === 'HIGH')
.map((issue) => `- \`${issue.test_name}\` in \`${issue.filename}:${issue.line_number}\``)
);
const semgrepSection = renderSection(
'Semgrep — static analysis patterns',
'semgrep-report.json',
(data) => (data.results || []).map(
(issue) => `- \`${issue.check_id}\` in \`${issue.path}:${issue.start?.line ?? '?'}\``
)
);
const comment = [
'# 🔒 Security Scan Results',
'',
safetySection,
'',
banditSection,
'',
semgrepSection,
'',
'---',
'',
'*This security scan runs automatically on source-code PRs and bi-weekly (skipped for doc/markdown-only changes).*',
'',
'📊 **Security Policy**: CI fails on Safety vulnerabilities and Bandit HIGH-severity findings. Semgrep findings above are informational and do not block merge.',
].join('\n');
try {
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
body: comment,
});
console.log('✅ Security comment posted successfully');
} catch (error) {
+2 -2
View File
@@ -12,8 +12,8 @@ jobs:
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v5
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
with:
python-version: '3.11'
- run: pip install pip-audit
+28
View File
@@ -0,0 +1,28 @@
name: Verify Action Pins
on:
pull_request:
paths:
- '.github/workflows/**'
- '.github/scripts/verify-action-pins.sh'
push:
branches: [main]
paths:
- '.github/workflows/**'
- '.github/scripts/verify-action-pins.sh'
schedule:
- cron: '0 3 * * 1' # weekly, in case an upstream tag is deliberately moved
workflow_dispatch:
permissions:
contents: read
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- name: Verify pinned action SHAs match their tag comments
env:
GH_TOKEN: ${{ github.token }}
run: bash .github/scripts/verify-action-pins.sh
+256
View File
@@ -9,6 +9,262 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Embedded Oxigraph backend for `TripletStore`** (#838, closes #834) by @Linxiushen
- Added `OxigraphStore` (`semantica/triplet_store/oxigraph_store.py`), an in-process SPARQL 1.1 store via the optional `pyoxigraph` dependency — no external server (Blazegraph/Jena/RDF4J/Anzo) required, fixing the confusing plain connection-error failure `TripletStore` previously produced with no server running (no local Docker daemon, no Java, CI, or a fresh laptop)
- Runs fully in memory by default, or persists to a local directory via `TripletStore(backend="oxigraph", path=...)`; reopening the same directory resumes existing data
- Full CRUD, native batch loading (`Store.extend`), named-graph scoping (`graph=` on add/query), and SPARQL SELECT/ASK/CONSTRUCT/DESCRIBE result mapping matching the existing backend contract; reuses `sparql_escaping.py` for datatype-IRI resolution instead of reimplementing it, and preserves RDF literal datatype/language metadata across writes, reads, and query results
- New optional `semantica[tripletstore-oxigraph]` extra (`pyoxigraph>=0.5.0`), included in the `all` extra; the import is lazy, so `TripletStore` and the rest of Semantica keep working without `pyoxigraph` installed
- Wired into `TripletStore` (`backend="oxigraph"`, added to `SUPPORTED_BACKENDS` and `NAMED_GRAPH_CAPABLE_BACKENDS`) and exported from `semantica.triplet_store`; README, module reference, glossary, and usage guide updated with install/configuration examples
- **Fixed along the way**: a missing `pyoxigraph` install surfaced as a generic wrapped `ProcessingError` instead of the underlying `ImportError` and its install hint, because `TripletStore._initialize_store_backend()`'s broad `except Exception` caught and rewrapped it; `ImportError` is now re-raised as-is so the `pip install "semantica[tripletstore-oxigraph]"` hint reaches the caller
- New integration tests in `tests/triplet_store/test_oxigraph_store.py` covering persistence/reopen, named-graph isolation, SELECT/ASK/CONSTRUCT result shapes, and the missing-dependency error message; skipped automatically when `pyoxigraph` isn't installed, and not yet exercised in CI since it doesn't install the optional extra or run the Python test suite
- **PROV-O trust blockers and general spec completeness for `ProvenanceManager`** (#825) by @KaifAhmad1
- **Invalidation instead of hard delete**: new `ProvenanceManager.invalidate(entity_id, agent_id, reason=None)` tombstones an entry — archives its pre-invalidation state under a stable versioned key, then appends the invalidated entry (`invalidated`, `invalidated_at_time`, `invalidated_by`, `invalidation_reason`) — instead of mutating or deleting it, so an audit can prove a fact existed, was reviewed, and was retracted. `ProvenanceManager.clear()` remains the bulk dev/test store-reset utility it always was; it was not repurposed
- **Hash-chained integrity**: every entry now carries `sequence_id`/`previous_checksum`, chaining it to the entry immediately before it in insertion order. New `ProvenanceManager.verify_chain()` walks the chain and reports any break, including a row hard-deleted directly from the underlying table — something a lone per-row SHA-256 checksum can never detect on its own. `compute_checksum()` now also covers `agent_id`/`agent_type`, the lineage-link fields, and the invalidation fields, closing several fields that previously weren't tamper-evident
- **Typed Agent/Activity**: `agent_id` was a dead field — no `track_*` method read it from kwargs, so it was always the `"semantica"` default regardless of what callers passed; fixed, and paired with new `AgentRecord(id, agent_type, is_automated)` / `ActivityRecord(id, activity_type, started_at_time, ended_at_time)` dataclasses (pass via `agent=`/`activity=` kwargs) so a human reviewer, an LLM call, and an automated pipeline stage are now distinguishable, and activities carry real start/end timing. Wired through all 18 `*_provenance.py` wrapper modules and `track_entity`/`track_relationship`/`track_chunk`/`track_property_source`
- **Versioning vs. derivation split**: new `previous_version_id` ("this corrects a prior version of the same fact") and `derived_from_id` ("this was derived from a different source entity") fields, additive alongside the legacy combined `parent_entity_id` so existing readers are unaffected
- **Downstream lineage traversal**: new `get_descendants()`/`trace_descendants()` (reverse BFS in both `InMemoryStorage` and `SQLiteStorage`), closing the gap flagged in `semantica/explorer/routes/provenance.py` where `direction="downstream"` was dead code with no reverse lookup to feed it; the Explorer's `/api/provenance` lineage response now merges both directions
- **W3C PROV-O qualified relations**: `export_prov()` now emits `prov:qualifiedAssociation`/`hadRole` (distinguishing "approved by" from "generated by" for sign-off workflows), `qualifiedGeneration`/`Generation`, `qualifiedUsage`/`Usage`, `qualifiedDerivation`/`Derivation`, `qualifiedInvalidation`/`Invalidation`, `wasAssociatedWith` (Activity→Agent), `actedOnBehalfOf` (Agent→Agent delegation), and `wasInformedBy` (Activity→Activity, via a new `informed_by=[...]` kwarg), alongside the existing plain triples
- **Bitemporal + Bundle support**: `revision_type`/`supersedes`/`valid_from`/`valid_until` fields (plain caller-supplied passthrough, matching the deprecated `kg.ProvenanceTracker`'s actual contract) plus new `revision_history()` and `query_recorded_between()` methods, closing the two "no direct equivalent yet" rows in `docs/migration/kg-provenance-tracker.md`; `bundle_id` emits `prov:Bundle`/`hadMember` membership triples to partition provenance by source/dataset/ingestion-run
- **Configurable, interlinked namespace**: `export_prov(base_uri=...)` / `--base-uri` CLI flag, defaulting to a new `ProvenanceManager.DEFAULT_BASE_URI` (`https://semantica.dev/ns#`) that `RDFExporter`'s `NamespaceManager` and `OWLExporter`'s default `ontology_uri` now both reuse, so KG-exported, OWL-exported, and PROV-exported URIs for the same `entity_id` co-resolve instead of three independently-hardcoded placeholder domains
- New CLI commands: `semantica provenance invalidate|verify-chain|descendants`
- **Fixed along the way**: `track_entities_batch()` silently absorbed batch-level typed kwargs (`agent_id`, `entity_type`, `activity_id`) into the opaque `metadata` JSON blob instead of forwarding them, so the documented banking example in `docs/guides/provenance.md` never actually worked as written
- **Fixed along the way**: `compute_checksum()` had to exclude `entity_id` itself from the hash — `track_entity()`'s versioning archives a prior value by copying it to a new key (`"X"``"X:v:<timestamp>"`), and hashing `entity_id` meant that legitimate relabel permanently orphaned any other entry that had already chained its `previous_checksum` from the pre-relabel value, surfacing as a false-positive "broken chain." Archival and invalidation are now always a pure relabel (unchanged checksum/sequence position) followed by a fresh chained append, never an in-place mutation of an already-chained entry
- **Fixed along the way**: `InMemoryStorage.get_chain_head()` ignored the already-committed chain head whenever the current transaction had staged any entries, understating the head and corrupting the next append's chain link
- **Fixed along the way**: several new `ProvenanceEntry` fields were initially wired into the dataclass and `export_prov()` but not into `SQLiteStorage`'s DDL/INSERT/row-mapping — `InMemoryStorage` stores the dataclass directly so it masked the gap. Added a permanent regression test (`test_all_fields_round_trip_through_sqlite`) asserting every field survives a SQLite round trip, to catch this class of bug for any future field additions
- Flagged, not fixed (separate, pre-existing issues independent of #825): `semantica/pipeline/pipeline_provenance.py` imports a nonexistent module and wraps a `Pipeline` dataclass with no `run()` method, so `PipelineWithProvenance` has never worked; most of the 18 wrapper modules' backing classes are themselves missing or incomplete (e.g. `context.context_manager`, `deduplication.deduplicator`, `normalize.normalizer` don't exist; `EmbeddingGenerator` exists but has no `.embed()`); `kg_provenance.py` passes `entity_type` inside its `metadata={}` dict instead of as a top-level `track_entity()` kwarg across most of its ~30 call sites, so it never actually populates the real field
- Extensive new test coverage across `tests/provenance/test_manager.py`, `test_schemas.py`, and `test_storage.py` (invalidation, hash-chain verification including a simulated hard-delete-detection case and an interleaved-chaining stress test, agent/activity typing, versioning/derivation split, downstream lineage, qualified export triples, bitemporal methods, Bundle export, and namespace interlinking)
- **Altair Anzo triplet store backend** (#813) by @KaifAhmad1
- Added `AnzoStore` (`semantica/triplet_store/anzo_store.py`), a fourth peer to `BlazegraphStore`/`RDF4JStore`/`JenaStore` speaking plain SPARQL 1.1 over HTTP — no new dependency, since Anzo has no official Python SDK but needs none
- The one structural difference from the existing backends: Anzo addresses data by a dataset/graphmart **URI** (`dataset_uri`, required) rather than a short namespace/repository name, so the endpoint path (`<endpoint>/sparql/<store_type>/<url-encoded_dataset_uri>`) percent-encodes it; `store_type` defaults to `"graphmart"` and can be set to `"dataset"`
- Reuses the shared `sparql_escaping.py` literal-escaping, datatype-IRI resolution, and CONSTRUCT-detection helpers rather than reimplementing them, matching `BlazegraphStore`'s CONSTRUCT/bindings `execute_sparql` contract exactly
- Wired into `TripletStore` (`backend="anzo"`, added to `SUPPORTED_BACKENDS` and `NAMED_GRAPH_CAPABLE_BACKENDS`) and `config.py` (`TRIPLET_STORE_ANZO_ENDPOINT` env var / `anzo_endpoint` config key), and exported from `semantica.triplet_store`
- 32 new tests in `tests/triplet_store/test_anzo_store.py` (mocked HTTP, no live Anzo instance needed), including dataset-URI percent-encoding cases that don't apply to the other backends
- Bulk loading uses SPARQL `INSERT DATA` (the same approach `BlazegraphStore` uses) rather than Anzo's separate HTTP Client Interface, keeping the `bulk_load()` contract identical across backends
- **Comprehensive unit and security test suite for the `/api/sparql` Explorer route** (#773) by @Sameer6305
- Added `tests/explorer/test_sparql_route.py` (34 tests) covering the SPARQL Explorer route (`semantica/explorer/routes/sparql.py`), which executes arbitrary SPARQL queries against an in-memory rdflib projection of the live graph and previously had zero test coverage
- Verified read-only allowlist enforcement against write and mutation queries (`INSERT DATA`, `DELETE DATA`, `DELETE WHERE`, `DROP ALL`, `CLEAR ALL`, `LOAD`, `CREATE GRAPH`, `MODIFY`, comments, and multi-statement injections like `SELECT ... ; DROP ALL`), confirming rejected queries short-circuit before any graph is built or queried
- Verified resource-limiting behavior, confirming row capping (`_SPARQL_MAX_ROWS`) truncates results and sets `truncated: true`, query timeout (`_SPARQL_TIMEOUT_S`) returns a clean error message without crashing, and concurrency semaphore (`_SPARQL_MAX_CONCURRENT`) prevents thread starvation under load
- Verified RDF projection fidelity for node properties and edge relationships, and error formatting for malformed SPARQL syntax with line and column extraction
- Follow-up review fixes (#805): extracted the duplicated row-cap-and-truncate loop (previously copy-pasted between the `CONSTRUCT`/`DESCRIBE` and `SELECT` branches) into a shared `_cap_rows()` helper so the `_SPARQL_MAX_ROWS` cap is enforced identically by both; added `test_row_cap_truncates_construct_results`, since the truncation path for `CONSTRUCT`/`DESCRIBE` results had no direct test coverage even though `SELECT` truncation did
- **Global default persistent storage for `ProvenanceManager`, plus a working `provenance` CLI** (#795, #802) by @Sameer6305 and @KaifAhmad1
- Every ingestion/processing module (`kg_provenance.py`, `pipeline_provenance.py`, and 20+ other call sites) instantiated its own `ProvenanceManager()` with no `storage_path`, so all of them silently fell back to `InMemoryStorage` and the SQLite audit trail was never actually written. `ProvenanceManager.set_default_storage_path(path)` now sets a class-level default that every no-arg instantiation picks up, and `Semantica.__init__` wires `config.provenance.storage_path` into it automatically during orchestrator init
- Added the thread-safe `default_storage_path(path)` context manager (`semantica.provenance.default_storage_path`) for test isolation — it stacks nested overrides and guarantees restoration of the previous default on exit, even on exception, so tests can't leak global state into each other
- Fixed `ProvenanceManager.__init__` raising `TypeError` on the CLI's `config=` kwarg, and implemented the four methods the CLI already called but that didn't exist on the class: `lineage()`, `audit_log()`, `export_prov()` (W3C PROV-O turtle/ntriples/jsonld via `rdflib`), and `check()` — unblocking `semantica provenance lineage|audit|export|check` end-to-end
- Follow-up review fixes: `track_entity` no longer aliases a caller-supplied `used_entities` list (it copied the reference and later mutated it in place via `.append()`, which could corrupt a list the caller still held); removed dead fallback branches in `orchestrator.py`/`manager.py` left over from not realizing `Config.get()` already resolves dotted paths; added a `--dry-run` option to `provenance audit` to match `provenance export` (previously only the global `--dry-run` flag worked, not a local one); and `provenance check --strict` no longer prints a green "✓" success line immediately before failing — a failing check now renders as a warning before the `ClickException` is raised
- **Markdown round-trip export/import for `AgentMemory`** (#765, #786) by @SaurabhScripts and @Sameer6305
- `AgentMemory.export(format="markdown")` and `import_data(format="markdown")` add a human-editable, diff-friendly alternative to the existing JSON/dict serialization: one Markdown file per memory item, with `id`, `created_at`, `updated_at`, and `type`/`kind` in required YAML frontmatter and the memory content as the Markdown body
- Exporting without a `destination` returns a single memory as a Markdown string; exporting a set requires a destination directory and writes one stable, content-hashed filename per memory ID, so re-exporting an unchanged set is byte-for-byte idempotent
- Importing upserts by ID: unknown IDs create new memories, known IDs replace them atomically (local state and vector store are only mutated after the whole batch validates cleanly), and unchanged re-imports are a deterministic no-op
- Malformed frontmatter, duplicate IDs within an import batch, and duplicate YAML keys are all rejected before any memory is mutated, with actionable error messages
- Export refuses to overwrite symbolic links and replaces files atomically; import safely compares timezone-aware and timezone-naive timestamps so retention, recency sorting, and date filters stay correct across both
- Entities and relationships round-trip as memory-local provenance only — Markdown import intentionally does not write into `ContextGraph`, matching the MVP scope agreed on in #765
- Documented the file contract and workflow in `docs/reference/context.md`; 43 new tests in `tests/context/test_agent_memory_markdown.py` cover round-trip losslessness, idempotency, validation errors, rollback on failure, and vector-store sync ordering
### Fixed
- **`VectorStore.search_vectors()` returned inconsistent result shapes across backend implementations** (#853, closes #845) by @Sameer6305, reviewed by @KaifAhmad1
- Every built-in backend (FAISS, Milvus, pgvector, Pinecone, Qdrant, SQLite-vec, Weaviate, in-memory) now returns the same canonical `SearchResult` shape (`id`, `score`, `metadata`, `vector`, `distance`), instead of some backends omitting `vector`/`metadata`/`distance` or, for Weaviate, returning a backend-specific `properties` key instead of `metadata`
- Added a `SearchResult` `TypedDict` (`semantica/vector_store/vector_store.py`, exported from `semantica.vector_store`) documenting the contract; `metadata` now always defaults to `{}` rather than being absent, and `id` accepts `Union[str, int]` to accommodate Milvus/Qdrant's native integer IDs without casting
- **Review fix**: the score-normalization formula added for Pinecone and Qdrant (`1.0 / (1.0 + max(0.0, 1.0 - score))`) clamped every raw score `>= 1.0` to an identical `1.0`, silently collapsing result ranking whenever the raw score could exceed 1 — which happens routinely for dot-product-metric indexes (unbounded), as opposed to cosine (bounded to `[-1, 1]`). Replaced with `(score / (1 + |score|) + 1) / 2`, which is strictly monotonic and bounded in `(0, 1)` for any real input, so ranking order is preserved regardless of metric or vector normalization
- Added `test_qdrant_unbounded_dot_product_scores_preserve_ranking` and `test_pinecone_unbounded_dotproduct_scores_preserve_ranking` (`tests/vector_store/test_search_result_schema.py`) asserting normalized scores stay strictly ordered and bounded for raw scores well above 1.0, the case the original formula silently collapsed and the existing tests (which only used scores `< 1`) never exercised
- Left out of scope, per the original PR: Weaviate's `similarity_search()` still isn't wired into `VectorStore.search_vectors()`'s backend dispatch; Milvus's collection schema still has no metadata column so its results always return `metadata: {}`; and `include_vectors` support (populating the `vector` field) is not yet implemented for any backend
- **`DecisionEmbeddingPipeline.find_similar_decisions()` crashed with `AttributeError` for any `VectorStore` backend other than `inmemory`** (#842, closes #839) by @Sameer6305
- `_get_candidate_embeddings()` iterated `VectorStore.vectors`/`VectorStore.metadata` directly, internal dicts only populated for `backend="inmemory"`; every persistent backend (FAISS, Pinecone, Qdrant, Milvus, ...) raised `AttributeError`. It now fetches candidates via the backend-agnostic `VectorStore.search_vectors()`, reading metadata via a `res.get("metadata") or res.get("payload")` fallback for backends that key it differently
- Backends such as FAISS don't return the raw vector for each hit; `find_similar_decisions()` and `_find_semantic_similar()` now fall back to the search-provided score (normalized from `distance` when present) as the semantic similarity for those candidates instead of computing cosine similarity against a zero placeholder vector
- `get_decision_statistics()` had the identical bug iterating `store.metadata.values()`; it now returns a limited stats payload with an explanatory `warning` field for backends that don't expose a full in-memory metadata dict, instead of crashing
- **Fixed along the way**: `_get_candidate_embeddings()`'s expand-and-retry loop (which widens the search pool when post-filtering leaves too few matches) discarded every candidate it had found once the pool hit its cap (`limit * 10`) without ever collecting `limit` matches or getting a short page back from the backend — the loop fell through without executing the branch that assigns results, silently returning `[]` even when matching candidates existed. It now falls back to the last batch collected instead of dropping it
- Added end-to-end regression tests against real `inmemory` and `faiss` backends (no mocks) plus a targeted unit test for the expand-and-retry loop's fallback behavior
- **`QdrantStore.search_vectors()` returned results keyed by `"payload"` instead of `"metadata"`** (#841, closes #840) by @divyankshah
- `QdrantCollection.search_points()` built its result dicts as `{"id", "score", "payload"}`, while `PineconeStore.search_vectors()` and every other backend consumed by `HybridSearch` use `"metadata"`. This silently dropped Qdrant metadata from results and made `HybridSearch.filter_by_metadata()` reject every candidate whenever a filter was applied, since it looks up `result["metadata"]` and got nothing back
- Normalized `search_points()` to return `"metadata"` instead of `"payload"`, matching the existing convention; no other module reads the old key, so the rename is a straight fix rather than a partial one
- Extended `tests/vector_store/test_vector_store_deepdive.py::test_qdrant_store` to assert the returned key is `"metadata"` (not `"payload"`) and that `HybridSearch.filter_by_metadata()` correctly matches against Qdrant results end-to-end
- **Explorer Temporal panel never rendered after clicking the toolbar button** (#830, #836) by @Sameer6305
- The panel stayed permanently stuck on "Loading temporal…" in `npm run dev`, with repeating "Maximum update depth exceeded" errors in the browser console. Two independent render loops were responsible:
- **Diagnostics state churn**: `handleDiagnosticsChange` unconditionally called `setGraphDiagnosticsState` on every invocation. `buildEffectAvailability` (inside `GraphCanvas`'s diagnostics `useEffect`) always returns a new object, so each call scheduled a re-render that immediately retriggered the effect. Fixed by comparing the incoming snapshot field-by-field against the last accepted value via `lastDiagnosticsRef` before calling `setState`
- **scrubberTime churn**: React 18 concurrent mode re-ran `TimelinePanel`'s `useEffect` with a structurally-new `Date` object for the same timestamp when speculative renders discarded `useMemo` caches, causing repeated `setScrubberTime` calls that propagated into `temporalState` churn and retriggered the diagnostics effect. Fixed by deduplicating by millisecond value via `onTimeChange`/`lastScrubberMsRef`
- **Bonus**: `temporal-overlay`'s `shouldLoad` predicate was changed to gate strictly on `panelState["temporal-panel"]`, removing the `|| temporalState?.currentTime` branch that caused eager loading on every scrubber update and continuously cancelled in-flight `load()` completions
- **Bonus**: `temporalState` removed from the plugin-loading `useEffect` dependency array; predicates extracted into `pluginRegistryPredicates.ts` and wired through `GraphWorkspace.tsx` so regression tests exercise the production code rather than a local copy
- The `scrubberTime`-churn fix was also applied to the equivalent (but currently unused/unmounted) `GraphWorkspaceShell.tsx`, which shares the same `TimelinePanel` integration pattern but does not have the diagnostics-churn code path
- **Follow-up review fix**: the diagnostics dedup's `structureLayer` comparison now also covers `disabledReason`, `curveCount`, `bridgeCurveCount`, and `backboneCurveCount` (previously only `cacheKey`/`lastDrawAt`/`enabled` were compared, so a pure `disabledReason` transition could leave the dev-only diagnostics panel stale)
- **Follow-up review fix**: `test:graph-store`, `test:graph-workspace`, and the new `test:plugin-registry` regression test are now run in CI (`.github/workflows/ci.yml`) — previously none of the Explorer frontend's `node --test` suites executed anywhere in CI, only `npm run build`, so this fix's own regression coverage (and all prior frontend test coverage) provided no protection against silent regressions
- **`HybridSearch.search()` crashed with `AttributeError` for any `VectorStore` backend other than `inmemory`** (#833, #837) by @KaifAhmad1
- `HybridSearch.search()` read `self.vector_store.vectors` directly, an internal dict `VectorStore` only populates for `backend="inmemory"`; every other backend (faiss, weaviate, qdrant, milvus, pinecone, pgvector, sqlite) raised `AttributeError`, making `HybridSearch` unusable against any real store. It now delegates to `VectorStore.search_vectors()` (the backend-agnostic public API) for non-inmemory backends, applies `metadata_filter` as a post-filter over the returned candidates, and normalizes results to a consistent `{id, score, distance, metadata}` shape
- **Fixed along the way**: `vector_ids` could stay `None` when callers passed explicit `vectors`/`metadata` without `vector_ids`, crashing downstream list indexing — now defaulted to generated positional IDs
- **Fixed along the way**: a `query_vector` passed as a plain list crashed backend stores (e.g. `FAISSStore.search_similar`) that call `.ndim` on it — now normalized to a numpy array up front
- **Fixed along the way**: `VectorStore.store_vectors()` silently dropped metadata for FAISS (and any `add_vectors`-only backend) because it called `add_vectors(vectors, **options)` without forwarding `metadata`, even though `FAISSStore.add_vectors()` accepts it — this blocked `HybridSearch`'s metadata filtering from ever matching anything on FAISS
- **Follow-up review fixes**: the legacy `top_k` kwarg was read but left in `options`, then forwarded via `**options` into `VectorStore.search_vectors()`, colliding with backends (sqlite, pgvector) that pass an explicit `top_k=k` to their own `search()` and raising `TypeError: got multiple values for keyword argument 'top_k'` — now popped instead of just read; `VectorStore.search_vectors()`'s dispatch only recognized backend methods named `search`/`search_similar`, so delegation still hit `NotImplementedError` for qdrant/milvus/pinecone, which name their method `search_vectors()` with a differently-named count parameter (`limit` vs `k`) — added a third dispatch branch that binds the count positionally so it works regardless of the backend's parameter name; a missing `distance` in backend-delegated results defaulted to the raw `score`, silently reusing the local path's cosine-similarity convention (`distance = 1 - score`) even for backends using unrelated metrics (L2, inner product) — now left as `None` instead of a fabricated, metric-inconsistent value
- Verified across all 7 supported backends: `inmemory`/`faiss`/`sqlite` work live end-to-end; `pgvector`'s dispatch reaches `PgVectorStore.add()`/`.search()` (blocked only by no Postgres server in the verification sandbox); `qdrant`/`milvus`/`pinecone` now reach their real `search_vectors()` method instead of crashing, though their storage side (`store_vectors()`) still doesn't recognize `insert_vectors`/`upsert_vectors`, and `weaviate` remains entirely unwired (`add_objects`/`query_vectors`) on both sides — both are separate, pre-existing gaps independent of this fix, left for a follow-up
- **`VectorStore.store_vectors()` silently dropped metadata for FAISS (and any `add_vectors`-only) backend** (#832, #835) by @KaifAhmad1
- `store_vectors()` fell into a branch that called `self._backend_store.add_vectors(vectors, **options)` without `metadata` whenever the backend exposed `add_vectors()` but neither `add()` nor `store_vectors()` — true for `FAISSStore`, the backend most real usage configures for genuine ANN search. Every caller that stores vectors with metadata (e.g. `AgentMemory._store_memory_vector()`, used internally by `AgentContext.store()`) lost that metadata once it reached FAISS, with no error or warning
- Downstream, `ContextRetriever._retrieve_from_vector()` recovers a result's text via `metadata.get("content", "")`, which was always `""` for any vector stored this way; `_rank_and_merge()` then embedded that empty string, tripping `TextEmbedder.embed_text()`'s empty-text rejection and masking the real bug as a spurious `TextEmbedder` failure recorded by the progress tracker
- `store_vectors()` now forwards `metadata` to `add_vectors()`, but only when the backend's `add_vectors()` signature actually accepts it (checked via `inspect.signature`, accepting either an explicit `metadata` parameter or a `**kwargs` catch-all), so a future/custom backend with a stricter signature raises no `TypeError`
- **Follow-up review fix**: the `inspect.signature()` probe is wrapped in `try/except (ValueError, TypeError)`, consistent with the identical pattern already used in `ProvenanceManager.trace_lineage()`, so signature introspection failing on an unusual callable can no longer abort `store_vectors()` before it even attempts to call the backend
- **`AgnoDecisionKit.check_policy` silently treated unevaluable policy rules as compliant** (#778, #822) by @Sameer6305
- `_eval_rule()` previously `return`ed `True` when a rule referenced a field missing from the decision payload, or when the rule string didn't match the expected `<field> <op> <value>` format — the docstring's claim that exceptions never silently return `compliant=True` didn't cover this, since neither path raised
- Both cases now raise `ValueError` instead, which routes through `check_policy`'s existing exception handler and records a `warnings` entry (e.g. `"Could not evaluate rule 'minimum_score >= 0.9': rule references undefined field 'minimum_score'"`) instead of disappearing with no signal
- `violations`/`compliant` are unaffected — an unevaluable rule is not counted as a violation, since it's genuinely unknown whether it would have passed; this matches the existing `compliant`/`violations`/`warnings` shape already used by `ContextGraph.enforce_decision_policy`
- This is additive: `warnings` was already part of the return contract and populated for other exception cases, so no caller that only checks `compliant` is affected, and no existing test asserts `warnings == []` for a payload that hits either of these paths
- **Follow-up review fix**: `check_policy` decoded `policy_rules` with `json.loads` and iterated the result without checking it was actually a list; a JSON-encoded bare string (e.g. `policy_rules='"confidence >= 0.7"'`) decodes to a `str`, so iterating it evaluated one "rule" per character — combined with the fix above, an 18-character rule string produced 17 warnings instead of being treated as the single rule it was meant to be. A decoded string is now wrapped as a single-element rule list; any other non-list shape (number, object, etc.) or non-string list element now produces exactly one `warnings` entry instead of silently misbehaving or being iterated character-by-character
- **Follow-up review fix**: `_eval_rule` used `data.get(field) is None` to detect a missing field, which can't distinguish a genuinely absent key from a key explicitly present with a JSON `null` value — both produced the same "undefined field" warning, misdiagnosing nullable fields. Field presence is now checked with `field not in data` first; a present-but-`null` value now raises a distinct `"field {field!r} is null — cannot evaluate rule"` message instead of the misleading "undefined field" one
- **Follow-up review fix**: `check_policy` only checked that `decision_data` was valid JSON, not that it decoded to an object. When it decoded to a list, `field not in data` silently became list-*membership* testing instead of a key check (e.g. `"confidence" not in ["confidence", 0.95]` is `False`), so a matching rule fell through to `data["confidence"]`, which raised a raw, confusing `TypeError: list indices must be integers or slices, not str` instead of any meaningful diagnostic; numbers/strings/bools produced similarly opaque `TypeError`s. `check_policy` now rejects any `decision_data` that doesn't decode to a JSON object upfront with a single clear `violations` entry, the same way it already rejects malformed JSON
- Added 15 tests to `tests/integrations/agno/test_decision_kit.py` covering the missing-field case (the issue's traced example), the malformed-rule-string case, the bare-JSON-string `policy_rules` amplification case, non-list/non-string `policy_rules` shapes, the missing-key-vs-null-value distinction, non-object `decision_data` shapes (list/number/string/bool/null), and regression checks confirming normal rule evaluation on present fields is unchanged
- **No cycle detection for SKOS concepts at write time** (#774, #819) by @mikemikimike, reviewed by @Sameer6305 and @KaifAhmad1
- Added cycle detection (`validate_skos_hierarchy`) for `skos:broader` and `skos:narrower` relationships in `ContextGraph.add_edge()` and `ContextGraph.add_edges()`, preventing direct 2-node cycles, self-loops, and multi-hop hierarchy cycles
- Added `GraphSession.add_nodes_and_edges()` to validate SKOS hierarchy edges upfront under lock before node insertion, preventing partial-write leaks where nodes remain after a cyclic edge is rejected
- Updated vocabulary, ontology (`/api/ontology/load`, `/api/ontology/create`), and JSON/CSV import routes to use `add_nodes_and_edges()` and return HTTP 422 with actionable error messages when a cycle is detected
- Follow-up fix by @KaifAhmad1: `validate_skos_hierarchy()` previously re-walked *every* SKOS hierarchy edge already in the graph on each write, so one pre-existing cycle anywhere (e.g. legacy data) blocked all unrelated future writes; it now only traverses concepts touched by the edges being written, while still checking against existing edges for cycles that span old and new data
- Follow-up fix by @KaifAhmad1: in `/api/ontology/load`, `except HTTPException: raise` was unreachable because a broader `except Exception` clause above it already matched `HTTPException`, so a 422 raised after a successful `OntologyIngestor` parse was silently swallowed and reprocessed via the fallback RDF parser; reordered the clauses so the deliberate 422 always propagates
- Follow-up fix (#775): `/api/ontology/{uri}/refresh` was missed by the original sweep and still called `session.add_nodes()` then `session.add_edges()` as two independent operations, so a cyclic SKOS edge rejected by `add_edges()` left the nodes from the preceding `add_nodes()` call committed to the graph; switched to `session.add_nodes_and_edges()` with the same `except ValueError` → HTTP 422 handling already used by `/api/ontology/load` and `/api/ontology/create`. Audited every other `add_nodes()`/`add_edges()` pairing in the repo (`GraphStore`, `graph_builder.py`, `agent_memory.py`, `context_graph.py.load()`, `enrich.py`) — none share `GraphSession`'s SKOS-cycle-validation write path, so none were changed
- **Agno `_AgentScopedStore.upsert_memory` silently swallowed decision recording failures** (#779)
- `upsert_memory()` now logs `logger.warning("[%s] record_decision failed: %s", self._role, exc, exc_info=True)` when `record_decision()` fails, matching the error-logging convention used for `store()` in the same method with traceback context preserved
- Preserves graceful fallback behavior: `record_decision()` remains optional and `upsert_memory()` continues without propagating the exception
- Added regression coverage in `tests/integrations/agno/test_shared_context.py` for both `store()` and `record_decision()` warning paths
- **`AgnoDecisionKit`/`AgnoKGToolkit` silently swallowed Agno tool registration failures** (#780, #818) by @Sameer6305 and @KaifAhmad1
- Removed the `try/except: pass` wrapped around `self.register(fn)` in both toolkits' `__init__`; when Agno is installed, a registration failure now propagates immediately instead of leaving the toolkit half-registered with no signal to the caller
- Graceful degradation when Agno isn't installed (`AGNO_AVAILABLE=False`) is unchanged — `_tools` is still populated so callers can introspect available tools without the package
- Fixed a related duplicate-entry bug: `self._tools` was appended to unconditionally *before* `register()` ran, which could double-count a tool when Agno's own `Toolkit.register()` also tracks it in `self._tools`
- This is a behavior change for callers that construct these toolkits expecting instantiation to always succeed — audited: no in-repo call site relies on the old silent-failure behavior
- Expanded `tests/integrations/agno/test_decision_kit.py` and `test_kg_toolkit.py` with coverage for registration invocation counts, failure propagation, graceful degradation, and no-duplicate-`_tools` assertions
- **`ProvenanceManager` tracking methods silently swallowed failures without logging and returned fabricated entries** (#783)
- `track_relationship()`, `track_chunk()`, and `track_property_source()` now return `Optional[ProvenanceEntry]` (`None` on storage failure, consistent with #782's `track_entity` fix) instead of a fabricated populated object
- `_save_entry()` now always logs on any storage failure, including previously-silent per-item batch failures
- `track_entities_batch()` and `track_chunks_batch()`'s rare block-level transaction failures are now logged too
- `source_tracker.py`'s `track_sources_batch()` no longer counts failed tracking calls in its stats
- **MCP `handle_get_causal_chain` returned an empty-but-valid-looking response when both `CausalChainAnalyzer` and the graph fallback were unavailable** (#781, #817) by @Sameer6305 and @KaifAhmad1
- Returns an explicit `{"error": "Causal chain analysis is not supported on this graph backend", "chain": []}` instead of `{"chain": [], "count": 0, "direction": ...}`, letting clients distinguish "unsupported" from a legitimately empty chain
- The fallback path now introspects `graph.get_causal_chain`'s signature to forward `direction`/`max_depth` (or a `depth` kwarg, or nothing, depending on what the backend accepts) instead of always calling with just `decision_id`, matching the primary analyzer path's behavior
- Hardened input handling: non-dict `args`, non-string `decision_id` (previously a latent `AttributeError` on `.strip()`), and `max_depth` clamped to `(0, 100]` with a safe default on invalid input
- Added `tests/test_mcp_decisions_causal_chain.py` (11 tests) covering the unsupported-backend, fallback-forwarding, and validation/exception paths across multiple backend signature shapes
- **Follow-up review fix**: the signature-detection try/except previously caught the *actual call*'s exceptions in the same block used for introspection failures, so a genuine bug inside a backend's `get_causal_chain` (raising an unrelated `TypeError`) was misread as a signature mismatch and the backend was invoked a second time with identical arguments before the real error surfaced. Signature introspection and the resulting call are now split into separate try/excepts so a successfully-introspected call is made exactly once; added `test_internal_typeerror_calls_backend_only_once` to lock this in
- **`ProvenanceManager.track_entity` persisted partial history and returned fabricated entries on storage failure** (#782, #816) by @Sameer6305 and @KaifAhmad1
- `track_entity()`'s two-step write (history archive + primary update) is now atomic — if either write fails, the whole operation rolls back via the existing #807 `transaction()` mechanism, instead of silently persisting a partial state
- `track_entity()`'s return type is now `Optional[ProvenanceEntry]`: on failure it returns a safe deep copy of the pre-failure existing entry (if one existed) or `None` (if this was a brand-new, never-successfully-tracked entity) — never a fabricated object claiming values that were never actually persisted
- This is a behavior change for callers that inspect the return value without checking for `None` first — audited: 0 of 47 production call sites in the repo currently dereference the return value, so this is safe today, but any NEW caller must handle `None`
- `InMemoryStorage` gained real transactional rollback (staging-buffer based) to match this guarantee — previously `transaction()` was a no-op
- **`ProvenanceManager` duplicated the same checksum/persist/exception-swallow block across 4 tracking methods** (#784, #815) by @Sameer6305 and @KaifAhmad1
- Consolidated the repeated `entry.checksum = compute_checksum(entry)` / `try: self.storage.store(entry) except Exception: pass` block used by `track_entity`, `track_relationship`, `track_chunk`, and `track_property_source` into a single `ProvenanceManager._save_entry()` helper, preserving the existing graceful-failure behavior and the batch `_conn`/re-raise semantics from #807
- Added 4 regression tests (`tests/provenance/test_manager.py`) covering storage-failure swallowing for each of the four tracking methods, none of which had coverage for this path before
- **Follow-up review fix**: the initial refactor of `track_entity`'s exception fallback (the branch that runs when a failure happens *before* the entry is built, e.g. a retrieve error inside the atomic transaction) routed through `_save_entry()`, which made a new `self.storage.store(entry)` call outside the already-failed transaction — a real behavioral change from the original code (which only computed a checksum on that path) that could have reintroduced the exact race #807's `BEGIN IMMEDIATE` transaction serialization was meant to prevent. Reverted that branch to only compute the checksum, and added `test_track_entity_pre_build_failure_fallback_skips_store` asserting `storage.store` is never called on that path
- **`SQLiteStorage` and `ProvenanceManager` connection churn, non-atomic writes, and batch tracking overhead** (#807) by @Sameer6305
- Scoped a single SQLite connection to the full duration of each public storage method call (`track_entity()`, `store()`, `retrieve_all()`, `clear()`) instead of opening independent connections per internal SQL statement, reducing connection churn by ~67% while closing the handle before the public method returns to preserve Windows filesystem unlink safety
- Implemented the `SQLiteStorage.transaction()` context manager with Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `busy_timeout=5000`, `synchronous=NORMAL`, and immediate write transactions (`BEGIN IMMEDIATE`), ensuring concurrent read-modify-write sequences (including history version ID generation) are serialized without lock contention or data loss
- Added block-level transaction sharing to `track_entities_batch()` and `track_chunks_batch()`, reducing SQLite commit overhead by ~99.9% for large batches and deferring `tracked_count` increments until successful commit so rolled-back items are never reported as successes
- Preserved 100% backward compatibility for custom storage backends overriding `trace_lineage(self, entity_id)` by inspecting signatures dynamically before passing `max_depth`, and optimized BFS lineage queries with batched IN-clause lookups per frontier level
- **Follow-up fix**: `retrieve()` and `trace_lineage()` were initially routed through `transaction()` too, so plain reads took the same `BEGIN IMMEDIATE` writer lock as read-modify-write calls, serializing every read behind every other read/write and defeating the WAL concurrency this PR was meant to add. They now use a dedicated `_read_connection()` (configured, no explicit `BEGIN`) so reads no longer contend for the writer lock
- **Follow-up fix**: `track_entity()`/`track_chunk()` caught all internal storage exceptions unconditionally, so when called from `track_entities_batch()`/`track_chunks_batch()`'s shared per-block transaction, a single item's storage failure (e.g. non-JSON-serializable metadata) was swallowed inside the call and never surfaced to the batch loop's per-item `except`, inflating `tracked_count` for entries that were never persisted. Both methods now re-raise when invoked with a shared `_conn` (batch context) while still degrading gracefully on standalone calls, so batch counts match what's actually committed
- Added 8 dedicated regression tests in `tests/provenance/test_sqlite_storage_performance_807.py` covering PRAGMA configuration, Windows unlink safety, batch transaction sharing, BFS `max_depth`, rollback count accuracy, custom storage backward compatibility, concurrent read-modify-write serialization, and connection cleanup guards on configuration error
- **Closed remaining `ProvenanceManager` storage-failure test-coverage gaps identified by a #785 audit** (#785)
- An audit of `tests/provenance/` (filed against a claim that zero tests exercised `storage.store()` failures) found #782/#783/#784/#807 had already closed most of the gap, but two residual surfaces had no test: `track_relationship()`, `track_chunk()`, and `track_property_source()`'s storage-failure-swallowing contract (returns `None`, logs, persists nothing) was only verified against `InMemoryStorage`, never `SQLiteStorage`; and `track_chunks_batch()` had no test for per-item `_save_entry` failure logging or for the block-level transaction-failure log message, even though `track_entities_batch()` had both
- No production code changed — #782/#783/#784/#807 already implemented the correct behavior; this closes the coverage gap proving it holds on both backends
- Added `test_track_relationship_storage_error_swallowed_sqlite`, `test_track_chunk_storage_error_swallowed_sqlite`, `test_track_property_source_storage_error_swallowed_sqlite`, `test_chunks_batch_logs_per_item_failure_memory`, and `test_track_chunks_batch_block_level_transaction_failure_logs` to `tests/provenance/test_manager.py`
- Read-path failure coverage (`get_lineage()`/`trace_lineage()`/`get_provenance()`/`clear()` propagating a raised storage exception) remains untested and is a candidate for a follow-up issue, since none of those methods currently wrap the underlying storage call in a try/except
- **Explorer's Provenance UI used a naive 2-hop graph traversal instead of the audit-grade `ProvenanceManager` backend** (#792, #809) by @Sameer6305
- `semantica/explorer/routes/provenance.py` never imported or called `ProvenanceManager` (`semantica/provenance/manager.py`); `/api/provenance` and `/api/provenance/report` built their lineage response entirely from a naive 2-hop networkx traversal over the live graph instead of querying the SQLite-backed, checksummed audit log. Both endpoints now query `session.provenance_manager.get_lineage(node_id)` first, and a new `_transform_audit_lineage()` maps the W3C PROV-O entries into the exact `{"nodes": [...], "edges": [...]}` shape `LineageDiagram.tsx` already expects — no frontend changes required
- Falls back to the original 2-hop traversal, never a 500: no audit records for a node, a `ProvenanceManager` storage failure (corrupted DB, permissions), or a failed SHA-256 integrity check on any entry in the lineage chain all degrade cleanly to the naive path. A new `source: "audit" | "graph_traversal"` field on the response discloses which path actually served the data
- `ProvenanceManager.get_lineage()` now returns `integrity_verified`, computed by re-verifying every entry's checksum before it's trusted; a single tampered or corrupted entry anywhere in the lineage chain now falls the *entire* response back to graph traversal rather than serving partially-verified audit data
- Replaced an initial classmethod-based `ProvenanceManager.set_default_storage_path()` approach (caught in review before merge — it would have let any two sessions/apps in the same process silently share and overwrite each other's storage path, including across unrelated test runs) with `provenance_storage_path` threaded through `GraphSession.__init__` and `create_app(...)`, so each session's `ProvenanceManager` is independently scoped
- Disclosed limitation: `ProvenanceManager.trace_lineage()`/`get_lineage()` only walk `parent_entity_id`/`used_entities` backward, so the audit path currently surfaces upstream lineage only — the naive fallback remains the only source for downstream/descendant relationships until `ProvenanceManager` gains a reverse lookup
- New `tests/explorer/test_provenance_manager_wiring.py` (8 tests): the audit path via a real multi-hop `track_entity()` chain, empty-record fallback, simulated storage-failure degradation (asserts `200`, not `500`), checksum-tamper fallback, evidence-field preservation, `create_app()` storage-path wiring, and cross-session storage isolation, confirmed order-invariant across `tests/explorer/` and `tests/provenance/` in both execution orders
- **`POST /shacl/validate` and the `/health` SHACL dimension never ran live SHACL validation** (#772, #804) by @Sameer6305 and @KaifAhmad1
- `/shacl/validate` had no data graph to validate submitted shapes against — only a Turtle syntax check. Added `_data_graph_turtle_for_uri()`, which serializes the loaded ontology's nodes/edges into an RDF/Turtle instance graph (CURIE resolution across owl/rdfs/skos/dct/dc, arbitrary node-property projection, typed individuals) and wires both `/shacl/validate` and the `/health` SHACL dimension to `OntologyEngine.validate_graph()` via pySHACL, returning real `conforms`/violations instead of a hardcoded `status="unavailable"` stub
- Fixed a cross-ontology namespace leak in `_node_belongs_to_ontology`: its prefix fallback (`_extract_namespace()`) split only on the last `/`, so sibling ontologies sharing a domain (e.g. `.../onto-a` and `.../onto-b`) could match entities across ontologies that shouldn't be related; fixed by comparing against the full URI stem via the new `_ontology_namespace()` helper
- Added resource guardrails to `/shacl/validate` to close a DoS risk flagged in review: a submitted-Turtle byte cap (`SEMANTICA_MAX_SHACL_TURTLE_BYTES`, default 256 KB), a parsed-triple cap (`SEMANTICA_MAX_SHACL_TRIPLES`, default 1,000), a validation timeout (`SEMANTICA_MAX_SHACL_TIMEOUT`, default 15s), and a global concurrency semaphore (`SEMANTICA_MAX_SHACL_CONCURRENCY`, default 4)
- Fixed `HealthDimension.status` being set to `"error"` on a real (non-`ImportError`) validation exception, which isn't a valid value on that model — Pydantic construction raised and turned the whole `/health` endpoint into a 422 on any real bug; now reports `status="critical"` (already a valid value) with a regression test forcing this exact path
- Follow-up review fixes: reverted an unrelated regression that had crept into this PR — `POST /api/ontology/create` had gone back to silently swallowing `OntologyEngine.from_data`/`from_text` failures into a near-empty "minimal" ontology instead of raising `HTTPException(500)`, undoing the earlier #770/#787 fix for the same endpoint (and breaking `TestOntologyCreateFailures`, which wasn't run before this PR's initial merge request); `sh:Warning`/`sh:Info`-severity pySHACL results were silently dropped from the `/shacl/validate` response — a shape using non-`Violation` severities could report `conforms=False` with an empty `violations` list and no explanation, so warnings/infos are now folded into the response's `violations` array; and `/health` was independently re-fetching and re-truncation-checking the same ontology's nodes/edges once for the generated SHACL shapes and once for the data graph — both now share a single fetch via `_fetch_analysis_graph()`
- New regression tests: `TestOntologyCreateFailures` (pre-existing, now passing again), `test_shacl_validate_surfaces_warning_severity_results`, `test_health_dedupes_node_edge_fetch`, plus the existing 26-test `tests/explorer/test_ontology_subissue3.py` suite (28/28 passing) and the pre-existing `tests/ontology/` suite (83/83 passing)
- **Neptune cookbook CloudFormation stack exposed the database port to the entire internet and had no network audit trail** ([code scanning alert #28](https://github.com/semantica-agi/semantica/security/code-scanning/28), [#26](https://github.com/semantica-agi/semantica/security/code-scanning/26), [#27](https://github.com/semantica-agi/semantica/security/code-scanning/27), `AC_AWS_0276`/`AC_AWS_0369`/`AC_AWS_0148`) by @KaifAhmad1
- `cookbook/introduction/neptune-setup.yaml`'s security group let anyone on `0.0.0.0/0` reach the Neptune Bolt/OpenCypher port (8182); it now requires a `ClientCidr` parameter (CIDR-validated, no default) so the stack can't be created without the deployer explicitly scoping access to their own IP or VPN/office range
- Added `AWS::EC2::FlowLog` plus a dedicated CloudWatch Logs group and IAM role so all traffic in the stack's VPC is now logged
- Left the account-wide IAM password policy check (`AC_AWS_0148`) unimplemented as a stack resource on purpose: `AWS::IAM::AccountPasswordPolicy` is an account singleton, and wiring it into a disposable per-learner tutorial stack would mean creating or deleting this stack also mutates or removes the account's real password policy — suppressed with a documented `ts:skip=AC_AWS_0148` explaining why, rather than "fixed"
- Updated `21_Amazon_Neptune_Store.ipynb`'s `aws cloudformation create-stack` instructions, prerequisites, and cost table to match the new required `ClientCidr` parameter and flow-log line item
- **Follow-up to the knowledge-explorer Helm chart default-namespace/seccomp scanner findings reopening** ([code scanning alert #846](https://github.com/semantica-agi/semantica/security/code-scanning/846), [#847](https://github.com/semantica-agi/semantica/security/code-scanning/847), [#848](https://github.com/semantica-agi/semantica/security/code-scanning/848), [#68](https://github.com/semantica-agi/semantica/security/code-scanning/68), [#63](https://github.com/semantica-agi/semantica/security/code-scanning/63), `CKV_K8S_21`/`AC_K8S_0086`/`AC_K8S_0080`) by @KaifAhmad1
- The `checkov.io/skip1` metadata annotation added previously (see the `CKV_K8S_21` entry below) evidently isn't being honored by the Microsoft Defender for DevOps scan — the same finding reopened under new alert numbers on the current `main`. Added the more standard `# checkov:skip=CKV_K8S_21` and `# ts:skip=AC_K8S_0086` inline comments at the top of `templates/deployment.yaml`, `templates/service.yaml`, and `templates/configmap.yaml` as a second suppression path (matching the convention already used in `deploy/gcp/cloudrun-service.yaml`), plus `# ts:skip=AC_K8S_0080` on `templates/deployment.yaml` for the seccomp finding, which trips for the same root cause: terrascan's static template scan never resolves `{{ toYaml .Values.podSecurityContext }}`, even though `values.yaml` sets `seccompProfile.type: RuntimeDefault` correctly
- Confirmed the `deploy/kubernetes/*` (non-Helm) manifests already had TLS and seccomp configured correctly, so no code change was needed there for the corresponding alerts (#61 and the non-Helm seccomp finding) — expected to close on the next scan
- Documented both suppression mechanisms and the reasoning in `.checkov.yaml`
- Residual risk: this environment could not run checkov/terrascan locally to confirm the inline comments are actually honored during a Helm-rendered scan; if the alerts are still open after the next scan, the reliable fallback is splitting the CI checkov/terrascan invocation so `deploy/helm/` is scanned with these specific checks excluded via `--skip-check` instead of relying on in-file suppression
- **`react-hooks/set-state-in-effect` cascading renders across 12 Explorer workspace files** (#769, #796) by @Sameer6305 and @KaifAhmad1
- Replaced synchronous `setState` calls inside `useEffect` bodies with React's recommended "adjust state during render" pattern (`if (x !== prevX) { setPrevX(x); ...setState... }`) across `OntologyWorkspace`, `ManageWorkspace`, `LineageWorkspace`, and `GraphWorkspace`, and inlined async data-fetching effects with `ignore` flags to prevent race conditions and stale writes after unmount
- Fixed a regression the inlining itself introduced: `AlignmentsTab.tsx`, `KGOverviewTab.tsx`, `OntologyManager.tsx`, and `VersionsTab.tsx` each duplicated their existing fetch callback (`reload` / `fetchOverview` / `fetchRegistry` / `loadVersions`+`loadProposals`) into a second, inline copy for the mount effect, and the copy silently dropped the `setError`/`flashMsg` calls the original had — re-introducing, on the very first page load, the exact error-swallowing behavior that #767/#790 had already fixed for these same files. The inline copies now mirror the original's error handling (including `207` partial-success messages) exactly
- Fixed `LineageDiagram.tsx` only clearing the previously-rendered nodes/edges when the new `activeId` was falsy instead of on every id change, so switching directly between two lineage views briefly kept showing the *previous* view's stale diagram instead of clearing before the new fetch resolved
- `GraphWorkspace.tsx` and `GraphLoadingOverlay.tsx` still have unrelated `react-hooks/set-state-in-effect` violations outside this PR's 12-file scope (confirmed via `npx eslint .`); left as follow-up work rather than expanding this PR further
- **Checkov flagged the knowledge-explorer Helm chart for using the default Kubernetes namespace** ([code scanning alert #779](https://github.com/semantica-agi/semantica/security/code-scanning/779), [#778](https://github.com/semantica-agi/semantica/security/code-scanning/778), [#777](https://github.com/semantica-agi/semantica/security/code-scanning/777), `CKV_K8S_21`) by @KaifAhmad1
- `templates/service.yaml`, `templates/deployment.yaml`, and `templates/configmap.yaml` all already set `metadata.namespace` to `{{ .Release.Namespace }}`, which is only bound at `helm install`/`helm template` time; Checkov's helm framework renders the chart without a namespace override, so it always resolves to `default` and trips `CKV_K8S_21` even though the chart is namespace-agnostic by design
- Added a `checkov.io/skip1: CKV_K8S_21` metadata annotation to each of the three files to suppress the scanner artifact false-positive properly in Helm templates, and documented the reasoning in `.checkov.yaml`
- **No React error boundaries around lazy-loaded Explorer workspaces — a single render error crashed the whole app** (#768, #794) by @Sameer6305
- Added an `ErrorBoundary` class component (`explorer/src/ErrorBoundary.tsx`) and wrapped each lazy-loaded workspace's `<Suspense>` block in `App.tsx` with it, keyed on the active sub-view so navigating away from and back to a crashed tab remounts it cleanly
- Failed retries are capped at 3 before the fallback UI switches from "Try Again" to a "Reload Application" dead-end, preventing infinite retry loops on deterministic crashes; raw error/stack details are logged via `console.error` only and never rendered into the fallback UI
- Fixed the retry counter so it resets after a retry actually succeeds and stays error-free for a few seconds, instead of never resetting (which could permanently exhaust the retry budget on unrelated, individually-recoverable transient errors) or resetting on the very next commit (which could fire prematurely while `Suspense` was still showing its fallback)
- **Explorer frontend workspaces silently swallowed network/server errors** (#767, #790) by @Sameer6305
- `ShaclStudio.tsx`, `VersionsTab.tsx`, `SKOSVocabularyManager.tsx`, `EntityResolutionTab.tsx`, `LineageDiagram.tsx`, `DecisionWorkspace.tsx`, `KGOverviewTab.tsx`, `OntologyManager.tsx`, `OntologySearch.tsx`, `ReasoningWorkspace.tsx`, and `SparqlWorkspace.tsx` now render a visible error banner instead of only `console.error()`-ing failed fetches
- Added explicit `response.status === 207` (Multi-Status) handling across these workspaces so partial backend failures surface a warning instead of reading as a full success (`response.ok` is `true` for all 2xx codes, including 207)
- Added defensive JSON parsing so an unexpected non-JSON (e.g. HTML 500) response body no longer crashes the app with `SyntaxError: Unexpected token < in JSON`
- Fixed `KGOverviewTab.tsx` dropping the `/api/graph/nodes` partial-success warning whenever `/api/graph/stats` also returned 207 — both warnings are now shown (appended) instead of one being silently discarded
- Fixed `HealthTab.tsx`'s registry load still using a bare `.catch(() => {})` that swallowed errors identically to the pattern fixed elsewhere in this same folder; failures now populate the existing error banner
- Fixed `AlignmentsTab.tsx`'s `reload()` using `Promise.allSettled` but never handling the `"rejected"` branches for the registry/alignments fetches, so both failures previously vanished with no error surfaced and no logging
- **`tests/explorer/test_explorer_api.py` failed with `TypeError: Client.__init__() got an unexpected keyword argument 'app'` on current httpx** (#788, #789) by @Sameer6305
- `httpx>=0.28.0` removed the `app=` kwarg that Starlette's `TestClient` relies on to wrap a FastAPI app for testing; `httpx` wasn't pinned anywhere in `pyproject.toml`, so different environments could independently resolve an incompatible transitive version and hit the same break
- Added an explicit `httpx<0.28.0` constraint to the main `[project.dependencies]` array (not just a dev extra), so it applies globally across production, dev, and CI installs
- Without the pin, the full test suite fails to even complete collection (fails immediately on `tests/explorer/test_vocabulary.py` with the same `TestClient` error); with it, `tests/explorer/test_explorer_api.py` goes from 7 failed/12 passed/58 errors to 77 passed, 0 errors
- **Explorer backend routes returned HTTP 200 with error/empty bodies on failure, defeating frontend error handling** (#770, #787) by @Sameer6305 and @KaifAhmad1
- `GET /api/temporal/patterns` now raises `HTTPException(500)` on a genuine computation failure instead of silently returning an empty-but-valid `TemporalPatternResponse`; the `ImportError` fallback (optional `kg` extra not installed) is unchanged and still degrades gracefully to an empty list
- `POST /api/ontology/create` now raises `HTTPException(500)` when ontology generation fails in either the `sample_data` or `schema_text` mode, instead of silently falling back to a partial/minimal ontology with a misleading `nodes_added` count
- `GET /api/analytics` sets `response.status_code = 207` (Multi-Status) when some, but not all, of the requested metrics fail, and raises `HTTPException(500)` when every requested metric fails — a plain 2xx (including 207) reads as success to callers that only check `response.ok`, so an all-failed request now surfaces as a hard error rather than a body full of `{"error": ...}`
- Added regression tests covering all three failure paths (`test_patterns_failure_returns_500`, `test_analytics_partial_failure_returns_207`, `test_analytics_total_failure_returns_500`, and two `TestOntologyCreateFailures` cases)
### Security
- **CI/CD supply-chain hardening against mutable-tag Action compromise (LiteLLM/Trivy-class attack)** (#824) by @KaifAhmad1
- Every third-party GitHub Action across all 8 workflows is now pinned to a full commit SHA instead of a mutable tag (`@v7``@3d3c42e... # v7`), closing the exact vector used against LiteLLM in March 2026 (a compromised Trivy Action tag stole a long-lived publishing token)
- Added `verify-action-pins.yml` + `.github/scripts/verify-action-pins.sh`: a CI check that fails closed on any `uses:` reference that isn't a full SHA (catching a newly introduced mutable tag, not just auditing existing pins) and re-verifies every pin against the GitHub API on each workflow change, on push to `main`, and weekly; an unresolvable API lookup is treated as a failure rather than a silent skip
- `release.yml`: scoped `permissions` to the job level (workflow default is now `contents: read`), added a `concurrency` group so simultaneous tag pushes can't race the publish job, and added SLSA build provenance attestation (`actions/attest-build-provenance`) for every released wheel
- Created a protected `pypi` GitHub Environment (required reviewer, restricted to `v*` tag deployments) and enabled branch protection on `main` (required PR review with stale-approval dismissal, required status checks, no force-push/deletion, required conversation resolution) — PyPI publishing already used Trusted Publishing (OIDC) with no long-lived token
- Grouped Dependabot's `github-actions` updates into a single PR
- **`security-scan.yml`'s Safety dependency-vulnerability check was silently non-functional** (#824) by @KaifAhmad1
- `safety check --json --output safety-report.json` is invalid in Safety 3.x (`--output` now selects a console format, not a file path); the command errored on every run, swallowed by `|| true`, so no report was ever produced and the job always fell back to a generic "scan completed" message with the vulnerability count hardcoded to 0
- Switched to `--save-json`, the correct flag for writing a JSON report to disk; also fixed `vuln.package``vuln.package_name` and Semgrep's `issue.rule_id``issue.check_id` (both produced `undefined` in the PR comment)
- The job never installed Semantica's own dependencies before scanning, so Safety was auditing the scanner tools' own transitive deps, not the project's; added `pip install -e ".[llm-litellm]"` so the actual dependency tree — including the LiteLLM extra — is what gets scanned
- Rewrote the PR-comment builder: every line previously used `\\n` inside JS template literals, which renders as the literal text `\n` rather than a newline, producing an unreadable wall of text; now builds real line arrays and collapses long finding lists into a `<details>` block
- Added the `pull-requests: write` permission the comment-posting step was missing (silently failing via its own try/catch on every prior run)
- **`pypdf2==3.0.1` removed (CVE-2023-36464)** (#824) by @KaifAhmad1
- Surfaced by the Safety fix above: PyPDF2 is a discontinued project (merged into `pypdf`) permanently frozen at the vulnerable 3.0.1 with no patched release possible. `grep -rn "import PyPDF2"` found zero real usages anywhere in the codebase — it was only referenced in docstrings describing a `PyPDF2.PdfReader()` fallback for PDF parsing that was never actually implemented (`pdfplumber` does the real work). Removed the dependency and corrected the stale docstrings in `parse/__init__.py`, `parse/methods.py`, `parse/pdf_parser.py`, and `ingest/email_ingestor.py`
- **10 Bandit B324 false positives suppressed (non-cryptographic MD5 use)** (#824) by @KaifAhmad1
- Surfaced by the same Safety fix restoring a working CI gate: Bandit's HIGH-severity check was blocking on 10 pre-existing `hashlib.md5()` calls, all generating short deterministic cache keys, entity IDs, or IRI suffixes from non-secret input — none used for passwords, tokens, or verifying untrusted data
- Bandit's own message suggests `usedforsecurity=False`, but that keyword argument needs Python 3.9+ and `pyproject.toml` declares `requires-python = ">=3.8"`; used a targeted `# nosec B324` with a one-line justification instead, which suppresses only this check with no runtime behavior change on any supported Python version
## [0.6.0] - 2026-07-21
### Added
+24
View File
@@ -19,6 +19,30 @@ Thank you for your interest in contributing! Every contribution, no matter how s
---
## 🗂️ Working on an Existing Issue
If you want to work on an open GitHub issue, please follow these steps to keep things coordinated and avoid duplicate effort:
1. **Check the issue.** Look at the issue's assignees and recent comments. If someone is already actively working on it, consider a different issue or ask in the comments whether help is welcome.
2. **Comment before you start.** Leave a comment on the issue saying you'd like to work on it — something like *"I'd like to take this on"* is enough. This gives maintainers the context they need to assign the issue appropriately.
3. **Wait for assignment.** A maintainer will review the request and assign the issue when appropriate. Please wait for this before investing significant time in implementation, as priorities and approaches can shift.
4. **Create a branch and implement.** Once assigned, fork the repository (if you haven't already), create a dedicated branch, and begin your work.
```bash
git checkout -b fix/short-description # or feature/short-description
```
5. **Open a focused PR and link the issue.** When you're ready, open a pull request and reference the issue in the description (e.g., `Closes #123`). Keep the PR scoped to the work described in the issue.
> **Why this matters:** Commenting before opening a PR helps maintainers track who is working on what, assign issues correctly, and prevent two contributors from solving the same problem independently. It also gives you a chance to align on the expected approach before writing code.
Not sure where to start? Try a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or ask in [Discord](https://discord.gg/sV34vps5hH).
---
## 🎯 Ways to Contribute
### 💻 Code
+45 -6
View File
@@ -2,6 +2,8 @@
<img src="Semantica Logo.png" alt="Semantica" width="420"/>
<a href="https://trendshift.io/repositories/18986?utm_source=repository-badge&amp;utm_medium=badge&amp;utm_campaign=badge-repository-18986" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/18986" alt="semantica-agi%2Fsemantica | Trendshift" width="250" height="55"/></a>
### Graph-Native Infrastructure for Context and Accountable AI Systems
#### *The Open Source Palantir for AI Agents*
@@ -20,6 +22,10 @@
[![Website](https://img.shields.io/badge/Website-getsemantica.ai-000000?style=flat-square&logo=googlechrome&logoColor=white)](https://getsemantica.ai/) [![Docs](https://img.shields.io/badge/Docs-docs.getsemantica.ai-0099FF?style=flat-square&logo=readthedocs&logoColor=white)](https://docs.getsemantica.ai/) [![Discord](https://img.shields.io/badge/Discord-Join%20Community-5865F2?style=flat-square&logo=discord&logoColor=white)](https://discord.gg/sV34vps5hH) [![Twitter/X](https://img.shields.io/badge/Follow-%40BuildSemantica-000000?style=flat-square&logo=x&logoColor=white)](https://x.com/BuildSemantica) [![YouTube](https://img.shields.io/badge/YouTube-Watch%20Demos-FF0000?style=flat-square&logo=youtube&logoColor=white)](https://www.youtube.com/watch?v=QfnNZg4-dZA) [![Changelog](https://img.shields.io/badge/Changelog-View-6E40C9?style=flat-square&logo=keepachangelog&logoColor=white)](CHANGELOG.md)
```bash
pip install semantica
```
</div>
---
@@ -49,6 +55,7 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
**Who it's for:**
- **AI/ML platform teams** shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index
- **Data platform teams on Databricks or Snowflake** who need to turn tables already sitting in Unity Catalog or a Snowflake warehouse into a governed, lineage-tracked knowledge graph, without exporting that data to a third-party SaaS first
- **Compliance, risk, and audit teams** who need a straight answer to "why did the AI do that?" in a format a regulator will actually accept
- **Regulated enterprises** (finance, healthcare, legal, government, defense) that can't ship a black box, and can't send their data to someone else's SaaS to get one
- **Platform and infra engineers** who want the KG, reasoning, and provenance stack self-hosted and swappable, not locked to one vendor's backend
@@ -66,8 +73,9 @@ Semantica sits underneath your LLM, vector store, and agent framework as a deter
- **Full Auditability:** W3C PROV-O provenance on every fact, with audit trails exportable to JSON, CSV, or RDF
- **Deterministic Reasoning:** Forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes
- **Knowledge Pipeline:** Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and knowledge graph construction, with semantic deduplication and provenance-preserving merges throughout
- **Enterprise Data Platforms:** Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection) and Snowflake (warehouse/database/schema, key-pair and OAuth auth), so tables already living in your lakehouse or warehouse become graph nodes with provenance, not another export/import hop
- **Graph Analytics:** Centrality, community detection, link prediction, and shortest-path queries over the graph you just built
- **Polyglot Graph Storage:** Native RDF (Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
- **Polyglot Graph Storage:** Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code
- **Visualization:** Explore any graph, ontology, or timeline in an interactive browser workbench
- **Drop-in Integrations:** Native Agno support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
@@ -154,7 +162,7 @@ Sources → Ingest → Parse → Normalize → Split → Extract → Conflict De
- **Extract → Conflict Detection → Deduplication:** NER, relations, events, triplets; conflicting facts flagged and resolved before they merge
- **Knowledge Graph:** `GraphBuilder` constructs the graph; bi-temporal facts and full graph analytics (centrality, communities, link prediction) run on top of it
- **Ontology · Reasoning · Provenance · Decisions:** the intelligence layer sitting on the KG, with SHACL/OWL governance, Rete/Datalog/SPARQL inference, W3C PROV-O lineage, and first-class decision records
- **Storage:** polyglot by design, with RDF triple stores (Blazegraph, Apache Jena, Eclipse RDF4J), Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune), and vector stores, all swappable without touching your code
- **Storage:** polyglot by design, with RDF triple stores (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J), Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune), and vector stores, all swappable without touching your code
- **Outputs:** export (RDF, OWL, Parquet, Cypher, JSON-LD), interactive visualization, and access via REST API, MCP server, or CLI
**→ [Full Mermaid diagrams for the pipeline and the decision intelligence lifecycle](ARCHITECTURE.md)**
@@ -309,7 +317,7 @@ Every module below is independently importable, with working code samples verifi
| Module | What it does |
| --- | --- |
| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Snowflake, MCP |
| [`semantica.ingest`](#semanticaingest-multi-source-ingestion) | Files, web, databases, APIs, streams, email, Git, Parquet, Databricks, Snowflake, MCP |
| [`semantica.semantic_extract`](#semanticasemantic_extract-ner-relations-events-triplets) | NER, relation extraction, event detection, triplet generation |
| [`semantica.kg`](#semanticakg-knowledge-graph-construction--analysis) | Graph construction, centrality, communities, link prediction |
| [`semantica.reasoning`](#semanticareasoning-forward-chaining-rete-datalog-sparql) | Forward chaining, Rete, Datalog, SPARQL, fully explainable |
@@ -360,9 +368,38 @@ rows = DBIngestor().ingest_database(
)
```
**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`)
```python
# Enterprise data platforms - pull tables straight out of your lakehouse
# or warehouse, with lineage, instead of exporting to CSV first
from semantica.ingest import DatabricksIngestor, SnowflakeIngestor
Elasticsearch and Google Drive ingestion also ship (`ElasticIngestor`, `GDriveIngestor`) but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly: `from semantica.ingest.elastic_ingestor import ElasticIngestor`.
# pip install "semantica[db-databricks]"
databricks = DatabricksIngestor(
host="https://adb-xxx.azuredatabricks.net",
token="dapi-xxxxxxxx", # or client_id/client_secret for OAuth M2M
http_path="/sql/1.0/warehouses/xxxxxxxx",
catalog="main",
)
customers = databricks.ingest_table("customers", limit=10_000)
sales = databricks.ingest_query("SELECT * FROM sales WHERE region = 'EMEA'")
table_lineage = databricks.get_table_lineage("customers", catalog="main", schema="default") # Unity Catalog lineage
# pip install semantica[db-snowflake]
snowflake = SnowflakeIngestor(
account="myaccount",
user="myuser",
password="mypassword", # or private_key=... for key-pair; use authenticator="oauth", token=... for OAuth
warehouse="COMPUTE_WH",
database="MYDB",
)
orders = snowflake.ingest_table("ORDERS", limit=10_000)
```
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
**Supported sources:** Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Databricks (Unity Catalog + Delta Lake) · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (`ArrowIngestor`)
DuckDB, Elasticsearch, Google Drive, HuggingFace, MongoDB, and Pandas ingestion also ship (`DuckDBIngestor`, `ElasticIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MongoIngestor`, `PandasIngestor`) but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly: `from semantica.ingest.duckdb_ingestor import DuckDBIngestor`.
</details>
@@ -1110,7 +1147,8 @@ if report.valid:
| **Ontology Hub** | SHACL Studio · visual editor · cross-ontology alignments · health dashboard |
| **Vector Store** | FAISS · Pinecone · Weaviate · Qdrant · Milvus · PgVector · hybrid + filtered search |
| **Graph Databases (LPG)** | Neo4j · FalkorDB · Apache AGE · AWS Neptune |
| **Triple Stores (RDF)** | Blazegraph · Apache Jena · Eclipse RDF4J · unified `TripletStore` interface · SPARQL query & bulk load |
| **Triple Stores (RDF)** | Oxigraph (embedded) · Blazegraph · Apache Jena · Eclipse RDF4J · unified `TripletStore` interface · SPARQL query & bulk load |
| **Enterprise Data Platforms** | Databricks (`DatabricksIngestor`: Unity Catalog + Delta Lake, PAT/OAuth M2M, table/query ingestion, catalog/schema/table/lineage introspection) · Snowflake (`SnowflakeIngestor`: warehouse/database/schema, password/key-pair/OAuth auth) |
| **LLM Providers** | **All already supported today:** OpenAI (GPT-4o, o1, o3) · Anthropic (Claude) · Google Gemini · Mistral · Meta Llama · Groq · Cohere · Azure OpenAI · AWS Bedrock · Ollama · DeepSeek · Perplexity · Together AI · Fireworks AI · Replicate · HuggingFace · via `semantica.llms` and LiteLLM |
---
@@ -1475,6 +1513,7 @@ pip install semantica[graph-neo4j] # Neo4j graph store (LPG)
pip install semantica[graph-falkordb] # FalkorDB graph store (LPG)
pip install semantica[graph-apache-age] # Apache AGE graph store (LPG)
pip install semantica[graph-amazon-neptune] # AWS Neptune graph store (LPG)
pip install semantica[tripletstore-oxigraph] # Embedded in-memory/on-disk RDF store
# RDF triple stores (Blazegraph, Apache Jena, Eclipse RDF4J) need no extra:
# semantica.triplet_store talks SPARQL over HTTP using the core `requests` dependency
pip install semantica[vectorstore-qdrant] # Qdrant vector store
+98 -3
View File
@@ -24,7 +24,7 @@ Security vulnerabilities should be reported privately to prevent potential explo
### 2. Report Security Issue
Create a [GitHub Security Advisory](https://github.com/semantica-agi/semantica/security/advisories/new) or contact us through [GitHub Issues](https://github.com/semantica-agi/semantica/issues) with "[SECURITY]" prefix.
Create a [GitHub Security Advisory](https://github.com/semantica-agi/semantica/security/advisories/new) or contact us via the security email listed in `SUPPORT.md`.
Include the following information:
@@ -37,7 +37,7 @@ Include the following information:
### 3. Response Timeline
- **Initial Response**: Within 48 hours
- **Initial Response**: Within 24 hours for critical issues; within 48 hours for non-critical issues
- **Status Update**: Within 7 days
- **Resolution**: Depends on severity and complexity
@@ -112,6 +112,101 @@ We regularly update dependencies to address security vulnerabilities. However, y
- Be cautious with external API calls
- Implement proper authentication and authorization
## CI/CD Supply-Chain Security
Semantica's build and release pipeline is explicitly hardened against
CI/CD supply-chain attacks — the class of attack behind the March 2026
LiteLLM/Trivy incident, where a compromised third-party Action with a
**mutable tag** was used to steal a long-lived publishing token, after which
malicious packages were pushed straight to PyPI without ever touching the
source repository. Every control below maps directly to closing one step of
that attack chain.
### Immutable build inputs
- **Risk**: a tag (`@v4`, `@release/v1`) is re-pointed by a compromised upstream maintainer or account, silently changing what every consumer's CI runs.
**Control**: every third-party GitHub Action in every workflow is pinned to a full 40-character commit SHA, with the human-readable tag kept only as a trailing comment (e.g. `actions/checkout@3d3c42e... # v7`).
- **Risk**: a SHA pin drifts out of sync with its own comment over time, or is mistyped.
**Control**: `verify-action-pins.yml` fails closed on any `uses:` reference that isn't a full commit SHA (catching a newly added mutable tag, not just auditing existing pins), resolves every pinned tag via the GitHub API on each workflow change, on every push to `main`, and weekly, and fails if the SHA no longer matches the tag it claims to be — an API lookup that can't be resolved is treated as a failure, not a silent skip.
- **Risk**: manually re-pinning ~15 actions across 8 workflow files on every upstream release is error-prone.
**Control**: Dependabot (`github-actions` ecosystem) opens a grouped PR that bumps the SHA *and* the tag comment together whenever an action releases — pins never require hand-editing.
### Publishing pipeline (highest-privilege path)
- **Risk**: a long-lived `PYPI_TOKEN` sitting in repo/org secrets is exfiltrated by any compromised step.
**Control**: PyPI publishing uses Trusted Publishing (OIDC) (`id-token: write`) — there is no long-lived PyPI credential anywhere in this repository to steal.
- **Risk**: a compromised CI run publishes to PyPI with no human in the loop.
**Control**: the publish job runs only inside a protected `pypi` GitHub Environment with a required human reviewer — every release needs manual approval in the Actions UI before it runs.
- **Risk**: the release job could be triggered from an arbitrary branch/ref.
**Control**: the `pypi` environment's deployment-branch policy is restricted to `v*` tags only.
- **Risk**: a scanner or unrelated job inherits publish-level credentials.
**Control**: `release.yml` sets `permissions: contents: read` at the workflow level; `contents: write` / `id-token: write` / `attestations: write` are granted only to the release job, never workflow-wide.
- **Risk**: two tag pushes race through the publish pipeline simultaneously.
**Control**: `concurrency: group: release-${{ github.ref }}` serializes releases per tag.
- **Risk**: a consumer can't verify a wheel on PyPI actually came from this repo's CI.
**Control**: SLSA build provenance is attested for every release via `actions/attest-build-provenance`, producing a signed, verifiable record of the exact commit and workflow run that produced the artifact (checkable with `gh attestation verify`).
### Repository controls
- **Risk**: unreviewed or force-pushed changes land on `main`.
**Control**: `main` requires 1 approving PR review (stale approvals dismissed on new pushes), resolved conversations, and blocks force-pushes and branch deletion.
- **Risk**: a PR merges without its security/CI checks passing.
**Control**: merges require the `build`, `Analyze Python` (CodeQL), and `security-scan` checks to pass, in strict mode (checks must be re-run against the latest `main`).
- **Risk**: a compromised scanner job reaches secrets or write access.
**Control**: scanning jobs (`CodeQL`, `security-scan.yml`, `security.yml`, `defender-for-devops.yml`) run with read-only, least-privilege permissions (typically `contents: read` + `security-events: write` only) and never share a job, environment, or secret scope with the publish job.
- **Risk**: secrets are committed accidentally.
**Control**: GitHub secret scanning and push protection are both enabled at the repository level, rejecting pushes that contain recognizable credential patterns before they land in history.
## Automated Security Scanning
Every scan below runs continuously in CI, not just at release time:
- **CodeQL** (`security-and-quality` query pack) — Python source: injection, unsafe deserialization, and other code-level vulnerability classes. Runs in `codeql.yml` on every push/PR to `main` and weekly.
- **Bandit** — Python-specific security anti-patterns (hardcoded secrets, unsafe `eval`/`pickle`, weak crypto, etc.); CI fails on any HIGH-severity finding. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **Semgrep** (`p/security` ruleset) — cross-language static-analysis security patterns. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **Safety** — known CVEs in Semantica's own installed dependencies, including optional LLM-provider extras such as LiteLLM; CI fails on any match. Runs in `security-scan.yml` on every push/PR to `main` and twice weekly.
- **pip-audit** — independent, PyPA-maintained vulnerability database cross-check against installed dependencies (Safety and pip-audit use different advisory sources, so both run). Runs in `security.yml` weekly.
- **Microsoft Defender for DevOps** (`eslint`, `templateanalyzer`, `terrascan`) — JavaScript/TypeScript lint-security rules and infrastructure-as-code misconfigurations. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
- **Checkov** — Kubernetes, Helm, Dockerfile, GitHub Actions, and secrets-pattern IaC scanning; results upload to the same Security tab as CodeQL. Runs in `defender-for-devops.yml` on every push/PR to `main` and weekly.
- **GitGuardian** — secret-detection check on every pull request, installed as a GitHub App integration (not a repo-local workflow). Runs on every PR.
- **GitHub secret scanning + push protection** — blocks known credential patterns before they're pushed, and continuously scans existing history. Platform-level, continuous.
- **Dependabot** — version/security PRs for Python, Docker, and GitHub Actions dependencies, grouped where relevant to reduce review noise. Configured in `.github/dependabot.yml`, runs weekly for security-relevant packages and monthly for docs dependencies.
- **`verify-action-pins.yml`** — enforces that every Action reference is a full commit SHA (failing on a newly introduced mutable tag) and confirms each SHA still matches the tag it claims to be. Runs on every workflow change, every push to `main`, and weekly.
All SARIF-producing scanners (CodeQL, Checkov, Microsoft Defender) publish
findings to the repository's **Security → Code scanning alerts** tab, giving
a single audit trail across tools rather than scattered per-tool reports.
### Adopting this posture in a fork or downstream deployment
Teams standing up their own instance of Semantica, or forking it for an
internal/regulated deployment, can reuse this posture directly:
1. Keep Dependabot's `github-actions` ecosystem entry — it is what keeps
SHA pins current without manual maintenance.
2. Re-run `verify-action-pins.yml` after re-pointing the repository's Actions
at your own mirrors, if you do so.
3. If you publish your own PyPI package from a fork, configure your own
Trusted Publishing trust relationship on PyPI (Trusted Publishing is
scoped to a specific `owner/repo` + workflow filename) and your own
protected environment with your own required reviewers — these are not
transferable from this repository.
4. Branch protection, environment protection, and repository secret
scanning are repository *settings*, not workflow files — cloning or
forking the repo does **not** copy them. They must be re-applied via
the GitHub UI or API on the new repository.
5. GitHub secret scanning and push protection are repository settings that
don't carry over to a fork either — re-enable both under the new
repository's Security settings, not just Dependabot.
6. GitGuardian runs as a GitHub App installation scoped to this specific
repository, not a workflow file — a fork gets no secret-detection
coverage from it until the app is installed separately on the new repo.
7. CodeQL's `upload-sarif` step in `codeql.yml` only runs meaningfully if
Default Setup is *not* already enabled for the repository (it's designed
to skip gracefully otherwise) — check whether Default Setup or Advanced
Setup is active on the new repository and adjust expectations for where
CodeQL findings show up accordingly.
## Dependency Security Policy
### Regular Updates
@@ -156,7 +251,7 @@ We appreciate responsible disclosure. Security researchers who help us improve t
For security-related questions or concerns:
- **GitHub Issues**: [Create an issue](https://github.com/semantica-agi/semantica/issues) with "[SECURITY]" prefix
- **Private Reporting**: Please do not report vulnerabilities in public issues.
- **GitHub Security Advisories**: [Report vulnerability](https://github.com/semantica-agi/semantica/security/advisories/new)
## Additional Resources
@@ -3,81 +3,7 @@
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Amazon Neptune Graph Store\n",
"\n",
"## Overview\n",
"\n",
"This notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n",
"\n",
"### Key Features\n",
"\n",
"- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n",
"- **OpenCypher Support**: Query using standard OpenCypher syntax\n",
"- **Bolt Protocol**: Uses Neo4j Bolt driver for efficient binary communication\n",
"- **Native ~id Support**: Leverages Neptune's native element ID handling\n",
"- **Full CRUD Operations**: Create, read, update, delete nodes and relationships\n",
"- **Automatic Retry**: Built-in retry logic with exponential backoff for transient errors\n",
"\n",
"### Prerequisites\n",
"\n",
"- An Amazon Neptune Database cluster\n",
"- AWS credentials configured (boto3, environment variables, or IAM role)\n",
"- Network access to your Neptune cluster (VPC, security groups)\n",
"\n",
"#### Quick Setup with CloudFormation\n",
"\n",
"If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n",
"\n",
"```bash\n",
"# Deploy the Neptune stack (takes ~15-20 minutes)\n",
"aws cloudformation create-stack \\\n",
" --stack-name semantica-neptune \\\n",
" --template-body file://neptune-setup.yaml \\\n",
" --capabilities CAPABILITY_NAMED_IAM\n",
"\n",
"# Wait for stack creation to complete\n",
"aws cloudformation wait stack-create-complete --stack-name semantica-neptune\n",
"\n",
"# Get the outputs (endpoint, port, credentials)\n",
"aws cloudformation describe-stacks --stack-name semantica-neptune \\\n",
" --query 'Stacks[0].Outputs' --output table\n",
"```\n",
"\n",
"The template creates:\n",
"- VPC with public subnets and Internet Gateway\n",
"- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
"- IAM user with least-privilege access for OpenCypher queries\n",
"- Security group allowing Bolt protocol (port 8182) access\n",
"\n",
"> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
"\n",
"**Outputs:**\n",
"- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
"- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
"- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
"- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
"- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
"\n",
"**Cleanup:**\n",
"```bash\n",
"aws cloudformation delete-stack --stack-name semantica-neptune\n",
"```\n",
"\n",
"**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n",
"\n",
"| Resource | Cost (USD) |\n",
"| --- | --- |\n",
"| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n",
"| Storage (10 GB) | ~1/month |\n",
"| I/O requests | ~1-5/month |\n",
"| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
"| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
"\n",
"> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
"\n",
"---"
]
"source": "# Amazon Neptune Graph Store\n\n## Overview\n\nThis notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n\n### Key Features\n\n- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n- **OpenCypher Support**: Query using standard OpenCypher syntax\n- **Bolt Protocol**: Uses Neo4j Bolt driver for efficient binary communication\n- **Native ~id Support**: Leverages Neptune's native element ID handling\n- **Full CRUD Operations**: Create, read, update, delete nodes and relationships\n- **Automatic Retry**: Built-in retry logic with exponential backoff for transient errors\n\n### Prerequisites\n\n- An Amazon Neptune Database cluster\n- AWS credentials configured (boto3, environment variables, or IAM role)\n- Network access to your Neptune cluster (VPC, security groups)\n- Your public IP address or VPN/office CIDR (run `curl ifconfig.me` to find your public IP), used below to restrict database access\n\n#### Quick Setup with CloudFormation\n\nIf you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n\n```bash\n# Deploy the Neptune stack (takes ~15-20 minutes)\n# Replace 203.0.113.25/32 with your own public IP (run `curl ifconfig.me` to find it)\n# or your office/VPN CIDR. This restricts who can reach the database on the\n# network level - never widen it to 0.0.0.0/0 outside of a short-lived local experiment.\naws cloudformation create-stack \\\n --stack-name semantica-neptune \\\n --template-body file://neptune-setup.yaml \\\n --parameters ParameterKey=ClientCidr,ParameterValue=203.0.113.25/32 \\\n --capabilities CAPABILITY_NAMED_IAM\n\n# Wait for stack creation to complete\naws cloudformation wait stack-create-complete --stack-name semantica-neptune\n\n# Get the outputs (endpoint, port, credentials)\naws cloudformation describe-stacks --stack-name semantica-neptune \\\n --query 'Stacks[0].Outputs' --output table\n```\n\nThe template creates:\n- VPC with public subnets, Internet Gateway, and VPC Flow Logs (to CloudWatch Logs)\n- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n- IAM user with least-privilege access for OpenCypher queries\n- Security group allowing Bolt protocol (port 8182) access only from the `ClientCidr` you specify\n\n> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager. The `ClientCidr` parameter is required (no default) precisely so the database is never silently exposed to the whole internet.\n\n**Outputs:**\n- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n\n**Cleanup:**\n```bash\naws cloudformation delete-stack --stack-name semantica-neptune\n```\n\n**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n\n| Resource | Cost (USD) |\n| --- | --- |\n| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n| Storage (10 GB) | ~1/month |\n| I/O requests | ~1-5/month |\n| Public IPv4 address | ~3.60/month (0.005/hr) |\n| VPC Flow Logs (CloudWatch Logs) | ~1-2/month depending on traffic |\n| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n\n> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n\n---"
},
{
"cell_type": "markdown",
@@ -722,4 +648,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
}
}
+71 -3
View File
@@ -1,7 +1,14 @@
# ts:skip=AC_AWS_0148 IAM password policy is an AWS-account-wide singleton, not a
# per-stack resource. Managing it here would mean every learner who deploys or
# deletes this cookbook stack also mutates (or removes) their account's password
# policy as a side effect. Account password policy should be set once, out of
# band, by the account owner - not by a disposable tutorial stack.
AWSTemplateFormatVersion: '2010-09-09'
Description: >
Amazon Neptune cluster with public endpoint, IAM authentication, and least-privilege
IAM user for Semantica cookbook. Uses db.t3.medium (most cost-effective Neptune instance type).
Network access to the Bolt/OpenCypher port is restricted to an operator-supplied CIDR
(see ClientCidr) - do not widen this to 0.0.0.0/0 outside of a short-lived local experiment.
Parameters:
EnvironmentName:
@@ -9,6 +16,16 @@ Parameters:
Default: semantica-neptune
Description: Environment name prefix for resource naming
ClientCidr:
Type: String
Description: >-
CIDR block allowed to reach the Neptune Bolt/OpenCypher endpoint (port 8182) - e.g. your
workstation's public IP as "x.x.x.x/32", or your office/VPN CIDR. Required: there is no
default, so you must explicitly choose a range. Passing 0.0.0.0/0 is possible but exposes
the database to the entire internet and is strongly discouraged beyond a brief local test.
AllowedPattern: '^((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])/(3[0-2]|[12]?[0-9])$'
ConstraintDescription: Must be a valid IPv4 CIDR block with octets 0-255 and prefix 0-32, e.g. 203.0.113.25/32
Resources:
# =============================================================================
# VPC & NETWORKING
@@ -87,6 +104,57 @@ Resources:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet2
# =============================================================================
# VPC FLOW LOGS
# =============================================================================
FlowLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub /aws/vpc/${EnvironmentName}-flow-logs
RetentionInDays: 30
FlowLogRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${EnvironmentName}-flow-log-role
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: vpc-flow-logs.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: flow-log-publish
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:DescribeLogGroups
- logs:DescribeLogStreams
Resource: "*"
- Effect: Allow
Action:
- logs:CreateLogStream
- logs:PutLogEvents
Resource: !GetAtt FlowLogGroup.Arn
VPCFlowLog:
Type: AWS::EC2::FlowLog
Properties:
ResourceType: VPC
ResourceId: !Ref VPC
TrafficType: ALL
LogDestinationType: cloud-watch-logs
LogGroupName: !Ref FlowLogGroup
DeliverLogsPermissionArn: !GetAtt FlowLogRole.Arn
Tags:
- Key: Name
Value: !Sub ${EnvironmentName}-vpc-flow-log
# =============================================================================
# SECURITY GROUP
# =============================================================================
@@ -95,14 +163,14 @@ Resources:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub ${EnvironmentName}-neptune-sg
GroupDescription: Security group for Neptune cluster - allows Bolt protocol access
GroupDescription: Security group for Neptune cluster - allows Bolt protocol access from ClientCidr only
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 8182
ToPort: 8182
CidrIp: 0.0.0.0/0
Description: Allow Bolt protocol access from anywhere
CidrIp: !Ref ClientCidr
Description: Allow Bolt/OpenCypher protocol access from the operator-specified CIDR
SecurityGroupEgress:
- IpProtocol: -1
CidrIp: 0.0.0.0/0
+3
View File
@@ -9,7 +9,10 @@ flyctl launch --copy-config --config deploy/fly/fly.toml --no-deploy
# Fly.io private networking uses .internal hostnames — do not use localhost
# unless FalkorDB is a co-located process inside the same Machine.
flyctl secrets set FALKORDB_HOST=<falkordb-app-name>.internal FALKORDB_PORT=6379
flyctl secrets set SEMANTICA_API_KEY=$(openssl rand -hex 32)
flyctl deploy --config deploy/fly/fly.toml
```
Change `app` in `fly.toml` before launch if the default app name is already taken.
Fly apps get a public `*.fly.dev` URL by default, so `SEMANTICA_API_KEY` is required — without it the Explorer refuses every protected route (503) rather than serving anonymously. Pass the same value as the `X-API-Key` header from any client that talks to the deployed API.
@@ -1,3 +1,4 @@
# checkov:skip=CKV_K8S_21:Namespace is bound via .Release.Namespace at helm install/template time; this chart is namespace-portable by design.
apiVersion: v1
kind: ConfigMap
metadata:
@@ -5,6 +6,9 @@ metadata:
namespace: {{ .Release.Namespace }}
labels:
{{- include "knowledge-explorer.labels" . | nindent 4 }}
annotations:
runterrascan.io/skip: '[{"rule": "AC_K8S_0086", "comment": "Namespace is bound via .Release.Namespace at helm install time"}]'
checkov.io/skip1: CKV_K8S_21=Namespace bound via .Release.Namespace at helm install/template time
data:
{{- range $key, $value := .Values.env }}
{{ $key }}: {{ $value | quote }}
@@ -5,6 +5,10 @@ metadata:
namespace: {{ .Release.Namespace }}
labels:
{{- include "knowledge-explorer.labels" . | nindent 4 }}
annotations:
runterrascan.io/skip: '[{"rule": "AC_K8S_0086", "comment": "Namespace is bound via .Release.Namespace at helm install time"}, {"rule": "AC_K8S_0080", "comment": "seccompProfile RuntimeDefault is set in values.yaml (podSecurityContext)"}]'
checkov.io/skip1: CKV_K8S_21=Namespace bound via .Release.Namespace at helm install/template time
checkov.io/skip2: CKV_K8S_31=seccompProfile RuntimeDefault set in values.yaml
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
@@ -19,8 +23,10 @@ spec:
{{- include "knowledge-explorer.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
runterrascan.io/skip: '[{"rule": "AC_K8S_0080", "comment": "seccompProfile RuntimeDefault is set in values.yaml (podSecurityContext)"}]'
checkov.io/skip1: CKV_K8S_31=seccompProfile RuntimeDefault set in values.yaml
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
@@ -1,3 +1,4 @@
# checkov:skip=CKV_K8S_21:Namespace is bound via .Release.Namespace at helm install/template time; this chart is namespace-portable by design.
apiVersion: v1
kind: Service
metadata:
@@ -5,6 +6,9 @@ metadata:
namespace: {{ .Release.Namespace }}
labels:
{{- include "knowledge-explorer.labels" . | nindent 4 }}
annotations:
runterrascan.io/skip: '[{"rule": "AC_K8S_0086", "comment": "Namespace is bound via .Release.Namespace at helm install time"}]'
checkov.io/skip1: CKV_K8S_21=Namespace bound via .Release.Namespace at helm install/template time
spec:
type: {{ .Values.service.type }}
ports:
+3
View File
@@ -9,7 +9,10 @@ railway add --database redis
railway variable --set "FALKORDB_HOST=${{Redis.REDISHOST}}"
railway variable --set "FALKORDB_PORT=${{Redis.REDISPORT}}"
railway variable --set "ALLOWED_ORIGINS=https://${{RAILWAY_PUBLIC_DOMAIN}}"
railway variable --set "SEMANTICA_API_KEY=$(openssl rand -hex 32)"
railway up
```
The Redis plugin variables are wired to the requested FalkorDB env names for deployment compatibility. The Explorer currently reads these settings but does not persist graph state to FalkorDB.
Railway exposes this service on a public domain, so `SEMANTICA_API_KEY` is required — without it the Explorer refuses every protected route (503) rather than serving anonymously. Pass the same value as the `X-API-Key` header from any client that talks to the deployed API.
+2
View File
@@ -9,3 +9,5 @@ render blueprint apply deploy/render/render.yaml
```
After creation, update `ALLOWED_ORIGINS` in the Render dashboard if you attach a custom domain.
`SEMANTICA_API_KEY` is auto-generated by the blueprint (`generateValue: true`) since this service gets a public `onrender.com` URL — without it the Explorer refuses every protected route (503) rather than serving anonymously. Find the generated value in the Render dashboard's environment tab and pass it as the `X-API-Key` header from any client that talks to the deployed API.
+2
View File
@@ -20,6 +20,8 @@ services:
type: keyvalue
name: semantica-explorer-redis
property: port
- key: SEMANTICA_API_KEY
generateValue: true
- type: keyvalue
name: semantica-explorer-redis
+2
View File
@@ -16,6 +16,8 @@ services:
ALLOWED_ORIGINS: http://localhost:5173,http://127.0.0.1:5173,http://localhost:8000,http://127.0.0.1:8000
FALKORDB_HOST: falkordb
FALKORDB_PORT: "6379"
# Local dev only: this compose file is not for public exposure.
SEMANTICA_ALLOW_ANONYMOUS: "true"
volumes:
- ./semantica:/app/semantica
- ./pyproject.toml:/app/pyproject.toml:ro
+5
View File
@@ -8,6 +8,11 @@ services:
FALKORDB_HOST: falkordb
FALKORDB_PORT: "6379"
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-http://localhost:8000,http://127.0.0.1:8000}
# Required for API access - the Explorer refuses all protected routes
# (503) until this is set. Generate one with `openssl rand -hex 32`.
SEMANTICA_API_KEY: ${SEMANTICA_API_KEY:-}
# Trusted local-only setups only: bypasses the API key entirely.
SEMANTICA_ALLOW_ANONYMOUS: ${SEMANTICA_ALLOW_ANONYMOUS:-false}
depends_on:
falkordb:
condition: service_started
+1 -1
View File
@@ -23,7 +23,7 @@ Loads data from any source into the pipeline as a unified `SourceDocument`.
| Parquet | `ingest.ParquetIngestor` | PyArrow, Hive-style partitions (v0.5.0) |
| XML | `ingest.XMLIngestor` | XXE-safe lxml, XSD/DTD validation (v0.5.0) |
| Web pages | `ingest.WebIngestor` | Configurable depth, link filtering |
| SQL / Snowflake | `ingest.DBIngestor` / `ingest.SnowflakeIngestor` | Custom SQL, schema introspection |
| SQL / Snowflake / Databricks | `ingest.DBIngestor` / `ingest.SnowflakeIngestor` / `ingest.DatabricksIngestor` | Custom SQL, schema introspection, Unity Catalog lineage |
| Kafka / streams | `ingest.StreamIngestor` | Real-time feed ingestion |
| Email | `ingest.EmailIngestor` | IMAP/SMTP with attachment extraction |
| Repositories | `ingest.RepoIngestor` | Git repos, code structure |
+1 -1
View File
@@ -18,7 +18,7 @@ Find your goal below. The **Module** column is your import path; **Key class** i
| Crawl a website | `ingest` | `WebIngestor` |
| Load Parquet files or partitioned datasets | `ingest` | `ParquetIngestor` |
| Ingest XML with schema validation | `ingest` | `XMLIngestor` |
| Ingest from SQL, Snowflake, Kafka, or email | `ingest` | `DBIngestor`, `SnowflakeIngestor`, `StreamIngestor` |
| Ingest from SQL, Snowflake, Databricks, Kafka, or email | `ingest` | `DBIngestor`, `SnowflakeIngestor`, `DatabricksIngestor`, `StreamIngestor` |
| Extract clean text and tables from a document | `parse` | `DocumentParser` |
| Parse complex PDFs with OCR or multi-column layout | `parse` | `DoclingParser` |
| Chunk text for embedding or RAG | `split` | `TextSplitter` |
+1 -1
View File
@@ -129,7 +129,7 @@ If you're on an older version, install extras individually: `pip install "semant
| :-------- | :------- |
| **Files** | PDF, DOCX, HTML, JSON, CSV, Excel, PPTX, Parquet (v0.5.0), XML (v0.5.0), archives |
| **Web** | `WebIngestor` crawl, RSS feeds, sitemaps |
| **Databases** | PostgreSQL, MySQL, Snowflake via `DBIngestor` / `SnowflakeIngestor` |
| **Databases** | PostgreSQL, MySQL, Snowflake, Databricks via `DBIngestor` / `SnowflakeIngestor` / `DatabricksIngestor` |
| **NoSQL** | MongoDB via `MongoIngestor`, DuckDB via `DuckDBIngestor` |
| **Streams** | Kafka, real-time ingestion via `StreamIngestor` |
| **Protocols** | MCP (Model Context Protocol) via `MCPIngestor` |
+1 -1
View File
@@ -149,7 +149,7 @@ A database optimized for storing and querying graph-structured data using node a
A retrieval strategy combining vector similarity search with keyword or metadata filtering: higher accuracy than either approach alone.
**Triplet Store**
A database designed specifically for storing and querying RDF `(subject, predicate, object)` triples. Semantica supports Blazegraph, Apache Jena, and RDF4J.
A database designed specifically for storing and querying RDF `(subject, predicate, object)` triples. Semantica supports embedded Oxigraph as well as Blazegraph, Apache Jena, and RDF4J.
**Vector Store**
A database optimized for storing and searching high-dimensional embedding vectors by similarity. Semantica supports FAISS, Pinecone, Weaviate, Qdrant, Milvus, and PgVector.
+86
View File
@@ -50,6 +50,7 @@ Use the ingest module when your data lives outside Semantica and you need to bri
- **Web content** — public documentation sites, regulatory publication pages, news feeds, or any URL you can crawl.
- **REST APIs** — internal platforms (SIEM, EDR, ITSM, CRM), threat intelligence feeds, or any paginated HTTP endpoint.
- **Databases** — existing SQL databases where relevant records can be fetched with a targeted query.
- **Enterprise data platforms** — tables already living in a Databricks lakehouse (Unity Catalog + Delta Lake) or a Snowflake warehouse, without exporting to CSV first.
- **Live streams** — Kafka or other message brokers where you need to process events as they arrive.
- **Git repositories** — source code, documentation, or configuration files tracked in version control.
@@ -298,6 +299,89 @@ for bundle in stix_xml_files:
print(f"{bundle.source_path}: {len(bundle.elements)} elements parsed")
```
## Source 6 — Enterprise Data Platforms (Databricks & Snowflake)
`DatabricksIngestor` and `SnowflakeIngestor` return wrapper objects (`DatabricksData` / `SnowflakeData`) whose `.data` field is `List[Dict]` — the same list-of-dicts row shape that `DBIngestor.execute_query()` returns directly, without a wrapper. The same "transform to text, then store" pattern from Source 3 applies: pull only the tables and columns you need with a targeted query, then build a sentence per record before handing it to `AgentContext.store()`.
```python
from semantica.ingest import DatabricksIngestor
# Unity Catalog + Delta Lake — PAT or OAuth M2M auth
databricks = DatabricksIngestor(
host="https://adb-xxx.azuredatabricks.net",
token="dapi-xxxxxxxx",
http_path="/sql/1.0/warehouses/xxxxxxxx",
catalog="main",
)
# .data is List[Dict] — one dict per row, same shape as DBIngestor.execute_query()
customers = databricks.ingest_query(
"SELECT customer_id, name, industry, arr FROM main.default.customers "
"WHERE churn_risk_score > 0.7"
)
customer_texts = [
f"Customer {r['customer_id']} ({r['name']}, {r['industry']}): "
f"ARR ${r['arr']:,}, flagged high churn risk"
for r in customers.data
]
# Unity Catalog lineage — build Table --DEPENDS_ON--> Table edges directly from
# Unity Catalog's own lineage tracking, instead of re-deriving them from query logs
lineage = databricks.get_table_lineage("customers", catalog="main", schema="default")
lineage_texts = [
f"Table main.default.customers depends on {upstream}"
for upstream in lineage["upstream"]
]
```
```python
from semantica.ingest import SnowflakeIngestor
snowflake = SnowflakeIngestor(
account="myaccount",
user="myuser",
password="mypassword", # or private_key=... for key-pair; use authenticator="oauth", token=... for OAuth
warehouse="COMPUTE_WH",
database="ANALYTICS",
schema="PUBLIC",
)
# Snowflake uppercases unquoted identifiers, so unquoted columns come back
# as ORDER_ID, PRODUCT, etc. unless the source table quotes them lowercase
orders = snowflake.ingest_query(
"SELECT order_id, product, region, amount FROM orders "
"WHERE order_date >= DATEADD(day, -30, CURRENT_DATE())"
)
order_texts = [
f"Order {r['ORDER_ID']}: {r['PRODUCT']} in {r['REGION']}, ${r['AMOUNT']}"
for r in orders.data
]
```
Feed the resulting text lists into `AgentContext.store()` exactly like any other structured source:
```python
from semantica.context import AgentContext, ContextGraph
from semantica.vector_store import VectorStore
graph = ContextGraph(advanced_analytics=True)
context = AgentContext(
vector_store = VectorStore(backend="faiss"),
knowledge_graph = graph,
)
context.store(
customer_texts + lineage_texts + order_texts,
extract_entities=True,
extract_relationships=True,
)
print(f"Enterprise data graph: {graph.stats()['node_count']} nodes")
```
For authentication details (PAT vs. OAuth M2M for Databricks; password vs. key-pair vs. OAuth for Snowflake), schema/catalog introspection, and troubleshooting, see the dedicated [Databricks Integration](../integrations/databricks) and [Snowflake Integration](../integrations/snowflake) guides.
> **Security Note:** Never hardcode credentials (`token`, `password`, `private_key`) in production code; pass them via environment variables (e.g., `DATABRICKS_TOKEN`, `SNOWFLAKE_PASSWORD`) or a secrets manager.
## Combining All Five Sources
Once you have text from each source, `AgentContext.store()` accepts a flat list of strings. Semantica embeds and indexes them together — the context graph has no concept of which string came from which source unless you add metadata explicitly.
@@ -831,3 +915,5 @@ print(f"Compliance graph: {graph.stats()['node_count']} nodes, "
- [Context Graphs](context-graphs) — storing and querying the entities you ingest as a typed property graph
- [Semantic Extraction](semantic-extraction) — NER, relation extraction, and triplet extraction from ingested text
- [Provenance](provenance) — tracking the origin document, confidence score, and ingestion timestamp for every extracted entity
- [Databricks Integration](../integrations/databricks) — Unity Catalog setup, PAT/OAuth M2M authentication, and lineage introspection
- [Snowflake Integration](../integrations/snowflake) — warehouse setup and password/key-pair/OAuth authentication
+25 -4
View File
@@ -83,9 +83,13 @@ prov = ProvenanceManager(storage=SQLiteStorage("audit.db"))
For any regulated deployment — security operations, clinical data, financial risk — use `storage_path`. A SQLite file can be backed up, versioned, and queried with standard tools without requiring a server.
<Note>
`SQLiteStorage` automatically configures Write-Ahead Logging (`WAL`), `busy_timeout=5000`, and `synchronous=NORMAL`, and executes read-modify-write operations (like `track_entity()`) in atomic immediate transactions (`BEGIN IMMEDIATE`); plain reads (`retrieve()`, `trace_lineage()`) use a separate connection without an explicit write lock so they don't serialize behind writers. Furthermore, `ProvenanceManager` automatically supports custom storage backends overriding only `trace_lineage(self, entity_id)` without requiring `max_depth` in their signature.
</Note>
## Recording provenance when ingesting data
The moment data enters your graph is the moment provenance must be recorded. `track_entity()` captures the source document, the timestamp, the operator or pipeline that ran the extraction, a verbatim quote from the source, and a confidence score. It returns a `ProvenanceEntry` with a SHA-256 checksum computed automatically.
The moment data enters your graph is the moment provenance must be recorded. `track_entity()` captures the source document, the timestamp, the operator or pipeline that ran the extraction, a verbatim quote from the source, and a confidence score. It returns an `Optional[ProvenanceEntry]` (`ProvenanceEntry` on success, or `None` if storage fails on a brand-new entity) with a SHA-256 checksum computed automatically.
```python
# Ingesting CVE-2024-3400 from NVD and a commercial feed
@@ -629,12 +633,29 @@ Every `ProvenanceEntry` maps directly to W3C PROV-O terms. If your compliance te
| :--- | :--- | :--- |
| `prov:Entity` | `entity_id` | The tracked object — entity, chunk, relationship, or property |
| `prov:Activity` | `activity_id` | The process that produced it — `"ner_extraction"`, `"bureau_parsing"` |
| `prov:Agent` | `agent_id` | Who ran the activity — pipeline name, analyst ID |
| `prov:wasDerivedFrom` | `parent_entity_id` | The previous version of this entity — enables version chaining |
| `prov:Agent` / `prov:Person` / `prov:SoftwareAgent` / `prov:Organization` | `agent_id`, `agent_type`, `is_automated` | Who — or what — ran the activity, and whether a human was directly accountable |
| `prov:qualifiedAssociation` + `prov:hadRole` | `role` | The agent's role for this specific entity — `"generator"` (default), `"approver"`, `"reviewer"` — for sign-off/four-eyes workflows |
| `prov:wasDerivedFrom` | `parent_entity_id` (legacy combined field) | The previous version or source of this entity |
| — | `previous_version_id` | This entry corrects/replaces a prior version of the *same* fact |
| `prov:wasDerivedFrom` | `derived_from_id` | This entry was derived from a *different* source entity |
| `prov:used` | `used_entities` | Entity IDs consumed to produce this one |
| `prov:generatedAtTime` | `timestamp` | ISO datetime, auto-set to `datetime.utcnow()` at write time |
| `prov:qualifiedInvalidation` | `invalidated`, `invalidated_at_time`, `invalidated_by`, `invalidation_reason` | A retraction/correction recorded as a tombstone via `ProvenanceManager.invalidate()`, never a hard delete |
| `prov:startedAtTime` / `prov:endedAtTime` | `activity_started_at_time`, `activity_ended_at_time` | Typed Activity timing — pass an `ActivityRecord` via the `activity=` kwarg to set these together with `activity_id` |
| `prov:qualifiedGeneration`/`Generation`, `qualifiedUsage`/`Usage`, `qualifiedDerivation`/`Derivation` | (derived from the fields above) | Additive qualified forms of `wasGeneratedBy`/`used`/`wasDerivedFrom`, emitted automatically alongside the plain triples |
| `prov:wasAssociatedWith` | (derived from `agent_id`) | Direct Activity→Agent link, distinct from the Entity→Agent `wasAttributedTo` |
| `prov:actedOnBehalfOf` | `acted_on_behalf_of` | Agent→Agent delegation — e.g. an automated agent acting on behalf of the human/organization that authorized it |
| `prov:wasInformedBy` | `informed_by_activities` (pass as `informed_by=[...]`) | Chains this entry's activity to prior activities it was informed by (e.g. a pipeline stage informed by the stage before it) |
| `prov:Bundle` + `prov:hadMember` | `bundle_id` | Groups entries by source/dataset/ingestion-run (membership triples, not true RDF named-graph partitioning) |
| — | `valid_from`, `valid_until`, `revision_type`, `supersedes` | Bitemporal fields merged from the deprecated `kg.ProvenanceTracker` — always caller-supplied (never auto-computed), surfaced via `ProvenanceManager.revision_history()`, which falls back to timestamp-based derivation for entries that don't set them explicitly |
The `checksum` field is not part of the PROV-O standard — it is Semantica's tamper-detection extension. Every entry's SHA-256 is computed from its content fields at write time and can be recomputed at any time to verify the record has not been modified.
`previous_version_id` and `derived_from_id` are additive alongside `parent_entity_id` — existing code reading `parent_entity_id` keeps working unchanged, while new code gets the two relations disambiguated.
The `checksum` field is not part of the PROV-O standard — it is Semantica's tamper-detection extension. Every entry's SHA-256 now also incorporates `previous_checksum` (the prior entry's checksum, by insertion order via `sequence_id`), chaining every entry to the one before it. `ProvenanceManager.verify_chain()` walks the full chain and reports any break — including a row that was hard-deleted from the underlying table, which a lone per-row checksum can't detect on its own.
Note: the banking example above passes `agent_id="credit_data_service_v2"` to `track_entities_batch()` — this now actually populates the entry's `agent_id` field (previously a bug caused batch-level typed kwargs like `agent_id`/`entity_type`/`activity_id` to be silently absorbed into the opaque `metadata` blob instead).
`export_prov()` mints entity/agent/activity URIs under `ProvenanceManager.DEFAULT_BASE_URI` (`https://semantica.dev/ns#` by default — the same namespace `RDFExporter`'s `NamespaceManager` uses for its `"semantica"` prefix, so KG-exported and PROV-exported URIs for the same `entity_id` co-resolve) unless overridden via `export_prov(base_uri=...)` or the CLI's `--base-uri` option.
## Related Guides
+4
View File
@@ -3,6 +3,10 @@ title: "Semantica"
description: "The Accountability and Context Layer for AI: Context Graphs · Decision Intelligence · Full Provenance"
---
```bash
pip install semantica
```
Your AI agent just made a decision. Now someone needs to explain it.
*What did it know at the time? Which facts shaped the outcome? Where did those facts come from? Has it made the same call before: and did that go well?*
+2 -2
View File
@@ -17,8 +17,8 @@ Every method on `kg.ProvenanceTracker` now emits a `DeprecationWarning` on use,
| `track_entity(entity_id, source, metadata)` | `track_entity(entity_id, source, metadata)` | Same call shape. `ProvenanceManager` additionally auto-links each update to its prior version via `parent_entity_id`. |
| `get_all_sources(entity_id)` | `get_all_sources(entity_id)` | Field name differs: the `kg` tracker returns each record's time under `"recorded_at"`; `ProvenanceManager` returns `"timestamp"`. |
| `clear(entity_id=None)` | `clear()` | `ProvenanceManager.clear()` clears all provenance data; there is no per-entity clear yet. |
| `query_recorded_between(start, end)` | *No direct equivalent yet* | Filter the entries returned by `get_lineage()` / `trace_lineage()` client-side in the meantime. |
| `revision_history(fact_id)` | *No direct equivalent yet* | `get_lineage(fact_id)["lineage_chain"]` returns the full chain of `ProvenanceEntry` records but not in the same versioned shape. |
| `query_recorded_between(start, end)` | `query_recorded_between(start, end)` | Same call shape; filters by `timestamp` (ISO 8601 string comparison) across all tracked entries, not just one entity. |
| `revision_history(fact_id)` | `revision_history(fact_id)` | Same call shape and return shape (`version`, `valid_from`, `valid_until`, `recorded_at`, `author`, optional `revision_type`/`supersedes`) — walks the entity's `previous_version_id` chain rather than a flat per-entity dict. |
| `export_audit_log(fact_ids, format)` | *No direct equivalent yet* | Build the export from `get_lineage()` output, or serialize `get_statistics()` for a summary view. |
Methods with no direct equivalent are not planned to be reimplemented on `kg.ProvenanceTracker` — they will need a small adapter in caller code, or a feature request against `ProvenanceManager` if you rely on them heavily.
+14 -6
View File
@@ -31,7 +31,7 @@ Semantica is organized into **27 modules** across six logical layers. Each modul
Loads data from files, web, databases, and streams into a unified `SourceDocument` format.
```python
from semantica.ingest import FileIngestor, WebIngestor, ParquetIngestor, XMLIngestor
from semantica.ingest import FileIngestor, WebIngestor, ParquetIngestor, XMLIngestor, DatabricksIngestor
# Files: PDF, DOCX, CSV, Excel, PPTX, JSON, HTML, archives
ingestor = FileIngestor()
@@ -39,18 +39,26 @@ documents = ingestor.ingest_directory("data/")
# Web crawl
web_ingestor = WebIngestor()
pages = web_ingestor.ingest_urls(["https://example.com"])
page = web_ingestor.ingest_url("https://example.com")
# Parquet: single file, partitioned directory, Hive-style (v0.5.0)
parquet = ParquetIngestor()
sources = parquet.ingest("data/events.parquet")
# XML with XSD/DTD validation, namespace handling (v0.5.0)
xml = XMLIngestor(validate_xsd="schema.xsd")
sources = xml.ingest("data/records/")
xml = XMLIngestor()
sources = xml.ingest("data/records/", schema_path="schema.xsd")
# Enterprise lakehouse/warehouse — Unity Catalog + Delta Lake, or a Snowflake warehouse
databricks = DatabricksIngestor(host="...", token="...", http_path="...")
customers = databricks.ingest_table("customers")
```
**Available ingestors:** `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor`, `RESTIngestor`, `PublicAPIIngestor`, `DBIngestor`, `DuckDBIngestor`, `ElasticIngestor`, `EmailIngestor`, `FeedIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MCPIngestor`, `MongoIngestor`, `OntologyIngestor`, `PandasIngestor`, `RepoIngestor`, `SnowflakeIngestor`, `StreamIngestor`
**Available ingestors:** `FileIngestor`, `WebIngestor`, `ParquetIngestor`, `XMLIngestor`, `RESTIngestor`, `PublicAPIIngestor`, `DBIngestor`, `DatabricksIngestor`, `SnowflakeIngestor`, `EmailIngestor`, `FeedIngestor`, `MCPIngestor`, `OntologyIngestor`, `RepoIngestor`, `StreamIngestor`, `ArrowIngestor`, `CloudStorageIngestor`
<Note>
`DuckDBIngestor`, `ElasticIngestor`, `GDriveIngestor`, `HuggingFaceIngestor`, `MongoIngestor`, and `PandasIngestor` also ship but aren't re-exported from the top-level `semantica.ingest` namespace yet — import them directly, e.g. `from semantica.ingest.duckdb_ingestor import DuckDBIngestor`.
</Note>
### Parse
@@ -243,7 +251,7 @@ store.add_triplets(subject, predicate, obj)
results = store.sparql("SELECT ?s ?p ?o WHERE { ?s ?p ?o }")
```
**Backends:** Blazegraph, Apache Jena, RDF4J
**Backends:** Oxigraph (embedded), Blazegraph, Apache Jena, RDF4J
## Quality Assurance
+45
View File
@@ -586,6 +586,51 @@ history = memory.get_conversation_history(conversation_id="conv_001", max_items=
| `max_memory_size` | `int` | `10000` | Max items before LRU eviction |
| `retention_policy` | `str` | `"unlimited"` | `"N_days"` (e.g. `"30_days"`) or `"unlimited"` |
### Markdown Round Trips
`AgentMemory` can export human-editable Markdown and import the edited files back.
Each file contains one memory item, with required metadata in YAML frontmatter and
the memory content in the Markdown body:
```markdown
---
id: mem_compliance_rule
created_at: '2026-07-22T09:00:00+00:00'
updated_at: '2026-07-22T10:30:00+00:00'
type: compliance
tags:
- trading
- approval
---
All trades must be pre-approved.
```
```python
from pathlib import Path
# A single selected memory can be returned as Markdown text.
document = memory.export(format="markdown", type="compliance")
# Export a memory set as one stable Markdown file per item.
memory.export(format="markdown", destination="memory_export/")
# New IDs create memories; existing IDs are updated in place.
count = memory.import_data(Path("memory_export/"), format="markdown")
```
The required frontmatter fields are `id`, `created_at`, `updated_at`, and either
`type` or `kind`. Optional metadata can be edited at the top level. Imports reject
malformed or duplicate fields before changing memory, and re-importing unchanged
files is idempotent. Memory-local `entities` and `relationships` are preserved as
provenance but are not applied to `ContextGraph` by Markdown import. Use a dedicated
export directory: matching files are overwritten, but unrelated or stale Markdown
files are not deleted automatically. Export refuses to overwrite symbolic links and
uses atomic file replacement. Timestamp offsets are preserved in Markdown and
normalized to UTC only for comparisons, so aware and local-naive records can be
queried together safely. Vector-store writes are deferred until the in-memory import
commits; adapter synchronization remains best-effort and logs failures.
## PolicyEngine
+6 -1
View File
@@ -258,11 +258,16 @@ Full interactive docs at `http://localhost:8000/docs`. All endpoints accept and
| `/api/vocabulary/hierarchy` | `GET` | Concept hierarchy tree |
| `/api/vocabulary/import` | `POST` | Import SKOS/RDF vocabulary file |
SKOS hierarchy writes reject cycles in both `skos:broader` and
`skos:narrower` relationships. Vocabulary imports validate the complete
batch before adding nodes, while direct graph/session edge writes apply
the same invariant at the graph storage boundary.
**SPARQL:**
| Endpoint | Method | Description |
| :-------- | :------ | :----------- |
| `/api/sparql` | `POST` | Execute a SPARQL SELECT or ASK query |
| `/api/sparql` | `POST` | Execute a read-only SPARQL query (`SELECT`, `ASK`, `CONSTRUCT`, or `DESCRIBE`); `CONSTRUCT`/`DESCRIBE` return triples as `subject`, `predicate`, `object` columns, and `ASK` returns a `result` boolean column |
</Accordion>
<Accordion title="Decisions, Provenance, Annotations & Export">
+2 -1
View File
@@ -6,7 +6,7 @@ icon: "database"
**`semantica.ingest`** is the **universal entry point** for loading data into Semantica:
- 15+ ingestion adapters: files, web, SQL, Snowflake, Kafka, MCP, Git repos, email
- 15+ ingestion adapters: files, web, SQL, Databricks, Snowflake, Kafka, MCP, Git repos, email
- PyArrow Parquet with column selection and partitioned dataset support
- XXE-safe lxml XML with optional XSD schema validation
- `ingest()` unified dispatcher: auto-detects source type from path or URL
@@ -29,6 +29,7 @@ icon: "database"
| `SnowflakeIngestor` | Snowflake data warehouse queries and table exports |
| `DatabricksIngestor` | Databricks Unity Catalog metadata, Delta table queries, and lineage |
| `ParquetIngestor` | Apache Parquet files and partitioned datasets with column selection |
| `ArrowIngestor` | Apache Arrow IPC and Feather file processing |
| `XMLIngestor` | XXE-safe XML parsing with optional XSD schema validation |
| `EmailIngestor` | IMAP/POP3 email ingestion with attachment extraction |
| `OntologyIngestor` | OWL/RDF/Turtle ontology file ingestion |
+11 -6
View File
@@ -155,13 +155,15 @@ prop_entry = manager.track_property_source(
### Batch Tracking
Batch tracking methods process items in blocks (default `batch_size=1000`) inside a shared transaction per block. Only entities or chunks that successfully commit to storage are added to the returned count, preventing rolled-back entries from inflating success counts.
```python
entities = [
{"id": "entity_1", "confidence": 0.9},
{"id": "entity_2", "confidence": 0.85},
]
count = manager.track_entities_batch(entities, source="doc_1")
# Returns the number of entities successfully tracked
# Returns the number of entities successfully tracked and committed
chunks = [
{"id": "chunk_0", "start_index": 0, "end_index": 500},
@@ -219,10 +221,10 @@ cleared = manager.clear()
| Method | Returns | Description |
| :------ | :------- | :----------- |
| `track_entity(entity_id, source, metadata, **kwargs)` | `ProvenanceEntry` | Record entity provenance; checksum set automatically |
| `track_relationship(relationship_id, source, metadata, **kwargs)` | `ProvenanceEntry` | Record relationship provenance |
| `track_chunk(chunk_id, source_document, ...)` | `ProvenanceEntry` | Record chunk provenance with char offsets |
| `track_property_source(entity_id, property_name, value, source)` | `ProvenanceEntry` | Record property-level source attribution |
| `track_entity(entity_id, source, metadata, **kwargs)` | `Optional[ProvenanceEntry]` | Record entity provenance atomically; returns `ProvenanceEntry` on success, or `None`/existing entry on storage failure |
| `track_relationship(relationship_id, source, metadata, **kwargs)` | `Optional[ProvenanceEntry]` | Record relationship provenance; returns `ProvenanceEntry` on success, or `None` on storage failure |
| `track_chunk(chunk_id, source_document, ...)` | `Optional[ProvenanceEntry]` | Record chunk provenance with char offsets; returns `ProvenanceEntry` on success, or `None` on storage failure |
| `track_property_source(entity_id, property_name, value, source)` | `Optional[ProvenanceEntry]` | Record property-level source attribution; returns `ProvenanceEntry` on success, or `None` on storage failure |
| `track_entities_batch(entities, source)` | `int` | Batch-track entities; returns success count |
| `track_chunks_batch(chunks, source_document)` | `int` | Batch-track chunks; returns success count |
| `get_lineage(entity_id)` | `Dict[str, Any]` | Full lineage as aggregated dict |
@@ -234,7 +236,7 @@ cleared = manager.clear()
## ProvenanceEntry Fields
`ProvenanceEntry` is the core dataclass. Every tracking method returns one:
`ProvenanceEntry` is the core dataclass. Every tracking method returns one on success (or `None` on storage failure):
```python
from semantica.provenance import ProvenanceEntry
@@ -322,6 +324,9 @@ manager = ProvenanceManager(storage_path="provenance.db")
`SQLiteStorage` creates the database and indexes automatically on first use.
- **Atomicity & Concurrency**: Configures Write-Ahead Logging (`PRAGMA journal_mode=WAL`), `PRAGMA busy_timeout=5000`, and `PRAGMA synchronous=NORMAL`. Read-modify-write methods (`track_entity()`, `store()`) open a single connection and execute inside an immediate write transaction (`BEGIN IMMEDIATE`), ensuring these sequences are serialized across concurrent connections without leaving open file handles across calls. Plain reads (`retrieve()`, `trace_lineage()`) use a separate connection with no explicit write lock, so concurrent reads don't serialize behind writers or each other.
- **Backward Compatibility**: Custom storage subclasses overriding `trace_lineage(self, entity_id)` remain backward compatible; `ProvenanceManager` inspects the override signature and automatically calls it with one argument if `max_depth` is unsupported.
## Tamper-Evident Checksums
`compute_checksum` and `verify_checksum` are auto-used by `track_entity` and all other tracking methods. You can also call them directly:
+35 -9
View File
@@ -1,6 +1,6 @@
---
title: "Triplet Store Module"
description: "RDF triple storage with SPARQL queries and bulk loading: Blazegraph, Apache Jena, and RDF4J."
description: "Embedded and server-backed RDF storage with SPARQL queries and bulk loading."
icon: "table"
---
@@ -16,14 +16,15 @@ icon: "table"
| `BlazegraphStore` | Blazegraph REST API: SPARQL 1.1 Update, namespace management |
| `JenaStore` | Apache Jena: rdflib-backed, SPARQL read support via remote endpoint |
| `RDF4JStore` | Eclipse RDF4J: REST API, transaction support |
| `OxigraphStore` | Embedded SPARQL 1.1 store with in-memory and on-disk modes |
## What You Get
- **TripletStore** — Unified interface across Blazegraph, Apache Jena, and RDF4J: swap backends with one parameter.
- **TripletStore** — Unified interface across embedded Oxigraph, Blazegraph, Apache Jena, and RDF4J: swap backends with one parameter.
- **SPARQL** — Full SPARQL SELECT, ASK, CONSTRUCT, and UPDATE query support via `execute_query()`.
- **Bulk Loading**`add_triplets()` batches writes with configurable batch size, retry logic, and progress tracking.
- **SKOS Vocabulary** — Built-in helpers: `add_skos_concept()` and `get_skos_concepts()` for controlled vocabulary management.
- **Named Graphs** — Blazegraph and RDF4J support named graph scoping via `graph=` on `execute_query()`.
- **Named Graphs** Oxigraph, Blazegraph, and RDF4J support named graph scoping via `graph=` on `execute_query()`.
- **Delta Computation**`compute_delta(old_graph_uri, new_graph_uri)` returns added and removed triples between two named graph snapshots.
## Getting Started
@@ -117,6 +118,25 @@ for row in result.bindings:
## Backends
<Tabs>
<Tab title="Oxigraph">
```bash
pip install "semantica[tripletstore-oxigraph]"
```
```python
# In-memory: no server process or files required
store = TripletStore(backend="oxigraph")
# Persistent: reopen the same directory to reuse the data
persistent_store = TripletStore(
backend="oxigraph",
path="./data/knowledge-graph",
)
```
**Best for:** local development, CI, desktop applications, and persistent
single-process workloads without external infrastructure.
</Tab>
<Tab title="Blazegraph">
```bash
pip install requests
@@ -172,6 +192,7 @@ for row in result.bindings:
| Backend | License | Named Graphs | Write via | Best For |
| :------- | :------- | :------------ | :--------- | :-------- |
| Oxigraph | Apache 2.0 / MIT | Yes | Embedded native API | Local, CI, on-disk |
| Blazegraph | Open source | Yes | SPARQL Update REST | High triple count, SPARQL 1.1 |
| Apache Jena | Apache 2.0 | No (rdflib backend) | rdflib in-process | Local dev, read queries |
| RDF4J | Eclipse 1.0 | Yes | REST API N-Triples | Enterprise Java, transactions |
@@ -180,7 +201,9 @@ for row in result.bindings:
</Tabs>
<Tip>
**Use Apache Jena for development, Blazegraph for production.** Jena initializes with rdflib in-memory: no server required for local testing. Switch to Blazegraph for high-throughput persistent workloads by changing `backend=`.
**Use Oxigraph for zero-infrastructure development and local persistence.**
Switch to a server-backed store for distributed production deployments by
changing `backend=`.
</Tip>
## Triplet Object
@@ -364,10 +387,10 @@ while True:
## Named Graph Scoping
Blazegraph and RDF4J support named graphs. Scope `execute_query()` to a named graph with the `graph=` parameter:
Oxigraph, Blazegraph, and RDF4J support named graphs. Scope `execute_query()` to a named graph with the `graph=` parameter:
```python
# Add a triplet: named graph stored in metadata or backend-specific API
# Add a triplet to a named graph
from semantica.semantic_extract.types import Triplet
t = Triplet(
@@ -375,7 +398,7 @@ t = Triplet(
predicate="http://example.org/p",
object="http://example.org/b",
)
store.add_triplet(t) # named graph targeting requires backend-specific API
store.add_triplet(t, graph="http://example.org/graph1")
# Query a named graph via FROM clause in SPARQL
result = store.execute_query("""
@@ -393,11 +416,14 @@ result = store.execute_query("""
```
<Note>
Named graph support is only available for Blazegraph and RDF4J backends. The `graph=` parameter is silently ignored for the Jena backend.
Named graph query scoping is available for Oxigraph, Blazegraph, and RDF4J.
The `graph=` query parameter is silently ignored for the Jena backend.
</Note>
<Tip>
**Use named graphs to isolate sources.** Pass `graph="http://example.org/source_A"` to `execute_query()` to scope a query to a specific named graph. Blazegraph and RDF4J support named graphs; Jena (rdflib backend) does not.
**Use named graphs to isolate sources.** Pass `graph="http://example.org/source_A"`
to writes and `execute_query()` to scope both storage and retrieval. Oxigraph,
Blazegraph, and RDF4J support named graph query scoping.
</Tip>
## Bulk Loading
+104 -360
View File
@@ -37,7 +37,7 @@
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^4.3.0",
"babel-plugin-react-compiler": "^1.0.0",
"eslint": "^9.39.4",
"eslint": "^10.8.0",
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.5.2",
"globals": "^17.4.0",
@@ -836,81 +836,44 @@
}
},
"node_modules/@eslint/config-array": {
"version": "0.21.2",
"resolved": "https://registry.npmjs.org/@eslint/config-array/-/config-array-0.21.2.tgz",
"integrity": "sha512-nJl2KGTlrf9GjLimgIru+V/mzgSK0ABCDQRvxw5BjURL7WfH5uoWmizbH7QB6MmnMBd8cIC9uceWnezL1VZWWw==",
"version": "0.23.5",
"resolved": "https://registry.npmjs.org/@eslint/config-array/-/config-array-0.23.5.tgz",
"integrity": "sha512-Y3kKLvC1dvTOT+oGlqNQ1XLqK6D1HU2YXPc52NmAlJZbMMWDzGYXMiPRJ8TYD39muD/OTjlZmNJ4ib7dvSrMBA==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"@eslint/object-schema": "^2.1.7",
"@eslint/object-schema": "^3.0.5",
"debug": "^4.3.1",
"minimatch": "^3.1.5"
"minimatch": "^10.2.4"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
}
},
"node_modules/@eslint/config-helpers": {
"version": "0.4.2",
"resolved": "https://registry.npmjs.org/@eslint/config-helpers/-/config-helpers-0.4.2.tgz",
"integrity": "sha512-gBrxN88gOIf3R7ja5K9slwNayVcZgK6SOUORm2uBzTeIEfeVaIhOpCtTox3P6R7o2jLFwLFTLnC7kU/RGcYEgw==",
"version": "0.7.0",
"resolved": "https://registry.npmjs.org/@eslint/config-helpers/-/config-helpers-0.7.0.tgz",
"integrity": "sha512-DObd/KKUsU+FaFv4PLxSRenpXfQWmPXXP3pPZ6/K1PCrMu2vQpMDMuQe/BqYeoLcz8ro0bVDF1RxOJgfVEdhUw==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"@eslint/core": "^0.17.0"
"@eslint/core": "^1.2.1"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
}
},
"node_modules/@eslint/core": {
"version": "0.17.0",
"resolved": "https://registry.npmjs.org/@eslint/core/-/core-0.17.0.tgz",
"integrity": "sha512-yL/sLrpmtDaFEiUj1osRP4TI2MDz1AddJL+jZ7KSqvBuliN4xqYY54IfdN8qD8Toa6g1iloph1fxQNkjOxrrpQ==",
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/@eslint/core/-/core-1.2.1.tgz",
"integrity": "sha512-MwcE1P+AZ4C6DWlpin/OmOA54mmIZ/+xZuJiQd4SyB29oAJjN30UW9wkKNptW2ctp4cEsvhlLY/CsQ1uoHDloQ==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"@types/json-schema": "^7.0.15"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
}
},
"node_modules/@eslint/eslintrc": {
"version": "3.3.5",
"resolved": "https://registry.npmjs.org/@eslint/eslintrc/-/eslintrc-3.3.5.tgz",
"integrity": "sha512-4IlJx0X0qftVsN5E+/vGujTRIFtwuLbNsVUe7TO6zYPDR1O6nFwvwhIKEKSrl6dZchmYBITazxKoUYOjdtjlRg==",
"dev": true,
"license": "MIT",
"dependencies": {
"ajv": "^6.14.0",
"debug": "^4.3.2",
"espree": "^10.0.1",
"globals": "^14.0.0",
"ignore": "^5.2.0",
"import-fresh": "^3.2.1",
"js-yaml": "^4.1.1",
"minimatch": "^3.1.5",
"strip-json-comments": "^3.1.1"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
},
"funding": {
"url": "https://opencollective.com/eslint"
}
},
"node_modules/@eslint/eslintrc/node_modules/globals": {
"version": "14.0.0",
"resolved": "https://registry.npmjs.org/globals/-/globals-14.0.0.tgz",
"integrity": "sha512-oahGvuMGQlPw/ivIYBjVSrWAfWLBeku5tpPE2fOPLi+WHffIWbuh2tCjhyQhTBPMf5E9jDEH4FOmTYgYwbKwtQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=18"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
"node": "^20.19.0 || ^22.13.0 || >=24"
}
},
"node_modules/@eslint/js": {
@@ -927,27 +890,27 @@
}
},
"node_modules/@eslint/object-schema": {
"version": "2.1.7",
"resolved": "https://registry.npmjs.org/@eslint/object-schema/-/object-schema-2.1.7.tgz",
"integrity": "sha512-VtAOaymWVfZcmZbp6E2mympDIHvyjXs/12LqWYjVw6qjrfF+VK+fyG33kChz3nnK+SU5/NeHOqrTEHS8sXO3OA==",
"version": "3.0.5",
"resolved": "https://registry.npmjs.org/@eslint/object-schema/-/object-schema-3.0.5.tgz",
"integrity": "sha512-vqTaUEgxzm+YDSdElad6PiRoX4t8VGDjCtt05zn4nU810UIx/uNEV7/lZJ6KwFThKZOzOxzXy48da+No7HZaMw==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
}
},
"node_modules/@eslint/plugin-kit": {
"version": "0.4.1",
"resolved": "https://registry.npmjs.org/@eslint/plugin-kit/-/plugin-kit-0.4.1.tgz",
"integrity": "sha512-43/qtrDUokr7LJqoF2c3+RInu/t4zfrpYdoSDfYyhg52rwLV6TnOvdG4fXm7IkSB3wErkcmJS9iEhjVtOSEjjA==",
"version": "0.7.2",
"resolved": "https://registry.npmjs.org/@eslint/plugin-kit/-/plugin-kit-0.7.2.tgz",
"integrity": "sha512-+CNAzxglkrpNf/kKywqQfk74QjtceuOE7Qm+AF8miRvPF/wmmK5+OJOgVh3AVTT3RP2mH3+FOaxlE5v72owk0A==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"@eslint/core": "^0.17.0",
"@eslint/core": "^1.2.1",
"levn": "^0.4.1"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
}
},
"node_modules/@humanfs/core": {
@@ -1602,6 +1565,13 @@
"@types/d3-selection": "*"
}
},
"node_modules/@types/esrecurse": {
"version": "4.3.1",
"resolved": "https://registry.npmjs.org/@types/esrecurse/-/esrecurse-4.3.1.tgz",
"integrity": "sha512-xJBAbDifo5hpffDBuHl0Y8ywswbiAp/Wi7Y/GtAgSlZyIABppyurxVueOPE8LUQOxdlgi6Zqce7uoEpqNTeiUw==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/estree": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz",
@@ -1849,45 +1819,6 @@
"typescript": ">=4.8.4 <6.1.0"
}
},
"node_modules/@typescript-eslint/typescript-estree/node_modules/balanced-match": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz",
"integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==",
"dev": true,
"license": "MIT",
"engines": {
"node": "18 || 20 || >=22"
}
},
"node_modules/@typescript-eslint/typescript-estree/node_modules/brace-expansion": {
"version": "5.0.6",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.6.tgz",
"integrity": "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==",
"dev": true,
"license": "MIT",
"dependencies": {
"balanced-match": "^4.0.2"
},
"engines": {
"node": "18 || 20 || >=22"
}
},
"node_modules/@typescript-eslint/typescript-estree/node_modules/minimatch": {
"version": "10.2.5",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz",
"integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==",
"dev": true,
"license": "BlueOak-1.0.0",
"dependencies": {
"brace-expansion": "^5.0.5"
},
"engines": {
"node": "18 || 20 || >=22"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/@typescript-eslint/typescript-estree/node_modules/semver": {
"version": "7.7.4",
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz",
@@ -1943,19 +1874,6 @@
"url": "https://opencollective.com/typescript-eslint"
}
},
"node_modules/@typescript-eslint/visitor-keys/node_modules/eslint-visitor-keys": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-5.0.1.tgz",
"integrity": "sha512-tD40eHxA35h0PEIZNeIjkHoDR4YjjJp34biM0mDvplBe//mB+IHCqHDGV7pxF+7MklTvighcCPPZC7ynWyjdTA==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": "^20.19.0 || ^22.13.0 || >=24"
},
"funding": {
"url": "https://opencollective.com/eslint"
}
},
"node_modules/@vitejs/plugin-react": {
"version": "4.7.0",
"resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-4.7.0.tgz",
@@ -2016,9 +1934,9 @@
"license": "MIT"
},
"node_modules/acorn": {
"version": "8.16.0",
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.16.0.tgz",
"integrity": "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw==",
"version": "8.17.0",
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.17.0.tgz",
"integrity": "sha512-xRQbDb9BnwDafYNn6Vwl839DYVjqXYb1XVGtWAZ1kcDc6iwAL4hg3B1dZlRiuENFeO2H53gFG3in621AdERVAg==",
"dev": true,
"license": "MIT",
"bin": {
@@ -2039,9 +1957,9 @@
}
},
"node_modules/ajv": {
"version": "6.14.0",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.14.0.tgz",
"integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==",
"version": "6.15.0",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.15.0.tgz",
"integrity": "sha512-fgFx7Hfoq60ytK2c7DhnF8jIvzYgOMxfugjLOSMHjLIPgenqa7S7oaagATUq99mV6IYvN2tRmC0wnTYX6iPbMw==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -2055,29 +1973,6 @@
"url": "https://github.com/sponsors/epoberezkin"
}
},
"node_modules/ansi-styles": {
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
"dev": true,
"license": "MIT",
"dependencies": {
"color-convert": "^2.0.1"
},
"engines": {
"node": ">=8"
},
"funding": {
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
}
},
"node_modules/argparse": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
"integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==",
"dev": true,
"license": "Python-2.0"
},
"node_modules/attr-accept": {
"version": "2.2.5",
"resolved": "https://registry.npmjs.org/attr-accept/-/attr-accept-2.2.5.tgz",
@@ -2098,11 +1993,14 @@
}
},
"node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz",
"integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==",
"dev": true,
"license": "MIT"
"license": "MIT",
"engines": {
"node": "18 || 20 || >=22"
}
},
"node_modules/baseline-browser-mapping": {
"version": "2.10.20",
@@ -2118,14 +2016,16 @@
}
},
"node_modules/brace-expansion": {
"version": "1.1.14",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.14.tgz",
"integrity": "sha512-MWPGfDxnyzKU7rNOW9SP/c50vi3xrmrua/+6hfPbCS2ABNWfx24vPidzvC7krjU/RTo235sV776ymlsMtGKj8g==",
"version": "5.0.8",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.8.tgz",
"integrity": "sha512-JZyDyq3D4AUifKTPOB7DELf6XsB3WdPuNxCtob1vFXPsSXhdAiHBWJ/tJ8HAc9aH84BK+5JFZLNkJKx3G9kzQg==",
"dev": true,
"license": "MIT",
"dependencies": {
"balanced-match": "^1.0.0",
"concat-map": "0.0.1"
"balanced-match": "^4.0.2"
},
"engines": {
"node": "20 || >=22"
}
},
"node_modules/browserslist": {
@@ -2162,16 +2062,6 @@
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
}
},
"node_modules/callsites": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz",
"integrity": "sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/caniuse-lite": {
"version": "1.0.30001788",
"resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001788.tgz",
@@ -2193,49 +2083,12 @@
],
"license": "CC-BY-4.0"
},
"node_modules/chalk": {
"version": "4.1.2",
"resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz",
"integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==",
"dev": true,
"license": "MIT",
"dependencies": {
"ansi-styles": "^4.1.0",
"supports-color": "^7.1.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/chalk?sponsor=1"
}
},
"node_modules/classcat": {
"version": "5.0.5",
"resolved": "https://registry.npmjs.org/classcat/-/classcat-5.0.5.tgz",
"integrity": "sha512-JhZUT7JFcQy/EzW605k/ktHtncoo9vnyW/2GspNYwFlN1C/WmjuV/xtS04e9SOkL2sTdw0VAZ2UGCcQ9lR6p6w==",
"license": "MIT"
},
"node_modules/color-convert": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
"integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"color-name": "~1.1.4"
},
"engines": {
"node": ">=7.0.0"
}
},
"node_modules/color-name": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
"dev": true,
"license": "MIT"
},
"node_modules/commander": {
"version": "2.20.3",
"resolved": "https://registry.npmjs.org/commander/-/commander-2.20.3.tgz",
@@ -2253,13 +2106,6 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/concat-map": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
"integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==",
"dev": true,
"license": "MIT"
},
"node_modules/convert-source-map": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz",
@@ -2447,9 +2293,9 @@
}
},
"node_modules/dompurify": {
"version": "3.4.11",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.11.tgz",
"integrity": "sha512-zhlUV12GsaRzMsf9q5M254YhA4+VuF0fG+QFqu6aYpoGlKtz+w8//jBcGVYBgQkR5GHjUomejY84AV+/uPbWdw==",
"version": "3.4.13",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.13.tgz",
"integrity": "sha512-2vmYIoqjze2d+kakP8S/nS5shfsl587kzwEjcGlTdiksUVgFHnFCsLYDVj/JNqJVOQZGSYBTmuycv0PodwmnMQ==",
"license": "(MPL-2.0 OR Apache-2.0)",
"peer": true,
"optionalDependencies": {
@@ -2529,33 +2375,33 @@
}
},
"node_modules/eslint": {
"version": "9.39.4",
"resolved": "https://registry.npmjs.org/eslint/-/eslint-9.39.4.tgz",
"integrity": "sha512-XoMjdBOwe/esVgEvLmNsD3IRHkm7fbKIUGvrleloJXUZgDHig2IPWNniv+GwjyJXzuNqVjlr5+4yVUZjycJwfQ==",
"version": "10.8.0",
"resolved": "https://registry.npmjs.org/eslint/-/eslint-10.8.0.tgz",
"integrity": "sha512-nuKKvN+oIBO0koN7Tm7dlkmnkc21mtt0QJLwAKzjLq14y6lRTdVG36MZHJ8eQHwdJMwZbQNMlPOYedMq/oVJvQ==",
"dev": true,
"license": "MIT",
"workspaces": [
"packages/*"
],
"dependencies": {
"@eslint-community/eslint-utils": "^4.8.0",
"@eslint-community/regexpp": "^4.12.1",
"@eslint/config-array": "^0.21.2",
"@eslint/config-helpers": "^0.4.2",
"@eslint/core": "^0.17.0",
"@eslint/eslintrc": "^3.3.5",
"@eslint/js": "9.39.4",
"@eslint/plugin-kit": "^0.4.1",
"@eslint-community/regexpp": "^4.12.2",
"@eslint/config-array": "^0.23.5",
"@eslint/config-helpers": "^0.7.0",
"@eslint/core": "^1.2.1",
"@eslint/plugin-kit": "^0.7.2",
"@humanfs/node": "^0.16.6",
"@humanwhocodes/module-importer": "^1.0.1",
"@humanwhocodes/retry": "^0.4.2",
"@types/estree": "^1.0.6",
"ajv": "^6.14.0",
"chalk": "^4.0.0",
"cross-spawn": "^7.0.6",
"debug": "^4.3.2",
"escape-string-regexp": "^4.0.0",
"eslint-scope": "^8.4.0",
"eslint-visitor-keys": "^4.2.1",
"espree": "^10.4.0",
"esquery": "^1.5.0",
"eslint-scope": "^9.1.2",
"eslint-visitor-keys": "^5.0.1",
"espree": "^11.2.0",
"esquery": "^1.7.0",
"esutils": "^2.0.2",
"fast-deep-equal": "^3.1.3",
"file-entry-cache": "^8.0.0",
@@ -2565,8 +2411,7 @@
"imurmurhash": "^0.1.4",
"is-glob": "^4.0.0",
"json-stable-stringify-without-jsonify": "^1.0.1",
"lodash.merge": "^4.6.2",
"minimatch": "^3.1.5",
"minimatch": "^10.2.5",
"natural-compare": "^1.4.0",
"optionator": "^0.9.3"
},
@@ -2574,7 +2419,7 @@
"eslint": "bin/eslint.js"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
},
"funding": {
"url": "https://eslint.org/donate"
@@ -2619,48 +2464,50 @@
}
},
"node_modules/eslint-scope": {
"version": "8.4.0",
"resolved": "https://registry.npmjs.org/eslint-scope/-/eslint-scope-8.4.0.tgz",
"integrity": "sha512-sNXOfKCn74rt8RICKMvJS7XKV/Xk9kA7DyJr8mJik3S7Cwgy3qlkkmyS2uQB3jiJg6VNdZd/pDBJu0nvG2NlTg==",
"version": "9.1.2",
"resolved": "https://registry.npmjs.org/eslint-scope/-/eslint-scope-9.1.2.tgz",
"integrity": "sha512-xS90H51cKw0jltxmvmHy2Iai1LIqrfbw57b79w/J7MfvDfkIkFZ+kj6zC3BjtUwh150HsSSdxXZcsuv72miDFQ==",
"dev": true,
"license": "BSD-2-Clause",
"dependencies": {
"@types/esrecurse": "^4.3.1",
"@types/estree": "^1.0.8",
"esrecurse": "^4.3.0",
"estraverse": "^5.2.0"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
},
"funding": {
"url": "https://opencollective.com/eslint"
}
},
"node_modules/eslint-visitor-keys": {
"version": "4.2.1",
"resolved": "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-4.2.1.tgz",
"integrity": "sha512-Uhdk5sfqcee/9H/rCOJikYz67o0a2Tw2hGRPOG2Y1R2dg7brRe1uG0yaNQDHu+TO/uQPF/5eCapvYSmHUjt7JQ==",
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-5.0.1.tgz",
"integrity": "sha512-tD40eHxA35h0PEIZNeIjkHoDR4YjjJp34biM0mDvplBe//mB+IHCqHDGV7pxF+7MklTvighcCPPZC7ynWyjdTA==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
},
"funding": {
"url": "https://opencollective.com/eslint"
}
},
"node_modules/espree": {
"version": "10.4.0",
"resolved": "https://registry.npmjs.org/espree/-/espree-10.4.0.tgz",
"integrity": "sha512-j6PAQ2uUr79PZhBjP5C5fhl8e39FmRnOjsD5lGnWrFU8i2G776tBK7+nP8KuQUTTyAZUwfQqXAgrVH5MbH9CYQ==",
"version": "11.2.0",
"resolved": "https://registry.npmjs.org/espree/-/espree-11.2.0.tgz",
"integrity": "sha512-7p3DrVEIopW1B1avAGLuCSh1jubc01H2JHc8B4qqGblmg5gI9yumBgACjWo4JlIc04ufug4xJ3SQI8HkS/Rgzw==",
"dev": true,
"license": "BSD-2-Clause",
"dependencies": {
"acorn": "^8.15.0",
"acorn": "^8.16.0",
"acorn-jsx": "^5.3.2",
"eslint-visitor-keys": "^4.2.1"
"eslint-visitor-keys": "^5.0.1"
},
"engines": {
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
"node": "^20.19.0 || ^22.13.0 || >=24"
},
"funding": {
"url": "https://opencollective.com/eslint"
@@ -2972,16 +2819,6 @@
"graphology-types": ">=0.23.0"
}
},
"node_modules/has-flag": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz",
"integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/hermes-estree": {
"version": "0.25.1",
"resolved": "https://registry.npmjs.org/hermes-estree/-/hermes-estree-0.25.1.tgz",
@@ -3018,23 +2855,6 @@
"node": ">= 4"
}
},
"node_modules/import-fresh": {
"version": "3.3.1",
"resolved": "https://registry.npmjs.org/import-fresh/-/import-fresh-3.3.1.tgz",
"integrity": "sha512-TR3KfrTZTYLPB6jUjfx6MF9WcWrHL9su5TObK4ZkYgBdWKPOFoSoQIdEuTuR82pmtxH2spWG9h6etwfr1pLBqQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"parent-module": "^1.0.0",
"resolve-from": "^4.0.0"
},
"engines": {
"node": ">=6"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/imurmurhash": {
"version": "0.1.4",
"resolved": "https://registry.npmjs.org/imurmurhash/-/imurmurhash-0.1.4.tgz",
@@ -3081,29 +2901,6 @@
"integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==",
"license": "MIT"
},
"node_modules/js-yaml": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.2.0.tgz",
"integrity": "sha512-ePWsvanv0DWuDRsW8dnt+R4jQ31SCRCQ7hhNcPXZPsoBZiemuZNYGf7adZdqX2D86j6rvKp3RpCxVTSb8WQlOw==",
"dev": true,
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/puzrin"
},
{
"type": "github",
"url": "https://github.com/sponsors/nodeca"
}
],
"license": "MIT",
"dependencies": {
"argparse": "^2.0.1"
},
"bin": {
"js-yaml": "bin/js-yaml.js"
}
},
"node_modules/jsesc": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz",
@@ -3198,13 +2995,6 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/lodash.merge": {
"version": "4.6.2",
"resolved": "https://registry.npmjs.org/lodash.merge/-/lodash.merge-4.6.2.tgz",
"integrity": "sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ==",
"dev": true,
"license": "MIT"
},
"node_modules/loose-envify": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/loose-envify/-/loose-envify-1.4.0.tgz",
@@ -3256,16 +3046,19 @@
"license": "MIT"
},
"node_modules/minimatch": {
"version": "3.1.5",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz",
"integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==",
"version": "10.2.5",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz",
"integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==",
"dev": true,
"license": "ISC",
"license": "BlueOak-1.0.0",
"dependencies": {
"brace-expansion": "^1.1.7"
"brace-expansion": "^5.0.5"
},
"engines": {
"node": "*"
"node": "18 || 20 || >=22"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/mnemonist": {
@@ -3306,9 +3099,9 @@
"license": "MIT"
},
"node_modules/nanoid": {
"version": "3.3.11",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz",
"integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==",
"version": "3.3.16",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.16.tgz",
"integrity": "sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q==",
"dev": true,
"funding": [
{
@@ -3412,19 +3205,6 @@
"mnemonist": "^0.39.2"
}
},
"node_modules/parent-module": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/parent-module/-/parent-module-1.0.1.tgz",
"integrity": "sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g==",
"dev": true,
"license": "MIT",
"dependencies": {
"callsites": "^3.0.0"
},
"engines": {
"node": ">=6"
}
},
"node_modules/path-exists": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz",
@@ -3510,9 +3290,9 @@
}
},
"node_modules/postcss": {
"version": "8.5.10",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.10.tgz",
"integrity": "sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==",
"version": "8.5.23",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.23.tgz",
"integrity": "sha512-g50586zr4bZmwFiTlflMu8E0bDTb5I5gertgwAKmsdUlTQIhZtunzUlD1WSzwcVWPoAVpsrA6vlfCD7oXvRwgg==",
"dev": true,
"funding": [
{
@@ -3530,7 +3310,7 @@
],
"license": "MIT",
"dependencies": {
"nanoid": "^3.3.11",
"nanoid": "^3.3.16",
"picocolors": "^1.1.1",
"source-map-js": "^1.2.1"
},
@@ -3712,16 +3492,6 @@
"integrity": "sha512-M9/ELqF6fy8FwmkpnF0S3YKOqMyoWJ4+CS5Efg2ct3oY9daQvd/Pc71FpGZsVsbl3Cpb+IIcjBDUnnyBdQbq4w==",
"license": "MIT"
},
"node_modules/resolve-from": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-4.0.0.tgz",
"integrity": "sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=4"
}
},
"node_modules/rollup": {
"version": "4.60.2",
"resolved": "https://registry.npmjs.org/rollup/-/rollup-4.60.2.tgz",
@@ -3832,32 +3602,6 @@
"integrity": "sha512-HTEHMNieakEnoe33shBYcZ7NX83ACUjCu8c40iOGEZsngj9zRnkqS9j1pqQPXwobB0ZcVTk27REb7COQ0UR59w==",
"license": "MIT"
},
"node_modules/strip-json-comments": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.1.1.tgz",
"integrity": "sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/supports-color": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
"integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==",
"dev": true,
"license": "MIT",
"dependencies": {
"has-flag": "^4.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/tinyglobby": {
"version": "0.2.16",
"resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz",
+3 -2
View File
@@ -9,7 +9,8 @@
"lint": "eslint .",
"preview": "vite preview",
"test:graph-store": "node --test tests/graphStore.multi-edge.test.mjs",
"test:graph-workspace": "node --import tsx --test tests/graphSceneState.display.test.ts"
"test:graph-workspace": "node --import tsx --test tests/graphSceneState.display.test.ts",
"test:plugin-registry": "node --import tsx --test tests/pluginRegistry.temporal.test.mjs"
},
"dependencies": {
"@monaco-editor/react": "^4.7.0",
@@ -41,7 +42,7 @@
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^4.3.0",
"babel-plugin-react-compiler": "^1.0.0",
"eslint": "^9.39.4",
"eslint": "^10.8.0",
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.5.2",
"globals": "^17.4.0",
+51 -38
View File
@@ -1,4 +1,4 @@
import { lazy, Suspense, useEffect, useState, type ReactNode } from 'react';
import { lazy, Suspense, useEffect, useState, type ReactNode } from 'react';
import { QueryClient, QueryClientProvider } from '@tanstack/react-query';
import {
ArrowRight,
@@ -16,6 +16,7 @@ import {
ShieldCheck,
type LucideIcon,
} from 'lucide-react';
import { ErrorBoundary } from './ErrorBoundary';
const DecisionWorkspace = lazy(() => import('./workspaces/DecisionWorkspace/DecisionWorkspace').then((module) => ({ default: module.DecisionWorkspace })));
const DiffMergeWorkspace = lazy(() => import('./workspaces/DiffMergeWorkspace/DiffMergeWorkspace').then((module) => ({ default: module.DiffMergeWorkspace })));
@@ -1800,14 +1801,16 @@ export default function App() {
</>
}
>
<Suspense fallback={<WorkspaceFallback />}>
{exploreView === 'graph' ? (
<GraphWorkspace
externalFocusNodeId={graphFocusRequest?.nodeId}
externalFocusToken={graphFocusRequest?.token}
/>
) : <VocabularyWorkspace />}
</Suspense>
<ErrorBoundary key={`explore-${exploreView}`}>
<Suspense fallback={<WorkspaceFallback />}>
{exploreView === 'graph' ? (
<GraphWorkspace
externalFocusNodeId={graphFocusRequest?.nodeId}
externalFocusToken={graphFocusRequest?.token}
/>
) : <VocabularyWorkspace />}
</Suspense>
</ErrorBoundary>
</WorkspaceShell>
);
}
@@ -1829,9 +1832,11 @@ export default function App() {
</>
}
>
<Suspense fallback={<WorkspaceFallback />}>
{analyzeView === 'reasoning' ? <ReasoningWorkspace /> : <SparqlWorkspace />}
</Suspense>
<ErrorBoundary key={`analyze-${analyzeView}`}>
<Suspense fallback={<WorkspaceFallback />}>
{analyzeView === 'reasoning' ? <ReasoningWorkspace /> : <SparqlWorkspace />}
</Suspense>
</ErrorBoundary>
</WorkspaceShell>
);
}
@@ -1843,9 +1848,11 @@ export default function App() {
subtitle="Inspect decision chains, causal context, and precedent matches."
kicker="Decision Intelligence"
>
<Suspense fallback={<WorkspaceFallback />}>
<DecisionWorkspace />
</Suspense>
<ErrorBoundary key="decisions">
<Suspense fallback={<WorkspaceFallback />}>
<DecisionWorkspace />
</Suspense>
</ErrorBoundary>
</WorkspaceShell>
);
}
@@ -1873,12 +1880,14 @@ export default function App() {
</>
}
>
<Suspense fallback={<WorkspaceFallback />}>
{enrichView === 'import' ? <ImportExportWorkspace /> :
enrichView === 'merge' ? <DiffMergeWorkspace /> :
enrichView === 'resolve' ? <EntityResolutionTab /> :
<RegistryTab />}
</Suspense>
<ErrorBoundary key={`enrich-${enrichView}`}>
<Suspense fallback={<WorkspaceFallback />}>
{enrichView === 'import' ? <ImportExportWorkspace /> :
enrichView === 'merge' ? <DiffMergeWorkspace /> :
enrichView === 'resolve' ? <EntityResolutionTab /> :
<RegistryTab />}
</Suspense>
</ErrorBoundary>
</WorkspaceShell>
);
}
@@ -1891,15 +1900,17 @@ export default function App() {
kicker="Schema Governance"
compact
>
<Suspense fallback={<WorkspaceFallback />}>
<OntologyWorkspace
onJumpToGraphNode={(nodeId: string) => {
setGraphFocusRequest({ nodeId, token: Date.now() });
setActiveWorkspace('explore');
setExploreView('graph');
}}
/>
</Suspense>
<ErrorBoundary key="ontology-hub">
<Suspense fallback={<WorkspaceFallback />}>
<OntologyWorkspace
onJumpToGraphNode={(nodeId: string) => {
setGraphFocusRequest({ nodeId, token: Date.now() });
setActiveWorkspace('explore');
setExploreView('graph');
}}
/>
</Suspense>
</ErrorBoundary>
</WorkspaceShell>
);
}
@@ -1923,14 +1934,16 @@ export default function App() {
</>
}
>
<Suspense fallback={<WorkspaceFallback />}>
{manageView === 'lineage' ? <LineageDiagram /> :
manageView === 'kg-overview' ? <KGOverviewTab /> :
<OntologySummaryTab onOpenVocabularyBrowser={() => {
setActiveWorkspace('explore');
setExploreView('vocabulary');
}} />}
</Suspense>
<ErrorBoundary key={`manage-${manageView}`}>
<Suspense fallback={<WorkspaceFallback />}>
{manageView === 'lineage' ? <LineageDiagram /> :
manageView === 'kg-overview' ? <KGOverviewTab /> :
<OntologySummaryTab onOpenVocabularyBrowser={() => {
setActiveWorkspace('explore');
setExploreView('vocabulary');
}} />}
</Suspense>
</ErrorBoundary>
</WorkspaceShell>
);
};
+112
View File
@@ -0,0 +1,112 @@
import { Component, type ErrorInfo, type ReactNode } from 'react';
import { AlertCircle } from 'lucide-react';
interface ErrorBoundaryProps {
children: ReactNode;
}
interface ErrorBoundaryState {
hasError: boolean;
error: Error | null;
retryCount: number;
}
const RETRY_SETTLE_MS = 5000;
export class ErrorBoundary extends Component<ErrorBoundaryProps, ErrorBoundaryState> {
private settleTimer: ReturnType<typeof setTimeout> | null = null;
constructor(props: ErrorBoundaryProps) {
super(props);
this.state = { hasError: false, error: null, retryCount: 0 };
}
static getDerivedStateFromError(error: Error): Partial<ErrorBoundaryState> {
return { hasError: true, error };
}
componentDidCatch(error: Error, errorInfo: ErrorInfo) {
console.error("ErrorBoundary caught an error:", error, errorInfo);
this.clearSettleTimer();
}
componentWillUnmount() {
this.clearSettleTimer();
}
private clearSettleTimer() {
if (this.settleTimer !== null) {
clearTimeout(this.settleTimer);
this.settleTimer = null;
}
}
resetErrorBoundary = () => {
this.clearSettleTimer();
this.setState((prev) => ({
hasError: false,
error: null,
retryCount: prev.retryCount + 1
}));
// Only clear the retry count once the workspace has stayed error-free for a
// sustained period, rather than on the next committed render (which can fire
// while Suspense is still showing its fallback) or immediately on retry
// (which would allow an unbounded number of clicks on a deterministic crash).
this.settleTimer = setTimeout(() => {
this.settleTimer = null;
this.setState({ retryCount: 0 });
}, RETRY_SETTLE_MS);
};
render() {
if (this.state.hasError) {
const maxRetriesReached = this.state.retryCount >= 3;
return (
<div
className="workspace-loading"
style={{
flexDirection: 'column',
gap: 12,
color: 'var(--ws-red)'
}}
>
<AlertCircle size={32} style={{ marginBottom: 4, opacity: 0.8 }} />
<div style={{ fontWeight: 500, fontSize: '15px' }}>
Something went wrong in this view.
</div>
<div style={{ fontSize: '13px', opacity: 0.7, maxWidth: 450, textAlign: 'center', marginBottom: 8, lineHeight: 1.5 }}>
{maxRetriesReached
? "This view continues to encounter a critical error. Please switch to another workspace or reload the page to restore functionality."
: "An unexpected problem occurred while rendering this workspace. Your data is safe, but this view cannot be displayed."}
</div>
{!maxRetriesReached ? (
<button
className="ws-btn ws-btn--ghost"
style={{
borderColor: 'var(--ws-red-soft)',
color: 'var(--ws-red)'
}}
onClick={this.resetErrorBoundary}
>
Try Again
</button>
) : (
<button
className="ws-btn ws-btn--ghost"
style={{
borderColor: 'var(--ws-border)',
color: 'var(--ws-text)'
}}
onClick={() => window.location.reload()}
>
Reload Application
</button>
)}
</div>
);
}
return this.props.children;
}
}
@@ -92,6 +92,7 @@ export function DecisionWorkspace() {
const [chainLoading, setChainLoading] = useState(false);
const [listLoading, setListLoading] = useState(true);
const [filter, setFilter] = useState("");
const [error, setError] = useState("");
// Tracks the active chain request so stale responses from rapid selections are ignored.
const chainCtrlRef = useRef<AbortController | null>(null);
@@ -99,14 +100,24 @@ export function DecisionWorkspace() {
useEffect(() => {
const ctrl = new AbortController();
setListLoading(true);
setError("");
fetch("/api/decisions", { signal: ctrl.signal })
.then((r) => r.ok ? r.json() : Promise.reject(r.status))
.then(async (r) => {
if (!r.ok) throw new Error(`HTTP ${r.status}`);
const data = await r.json();
if (r.status === 207) setError(data.message || "Warning: Partial success loading decisions.");
return data;
})
.then((data) => {
if (ctrl.signal.aborted) return;
setDecisions(data);
if (data.length > 0) void loadChain(data[0]);
})
.catch((e) => { if (e?.name !== "AbortError") console.error(e); })
.catch((e) => {
if (e?.name !== "AbortError") {
setError(e instanceof Error ? e.message : "Failed to load decisions.");
}
})
.finally(() => {
if (!ctrl.signal.aborted) setListLoading(false);
});
@@ -125,13 +136,19 @@ export function DecisionWorkspace() {
setSelected(d);
setChainLoading(true);
setChain([]);
setError("");
try {
const res = await fetch(`/api/decisions/${encodeURIComponent(d.decision_id)}/chain`, { signal: ctrl.signal });
if (!res.ok) throw new Error(`${res.status}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.json();
if (res.status === 207 && !ctrl.signal.aborted) {
setError(data.message || "Warning: Partial success loading chain.");
}
if (!ctrl.signal.aborted) setChain(data.chain || []);
} catch (e) {
if (e instanceof Error && e.name !== "AbortError") console.error(e);
if (e instanceof Error && e.name !== "AbortError") {
setError(e.message);
}
} finally {
if (!ctrl.signal.aborted) setChainLoading(false);
}
@@ -213,6 +230,12 @@ export function DecisionWorkspace() {
<div style={{ flex: 1, display: "flex", flexDirection: "column", overflow: "hidden", position: "relative" }}>
<div style={{ position: "absolute", inset: 0, background: "radial-gradient(ellipse 60% 40% at 70% 20%, rgba(74,163,255,0.04), transparent 55%)", pointerEvents: "none" }} />
{error ? (
<div style={{ padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)", margin: "16px 16px 0 16px", zIndex: 2, position: "relative" }}>
{error}
</div>
) : null}
{selected ? (
<div className="ws-scroll ws-padded ws-animate-in" style={{ position: "relative", zIndex: 1 }}>
{/* Decision header */}
@@ -192,6 +192,7 @@ export function EntityResolutionTab() {
}, [threshold]);
const handleMerge = useCallback(async (primaryId: string, duplicateId: string) => {
setScanError("");
try {
const res = await fetch("/api/enrich/merge", {
method: "POST",
@@ -200,6 +201,9 @@ export function EntityResolutionTab() {
});
if (!res.ok) throw new Error(`Merge failed (${res.status})`);
const data = await res.json();
if (res.status === 207) {
setScanError(data.message || "Warning: Partial merge.");
}
logEvent("merge", `Merged ${duplicateId}${primaryId} · ${data.edges_updated ?? 0} edges redirected`, {
primary: primaryId,
duplicate: duplicateId,
@@ -207,7 +211,7 @@ export function EntityResolutionTab() {
});
setPairs((prev) => prev.filter((p) => !(p.a.id === primaryId && p.b.id === duplicateId)));
} catch (err) {
console.error("[EntityResolution] merge failed", err);
setScanError(err instanceof Error ? err.message : "Merge failed");
}
}, []);
@@ -38,6 +38,7 @@ import {
type GraphPluginPanelDescriptor,
type GraphPluginToolbarItem,
} from "./plugins";
import { explorationEffectsShouldLoad, neighborhoodPanelShouldLoad, temporalOverlayShouldLoad } from "./pluginRegistryPredicates";
import type { LinkPrediction, PathResponse } from "./GraphInspectorPanel";
import type { GraphSceneHandle, GraphSceneRuntime } from "./scene";
import type {
@@ -126,7 +127,7 @@ type LazyPluginRegistryEntry = {
load: () => Promise<GraphPlugin>;
shouldLoad: (context: {
panelState: Record<string, boolean>;
temporalState: GraphTemporalState | null;
temporalState?: GraphTemporalState | null;
}) => boolean;
};
@@ -1119,6 +1120,18 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
const [activeNodeCount, setActiveNodeCount] = useState<number | null>(null);
const [temporalBounds, setTemporalBounds] = useState<TemporalBounds | null>(null);
const [scrubberTime, setScrubberTime] = useState<Date | null>(null);
// Deduplicates setScrubberTime calls by millisecond value so that React 18
// concurrent-mode re-renders with a new Date object for the same timestamp
// do not churn temporalState and retrigger the diagnostics effect (issue #830).
const lastScrubberMsRef = useRef<number | null>(null);
const onTimeChange = useCallback((time: Date) => {
const ms = time.getTime();
if (ms === lastScrubberMsRef.current) {
return;
}
lastScrubberMsRef.current = ms;
setScrubberTime(time);
}, []);
const [loadingProgress, setLoadingProgress] = useState<GraphLoadProgress | null>(null);
const [pluginPanelState, setPluginPanelState] = useState<Record<string, boolean>>({
"effects-panel": false,
@@ -1129,6 +1142,9 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
const [pluginRuntimeVersion, setPluginRuntimeVersion] = useState(0);
const [effectsState, setEffectsState] = useState<GraphEffectsState>(DEFAULT_EFFECTS_STATE);
const [graphDiagnosticsState, setGraphDiagnosticsState] = useState<GraphRuntimeDiagnosticsSnapshot | null>(null);
// Tracks the last accepted diagnostics outside React's state cycle, allowing
// handleDiagnosticsChange to compare synchronously before calling setState.
const lastDiagnosticsRef = useRef<GraphRuntimeDiagnosticsSnapshot | null>(null);
const [graphAnalyticsState, setGraphAnalyticsState] = useState<GraphAnalyticsSnapshot | null>(null);
const [loadedPlugins, setLoadedPlugins] = useState<Record<string, GraphPlugin>>({});
@@ -2056,7 +2072,7 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
title: "Open exploration effects controls",
order: 18,
load: loadExplorationEffectsPlugin,
shouldLoad: ({ panelState }) => Boolean(panelState["effects-panel"]),
shouldLoad: explorationEffectsShouldLoad,
},
{
id: "neighborhood-panel",
@@ -2065,7 +2081,7 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
title: "Toggle neighborhood panel",
order: 30,
load: loadNeighborhoodPanelPlugin,
shouldLoad: ({ panelState }) => Boolean(panelState["neighborhood-panel"]),
shouldLoad: neighborhoodPanelShouldLoad,
},
{
id: "temporal-overlay",
@@ -2074,7 +2090,7 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
title: "Toggle temporal context panel",
order: 40,
load: loadTemporalOverlayPlugin,
shouldLoad: ({ panelState, temporalState }) => Boolean(panelState["temporal-panel"] || temporalState?.currentTime),
shouldLoad: temporalOverlayShouldLoad,
},
],
[],
@@ -2092,7 +2108,7 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
return;
}
if (!entry.shouldLoad({ panelState: pluginPanelState, temporalState })) {
if (!entry.shouldLoad({ panelState: pluginPanelState })) {
return;
}
@@ -2111,7 +2127,7 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
return () => {
cancelled = true;
};
}, [loadedPlugins, pluginPanelState, pluginRegistry, temporalState]);
}, [loadedPlugins, pluginPanelState, pluginRegistry]);
const setEffectToggle = useCallback((effect: GraphEffectToggle, enabled: boolean | ((current: boolean) => boolean)) => {
setEffectsState((current) => {
@@ -2274,6 +2290,55 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
if (!GRAPH_THEME.effects.diagnostics.enabledInDev) {
return;
}
// Compare against the last accepted snapshot synchronously before calling
// setState. buildEffectAvailability always returns a new object, so an
// unconditional setGraphDiagnosticsState on every call created a
// render → diagnostics effect → setState → render cycle that exceeded
// React's max update depth in dev mode (issue #830).
const prev = lastDiagnosticsRef.current;
if (prev !== null) {
const EFFECT_KEYS = [
"pathPulse", "pathFlow", "lens", "temporalEmphasis", "semanticRegions",
"contours", "pathfinding", "communities", "centrality", "legend", "diagnostics",
] as const;
const prevEA = prev.effectAvailability;
const nextEA = diagnostics.effectAvailability;
const availabilityChanged = EFFECT_KEYS.some((key) => {
const p = prevEA[key];
const n = nextEA[key];
return (
p.enabled !== n.enabled ||
p.available !== n.available ||
p.reason !== n.reason ||
p.detail !== n.detail ||
p.visibleSegments !== n.visibleSegments ||
p.segmentCap !== n.segmentCap
);
});
const edgeClassesChanged =
prev.edgeClasses?.updatedAt !== diagnostics.edgeClasses?.updatedAt;
const structureLayerChanged =
prev.structureLayer?.cacheKey !== diagnostics.structureLayer?.cacheKey ||
prev.structureLayer?.lastDrawAt !== diagnostics.structureLayer?.lastDrawAt ||
prev.structureLayer?.enabled !== diagnostics.structureLayer?.enabled ||
prev.structureLayer?.disabledReason !== diagnostics.structureLayer?.disabledReason ||
prev.structureLayer?.curveCount !== diagnostics.structureLayer?.curveCount ||
prev.structureLayer?.bridgeCurveCount !== diagnostics.structureLayer?.bridgeCurveCount ||
prev.structureLayer?.backboneCurveCount !== diagnostics.structureLayer?.backboneCurveCount;
// distanceVisual is compared by reference: GraphCanvas passes the same
// object when distances haven't changed.
const distanceVisualChanged = prev.distanceVisual !== diagnostics.distanceVisual;
if (!availabilityChanged && !edgeClassesChanged && !structureLayerChanged && !distanceVisualChanged) {
return;
}
}
lastDiagnosticsRef.current = diagnostics;
setGraphDiagnosticsState(diagnostics);
}, []);
@@ -2351,16 +2416,17 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
showPluginDock: openDockPanels.length > 0,
};
useEffect(() => {
if (!openDockPanels.length) {
setActiveDockPanelId(null);
return;
}
const openDockPanelIdsString = openDockPanels.map((p) => p.id).join("|");
const [prevOpenDockPanelIdsString, setPrevOpenDockPanelIdsString] = useState(openDockPanelIdsString);
if (!activeDockPanelId || !openDockPanels.some((panel) => panel.id === activeDockPanelId)) {
if (openDockPanelIdsString !== prevOpenDockPanelIdsString) {
setPrevOpenDockPanelIdsString(openDockPanelIdsString);
if (!openDockPanels.length) {
if (activeDockPanelId !== null) setActiveDockPanelId(null);
} else if (!activeDockPanelId || !openDockPanels.some((panel) => panel.id === activeDockPanelId)) {
setActiveDockPanelId(openDockPanels[0].id);
}
}, [activeDockPanelId, openDockPanels]);
}
const viewModeItems = useMemo<GraphToolbarItem[]>(() => {
if (!hasGraphContent) {
@@ -2921,7 +2987,7 @@ export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: Grap
<div className="explore-scene-footer">
<Suspense fallback={<div style={timelineFallbackStyle}>Loading timeline</div>}>
<LazyTimelinePanel
onTimeChange={setScrubberTime}
onTimeChange={onTimeChange}
minDate={temporalBounds?.min ?? undefined}
maxDate={temporalBounds?.max ?? undefined}
/>
@@ -325,6 +325,17 @@ export function GraphWorkspaceShell() {
const [activeNodeCount, setActiveNodeCount] = useState<number | null>(null);
const [temporalBounds, setTemporalBounds] = useState<TemporalBounds | null>(null);
const [scrubberTime, setScrubberTime] = useState<Date | null>(null);
// Deduplicates setScrubberTime calls by millisecond value — same fix as
// GraphWorkspace.tsx (issue #830).
const lastScrubberMsRef = useRef<number | null>(null);
const onTimeChange = useCallback((time: Date) => {
const ms = time.getTime();
if (ms === lastScrubberMsRef.current) {
return;
}
lastScrubberMsRef.current = ms;
setScrubberTime(time);
}, []);
const [loadingProgress, setLoadingProgress] = useState<GraphLoadProgress | null>(null);
const [isGraphStageReady, setIsGraphStageReady] = useState(false);
const [layoutStatus, setLayoutStatus] = useState<GraphLayoutStatus>({
@@ -369,7 +380,9 @@ export function GraphWorkspaceShell() {
}
}, []);
useEffect(() => {
const [prevFetchedAt, setPrevFetchedAt] = useState(snapshot?.fetchedAt);
if (snapshot?.fetchedAt !== prevFetchedAt) {
setPrevFetchedAt(snapshot?.fetchedAt);
if (snapshot) {
setIsGraphStageReady(false);
setActiveNodeCount(null);
@@ -383,7 +396,7 @@ export function GraphWorkspaceShell() {
stableSamples: 0,
});
}
}, [snapshot?.fetchedAt]);
}
useEffect(() => {
let cancelled = false;
@@ -630,7 +643,7 @@ export function GraphWorkspaceShell() {
<Suspense fallback={<TimelineFallback min={temporalBounds?.min ?? null} max={temporalBounds?.max ?? null} />}>
<TimelinePanel
onTimeChange={setScrubberTime}
onTimeChange={onTimeChange}
minDate={temporalBounds?.min ?? undefined}
maxDate={temporalBounds?.max ?? undefined}
/>
@@ -0,0 +1,24 @@
/**
* shouldLoad predicates for the GraphWorkspace lazy plugin registry.
*
* Extracted into a pure module so the predicates can be unit-tested without
* importing the full GraphWorkspace React component. Each predicate gates
* whether a plugin's module is lazily imported; none reference temporalState
* so temporal scrubber updates never retrigger plugin loading (issue #830).
*/
export type PluginShouldLoadContext = {
panelState: Record<string, boolean>;
};
export function explorationEffectsShouldLoad({ panelState }: PluginShouldLoadContext): boolean {
return Boolean(panelState["effects-panel"]);
}
export function neighborhoodPanelShouldLoad({ panelState }: PluginShouldLoadContext): boolean {
return Boolean(panelState["neighborhood-panel"]);
}
export function temporalOverlayShouldLoad({ panelState }: PluginShouldLoadContext): boolean {
return Boolean(panelState["temporal-panel"]);
}
@@ -33,6 +33,7 @@ export function LineageDiagram() {
const [edges, setEdges] = useState<any[]>([]);
const [searchId, setSearchId] = useState("");
const [activeId, setActiveId] = useState("");
const [error, setError] = useState("");
const downloadReport = async (format: "json" | "markdown") => {
if (!activeId) return;
@@ -51,12 +52,17 @@ export function LineageDiagram() {
document.body.removeChild(anchor);
};
const [prevActiveId, setPrevActiveId] = useState(activeId);
if (activeId !== prevActiveId) {
setPrevActiveId(activeId);
setError("");
setNodes([]);
setEdges([]);
}
useEffect(() => {
if (!activeId) {
setNodes([]);
setEdges([]);
return;
}
let ignore = false;
if (!activeId) return;
const xLanes = [
{ id: "group_agent", type: "group", position: { x: 50, y: 50 }, style: { width: 800, height: 120 } },
@@ -65,22 +71,24 @@ export function LineageDiagram() {
];
const fetchLineage = async () => {
setError("");
try {
const res = await fetch("/api/provenance?node_id=" + encodeURIComponent(activeId));
if (!res.ok) {
const text = await res.text();
console.error(`HTTP ${res.status}: API Route missing or failed.`, text.substring(0, 100));
return;
throw new Error(`HTTP ${res.status}: API Route missing or failed. ${text.substring(0, 100)}`);
}
const contentType = res.headers.get("content-type");
if (!contentType || !contentType.includes("application/json")) {
console.error("Backend returned non-JSON response (likely an HTML fallback). Check FastAPI routing.");
return;
throw new Error("Backend returned non-JSON response (likely an HTML fallback).");
}
const data = await res.json();
if (res.status === 207) {
setError(data.message || "Warning: Partial success loading lineage.");
}
const counters: Record<string, number> = { "group_agent": 0, "group_activity": 0, "group_entity": 0 };
@@ -89,7 +97,7 @@ export function LineageDiagram() {
counters[n.parent_id] = c + 1;
return {
id: n.id,
data: { label: n.label + "\\n(" + n.prov_type + ")" },
data: { label: n.label + "\n(" + n.prov_type + ")" },
position: { x: 50 + c * 180, y: 30 },
parentId: n.parent_id,
extent: "parent",
@@ -106,13 +114,16 @@ export function LineageDiagram() {
style: { stroke: "#58a6ff" }
}));
setNodes([...xLanes, ...mappedNodes]);
setEdges(mappedEdges);
if (!ignore) {
setNodes([...xLanes, ...mappedNodes]);
setEdges(mappedEdges);
}
} catch (err) {
console.error(err);
setError(err instanceof Error ? err.message : "Failed to load lineage.");
}
};
fetchLineage();
void fetchLineage();
return () => { ignore = true; };
}, [activeId]);
return (
@@ -143,6 +154,12 @@ export function LineageDiagram() {
</button>
</div>
{error ? (
<div style={{ position: "absolute", top: 60, left: 14, right: 14, zIndex: 10, padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" }}>
{error}
</div>
) : null}
{activeId ? (
<ReactFlow nodes={nodes} edges={edges} fitView>
<Background color="rgba(74,163,255,0.08)" gap={24} />
@@ -81,15 +81,79 @@ export function KGOverviewTab() {
fetch("/api/graph/nodes?limit=500"),
]);
if (statsRes.ok) {
const statsData: KGStats = await statsRes.json();
setStats(statsData);
if (!statsRes.ok) throw new Error(`Stats fetch failed (${statsRes.status})`);
if (!nodesRes.ok) throw new Error(`Nodes fetch failed (${nodesRes.status})`);
const statsData: KGStats = await statsRes.json();
setStats(statsData);
if (statsRes.status === 207) {
setError((statsData as any).message || "Warning: Partial success loading stats.");
}
if (nodesRes.ok) {
const nodesData: NodeListResponse = await nodesRes.json();
const nodes = nodesData.nodes ?? [];
setNodeTypeMap(buildTypeMap(nodes, "type"));
if (nodesRes.status === 207) {
const nodesMessage = (nodesData as any).message || "Warning: Partial success loading nodes.";
setError((prev) => (prev ? `${prev} ${nodesMessage}` : nodesMessage));
}
// Simulate neighbor counts via edges fetch for top-N
const edgesRes = await fetch("/api/graph/edges?limit=2000");
if (edgesRes.ok) {
const edgesData = await edgesRes.json();
const edges: { source: string; target: string }[] = edgesData.edges ?? [];
const degreeMap: Record<string, number> = {};
for (const edge of edges) {
degreeMap[edge.source] = (degreeMap[edge.source] ?? 0) + 1;
degreeMap[edge.target] = (degreeMap[edge.target] ?? 0) + 1;
}
const sorted = nodes
.map((n) => ({ node: n, neighborCount: degreeMap[n.id] ?? 0 }))
.sort((a, b) => b.neighborCount - a.neighborCount)
.slice(0, 10);
setTopNodes(sorted);
}
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to load graph overview. Ensure the server is running.");
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
let ignore = false;
async function fetchInitial() {
if (!ignore) {
setLoading(true);
setError("");
}
try {
const [statsRes, nodesRes] = await Promise.all([
fetch("/api/graph/stats"),
fetch("/api/graph/nodes?limit=500"),
]);
if (!statsRes.ok) throw new Error(`Stats fetch failed (${statsRes.status})`);
if (!nodesRes.ok) throw new Error(`Nodes fetch failed (${nodesRes.status})`);
const statsData: KGStats = await statsRes.json();
if (!ignore) {
setStats(statsData);
if (statsRes.status === 207) {
setError((statsData as { message?: string }).message || "Warning: Partial success loading stats.");
}
}
const nodesData: NodeListResponse = await nodesRes.json();
const nodes = nodesData.nodes ?? [];
setNodeTypeMap(buildTypeMap(nodes, "type"));
if (!ignore) {
setNodeTypeMap(buildTypeMap(nodes, "type"));
if (nodesRes.status === 207) {
const nodesMessage = (nodesData as { message?: string }).message || "Warning: Partial success loading nodes.";
setError((prev) => (prev ? `${prev} ${nodesMessage}` : nodesMessage));
}
}
// Simulate neighbor counts via edges fetch for top-N
const edgesRes = await fetch("/api/graph/edges?limit=2000");
@@ -105,20 +169,18 @@ export function KGOverviewTab() {
.map((n) => ({ node: n, neighborCount: degreeMap[n.id] ?? 0 }))
.sort((a, b) => b.neighborCount - a.neighborCount)
.slice(0, 10);
setTopNodes(sorted);
if (!ignore) setTopNodes(sorted);
}
} catch (err) {
if (!ignore) setError(err instanceof Error ? err.message : "Failed to load graph overview. Ensure the server is running.");
} finally {
if (!ignore) setLoading(false);
}
} catch {
setError("Failed to load graph overview. Ensure the server is running.");
} finally {
setLoading(false);
}
void fetchInitial();
return () => { ignore = true; };
}, []);
useEffect(() => {
void fetchOverview();
}, [fetchOverview]);
const nodeTypeEntries = Object.entries(nodeTypeMap).sort((a, b) => b[1] - a[1]);
const edgeTypeEntries = stats?.edge_types
? Object.entries(stats.edge_types).sort((a, b) => b[1] - a[1])
@@ -135,6 +197,12 @@ export function KGOverviewTab() {
return (
<div className="ws-page">
{error ? (
<div style={{ margin: "16px 22px 0 22px", padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" }}>
{error}
</div>
) : null}
{/* Header */}
<div style={{ display: "flex", alignItems: "center", justifyContent: "space-between", padding: "16px 22px", borderBottom: "1px solid var(--ws-border)", flexShrink: 0 }}>
<div style={{ display: "flex", alignItems: "center", gap: 10 }}>
@@ -152,11 +220,7 @@ export function KGOverviewTab() {
</button>
</div>
{error && (
<div style={{ margin: "12px 22px", padding: "10px 14px", borderRadius: "var(--ws-radius-sm)", background: "var(--ws-red-soft)", border: "1px solid rgba(255,123,114,0.28)", color: "#fca5a5", fontSize: 13 }}>
{error}
</div>
)}
<div className="ws-scroll" style={{ flex: 1, padding: "18px 22px", display: "flex", flexDirection: "column", gap: 16 }}>
{/* Stat cards */}
@@ -54,20 +54,49 @@ export function AlignmentsTab() {
loadAlignments(),
]);
const errors: string[] = [];
if (registryResult.status === "fulfilled") {
setRegistry(registryResult.value);
setSourceOntology((current) => current || registryResult.value[0]?.uri || "");
setTargetOntology((current) => current || registryResult.value[1]?.uri || registryResult.value[0]?.uri || "");
} else {
errors.push(registryResult.reason instanceof Error ? registryResult.reason.message : "Failed to load ontology registry.");
}
if (alignmentResult.status === "fulfilled") {
setAlignments(alignmentResult.value);
} else {
errors.push(alignmentResult.reason instanceof Error ? alignmentResult.reason.message : "Failed to load alignments.");
}
if (errors.length) setError(errors.join(" "));
}, []);
useEffect(() => {
void reload();
}, [reload]);
let ignore = false;
async function fetchInitial() {
const [registryResult, alignmentResult] = await Promise.allSettled([
loadOntologyRegistry(),
loadAlignments(),
]);
if (ignore) return;
const errors: string[] = [];
if (registryResult.status === "fulfilled") {
setRegistry(registryResult.value);
setSourceOntology((current) => current || registryResult.value[0]?.uri || "");
setTargetOntology((current) => current || registryResult.value[1]?.uri || registryResult.value[0]?.uri || "");
} else {
errors.push(registryResult.reason instanceof Error ? registryResult.reason.message : "Failed to load ontology registry.");
}
if (alignmentResult.status === "fulfilled") {
setAlignments(alignmentResult.value);
} else {
errors.push(alignmentResult.reason instanceof Error ? alignmentResult.reason.message : "Failed to load alignments.");
}
if (errors.length) setError(errors.join(" "));
}
void fetchInitial();
return () => { ignore = true; };
}, []);
const relationCounts = useMemo(() => {
const counts = new Map<string, number>();
@@ -23,29 +23,40 @@ export function HealthTab({ onFixInEditor }: HealthTabProps) {
setRegistry(entries);
setSelectedUri((current) => current || entries[0]?.uri || "");
})
.catch(() => { /* backend unavailable — leave registry empty */ });
.catch((err) => {
if (cancelled) return;
setError(err instanceof Error ? err.message : "Failed to load ontology registry.");
});
return () => {
cancelled = true;
};
}, []);
const loadHealth = useCallback(async (uri: string) => {
if (!uri) return;
setLoading(true);
setError("");
try {
setHealth(await loadOntologyHealth(uri));
} catch {
// Backend unavailable — show "select an ontology" placeholder, not an error
setHealth(null);
} finally {
setLoading(false);
const [prevUri, setPrevUri] = useState(selectedUri);
if (selectedUri !== prevUri) {
setPrevUri(selectedUri);
if (selectedUri) {
setLoading(true);
setError("");
}
}, []);
}
useEffect(() => {
void loadHealth(selectedUri);
}, [selectedUri, loadHealth]);
let ignore = false;
async function fetchHealth() {
if (!selectedUri) return;
try {
const data = await loadOntologyHealth(selectedUri);
if (!ignore) setHealth(data);
} catch {
if (!ignore) setHealth(null);
} finally {
if (!ignore) setLoading(false);
}
}
void fetchHealth();
return () => { ignore = true; };
}, [selectedUri]);
const exportReport = useCallback(() => {
if (!health) return;
@@ -248,6 +248,18 @@ export function OntologyManager() {
const [rightPanel, setRightPanel] = useState<RightPanel>("none");
const [actionMsg, setActionMsg] = useState<{ type: "ok" | "err"; text: string } | null>(null);
const [prevSearchQ, setPrevSearchQ] = useState(searchQ);
if (searchQ !== prevSearchQ) {
setPrevSearchQ(searchQ);
setLoading(true);
setActionMsg(null);
}
const flashMsg = useCallback((type: "ok" | "err", text: string) => {
setActionMsg({ type, text });
setTimeout(() => setActionMsg(null), 3000);
}, []);
const fetchRegistry = useCallback(async () => {
setLoading(true);
setActionMsg(null);
@@ -255,26 +267,42 @@ export function OntologyManager() {
const params = new URLSearchParams();
if (searchQ) params.set("q", searchQ);
const res = await fetch(`/api/ontology/registry?${params}`);
if (res.ok) {
setEntries(await res.json());
} else {
setEntries([]);
}
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.json();
setEntries(data);
if (res.status === 207) flashMsg("err", data.message || "Warning: Partial success loading registry.");
} catch {
setEntries([]);
flashMsg("err", "Failed to load ontology registry");
} finally {
setLoading(false);
}
}, [searchQ, statusFilter]);
}, [searchQ, flashMsg]);
useEffect(() => {
fetchRegistry();
}, [fetchRegistry]);
const flashMsg = (type: "ok" | "err", text: string) => {
setActionMsg({ type, text });
setTimeout(() => setActionMsg(null), 3000);
};
let ignore = false;
async function fetchInitial() {
try {
const params = new URLSearchParams();
if (searchQ) params.set("q", searchQ);
const res = await fetch(`/api/ontology/registry?${params}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.json();
if (ignore) return;
setEntries(data);
if (res.status === 207) flashMsg("err", data.message || "Warning: Partial success loading registry.");
} catch {
if (!ignore) {
setEntries([]);
flashMsg("err", "Failed to load ontology registry");
}
} finally {
if (!ignore) setLoading(false);
}
}
void fetchInitial();
return () => { ignore = true; };
}, [searchQ, flashMsg]);
const handleToggle = useCallback(async (uri: string) => {
try {
@@ -178,17 +178,33 @@ function DetailPanel({
const [loading, setLoading] = useState(true);
const [error, setError] = useState("");
useEffect(() => {
const [prevUri, setPrevUri] = useState(uri);
if (uri !== prevUri) {
setPrevUri(uri);
setLoading(true);
setError("");
setDetail(null);
}
useEffect(() => {
let ignore = false;
fetch(`/api/ontology/entity/${encodeURIComponent(uri)}`)
.then((r) => {
.then(async (r) => {
if (!r.ok) throw new Error("Not found");
return r.json();
const data = await r.json();
if (r.status === 207) setError(data.message || "Warning: Partial success loading entity.");
return data;
})
.then(setDetail)
.catch((e) => setError(e.message))
.finally(() => setLoading(false));
.then((data) => {
if (!ignore) setDetail(data);
})
.catch((e) => {
if (!ignore) setError(e.message);
})
.finally(() => {
if (!ignore) setLoading(false);
});
return () => { ignore = true; };
}, [uri]);
return (
@@ -40,19 +40,6 @@ export function ProposalReview({ proposalId }: { proposalId: string }) {
const [selectedElement, setSelectedElement] = useState<string | null>(null);
const [commentText, setCommentText] = useState("");
const loadProposal = useCallback(async () => {
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}`);
if (response.ok) {
const data = await response.json();
setProposal(data);
generateDiff(data);
}
} catch (error) {
console.error("Failed to load proposal:", error);
}
}, [proposalId]);
const generateDiff = useCallback((prop: Proposal) => {
const changes: DiffChange[] = [];
@@ -75,9 +62,38 @@ export function ProposalReview({ proposalId }: { proposalId: string }) {
setDiff(changes);
}, []);
const loadProposal = useCallback(async () => {
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}`);
if (response.ok) {
const data = await response.json();
setProposal(data);
generateDiff(data);
}
} catch (error) {
console.error("Failed to load proposal:", error);
}
}, [proposalId, generateDiff]);
useEffect(() => {
loadProposal();
}, [loadProposal]);
let ignore = false;
async function fetchInitial() {
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}`);
if (response.ok) {
const data = await response.json();
if (!ignore) {
setProposal(data);
generateDiff(data);
}
}
} catch (error) {
console.error("Failed to load proposal:", error);
}
}
void fetchInitial();
return () => { ignore = true; };
}, [proposalId, generateDiff]);
const addComment = useCallback(async () => {
if (!selectedElement || !commentText || !proposal) return;
@@ -77,17 +77,35 @@ function ConceptDetailPanel({
const [loading, setLoading] = useState(true);
const [error, setError] = useState("");
useEffect(() => {
const [prevUri, setPrevUri] = useState(uri);
if (uri !== prevUri) {
setPrevUri(uri);
setLoading(true);
setError("");
setDetail(null);
}
useEffect(() => {
let ignore = false;
fetch(`/api/ontology/skos/concept/${encodeURIComponent(uri)}`)
.then((r) => {
.then(async (r) => {
if (!r.ok) throw new Error("Concept not found");
return r.json();
const data = await r.json();
if (r.status === 207) setError(data.message || "Warning: Partial success loading concept.");
return data;
})
.then(setDetail)
.catch((e) => setError(e.message))
.finally(() => setLoading(false));
.then((data) => {
if (!ignore) setDetail(data);
})
.catch((e) => {
if (!ignore) setError(e.message);
})
.finally(() => {
if (!ignore) setLoading(false);
});
return () => {
ignore = true;
};
}, [uri]);
const renderUriList = (label: string, uris: string[]) => {
@@ -322,16 +340,48 @@ function SchemePanel({
}) {
const [expanded, setExpanded] = useState(true);
const [hierarchy, setHierarchy] = useState<ConceptNode[]>([]);
const [loading, setLoading] = useState(false);
const [loading, setLoading] = useState(expanded);
const [error, setError] = useState("");
const [prevExpanded, setPrevExpanded] = useState(expanded);
const [prevSchemeUri, setPrevSchemeUri] = useState(scheme.uri);
if (expanded !== prevExpanded || scheme.uri !== prevSchemeUri) {
setPrevExpanded(expanded);
setPrevSchemeUri(scheme.uri);
if (expanded) {
setLoading(true);
setError("");
setHierarchy([]);
} else {
setLoading(false);
}
}
useEffect(() => {
let ignore = false;
if (!expanded) return;
setLoading(true);
fetch(`/api/vocabulary/hierarchy?scheme=${encodeURIComponent(scheme.uri)}`)
.then((r) => (r.ok ? r.json() : []))
.then(setHierarchy)
.catch(() => setHierarchy([]))
.finally(() => setLoading(false));
.then(async (r) => {
if (!r.ok) throw new Error(`HTTP ${r.status}`);
const data = await r.json();
if (r.status === 207 && !ignore) setError(data.message || "Warning: Partial success loading hierarchy.");
return data;
})
.then((data) => {
if (!ignore) setHierarchy(data);
})
.catch((err) => {
if (!ignore) {
setHierarchy([]);
setError(err instanceof Error ? err.message : "Failed to load hierarchy.");
}
})
.finally(() => {
if (!ignore) setLoading(false);
});
return () => {
ignore = true;
};
}, [scheme.uri, expanded]);
const totalConcepts = countConcepts(hierarchy);
@@ -366,6 +416,7 @@ function SchemePanel({
{expanded && (
<div style={{ paddingBottom: 8 }}>
{error ? <div style={errorStyle}>{error}</div> : null}
{loading ? (
<div style={{ padding: "10px 20px", display: "flex", alignItems: "center", gap: 8 }}>
<Loader2 size={12} color="#4aa3ff" style={{ animation: "spin 0.8s linear infinite" }} />
@@ -410,9 +461,13 @@ export function SKOSVocabularyManager({ schemeUri }: Props) {
const [selectedUri, setSelectedUri] = useState<string | null>(null);
useEffect(() => {
setLoading(true);
fetch("/api/ontology/skos/schemes")
.then((r) => (r.ok ? r.json() : []))
.then(async (r) => {
if (!r.ok) throw new Error(`Failed to load schemes (${r.status})`);
const data = await r.json();
if (r.status === 207) setError(data.message || "Warning: Partial success loading schemes.");
return data;
})
.then(setSchemes)
.catch((e) => setError(e.message))
.finally(() => setLoading(false));
@@ -424,6 +479,7 @@ export function SKOSVocabularyManager({ schemeUri }: Props) {
return (
<div style={managerShellStyle}>
{error ? <div style={errorStyle}>{error}</div> : null}
{/* Search bar */}
<div style={skosToolbarStyle}>
<div style={skosSearchBarStyle}>
@@ -628,6 +684,8 @@ const navLinkStyle: React.CSSProperties = {
textAlign: "left",
};
const errorStyle: React.CSSProperties = { padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)", margin: "0 10px 10px 10px", fontSize: 12 };
const centerStyle: React.CSSProperties = {
display: "flex",
flexDirection: "column",
@@ -33,38 +33,51 @@ export function ShaclStudio({ onJumpToNode }: ShaclStudioProps) {
setRegistry(entries);
setSelectedUri((current) => current || entries[0]?.uri || "");
})
.catch(() => { /* backend unavailable — leave registry empty */ });
.catch((err) => {
if (cancelled) return;
setError(err instanceof Error ? err.message : "Failed to load ontology registry.");
});
return () => {
cancelled = true;
};
}, []);
const loadShapes = useCallback(async (uri: string) => {
if (!uri) return;
setLoading(true);
setError("");
try {
const data = await loadShaclShapes(uri);
setShapes(data.shapes);
const turtle = data.shacl_turtle;
setFullShacl(turtle);
setShacl((current) => current || turtle);
setSelectedShapeId(null);
setValidation(null);
} catch {
// Shapes not yet generated or backend unavailable — show empty shape list
setShapes([]);
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
const [prevUri, setPrevUri] = useState(selectedUri);
if (selectedUri !== prevUri) {
setPrevUri(selectedUri);
setShacl("");
setFullShacl("");
setSelectedShapeId(null);
void loadShapes(selectedUri);
}, [selectedUri, loadShapes]);
setValidation(null);
setLoading(true);
setError("");
}
useEffect(() => {
let ignore = false;
async function fetchShapes() {
if (!selectedUri) return;
try {
const data = await loadShaclShapes(selectedUri);
if (!ignore) {
setShapes(data.shapes);
const turtle = data.shacl_turtle;
setFullShacl(turtle);
setShacl((current) => current || turtle);
setSelectedShapeId(null);
setValidation(null);
}
} catch {
if (!ignore) setShapes([]);
} finally {
if (!ignore) setLoading(false);
}
}
void fetchShapes();
return () => {
ignore = true;
};
}, [selectedUri]);
const handleGenerate = useCallback(async () => {
if (!selectedUri) return;
@@ -47,86 +47,151 @@ export function VersionsTab() {
const [comparePair, setComparePair] = useState<{ v1: string; v2: string } | null>(null);
const [compareResult, setCompareResult] = useState<Record<string, any> | null>(null);
const [isLoading, setIsLoading] = useState(false);
const [error, setError] = useState("");
const loadVersions = useCallback(async () => {
if (!ontologyUri) return;
setError("");
try {
const response = await fetch(`/api/ontology/versions/${encodeURIComponent(ontologyUri)}`);
if (response.ok) {
const data = await response.json();
setVersions(data);
if (response.status === 207) setError(data.message || "Warning: Partial success loading versions.");
} else {
setError(`Failed to load versions (${response.status})`);
}
} catch (error) {
console.error("Failed to load versions:", error);
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to load versions.");
}
}, [ontologyUri]);
const loadProposals = useCallback(async () => {
setError("");
try {
const response = await fetch("/api/ontology/proposals");
if (response.ok) {
const data = await response.json();
setProposals(data);
if (response.status === 207) setError(data.message || "Warning: Partial success loading proposals.");
} else {
setError(`Failed to load proposals (${response.status})`);
}
} catch (error) {
console.error("Failed to load proposals:", error);
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to load proposals.");
}
}, []);
useEffect(() => {
loadVersions();
loadProposals();
}, [loadVersions, loadProposals]);
let ignore = false;
async function fetchInitial() {
if (!ignore) setError("");
try {
const propRes = await fetch("/api/ontology/proposals");
if (propRes.ok) {
const propData = await propRes.json();
if (!ignore) {
setProposals(propData);
if (propRes.status === 207) setError(propData.message || "Warning: Partial success loading proposals.");
}
} else if (!ignore) {
setError(`Failed to load proposals (${propRes.status})`);
}
} catch (err) {
if (!ignore) setError(err instanceof Error ? err.message : "Failed to load proposals.");
}
if (!ontologyUri) return;
try {
const verRes = await fetch(`/api/ontology/versions/${encodeURIComponent(ontologyUri)}`);
if (verRes.ok) {
const verData = await verRes.json();
if (!ignore) {
setVersions(verData);
if (verRes.status === 207) setError(verData.message || "Warning: Partial success loading versions.");
}
} else if (!ignore) {
setError((prev) => prev || `Failed to load versions (${verRes.status})`);
}
} catch (err) {
if (!ignore) setError((prev) => prev || (err instanceof Error ? err.message : "Failed to load versions."));
}
}
void fetchInitial();
return () => { ignore = true; };
}, [ontologyUri]);
const approveProposal = useCallback(async (proposalId: string) => {
setError("");
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}/approve`, {
method: "POST",
});
if (response.ok) {
alert("Proposal approved");
if (response.status === 207) {
const data = await response.json().catch(() => ({}));
setError(data.message || "Warning: Partial success approving proposal.");
} else {
alert("Proposal approved");
}
loadProposals();
} else {
setError(`Failed to approve proposal (${response.status})`);
}
} catch (error) {
console.error("Failed to approve proposal:", error);
alert("Failed to approve proposal");
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to approve proposal.");
}
}, [loadProposals]);
const rejectProposal = useCallback(async (proposalId: string) => {
setError("");
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}/reject`, {
method: "POST",
});
if (response.ok) {
alert("Proposal rejected");
if (response.status === 207) {
const data = await response.json().catch(() => ({}));
setError(data.message || "Warning: Partial success rejecting proposal.");
} else {
alert("Proposal rejected");
}
loadProposals();
} else {
setError(`Failed to reject proposal (${response.status})`);
}
} catch (error) {
console.error("Failed to reject proposal:", error);
alert("Failed to reject proposal");
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to reject proposal.");
}
}, [loadProposals]);
const publishProposal = useCallback(async (proposalId: string) => {
setError("");
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}/publish`, {
method: "POST",
});
if (response.ok) {
alert("Proposal published");
if (response.status === 207) {
const data = await response.json().catch(() => ({}));
setError(data.message || "Warning: Partial success publishing proposal.");
} else {
alert("Proposal published");
}
loadProposals();
loadVersions();
} else {
setError(`Failed to publish proposal (${response.status})`);
}
} catch (error) {
console.error("Failed to publish proposal:", error);
alert("Failed to publish proposal");
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to publish proposal.");
}
}, [loadProposals, loadVersions]);
const runVersionComparison = useCallback(async () => {
if (!comparePair || !ontologyUri) return;
setIsLoading(true);
setError("");
try {
const response = await fetch(`/api/ontology/versions/${encodeURIComponent(ontologyUri)}/compare`, {
method: "POST",
@@ -138,11 +203,13 @@ export function VersionsTab() {
});
if (response.ok) {
const data = await response.json();
if (response.status === 207) setError(data.message || "Warning: Partial success comparing versions.");
setCompareResult(data);
} else {
setError(`Failed to compare versions (${response.status})`);
}
} catch (error) {
console.error("Failed to compare versions:", error);
alert("Failed to compare versions");
} catch (err) {
setError(err instanceof Error ? err.message : "Failed to compare versions.");
} finally {
setIsLoading(false);
}
@@ -265,6 +332,8 @@ export function VersionsTab() {
marginBottom: "12px",
};
const errorStyle: React.CSSProperties = { padding: "12px", borderRadius: "14px", color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)", marginBottom: "16px" };
return (
<div style={containerStyle}>
<div style={headerStyle}>
@@ -278,6 +347,8 @@ export function VersionsTab() {
/>
</div>
{error ? <div style={errorStyle}>{error}</div> : null}
<div style={sectionStyle}>
<h2 style={sectionTitleStyle}>
<Layers size={16} />
@@ -20,7 +20,11 @@ async function parseResponse<T>(response: Response): Promise<T> {
}
throw new Error(detail);
}
return response.json() as Promise<T>;
const data = await response.json();
if (response.status === 207) {
console.warn("Partial Success:", data.message || "Warning: 207 Multi-Status");
}
return data as T;
}
export async function loadOntologyRegistry(): Promise<OntologyEntry[]> {
@@ -57,6 +57,7 @@ export function ReasoningWorkspace() {
});
const data = await response.json();
if (!response.ok) throw new Error(data.detail || `Status ${response.status}`);
if (response.status === 207) setError(data.message || "Warning: Partial success reasoning.");
setResult(data);
if (data.mutated) queryClient.invalidateQueries({ queryKey: ["graph", "full-load"] });
} catch (e) {
@@ -72,7 +72,15 @@ export function SparqlWorkspace() {
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ query }),
});
if (!res.headers.get("content-type")?.includes("application/json")) {
const text = await res.text();
throw new Error(`HTTP ${res.status}: ${text.substring(0, 100)}`);
}
const data = await res.json();
if (res.status === 207) {
data.error = data.message || "Warning: Partial success running query.";
}
if (data.error && data.error_line && monaco && editorRef.current) {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
(monaco as any).editor.setModelMarkers((editorRef.current as any).getModel(), "sparql", [{
@@ -81,12 +89,14 @@ export function SparqlWorkspace() {
endLineNumber: data.error_line,
endColumn: 100,
message: data.error,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
severity: (monaco as any).MarkerSeverity.Error,
}]);
}
setResult(data);
} catch {
setResult({ error: "Network error — could not reach the SPARQL endpoint." });
} catch (e) {
const msg = e instanceof Error ? e.message : "Network error — could not reach the SPARQL endpoint.";
setResult({ error: msg });
} finally {
setIsLoading(false);
}
@@ -0,0 +1,90 @@
/**
* Regression tests for issue #830: plugin registry shouldLoad predicates.
*
* Imports the production predicates from pluginRegistryPredicates.ts so that
* a regression in GraphWorkspace.tsx is detected here. The key invariant: no
* predicate may read temporalState doing so caused a render loop because
* temporalState.currentTime is non-null from startup, which triggered eager
* plugin loads on every scrubber update and continuously cancelled in-flight
* load() calls before they could register the plugin.
*/
import test from "node:test";
import assert from "node:assert/strict";
import { createRequire } from "node:module";
const require = createRequire(import.meta.url);
const {
explorationEffectsShouldLoad,
neighborhoodPanelShouldLoad,
temporalOverlayShouldLoad,
} = require("../src/workspaces/GraphWorkspace/pluginRegistryPredicates.ts");
// ── temporal-overlay ─────────────────────────────────────────────────────────
test("temporal-overlay shouldLoad: false when panel is closed and no scrubber time", () => {
assert.equal(
temporalOverlayShouldLoad({ panelState: { "temporal-panel": false } }),
false,
);
});
test("temporal-overlay shouldLoad: false when panel is closed even if scrubber time is set", () => {
// Before the fix, a non-null currentTime caused an eager load on every scrubber update.
assert.equal(
temporalOverlayShouldLoad({
panelState: { "temporal-panel": false },
temporalState: { currentTime: new Date() },
}),
false,
);
});
test("temporal-overlay shouldLoad: true only when the panel is explicitly opened", () => {
assert.equal(
temporalOverlayShouldLoad({ panelState: { "temporal-panel": true } }),
true,
);
});
test("temporal-overlay shouldLoad: true when panel opened even without a scrubber time", () => {
assert.equal(
temporalOverlayShouldLoad({
panelState: { "temporal-panel": true },
temporalState: { currentTime: null },
}),
true,
);
});
// ── other entries — confirm they also gate only on panelState ─────────────────
test("exploration-effects shouldLoad: gates only on effects-panel state", () => {
assert.equal(explorationEffectsShouldLoad({ panelState: { "effects-panel": false } }), false);
assert.equal(explorationEffectsShouldLoad({ panelState: { "effects-panel": true } }), true);
});
test("neighborhood-panel shouldLoad: gates only on neighborhood-panel state", () => {
assert.equal(neighborhoodPanelShouldLoad({ panelState: { "neighborhood-panel": false } }), false);
assert.equal(neighborhoodPanelShouldLoad({ panelState: { "neighborhood-panel": true } }), true);
});
test("all three shouldLoad conditions are consistent: none reference temporalState", () => {
// A regressed predicate reading temporalState?.currentTime would return true
// for a closed panel when currentTime is set — detecting the loop bug.
const nonNullTemporalState = { currentTime: new Date(), activeNodeCount: 6 };
assert.equal(
temporalOverlayShouldLoad({ panelState: { "temporal-panel": false }, temporalState: nonNullTemporalState }),
false,
"temporal-overlay must not load when panel is closed, regardless of scrubber time",
);
assert.equal(
explorationEffectsShouldLoad({ panelState: { "effects-panel": false }, temporalState: nonNullTemporalState }),
false,
);
assert.equal(
neighborhoodPanelShouldLoad({ panelState: { "neighborhood-panel": false }, temporalState: nonNullTemporalState }),
false,
);
});
+70 -17
View File
@@ -123,12 +123,10 @@ class AgnoDecisionKit(_ToolkitBase): # type: ignore[misc]
tools_to_register.append(self.check_policy)
for fn in tools_to_register:
self._tools.append(fn)
if AGNO_AVAILABLE:
try:
self.register(fn)
except Exception:
pass
self.register(fn)
if fn not in self._tools:
self._tools.append(fn)
logger.info("AgnoDecisionKit initialised")
@@ -312,18 +310,33 @@ class AgnoDecisionKit(_ToolkitBase): # type: ignore[misc]
Rules are evaluated inline using simple comparison expressions. This
avoids misuse of ``PolicyEngine.check_compliance`` (which requires a
stored ``Decision`` + ``policy_id``) and ensures exceptions never
silently return ``compliant=True``.
silently return ``compliant=True``. A rule that references a field
missing from ``decision_data``, a field whose value is JSON ``null``,
or a rule that doesn't match the expected ``<field> <op> <value>``
format, cannot be evaluated it is recorded in ``warnings`` (not
``violations``) since we don't know whether it would have passed or
failed. These are reported as distinct messages (missing key vs.
null value) so the warning is actionable.
Parameters
----------
decision_data:
JSON string describing the decision (must include ``category``,
``outcome``, ``confidence`` keys at minimum).
``outcome``, ``confidence`` keys at minimum). Must decode to a
JSON object any other shape (list, number, string, bool) is
rejected with a single ``violations`` entry, the same as
malformed JSON, rather than being passed through to per-rule
evaluation where it would produce confusing internal errors.
policy_rules:
JSON list of rule strings, e.g.
``'["confidence >= 0.7", "category != \\"test\\""]'``.
Each rule is a simple comparison: ``<field> <op> <value>``
where op is one of ``>=``, ``<=``, ``!=``, ``==``, ``>``, ``<``.
A JSON-encoded bare string (e.g. ``'"confidence >= 0.7"'``) is
treated as a single rule. Any other decoded JSON shape (e.g. a
number or object), or a non-string list element, is recorded as
one ``warnings`` entry and otherwise ignored rather than being
iterated character-by-character.
Returns
-------
@@ -341,16 +354,47 @@ class AgnoDecisionKit(_ToolkitBase): # type: ignore[misc]
}
)
rules: List[str] = []
if policy_rules:
try:
rules = json.loads(policy_rules)
except json.JSONDecodeError:
rules = [r.strip() for r in policy_rules.split(",") if r.strip()]
if not isinstance(data, dict):
return json.dumps(
{
"compliant": False,
"violations": [
f"decision_data must decode to a JSON object, "
f"got {type(data).__name__}: {data!r}"
],
"warnings": [],
}
)
violations: List[str] = []
warnings: List[str] = []
rules: List[str] = []
if policy_rules:
try:
parsed_rules = json.loads(policy_rules)
except json.JSONDecodeError:
rules = [r.strip() for r in policy_rules.split(",") if r.strip()]
else:
if isinstance(parsed_rules, str):
# A single rule encoded as a bare JSON string, e.g.
# policy_rules='"confidence >= 0.7"'. Treat it as one
# rule rather than iterating it character-by-character.
rules = [parsed_rules]
elif isinstance(parsed_rules, list):
for item in parsed_rules:
if isinstance(item, str):
rules.append(item)
else:
warnings.append(
f"Ignoring non-string policy rule entry: {item!r}"
)
else:
warnings.append(
f"policy_rules must decode to a JSON list of rule strings, "
f"got {type(parsed_rules).__name__}: {parsed_rules!r}"
)
for rule in rules:
try:
if not self._eval_rule(rule, data):
@@ -369,14 +413,23 @@ class AgnoDecisionKit(_ToolkitBase): # type: ignore[misc]
)
def _eval_rule(self, rule: str, data: Dict[str, Any]) -> bool:
"""Evaluate a simple comparison rule (``field op value``) against data."""
"""
Evaluate a simple comparison rule (``field op value``) against data.
Raises ``ValueError`` when the rule cannot be evaluated (unrecognised
format, the referenced field is absent from ``data``, or the field's
value is JSON ``null``) so that ``check_policy`` records it as a
``warnings`` entry instead of silently treating it as passed.
"""
m = re.match(r"(\w+)\s*(>=|<=|!=|==|>|<)\s*(.+)", rule.strip())
if not m:
return True # unrecognised format — pass through
raise ValueError(f"unrecognised rule format: {rule!r}")
field, op, val_str = m.group(1), m.group(2), m.group(3).strip().strip("\"'")
actual = data.get(field)
if field not in data:
raise ValueError(f"rule references undefined field {field!r}")
actual = data[field]
if actual is None:
return True # field absent — cannot evaluate
raise ValueError(f"field {field!r} is null — cannot evaluate rule")
try:
val: Any = type(actual)(val_str)
except (ValueError, TypeError):
+3 -5
View File
@@ -122,12 +122,10 @@ class AgnoKGToolkit(_ToolkitBase): # type: ignore[misc]
self.export_subgraph,
]
for fn in tools_to_register:
self._tools.append(fn)
if AGNO_AVAILABLE:
try:
self.register(fn)
except Exception:
pass
self.register(fn)
if fn not in self._tools:
self._tools.append(fn)
logger.info("AgnoKGToolkit initialised (backend=%s)", graph_store_backend)
+3 -3
View File
@@ -83,7 +83,7 @@ class _AgentScopedStore(AgnoContextStore):
try:
self._context.store(mem_text, conversation_id=self.session_id)
except Exception as exc:
logger.warning("[%s] store failed: %s", self._role, exc)
logger.warning("[%s] store failed: %s", self._role, exc, exc_info=True)
if self.decision_tracking:
try:
@@ -94,8 +94,8 @@ class _AgentScopedStore(AgnoContextStore):
outcome="stored",
confidence=1.0,
)
except Exception:
pass
except Exception as exc:
logger.warning("[%s] record_decision failed: %s", self._role, exc, exc_info=True)
if hasattr(memory, "id"):
memory.id = mem_id
+80 -4
View File
@@ -91,11 +91,19 @@ def handle_find_precedents(args: dict) -> dict:
def handle_get_causal_chain(args: dict) -> dict:
"""Trace the upstream or downstream causal chain from a decision."""
decision_id = args.get("decision_id", "").strip()
if not isinstance(args, dict):
return {"error": "args must be a dictionary", "chain": []}
decision_id = str(args.get("decision_id") or "").strip()
if not decision_id:
return {"error": "decision_id is required", "chain": []}
direction = args.get("direction", "downstream")
max_depth = int(args.get("max_depth", 5))
direction = str(args.get("direction") or "downstream").strip()
try:
max_depth = int(args.get("max_depth", 5))
if max_depth <= 0:
max_depth = 5
max_depth = min(max_depth, 100)
except (ValueError, TypeError):
max_depth = 5
try:
graph = get_graph()
try:
@@ -105,7 +113,75 @@ def handle_get_causal_chain(args: dict) -> dict:
decision_id, direction=direction, max_depth=max_depth
)
except (ImportError, AttributeError):
chain = graph.get_causal_chain(decision_id) if hasattr(graph, "get_causal_chain") else []
if hasattr(graph, "get_causal_chain"):
import inspect
# Introspect the signature in its own try/except: only
# failure to introspect (ValueError/TypeError from
# inspect.signature itself, e.g. a C-extension callable)
# should fall through to the trial-and-error cascade below.
# A call made after a *successful* introspection must not be
# wrapped in that cascade's except block — otherwise a
# genuine bug inside get_causal_chain (raising an unrelated
# TypeError) gets misread as "wrong signature" and the
# backend is invoked a second time with identical arguments.
try:
params = inspect.signature(graph.get_causal_chain).parameters
except (ValueError, TypeError):
params = None
if params is not None:
has_var_kwargs = any(
p.kind == inspect.Parameter.VAR_KEYWORD
for p in params.values()
)
if has_var_kwargs or (
"direction" in params and "max_depth" in params
):
chain = graph.get_causal_chain(
decision_id,
direction=direction,
max_depth=max_depth,
)
elif "depth" in params:
chain = graph.get_causal_chain(
decision_id,
depth=max_depth,
)
else:
chain = graph.get_causal_chain(decision_id)
else:
try:
chain = graph.get_causal_chain(
decision_id,
direction=direction,
max_depth=max_depth,
)
except TypeError as exc:
if "unexpected keyword argument" in str(
exc
) or "positional" in str(exc):
try:
chain = graph.get_causal_chain(
decision_id,
depth=max_depth,
)
except TypeError as exc2:
if "unexpected keyword argument" in str(
exc2
) or "positional" in str(exc2):
chain = graph.get_causal_chain(decision_id)
else:
raise
else:
raise
else:
return {
"error": (
"Causal chain analysis is not supported on this graph"
" backend"
),
"chain": [],
}
result = chain if isinstance(chain, list) else list(chain)
return {"chain": result, "count": len(result), "direction": direction}
except Exception as exc:
+9 -5
View File
@@ -66,7 +66,6 @@ dependencies = [
"grpcio>=1.81.1",
"beautifulsoup4>=4.15.0",
"lxml>=6.1.1",
"pypdf2>=2.10.0",
"python-docx>=1.2.0",
"openpyxl>=3.1.5",
"pillow>=12.2.0",
@@ -85,7 +84,8 @@ dependencies = [
"python-dotenv>=1.2.1",
"loguru>=0.7.3",
"structlog>=22.1.0",
"gensim>=4.4.0"
"gensim>=4.4.0",
"httpx<0.29.0"
]
[project.urls]
@@ -146,6 +146,9 @@ graph-all = [
"semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune,graph-apache-age]"
]
# ---- Triplet Store Backends ----
tripletstore-oxigraph = ["pyoxigraph>=0.5.0"]
# ---- Vector Store Backends ----
vectorstore-qdrant = ["qdrant-client>=1.0.0"]
vectorstore-weaviate = ["weaviate-client>=4.0.0"]
@@ -230,7 +233,8 @@ explorer = [
"fastapi>=0.100.0",
"uvicorn[standard]>=0.22.0",
"websockets>=15.0.1",
"python-multipart>=0.0.6"
"python-multipart>=0.0.6",
"defusedxml>=0.7.1"
]
explorer-lite = [
"streamlit>=1.25.0",
@@ -239,8 +243,8 @@ explorer-lite = [
# Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
all = [
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,watch,llm-all,models-huggingface,split-all,graph-all,tripletstore-oxigraph,vectorstore-all,parse-docling,ingest-parquet,ingest-arrow,shacl,agno]"
]
# ---------------- ENTRYPOINTS ----------------
+142 -7
View File
@@ -2523,9 +2523,11 @@ def provenance_lineage(cli_ctx: CLIContext, entity_id: str, depth: int, local_js
default="table", show_default=True)
@click.option("--output", default=None, type=click.Path())
@click.option("--json", "local_json", is_flag=True, default=False)
@click.option("--dry-run", "local_dry", is_flag=True, default=False)
@click.pass_obj
def provenance_audit(cli_ctx: CLIContext, since: Optional[str], fmt: str,
output: Optional[str], local_json: bool) -> None:
output: Optional[str], local_json: bool,
local_dry: bool) -> None:
"""Export the audit log.
\b
@@ -2535,6 +2537,9 @@ def provenance_audit(cli_ctx: CLIContext, since: Optional[str], fmt: str,
cli_ctx = _require_ctx(cli_ctx)
def _action() -> None:
if _is_dry(cli_ctx, local_dry):
_dry(cli_ctx, "export audit log", since=since, format=fmt, output=output)
return
try:
from .provenance import ProvenanceManager
pm = ProvenanceManager(config=cli_ctx.config.to_dict())
@@ -2554,27 +2559,31 @@ def provenance_audit(cli_ctx: CLIContext, since: Optional[str], fmt: str,
@provenance.command("export")
@click.option("--format", "fmt", type=click.Choice(["turtle", "ntriples", "jsonld"]),
default="turtle", show_default=True)
@click.option("--base-uri", "base_uri", default=None,
help="Namespace URI entities/agents/activities are minted under "
"(default: ProvenanceManager.DEFAULT_BASE_URI).")
@click.option("--output", default=None, type=click.Path())
@click.option("--dry-run", "local_dry", is_flag=True, default=False)
@click.pass_obj
def provenance_export(cli_ctx: CLIContext, fmt: str, output: Optional[str],
local_dry: bool) -> None:
def provenance_export(cli_ctx: CLIContext, fmt: str, base_uri: Optional[str],
output: Optional[str], local_dry: bool) -> None:
"""Export provenance as W3C PROV-O RDF.
\b
Example:
semantica provenance export --format turtle --output prov.ttl
semantica provenance export --base-uri https://example.org/kg# --output prov.ttl
"""
cli_ctx = _require_ctx(cli_ctx)
def _action() -> None:
if _is_dry(cli_ctx, local_dry):
_dry(cli_ctx, "export provenance", format=fmt, output=output)
_dry(cli_ctx, "export provenance", format=fmt, base_uri=base_uri, output=output)
return
try:
from .provenance import ProvenanceManager
pm = ProvenanceManager(config=cli_ctx.config.to_dict())
data = pm.export_prov(format=fmt)
data = pm.export_prov(format=fmt, base_uri=base_uri)
except ImportError as exc:
raise click.ClickException(f"Provenance module not available: {exc}") from exc
if output:
@@ -2601,10 +2610,126 @@ def provenance_check(cli_ctx: CLIContext, strict: bool, local_json: bool) -> Non
result = pm.check(strict=strict)
except ImportError as exc:
raise click.ClickException(f"Provenance module not available: {exc}") from exc
is_valid = not isinstance(result, dict) or result.get("valid", True)
if _is_json(cli_ctx, local_json):
_jecho(result if isinstance(result, dict) else {"valid": bool(result)})
else:
elif is_valid:
_ok(cli_ctx, f"Provenance check: {result}")
else:
_warn(cli_ctx, f"Provenance check: {result}")
if strict and not is_valid:
raise click.ClickException(
f"Provenance integrity check failed: {result.get('errors')} error(s)"
)
_run_with_error_handling(_action)
@provenance.command("invalidate")
@click.argument("entity_id")
@click.option("--by", "agent_id", required=True, help="Agent responsible for the invalidation.")
@click.option("--reason", default=None, help="Human-readable reason for the invalidation.")
@click.option("--json", "local_json", is_flag=True, default=False)
@click.option("--dry-run", "local_dry", is_flag=True, default=False)
@click.pass_obj
def provenance_invalidate(cli_ctx: CLIContext, entity_id: str, agent_id: str,
reason: Optional[str], local_json: bool,
local_dry: bool) -> None:
"""Mark a tracked entity as invalidated (tombstone, not a hard delete).
\b
Example:
semantica provenance invalidate entity_alice --by reviewer_jane --reason "Source retracted"
"""
cli_ctx = _require_ctx(cli_ctx)
def _action() -> None:
if _is_dry(cli_ctx, local_dry):
_dry(cli_ctx, "invalidate provenance entity", entity_id=entity_id, by=agent_id, reason=reason)
return
try:
from .provenance import ProvenanceManager
pm = ProvenanceManager(config=cli_ctx.config.to_dict())
result = pm.invalidate(entity_id, agent_id=agent_id, reason=reason)
except ImportError as exc:
raise click.ClickException(f"Provenance module not available: {exc}") from exc
except ValueError as exc:
raise click.ClickException(str(exc)) from exc
if _is_json(cli_ctx, local_json):
_jecho(result.to_dict())
else:
_ok(cli_ctx, f"Invalidated {entity_id} (by {agent_id})")
_run_with_error_handling(_action)
@provenance.command("verify-chain")
@click.option("--json", "local_json", is_flag=True, default=False)
@click.pass_obj
def provenance_verify_chain(cli_ctx: CLIContext, local_json: bool) -> None:
"""Verify the hash chain across all provenance entries.
Detects wholesale row deletion: per-row checksums alone only prove a
surviving row wasn't edited in place, not that no row is missing.
\b
Example:
semantica provenance verify-chain
"""
cli_ctx = _require_ctx(cli_ctx)
def _action() -> None:
try:
from .provenance import ProvenanceManager
pm = ProvenanceManager(config=cli_ctx.config.to_dict())
result = pm.verify_chain()
except ImportError as exc:
raise click.ClickException(f"Provenance module not available: {exc}") from exc
if _is_json(cli_ctx, local_json):
_jecho(result)
elif result.get("valid"):
_ok(cli_ctx, f"Chain verified: {result.get('total_entries')} entries, no breaks")
else:
_warn(cli_ctx, f"Chain verification failed: {len(result.get('broken_links', []))} broken link(s)")
for link in result.get("broken_links", []):
click.echo(f" {link}")
_run_with_error_handling(_action)
@provenance.command("descendants")
@click.argument("entity_id")
@click.option("--depth", default=None, type=int, show_default=True)
@click.option("--json", "local_json", is_flag=True, default=False)
@click.pass_obj
def provenance_descendants(cli_ctx: CLIContext, entity_id: str, depth: Optional[int],
local_json: bool) -> None:
"""Show downstream descendants (reverse lineage) for an entity.
The counterpart to `provenance lineage`, which only traces upstream
ancestors. Answers "entity X was wrong — what downstream facts used it?"
\b
Example:
semantica provenance descendants entity_alice --depth 3
"""
cli_ctx = _require_ctx(cli_ctx)
def _action() -> None:
try:
from .provenance import ProvenanceManager
pm = ProvenanceManager(config=cli_ctx.config.to_dict())
if depth is not None:
entries = [e.to_dict() for e in pm.trace_descendants(entity_id, max_depth=depth)]
result = {"entity_id": entity_id, "depth": depth, "entries": entries}
else:
result = pm.get_descendants(entity_id) or {"entity_id": entity_id, "entries": []}
except ImportError as exc:
raise click.ClickException(f"Provenance module not available: {exc}") from exc
if _is_json(cli_ctx, local_json):
_jecho(result)
else:
_pprint(cli_ctx, result)
_run_with_error_handling(_action)
@@ -3940,7 +4065,7 @@ def server(ctx: click.Context) -> None:
@click.option("--port", default=8000, type=int, show_default=True)
@click.option("--workers", default=1, type=int, show_default=True)
@click.option("--reload", is_flag=True, default=False, help="Enable hot reload.")
@click.option("--host", default="0.0.0.0", show_default=True)
@click.option("--host", default="127.0.0.1", show_default=True)
@click.pass_obj
def server_start(cli_ctx: CLIContext, port: int, workers: int, reload: bool, host: str) -> None:
"""Start the REST API server.
@@ -3951,6 +4076,16 @@ def server_start(cli_ctx: CLIContext, port: int, workers: int, reload: bool, hos
"""
cli_ctx = _require_ctx(cli_ctx)
_LOOPBACK_HOSTS = {"127.0.0.1", "::1", "localhost"}
if host not in _LOOPBACK_HOSTS:
console.print(
f"[{_WARN_STY}] ⚠[/{_WARN_STY}] Binding to [cyan]{host}[/cyan] exposes "
"the server to the network. Set SEMANTICA_API_KEY before doing this "
"in any reachable environment — without it, protected routes refuse "
"all requests (503), and with SEMANTICA_ALLOW_ANONYMOUS=true they are "
"wide open."
)
def _action() -> None:
import subprocess as sp
cmd = [
+22 -6
View File
@@ -15,36 +15,52 @@ License: MIT
"""
from typing import Optional, Dict, Any, List
from datetime import datetime
class SourceTrackerWithUnifiedBackend:
"""SourceTracker using unified provenance backend."""
def __init__(self, **config):
def __init__(
self,
agent_id: Optional[str] = None,
is_automated: bool = True,
**config,
):
"""Initialize with unified backend or fallback to legacy."""
from .source_tracker import SourceTracker
self._agent_id = agent_id or self.__class__.__name__
self._is_automated = is_automated
try:
from semantica.provenance import ProvenanceManager
self._unified_manager = ProvenanceManager()
self._use_unified = True
except ImportError:
self._use_unified = False
self._original_tracker = SourceTracker(**config)
def track_property_source(self, entity_id: str, property_name: str, value: Any, source: Any, **metadata):
"""Track property source with unified backend."""
activity_started_at_time = datetime.utcnow().isoformat()
if self._use_unified:
from semantica.provenance import SourceReference
source_ref = SourceReference(
document=source.document if hasattr(source, 'document') else str(source),
page=getattr(source, 'page', None),
section=getattr(source, 'section', None),
confidence=getattr(source, 'confidence', 1.0)
)
metadata.setdefault("agent_id", self._agent_id)
metadata.setdefault("agent_type", "software_agent")
metadata.setdefault("is_automated", self._is_automated)
metadata.setdefault("activity_started_at_time", activity_started_at_time)
metadata.setdefault("activity_ended_at_time", datetime.utcnow().isoformat())
self._unified_manager.track_property_source(
entity_id=entity_id,
property_name=property_name,
+24 -9
View File
@@ -614,29 +614,44 @@ class SourceTracker:
if item_type == "entity":
entity_id = item.get("entity_id")
if entity_id:
self.track_entity_source(entity_id, source_ref, **metadata)
stats["entities_tracked"] += 1
stats["total_tracked"] += 1
ok = self.track_entity_source(entity_id, source_ref, **metadata)
if ok:
stats["entities_tracked"] += 1
stats["total_tracked"] += 1
else:
self.logger.warning(
f"Failed to track entity source for '{entity_id}' in item {i}"
)
elif item_type == "property":
entity_id = item.get("entity_id")
property_name = item.get("property_name")
value = item.get("value")
if entity_id and property_name is not None:
self.track_property_source(
ok = self.track_property_source(
entity_id, property_name, value, source_ref, **metadata
)
stats["properties_tracked"] += 1
stats["total_tracked"] += 1
if ok:
stats["properties_tracked"] += 1
stats["total_tracked"] += 1
else:
self.logger.warning(
f"Failed to track property source for '{entity_id}.{property_name}' in item {i}"
)
elif item_type == "relationship":
relationship_id = item.get("relationship_id")
if relationship_id:
self.track_relationship_source(
ok = self.track_relationship_source(
relationship_id, source_ref, **metadata
)
stats["relationships_tracked"] += 1
stats["total_tracked"] += 1
if ok:
stats["relationships_tracked"] += 1
stats["total_tracked"] += 1
else:
self.logger.warning(
f"Failed to track relationship source for '{relationship_id}' in item {i}"
)
else:
self.logger.warning(
File diff suppressed because it is too large Load Diff
+16 -1
View File
@@ -117,6 +117,7 @@ import uuid
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from ..utils.helpers import classify_path_distance
from ..utils.skos import is_skos_hierarchy_edge, validate_skos_hierarchy
from .entity_linker import EntityLinker
# Optional imports for advanced features
@@ -589,6 +590,14 @@ class ContextGraph:
"""
count = 0
with self._lock:
# Keep the SKOS hierarchy invariant at the lowest common write
# layer so direct graph users cannot bypass API/session checks.
hierarchy_edges = [edge for edge in edges if is_skos_hierarchy_edge(edge)]
if hierarchy_edges:
existing_edges = [
edge for edge in self.find_edges() if is_skos_hierarchy_edge(edge)
]
validate_skos_hierarchy(hierarchy_edges, existing_edges)
for raw_edge in edges:
if not isinstance(raw_edge, dict):
continue
@@ -948,6 +957,12 @@ class ContextGraph:
family_id=explicit_family_id,
)
with self._lock:
candidate = {"source": source_id, "target": target_id, "type": edge_type}
if is_skos_hierarchy_edge(candidate):
existing_edges = [
edge for edge in self.find_edges() if is_skos_hierarchy_edge(edge)
]
validate_skos_hierarchy([candidate], existing_edges)
return self._add_internal_edge(
ContextEdge(
edge_id=edge_id,
@@ -1676,7 +1691,7 @@ class ContextGraph:
# Fallback ID generation
import hashlib
entity_hash = hashlib.md5(
entity_hash = hashlib.md5( # nosec B324 - deterministic entity ID, not security-sensitive
f"{entity_text}_{entity_type}".encode()
).hexdigest()[:12]
entity_id = f"{entity_type.lower()}_{entity_hash}"
+21 -5
View File
@@ -12,36 +12,52 @@ License: MIT
"""
from typing import Optional, Any
from datetime import datetime
import uuid
class ContextManagerWithProvenance:
"""Context manager with provenance tracking."""
def __init__(self, provenance: bool = False, **config):
def __init__(
self,
provenance: bool = False,
agent_id: Optional[str] = None,
is_automated: bool = True,
**config,
):
"""Initialize context manager with optional provenance."""
from .context_manager import ContextManager
self.provenance = provenance
self._context_manager = ContextManager(**config)
self._prov_manager = None
self._agent_id = agent_id or self.__class__.__name__
self._is_automated = is_automated
if provenance:
try:
from semantica.provenance import ProvenanceManager
self._prov_manager = ProvenanceManager()
except ImportError:
self.provenance = False
def add_context(self, context: Any, source: Optional[str] = None, **kwargs):
"""Add context with provenance tracking."""
activity_started_at_time = datetime.utcnow().isoformat()
result = self._context_manager.add_context(context, **kwargs)
activity_ended_at_time = datetime.utcnow().isoformat()
if self.provenance and self._prov_manager:
self._prov_manager.track_entity(
entity_id=f"context_{uuid.uuid4().hex[:8]}",
source=source or "context_manager",
entity_type="context",
agent_id=self._agent_id,
agent_type="software_agent",
is_automated=self._is_automated,
activity_started_at_time=activity_started_at_time,
activity_ended_at_time=activity_ended_at_time,
metadata={"context_preview": str(context)[:100]}
)
+2 -2
View File
@@ -170,7 +170,7 @@ class EntityLinker:
uri = f"{self.base_uri}{uri_safe}"
else:
# Use hash of entity_id
entity_hash = hashlib.md5(entity_id.encode()).hexdigest()[:8]
entity_hash = hashlib.md5(entity_id.encode()).hexdigest()[:8] # nosec B324 - deterministic URI suffix, not security-sensitive
uri = f"{self.base_uri}{entity_hash}"
# Add type if available
@@ -500,7 +500,7 @@ class EntityLinker:
def _generate_entity_id(self, text: str, entity_type: str) -> str:
"""Generate entity ID from text and type."""
entity_hash = hashlib.md5(f"{text}_{entity_type}".encode()).hexdigest()[:12]
entity_hash = hashlib.md5(f"{text}_{entity_type}".encode()).hexdigest()[:12] # nosec B324 - deterministic entity ID, not security-sensitive
return f"{entity_type.lower()}_{entity_hash}"
def build_entity_web(self) -> Dict[str, Any]:
+2 -2
View File
@@ -51,7 +51,6 @@ Author: Semantica Contributors
License: MIT
"""
import json
import os
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
@@ -67,7 +66,6 @@ from ..utils.helpers import (
set_nested_value,
)
from ..utils.progress_tracker import get_progress_tracker
from ..utils.validators import validate_config
class Config:
@@ -157,6 +155,7 @@ class Config:
self.security = config_data.get(
"security", DEFAULT_CONFIG.get("security", {})
)
self.provenance = config_data.get("provenance", {})
self.custom = config_data.get("custom", {})
def _load_from_env(self, config_dict: Dict[str, Any]) -> None:
@@ -346,6 +345,7 @@ class Config:
"logging": self.logging,
"quality": self.quality,
"security": self.security,
"provenance": self.provenance,
"custom": self.custom,
}
+18 -6
View File
@@ -27,11 +27,11 @@ License: MIT
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ..utils.exceptions import ConfigurationError, ProcessingError
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger, log_execution_time
from ..utils.progress_tracker import get_progress_tracker
from .config_manager import Config, ConfigManager
from .lifecycle import LifecycleManager, SystemState
from .lifecycle import LifecycleManager
from .plugin_registry import PluginRegistry
@@ -90,6 +90,18 @@ class Semantica:
)
self.lifecycle_manager.register_component("config_manager", self.config_manager)
# Configure global provenance storage path if specified
try:
# Config.get() already resolves dotted paths against nested dicts
# (see get_nested_value), so this single lookup covers both
# top-level and config.provenance={"storage_path": ...} shapes.
prov_storage_path = self.config.get("provenance.storage_path")
if prov_storage_path:
from ..provenance import ProvenanceManager
ProvenanceManager.set_default_storage_path(prov_storage_path)
except Exception as e:
self.logger.warning(f"Failed to configure provenance storage: {e}")
# Module placeholders (to be initialized)
self._modules: Dict[str, Any] = {}
self._initialized: bool = False
@@ -631,10 +643,10 @@ class Semantica:
try:
# Import key modules to verify they're available
# These imports don't create instances, just verify module availability
from ..ingest import FileIngestor
from ..kg import GraphBuilder
from ..parse import DocumentParser
from ..pipeline import PipelineBuilder
from ..ingest import FileIngestor # noqa: F401
from ..kg import GraphBuilder # noqa: F401
from ..parse import DocumentParser # noqa: F401
from ..pipeline import PipelineBuilder # noqa: F401
self.logger.debug("Framework modules verified and available")
except (ImportError, OSError) as e:
@@ -13,37 +13,53 @@ Author: Semantica Contributors
License: MIT
"""
from typing import List, Any
from typing import List, Any, Optional
from datetime import datetime
import uuid
class DeduplicatorWithProvenance:
"""Deduplicator with provenance tracking."""
def __init__(self, provenance: bool = False, **config):
def __init__(
self,
provenance: bool = False,
agent_id: Optional[str] = None,
is_automated: bool = True,
**config,
):
from .deduplicator import Deduplicator
self.provenance = provenance
self._deduplicator = Deduplicator(**config)
self._prov_manager = None
self._agent_id = agent_id or self.__class__.__name__
self._is_automated = is_automated
if provenance:
try:
from semantica.provenance import ProvenanceManager
self._prov_manager = ProvenanceManager()
except ImportError:
self.provenance = False
def deduplicate(self, items: List[Any], source: str = None, **kwargs):
"""Deduplicate items with provenance tracking."""
activity_started_at_time = datetime.utcnow().isoformat()
unique_items = self._deduplicator.deduplicate(items, **kwargs)
activity_ended_at_time = datetime.utcnow().isoformat()
if self.provenance and self._prov_manager:
duplicates_found = len(items) - len(unique_items)
self._prov_manager.track_entity(
entity_id=f"dedup_{uuid.uuid4().hex[:8]}",
source=source or "deduplication",
entity_type="deduplication_operation",
agent_id=self._agent_id,
agent_type="software_agent",
is_automated=self._is_automated,
activity_started_at_time=activity_started_at_time,
activity_ended_at_time=activity_ended_at_time,
metadata={
"input_count": len(items),
"output_count": len(unique_items),
+23 -7
View File
@@ -13,36 +13,52 @@ Author: Semantica Contributors
License: MIT
"""
from typing import List
from typing import List, Optional
from datetime import datetime
import uuid
class EmbeddingGeneratorWithProvenance:
"""Embedding generator with provenance tracking."""
def __init__(self, provenance: bool = False, **config):
def __init__(
self,
provenance: bool = False,
agent_id: Optional[str] = None,
is_automated: bool = True,
**config,
):
from .embedding_generator import EmbeddingGenerator
self.provenance = provenance
self._generator = EmbeddingGenerator(**config)
self._prov_manager = None
self._agent_id = agent_id or self.__class__.__name__
self._is_automated = is_automated
if provenance:
try:
from semantica.provenance import ProvenanceManager
self._prov_manager = ProvenanceManager()
except ImportError:
self.provenance = False
def embed(self, texts: List[str], source: str = None, **kwargs):
"""Generate embeddings with provenance tracking."""
activity_started_at_time = datetime.utcnow().isoformat()
embeddings = self._generator.embed(texts, **kwargs)
activity_ended_at_time = datetime.utcnow().isoformat()
if self.provenance and self._prov_manager:
self._prov_manager.track_entity(
entity_id=f"embed_{uuid.uuid4().hex[:8]}",
source=source or "embedding_generation",
entity_type="embeddings",
agent_id=self._agent_id,
agent_type="software_agent",
is_automated=self._is_automated,
activity_started_at_time=activity_started_at_time,
activity_ended_at_time=activity_ended_at_time,
metadata={
"model": getattr(self._generator, 'model', 'unknown'),
"dimensions": len(embeddings[0]) if embeddings else 0,
+16 -6
View File
@@ -83,12 +83,22 @@ def main(argv=None):
_LOOPBACK_HOSTS = {"127.0.0.1", "::1", "localhost"}
if args.host not in _LOOPBACK_HOSTS:
_err.print(
f"[bold yellow]Warning:[/bold yellow] Binding to "
f"[cyan]{args.host}[/cyan] exposes the Explorer to the network. "
"The API has no authentication — all graph data will be readable "
"and writable by any host that can reach this port."
)
import os as _os
if _os.environ.get("SEMANTICA_ALLOW_ANONYMOUS", "").strip().lower() == "true":
_err.print(
f"[bold yellow]Warning:[/bold yellow] Binding to "
f"[cyan]{args.host}[/cyan] with SEMANTICA_ALLOW_ANONYMOUS=true "
"exposes the Explorer to the network with no authentication — "
"all graph data will be readable and writable by any host that "
"can reach this port."
)
elif not _os.environ.get("SEMANTICA_API_KEY"):
_err.print(
f"[bold yellow]Warning:[/bold yellow] Binding to "
f"[cyan]{args.host}[/cyan] but SEMANTICA_API_KEY is not set — "
"protected routes will refuse all requests (503) until it is "
"configured."
)
if not args.no_browser:
import threading
+24 -3
View File
@@ -17,6 +17,7 @@ from .. import __version__
from ..context.context_graph import ContextGraph
from .session import GraphSession
from .ws import ConnectionManager
from .auth import APIKeyAuthMiddleware, warn_if_unauthenticated
def _read_int_env(name: str, default: int) -> int:
@@ -45,6 +46,10 @@ def _read_explorer_settings() -> dict:
# ContextGraph and does not open a network connection to FalkorDB.
"falkordb_host": os.environ.get("FALKORDB_HOST", "localhost"),
"falkordb_port": _read_int_env("FALKORDB_PORT", 6379),
"provenance_storage_path": os.environ.get(
"SEMANTICA_PROVENANCE_DB",
os.environ.get("EXPLORER_PROVENANCE_DB"),
),
}
@@ -75,9 +80,21 @@ def _install_mutation_bridge(app: FastAPI, session: GraphSession) -> None:
session.graph.mutation_callback = on_mutation
def create_app(session: Optional[GraphSession] = None) -> FastAPI:
active_session = session or GraphSession(ContextGraph(advanced_analytics=False))
def create_app(
session: Optional[GraphSession] = None,
provenance_storage_path: Optional[str] = None,
) -> FastAPI:
settings = _read_explorer_settings()
prov_path = provenance_storage_path or settings.get("provenance_storage_path")
if session is None:
active_session = GraphSession(
ContextGraph(advanced_analytics=False),
provenance_storage_path=prov_path,
)
else:
active_session = session
if prov_path is not None:
active_session.set_provenance_storage_path(prov_path)
@asynccontextmanager
async def lifespan(app: FastAPI):
@@ -107,10 +124,14 @@ def create_app(session: Optional[GraphSession] = None) -> FastAPI:
allow_origins=settings["allowed_origins"],
allow_credentials=_allow_credentials,
allow_methods=["GET", "POST", "DELETE", "OPTIONS"],
allow_headers=["Content-Type", "Authorization"],
allow_headers=["Content-Type", "Authorization", "X-API-Key"],
max_age=600,
)
# API key authentication (opt-in via EXPLORER_API_KEY env var)
app.add_middleware(APIKeyAuthMiddleware)
warn_if_unauthenticated()
import logging as _logging
_logger = _logging.getLogger(__name__)
+102
View File
@@ -0,0 +1,102 @@
"""
Semantica Explorer : Authentication Middleware
Provides opt-in API key authentication for all Explorer API routes.
Enable by setting the ``EXPLORER_API_KEY`` environment variable. When set,
every request to ``/api/*`` must include either:
- An ``Authorization: Bearer <key>`` header, or
- An ``X-API-Key: <key>`` header.
When ``EXPLORER_API_KEY`` is not set, authentication is disabled and the
Explorer operates in open/development mode (with a startup warning).
"""
import hmac
import logging
import os
from typing import Optional
from fastapi import HTTPException, Request, status
from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
from starlette.responses import Response
_logger = logging.getLogger(__name__)
def _get_api_key() -> Optional[str]:
"""Read the configured API key from the environment."""
return os.environ.get("EXPLORER_API_KEY")
def _extract_token(request: Request) -> Optional[str]:
"""Extract the API key from the request headers."""
# Check Authorization: Bearer <key>
auth_header = request.headers.get("Authorization", "")
if auth_header.startswith("Bearer "):
return auth_header[7:].strip()
# Check X-API-Key: <key>
api_key_header = request.headers.get("X-API-Key", "")
if api_key_header:
return api_key_header.strip()
return None
class APIKeyAuthMiddleware(BaseHTTPMiddleware):
"""
Middleware that enforces API key authentication on ``/api/*`` routes.
Skips authentication for:
- Non-API routes (static files, health checks, WebSocket, docs)
- OPTIONS requests (CORS preflight)
- When ``EXPLORER_API_KEY`` is not configured (open mode)
"""
async def dispatch(
self, request: Request, call_next: RequestResponseEndpoint
) -> Response:
api_key = _get_api_key()
# If no API key is configured, allow all requests (open mode)
if not api_key:
return await call_next(request)
# Skip authentication for non-API paths
path = request.url.path
if not path.startswith("/api/"):
return await call_next(request)
# Skip CORS preflight
if request.method == "OPTIONS":
return await call_next(request)
# Validate the token
token = _extract_token(request)
if not token:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Missing API key. Provide via 'Authorization: Bearer <key>' or 'X-API-Key: <key>' header.",
headers={"WWW-Authenticate": "Bearer"},
)
# Constant-time comparison to prevent timing attacks
if not hmac.compare_digest(token, api_key):
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="Invalid API key.",
)
return await call_next(request)
def warn_if_unauthenticated() -> None:
"""Log a warning at startup if no API key is configured."""
if not _get_api_key():
_logger.warning(
"EXPLORER_API_KEY is not set. The Explorer API is running WITHOUT "
"authentication. Set EXPLORER_API_KEY to enable API key protection "
"for all /api/* endpoints."
)
+63 -2
View File
@@ -2,14 +2,75 @@
Semantica Explorer : FastAPI Dependencies
Provides ``Depends()``-compatible callables for injecting the
current ``GraphSession`` and ``ConnectionManager`` into route handlers.
current ``GraphSession`` and ``ConnectionManager`` into route handlers,
and for enforcing API-key authentication on protected routes.
"""
from fastapi import Request, HTTPException, status
import hmac
import os
from typing import Optional
from fastapi import Request, HTTPException, Security, status
from fastapi.security.api_key import APIKeyHeader
from .session import GraphSession
from .ws import ConnectionManager
_api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False)
def get_expected_api_key() -> Optional[str]:
"""Read the configured API key from the environment on every call.
Read fresh (not cached) so tests and ops tooling can rotate the key
without restarting the process.
"""
return os.environ.get("SEMANTICA_API_KEY") or None
def anonymous_access_allowed() -> bool:
return os.environ.get("SEMANTICA_ALLOW_ANONYMOUS", "").strip().lower() == "true"
def is_valid_api_key(candidate: Optional[str]) -> bool:
"""Return True if *candidate* matches the configured key, or if the
server has explicitly opted into anonymous access."""
if anonymous_access_allowed():
return True
expected = get_expected_api_key()
if not expected:
return False
return bool(candidate) and hmac.compare_digest(candidate, expected)
def require_auth(api_key: Optional[str] = Security(_api_key_header)) -> None:
"""Dependency enforcing the ``X-API-Key`` header on protected routes.
Every Explorer/API router (except health/info/static assets) should be
mounted with ``dependencies=[Depends(require_auth)]``. If
SEMANTICA_API_KEY is unset, requests are refused with 503 rather than
silently served unauthenticated SEMANTICA_ALLOW_ANONYMOUS=true opts
into that explicitly for local development.
"""
if anonymous_access_allowed():
return
expected = get_expected_api_key()
if not expected:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail=(
"Server is not configured for authentication. Set the "
"SEMANTICA_API_KEY environment variable, or explicitly opt "
"into unauthenticated access (development only) with "
"SEMANTICA_ALLOW_ANONYMOUS=true."
),
)
if not api_key or not hmac.compare_digest(api_key, expected):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or missing API key. Send it as the X-API-Key header.",
)
def get_session(request: Request) -> GraphSession:
"""Retrieve the GraphSession stored on ``app.state``."""
+20 -2
View File
@@ -1,11 +1,11 @@
"""
"""
Analytics routes for graph metrics and validation.
"""
import asyncio
from typing import Optional
from fastapi import APIRouter, Depends, Query
from fastapi import APIRouter, Depends, HTTPException, Query, Response
from ..dependencies import get_session
from ..schemas import AnalyticsResponse, ValidationIssue, ValidationReportResponse
@@ -16,6 +16,7 @@ router = APIRouter(prefix="/api/analytics", tags=["Analytics"])
@router.get("", response_model=AnalyticsResponse)
async def get_analytics(
response: Response,
metrics: Optional[str] = Query(
None,
description="Comma-separated metrics to compute: centrality,community,connectivity",
@@ -25,8 +26,11 @@ async def get_analytics(
requested = set((metrics or "centrality,community,connectivity").split(","))
graph_dict = await asyncio.to_thread(session.build_graph_dict)
result: dict = {}
attempted = 0
failed = 0
if "centrality" in requested and session.centrality is not None:
attempted += 1
try:
result["centrality"] = await asyncio.to_thread(
session.centrality.calculate_degree_centrality,
@@ -34,8 +38,10 @@ async def get_analytics(
)
except Exception as exc:
result["centrality"] = {"error": str(exc)}
failed += 1
if "community" in requested and session.community is not None:
attempted += 1
try:
result["community"] = await asyncio.to_thread(
session.community.detect_communities,
@@ -43,8 +49,10 @@ async def get_analytics(
)
except Exception as exc:
result["community"] = {"error": str(exc)}
failed += 1
if "connectivity" in requested and session.connectivity is not None:
attempted += 1
try:
result["connectivity"] = await asyncio.to_thread(
session.connectivity.analyze_connectivity,
@@ -52,6 +60,16 @@ async def get_analytics(
)
except Exception as exc:
result["connectivity"] = {"error": str(exc)}
failed += 1
if attempted and failed == attempted:
# Every requested metric raised: a plain 2xx (even 207) reads as
# success to callers that only check `response.ok`, so surface this
# as a hard failure rather than a body full of {"error": ...}.
raise HTTPException(status_code=500, detail="All requested analytics metrics failed to compute")
if failed:
response.status_code = 207
return AnalyticsResponse(**result)
+8 -4
View File
@@ -112,8 +112,10 @@ async def import_file(
}
)
nodes_added = session.add_nodes(nodes)
edges_added = session.add_edges(edges)
try:
nodes_added, edges_added = session.add_nodes_and_edges(nodes, edges)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
return _import_response(nodes_added, edges_added)
if filename.endswith(".csv"):
@@ -184,8 +186,10 @@ async def import_file(
detail="No valid nodes or edges could be parsed from the CSV payload.",
)
nodes_added = session.add_nodes(nodes)
edges_added = session.add_edges(edges)
try:
nodes_added, edges_added = session.add_nodes_and_edges(nodes, edges)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
return _import_response(nodes_added, edges_added)
raise HTTPException(
+598 -61
View File
@@ -11,7 +11,7 @@ import uuid
from datetime import datetime, UTC
from difflib import SequenceMatcher
from typing import Any, Dict, List, Optional, Tuple
from urllib.parse import urlparse
from urllib.parse import urljoin, urlparse
from typing_extensions import Literal
from fastapi import APIRouter, Depends, HTTPException, Query, Request
@@ -36,6 +36,12 @@ _MAX_FETCH_BYTES = 20 * 1024 * 1024 # 20 MB
_MAX_ANALYSIS_NODES = 5_000 # cap for health/suggest/shacl node scans to avoid OOM
_MAX_ENTITIES_PER_SIDE = 500 # per-ontology cap for the O(n²) pairwise suggestion loop
class GraphTruncationError(Exception):
"""Raised when a graph exceeds _MAX_ANALYSIS_NODES so analysis would be truncated."""
pass
_CLASS_TYPES = frozenset({
"owl:Class", "rdfs:Class",
"http://www.w3.org/2002/07/owl#Class",
@@ -107,6 +113,43 @@ _INGEST_FORMAT_SUFFIXES: Dict[str, str] = {
}
# ---------------------------------------------------------------------------
# SHACL Validation Resource Guardrails
# ---------------------------------------------------------------------------
_MAX_SHACL_TURTLE_BYTES: int = int(
os.environ.get("SEMANTICA_MAX_SHACL_TURTLE_BYTES", "262144")
) # 256 KB
_MAX_SHACL_TRIPLES: int = int(
os.environ.get("SEMANTICA_MAX_SHACL_TRIPLES", "1000")
) # 1,000 triples
_MAX_SHACL_TIMEOUT_SECONDS: float = float(
os.environ.get("SEMANTICA_MAX_SHACL_TIMEOUT", "15.0")
)
_MAX_SHACL_CONCURRENCY: int = int(
os.environ.get("SEMANTICA_MAX_SHACL_CONCURRENCY", "4")
)
_shacl_validation_semaphore: Optional[
Tuple[asyncio.AbstractEventLoop, asyncio.Semaphore]
] = None
def _get_shacl_semaphore() -> asyncio.Semaphore:
global _shacl_validation_semaphore
try:
current_loop = asyncio.get_running_loop()
except RuntimeError:
current_loop = None
if (
_shacl_validation_semaphore is None
or _shacl_validation_semaphore[0] != current_loop
):
_shacl_validation_semaphore = (
current_loop,
asyncio.Semaphore(_MAX_SHACL_CONCURRENCY),
)
return _shacl_validation_semaphore[1]
# ---------------------------------------------------------------------------
# Schemas
# ---------------------------------------------------------------------------
@@ -579,7 +622,17 @@ def _as_uri_list(value: Any) -> List[str]:
if value is None or value == "":
return []
if isinstance(value, list):
return [str(item) for item in value if item]
result = []
for item in value:
if item is None or item == "":
continue
if isinstance(item, dict):
uri = item.get("uri") or item.get("id") or item.get("@id")
if uri:
result.append(str(uri))
else:
result.append(str(item))
return result
if isinstance(value, dict):
uri = value.get("uri") or value.get("id") or value.get("@id")
return [str(uri)] if uri else []
@@ -670,6 +723,14 @@ def _extract_namespace(uri: str) -> Optional[str]:
return None
def _ontology_namespace(uri: str) -> str:
if "#" in uri:
return uri.rsplit("#", 1)[0] + "#"
if uri.endswith("/"):
return uri
return uri.rstrip("#/") + "#"
def _alignment_id(source_uri: str, relation: str, target_uri: str) -> str:
key = f"{source_uri}|{relation}|{target_uri}"
return str(uuid.uuid5(uuid.NAMESPACE_OID, key))
@@ -697,8 +758,8 @@ def _node_belongs_to_ontology(node: Dict[str, Any], ontology_uri: str) -> bool:
return True
if _node_source_ontology(node) == ontology_uri:
return True
namespace = _extract_namespace(ontology_uri)
return bool(namespace and nid.startswith(namespace))
stem = ontology_uri.rstrip("#/")
return nid.startswith((stem + "#", stem + "/"))
def _is_ontology_entity(node: Dict[str, Any]) -> bool:
@@ -785,6 +846,17 @@ def _ontology_entities(nodes: List[Dict[str, Any]], ontology_uri: Optional[str]
return result
def _data_graph_entities(nodes: List[Dict[str, Any]], ontology_uri: Optional[str] = None) -> List[Dict[str, Any]]:
result = []
for node in nodes:
if _classify_node_type(node.get("type", "")) not in {"class", "property", "individual", "concept", "scheme"}:
continue
if ontology_uri and not _node_belongs_to_ontology(node, ontology_uri):
continue
result.append(node)
return result
def _ontology_dict_from_nodes(uri: str, name: str, nodes: List[Dict[str, Any]], edges: List[Dict[str, Any]]) -> Dict[str, Any]:
classes = []
properties = []
@@ -820,14 +892,14 @@ def _ontology_dict_from_nodes(uri: str, name: str, nodes: List[Dict[str, Any]],
return {
"name": name,
"namespace": _extract_namespace(uri) or uri.rstrip("#/") + "#",
"namespace": _ontology_namespace(uri),
"classes": classes,
"properties": properties,
}
def _basic_shacl_turtle(uri: str, name: str, nodes: List[Dict[str, Any]]) -> str:
namespace = _extract_namespace(uri) or uri.rstrip("#/") + "#"
namespace = _ontology_namespace(uri)
lines = [
"@prefix sh: <http://www.w3.org/ns/shacl#> .",
"@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .",
@@ -906,7 +978,27 @@ def _normalize_format(fmt: Optional[str]) -> str:
return _FORMAT_ALIASES.get(lower, lower)
def _validate_fetch_url(url: str) -> None:
import threading
import urllib3.util.connection
if not hasattr(urllib3.util.connection, "_orig_create_connection"):
urllib3.util.connection._orig_create_connection = urllib3.util.connection.create_connection
_dns_pin_tls = threading.local()
def _patched_create_connection(address, *args, **kwargs):
host, port = address
pinned_host = getattr(_dns_pin_tls, 'pinned_host', None)
pinned_ip = getattr(_dns_pin_tls, 'pinned_ip', None)
if pinned_host and pinned_ip and host == pinned_host:
return urllib3.util.connection._orig_create_connection((pinned_ip, port), *args, **kwargs)
return urllib3.util.connection._orig_create_connection(address, *args, **kwargs)
urllib3.util.connection.create_connection = _patched_create_connection
def _validate_fetch_url(url: str) -> str:
"""Reject non-HTTP(S) schemes and private/loopback/link-local targets."""
parsed = urlparse(url)
if parsed.scheme not in ("http", "https"):
@@ -918,6 +1010,7 @@ def _validate_fetch_url(url: str) -> None:
addrinfos = socket.getaddrinfo(hostname, None)
except socket.gaierror as exc:
raise HTTPException(status_code=422, detail=f"Cannot resolve hostname '{hostname}': {exc}") from exc
safe_ips = []
for _family, _type, _proto, _canonname, sockaddr in addrinfos:
try:
ip = ipaddress.ip_address(sockaddr[0])
@@ -928,28 +1021,57 @@ def _validate_fetch_url(url: str) -> None:
status_code=422,
detail="Fetching from private, loopback, or reserved network addresses is not allowed.",
)
safe_ips.append(str(ip))
if not safe_ips:
raise HTTPException(status_code=422, detail="Could not resolve to a valid IP address.")
return safe_ips[0]
def _fetch_url_sync(url: str) -> bytes:
_validate_fetch_url(url)
import requests as _req
_MAX_REDIRECTS = 5
current_url = url
try:
resp = _req.get(
url,
headers={"Accept": "text/turtle, application/rdf+xml, application/ld+json, */*;q=0.1"},
timeout=30,
stream=True,
allow_redirects=True,
)
resp.raise_for_status()
chunks: List[bytes] = []
total = 0
for chunk in resp.iter_content(65536):
total += len(chunk)
if total > _MAX_FETCH_BYTES:
raise HTTPException(status_code=413, detail="Remote resource exceeds 20 MB limit.")
chunks.append(chunk)
return b"".join(chunks)
for _ in range(_MAX_REDIRECTS + 1):
safe_ip = _validate_fetch_url(current_url)
_dns_pin_tls.pinned_host = urlparse(current_url).hostname
_dns_pin_tls.pinned_ip = safe_ip
try:
resp = _req.get(
current_url,
headers={"Accept": "text/turtle, application/rdf+xml, application/ld+json, */*;q=0.1"},
timeout=30,
stream=True,
allow_redirects=False, # SECURITY: follow redirects manually
)
finally:
_dns_pin_tls.pinned_host = None
_dns_pin_tls.pinned_ip = None
if resp.is_redirect or resp.is_permanent_redirect:
redirect_url = resp.headers.get("Location")
resp.close() # Release the streamed connection before following the redirect
if not redirect_url:
raise HTTPException(status_code=502, detail="Redirect without Location header.")
# Resolve relative redirects (e.g. /ontology.ttl) against the current URL
redirect_url = urljoin(current_url, redirect_url)
# Re-validate the redirect target to prevent SSRF via
# open-redirect to internal/cloud-metadata endpoints.
_validate_fetch_url(redirect_url)
current_url = redirect_url
continue
try:
resp.raise_for_status()
chunks: List[bytes] = []
total = 0
for chunk in resp.iter_content(65536):
total += len(chunk)
if total > _MAX_FETCH_BYTES:
raise HTTPException(status_code=413, detail="Remote resource exceeds 20 MB limit.")
chunks.append(chunk)
return b"".join(chunks)
finally:
resp.close() # Release the streamed connection once fully read (or on error)
raise HTTPException(status_code=502, detail=f"Too many redirects (max {_MAX_REDIRECTS}).")
except HTTPException:
raise
except Exception as exc:
@@ -1217,6 +1339,8 @@ async def load_ontology(
temp_path,
format=fmt
)
if not ontology_data.data.get("classes") and not ontology_data.data.get("properties"):
raise ValueError("No OWL classes or properties found by OntologyIngestor")
# Convert to graph nodes/edges using ontology data
nodes, edges = await asyncio.to_thread(
@@ -1224,9 +1348,12 @@ async def load_ontology(
ontology_data.data
)
# Add nodes and edges to session
nodes_added = await asyncio.to_thread(session.add_nodes, nodes)
edges_added = await asyncio.to_thread(session.add_edges, edges)
try:
nodes_added, edges_added = await asyncio.to_thread(
session.add_nodes_and_edges, nodes, edges
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
# Register in registry
registry = _get_registry(request)
@@ -1262,21 +1389,29 @@ async def load_ontology(
except OSError as cleanup_exc:
logger.debug("Failed to remove temporary ontology file: %s", cleanup_exc)
except HTTPException:
# Re-raise HTTPExceptions we deliberately raised above (e.g. the 422 from
# SKOS cycle validation) instead of letting the broad `except Exception`
# below mask them as an ingestor failure and silently retry via the
# fallback parser.
raise
except Exception as ingest_exc:
logger.warning(f"OntologyIngestor failed, falling back to basic parsing: {ingest_exc}")
# Fallback to basic parsing
nodes, edges, metadata = await asyncio.to_thread(
_parse_rdf_sync, content_str.encode("utf-8"), fmt
)
except HTTPException:
raise
except Exception as exc:
raise HTTPException(status_code=422, detail=f"Could not parse ontology: {exc}") from exc
# Fallback path - use basic parsing
nodes_added = await asyncio.to_thread(session.add_nodes, nodes)
edges_added = await asyncio.to_thread(session.add_edges, edges)
try:
nodes_added, edges_added = await asyncio.to_thread(
session.add_nodes_and_edges, nodes, edges
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
registry = _get_registry(request)
ontology_uri = metadata.get("uri", f"temp:{uuid.uuid4().hex[:12]}")
@@ -1400,8 +1535,8 @@ async def create_ontology(
logger.info(f"Generated ontology from sample data with {len(nodes)} nodes, {len(edges)} edges")
except Exception as exc:
logger.exception("Failed to generate ontology from sample data; falling back to minimal ontology.")
logger.warning(f"OntologyEngine.from_data error: {exc}")
logger.exception("Failed to generate ontology from sample data; aborting ontology creation.")
raise HTTPException(status_code=500, detail=f"Ontology generation failed: {exc}") from exc
elif body.mode == "text" and body.schema_text:
try:
@@ -1470,11 +1605,15 @@ async def create_ontology(
logger.info(f"Generated ontology from text with {len(nodes)} nodes, {len(edges)} edges")
except Exception as exc:
logger.exception("Failed to generate ontology from schema text; falling back to minimal ontology.")
logger.warning(f"OntologyEngine.from_text error: {exc}")
logger.exception("Failed to generate ontology from schema text; aborting ontology creation.")
raise HTTPException(status_code=500, detail=f"Ontology generation failed: {exc}") from exc
nodes_added = await asyncio.to_thread(session.add_nodes, nodes)
edges_added = await asyncio.to_thread(session.add_edges, edges)
try:
nodes_added, edges_added = await asyncio.to_thread(
session.add_nodes_and_edges, nodes, edges
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
registry = _get_registry(request)
registry[onto_uri] = OntologyEntry(
@@ -2093,13 +2232,72 @@ async def ontology_health(
documentation_score = ((with_comment / total) * 80.0) + (20.0 if entry.version or entry.source_url else 0.0)
shacl_dimension = HealthDimension(
key="shacl",
label="SHACL Conformance",
score=0.0,
status="unavailable",
detail="Live SHACL validation is available in SHACL Studio when optional validation dependencies are installed.",
)
try:
health_nodes, health_edges = await _fetch_analysis_graph(session, uri, "shacl-health")
shacl_turtle, _ = await _generated_shacl_for_uri(
request, session, uri, nodes=health_nodes, edges=health_edges
)
data_graph_turtle = await _data_graph_turtle_for_uri(
request, session, uri, nodes=health_nodes, edges=health_edges
)
from ...ontology import OntologyEngine
engine = OntologyEngine()
report = await asyncio.to_thread(
engine.validate_graph,
data_graph_turtle,
shacl=shacl_turtle,
data_graph_format="turtle",
shacl_format="turtle",
)
warning_count = len(report.warnings)
if not report.conforms:
shacl_score = max(0.0, 100.0 - (report.violation_count * 20.0))
elif warning_count:
shacl_score = max(0.0, 100.0 - (warning_count * 5.0))
else:
shacl_score = 100.0
shacl_status = "ok" if report.conforms and not warning_count else "warning"
detail_parts = []
if not report.conforms:
detail_parts.append(f"{report.violation_count} SHACL violation(s)")
if warning_count:
detail_parts.append(f"{warning_count} SHACL warning(s)")
shacl_detail = (
"Graph conforms to all generated SHACL constraints."
if not detail_parts
else f"Graph has {', '.join(detail_parts)}."
)
shacl_dimension = HealthDimension(
key="shacl",
label="SHACL Conformance",
score=round(shacl_score, 1),
status=shacl_status,
detail=shacl_detail,
)
except GraphTruncationError as exc:
shacl_dimension = HealthDimension(
key="shacl",
label="SHACL Conformance",
score=0.0,
status="critical",
detail=str(exc),
)
except ImportError:
shacl_dimension = HealthDimension(
key="shacl",
label="SHACL Conformance",
score=0.0,
status="unavailable",
detail="Live SHACL validation is available in SHACL Studio when optional validation dependencies are installed.",
)
except Exception as exc:
shacl_dimension = HealthDimension(
key="shacl",
label="SHACL Conformance",
score=0.0,
status="critical",
detail=f"Live SHACL validation failed: {exc}",
)
dimensions = [
HealthDimension(
@@ -2145,19 +2343,46 @@ async def ontology_health(
)
async def _fetch_analysis_graph(
session: GraphSession,
uri: str,
log_tag: str,
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
"""Fetch nodes/edges for SHACL analysis, raising GraphTruncationError if oversized.
Shared by `_generated_shacl_for_uri` and `_data_graph_turtle_for_uri` so callers that
need both (e.g. `/health`) can fetch once and pass the results to both via `nodes=`/`edges=`.
"""
nodes, total_nodes = await asyncio.to_thread(session.get_nodes, skip=0, limit=_MAX_ANALYSIS_NODES)
edges, total_edges = await asyncio.to_thread(session.get_edges, skip=0, limit=_MAX_ANALYSIS_NODES)
if total_nodes > _MAX_ANALYSIS_NODES or total_edges > _MAX_ANALYSIS_NODES:
logger.warning(
"%s: graph for %s is too large to analyse (nodes=%d, edges=%d, limit=%d); skipping.",
log_tag, uri, total_nodes, total_edges, _MAX_ANALYSIS_NODES,
)
raise GraphTruncationError(
f"Graph size (nodes={total_nodes}, edges={total_edges}) exceeds maximum analysis limit ({_MAX_ANALYSIS_NODES}). "
"SHACL validation is skipped because the graph is too large to validate fully under current limits."
)
return nodes, edges
async def _generated_shacl_for_uri(
request: Request,
session: GraphSession,
uri: str,
quality_tier: str = "strict",
*,
nodes: Optional[List[Dict[str, Any]]] = None,
edges: Optional[List[Dict[str, Any]]] = None,
) -> tuple[str, List[ShaclShapeSummary]]:
registry = {entry.uri: entry for entry in await _registry_entries(request, session)}
entry = registry.get(uri)
if entry is None:
raise HTTPException(status_code=404, detail="Ontology not found in registry.")
nodes, _ = await asyncio.to_thread(session.get_nodes, skip=0, limit=_MAX_ANALYSIS_NODES)
edges, _ = await asyncio.to_thread(session.get_edges, skip=0, limit=_MAX_ANALYSIS_NODES)
if nodes is None or edges is None:
nodes, edges = await _fetch_analysis_graph(session, uri, "shacl-generate")
entities = _ontology_entities(nodes, uri)
ontology_dict = _ontology_dict_from_nodes(uri, entry.name, entities, edges)
@@ -2178,13 +2403,193 @@ async def _generated_shacl_for_uri(
return shacl_turtle, _summarize_shapes(shacl_turtle)
async def _data_graph_turtle_for_uri(
request: Request,
session: GraphSession,
uri: str,
*,
nodes: Optional[List[Dict[str, Any]]] = None,
edges: Optional[List[Dict[str, Any]]] = None,
) -> str:
try:
import rdflib
except ImportError as exc:
raise ImportError("rdflib is not installed.") from exc
registry = {entry.uri: entry for entry in await _registry_entries(request, session)}
if uri not in registry:
raise HTTPException(status_code=404, detail="Ontology not found in registry.")
if nodes is None or edges is None:
nodes, edges = await _fetch_analysis_graph(session, uri, "shacl-data-graph")
entities = _data_graph_entities(nodes, uri)
g = rdflib.Graph()
base_namespace = _ontology_namespace(uri)
OWL = rdflib.Namespace("http://www.w3.org/2002/07/owl#")
RDF = rdflib.RDF
RDFS = rdflib.RDFS
SKOS = rdflib.Namespace("http://www.w3.org/2004/02/skos/core#")
DCT = rdflib.Namespace("http://purl.org/dc/terms/")
DC = rdflib.Namespace("http://purl.org/dc/elements/1.1/")
SH = rdflib.Namespace("http://www.w3.org/ns/shacl#")
XSD = rdflib.XSD
ONTO = rdflib.Namespace(base_namespace)
g.bind("owl", OWL)
g.bind("rdf", RDF)
g.bind("rdfs", RDFS)
g.bind("skos", SKOS)
g.bind("dct", DCT)
g.bind("dc", DC)
g.bind("sh", SH)
g.bind("xsd", XSD)
g.bind("onto", ONTO)
def _resolve_uri(val: str, base_ns: str) -> rdflib.URIRef:
val_str = str(val).strip()
if val_str == "a":
return rdflib.RDF.type
if val_str.startswith(("http://", "https://", "urn:", "ftp://")):
return rdflib.URIRef(val_str)
prefix_map = {
"owl": "http://www.w3.org/2002/07/owl#",
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"skos": "http://www.w3.org/2004/02/skos/core#",
"dct": "http://purl.org/dc/terms/",
"dc": "http://purl.org/dc/elements/1.1/",
"sh": "http://www.w3.org/ns/shacl#",
"xsd": "http://www.w3.org/2001/XMLSchema#",
"onto": base_ns,
}
if ":" in val_str and not val_str.startswith("/"):
prefix, _, rest = val_str.partition(":")
if prefix in prefix_map:
return rdflib.URIRef(prefix_map[prefix] + rest)
return rdflib.URIRef(val_str)
if val_str.startswith("#"):
val_str = val_str.lstrip("#")
if base_ns.endswith(("/", "#")):
return rdflib.URIRef(base_ns + val_str.lstrip("/"))
return rdflib.URIRef(base_ns + "#" + val_str.lstrip("/"))
_shorthand_types = {
"class": "owl:Class",
"object_property": "owl:ObjectProperty",
"datatype_property": "owl:DatatypeProperty",
"annotation_property": "owl:AnnotationProperty",
"property": "rdf:Property",
"individual": "owl:NamedIndividual",
"concept": "skos:Concept",
"scheme": "skos:ConceptScheme",
"ontology": "owl:Ontology",
}
_uri_predicates = {
"rdf:type",
"a",
"rdfs:subClassOf",
"rdfs:domain",
"rdfs:range",
"owl:equivalentClass",
"owl:equivalentProperty",
"owl:sameAs",
"skos:exactMatch",
"skos:closeMatch",
"skos:broadMatch",
"skos:narrowMatch",
"skos:relatedMatch",
"sh:targetClass",
"sh:targetNode",
}
_literal_predicates = {
"rdfs:label",
"rdfs:comment",
"skos:definition",
"skos:prefLabel",
"skos:altLabel",
"dct:description",
"dct:title",
"dc:title",
"dc:description",
"version",
}
_skip_keys = {
"id",
"type",
"content",
"valid_from",
"valid_until",
"scheme_uri",
"ontology_uri",
"ontology",
}
entity_ids = set()
for node in entities:
nid_str = str(node.get("id", "")).strip()
if not nid_str:
continue
entity_ids.add(nid_str)
subj = _resolve_uri(nid_str, base_namespace)
node_type = node.get("type", "")
if node_type:
for t in _as_uri_list(node_type):
t_mapped = _shorthand_types.get(str(t).lower(), str(t))
g.add((subj, rdflib.RDF.type, _resolve_uri(t_mapped, base_namespace)))
content = str(node.get("content", "")).strip()
props = node.get("properties", {}) or {}
if content and "rdfs:label" not in props and "pref_label" not in props and "label" not in props:
g.add((subj, RDFS.label, rdflib.Literal(content)))
for key, val in props.items():
if key in _skip_keys or val is None or val == "":
continue
pred = _resolve_uri(str(key), base_namespace)
for item in _as_uri_list(val) if isinstance(val, (list, dict)) else [val]:
if item is None or item == "":
continue
if isinstance(item, dict):
item = item.get("uri") or item.get("id") or item.get("@id")
if not item:
continue
if str(key) in _uri_predicates or (
str(key) not in _literal_predicates
and str(item).strip().startswith(("http://", "https://", "urn:"))
):
g.add((subj, pred, _resolve_uri(str(item), base_namespace)))
else:
g.add((subj, pred, rdflib.Literal(str(item))))
for edge in edges:
source = str(edge.get("source", edge.get("source_id", ""))).strip()
target = str(edge.get("target", edge.get("target_id", ""))).strip()
pred_str = str(edge.get("type", "related_to")).strip()
if not source or not target:
continue
if source in entity_ids or target in entity_ids:
g.add((
_resolve_uri(source, base_namespace),
_resolve_uri(pred_str, base_namespace),
_resolve_uri(target, base_namespace),
))
return await asyncio.to_thread(g.serialize, format="turtle")
@router.post("/shacl/generate", response_model=ShaclGenerateResponse)
async def generate_shacl(
request: Request,
body: ShaclGenerateRequest,
session: GraphSession = Depends(get_session),
):
shacl_turtle, shapes = await _generated_shacl_for_uri(request, session, body.uri, body.quality_tier)
try:
shacl_turtle, shapes = await _generated_shacl_for_uri(request, session, body.uri, body.quality_tier)
except GraphTruncationError as exc:
raise HTTPException(status_code=413, detail=str(exc)) from exc
return ShaclGenerateResponse(
uri=body.uri,
shacl_turtle=shacl_turtle,
@@ -2199,7 +2604,10 @@ async def list_shacl_shapes(
uri: str = Query(...),
session: GraphSession = Depends(get_session),
):
shacl_turtle, shapes = await _generated_shacl_for_uri(request, session, uri)
try:
shacl_turtle, shapes = await _generated_shacl_for_uri(request, session, uri)
except GraphTruncationError as exc:
raise HTTPException(status_code=413, detail=str(exc)) from exc
return ShaclShapesResponse(
uri=uri,
shapes=shapes,
@@ -2209,15 +2617,43 @@ async def list_shacl_shapes(
@router.post("/shacl/validate", response_model=ShaclValidationResponse)
async def validate_shacl(body: ShaclValidateRequest):
async def validate_shacl(
request: Request,
body: ShaclValidateRequest,
session: GraphSession = Depends(get_session),
):
if not body.shacl_turtle.strip():
raise HTTPException(status_code=422, detail="SHACL Turtle cannot be empty.")
_shacl_bytes = body.shacl_turtle.encode("utf-8")
if len(_shacl_bytes) > _MAX_SHACL_TURTLE_BYTES:
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="error",
message=(
f"SHACL Turtle size ({len(_shacl_bytes)} bytes) "
f"exceeds maximum allowed size ({_MAX_SHACL_TURTLE_BYTES} bytes)."
),
violations=[],
)
# Syntax-check the submitted Turtle with rdflib before claiming anything about it.
try:
import rdflib # type: ignore
g = rdflib.Graph()
await asyncio.to_thread(g.parse, data=body.shacl_turtle, format="turtle")
if len(g) > _MAX_SHACL_TRIPLES:
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="error",
message=(
f"SHACL graph triple count ({len(g)}) "
f"exceeds maximum allowed limit ({_MAX_SHACL_TRIPLES})."
),
violations=[],
)
except ImportError:
pass # rdflib unavailable; skip syntax check
except Exception as exc:
@@ -2226,17 +2662,113 @@ async def validate_shacl(body: ShaclValidateRequest):
detail=f"Invalid Turtle syntax: {exc}",
) from exc
# Live data-graph validation requires pySHACL wired to OntologyEngine.validate_graph().
if not body.uri:
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="unavailable",
message=(
"Turtle parsed successfully. "
"No target ontology URI was provided — "
"specify 'uri' to execute live SHACL validation against an ontology data graph."
),
violations=[],
)
try:
data_graph_turtle = await _data_graph_turtle_for_uri(request, session, body.uri)
from ...ontology import OntologyEngine
engine = OntologyEngine()
semaphore = _get_shacl_semaphore()
async def _run_validation():
async with semaphore:
return await asyncio.to_thread(
engine.validate_graph,
data_graph_turtle,
shacl=body.shacl_turtle,
data_graph_format="turtle",
shacl_format="turtle",
)
report = await asyncio.wait_for(
_run_validation(),
timeout=_MAX_SHACL_TIMEOUT_SECONDS,
)
except (asyncio.TimeoutError, TimeoutError):
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="error",
message=(
f"SHACL validation timed out after {_MAX_SHACL_TIMEOUT_SECONDS} seconds."
),
violations=[],
)
except GraphTruncationError as exc:
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="unavailable",
message=str(exc),
violations=[],
)
except ImportError as exc:
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="unavailable",
message=(
"Turtle parsed successfully. "
f"Live SHACL validation is unavailable: {exc}"
),
violations=[],
)
except HTTPException:
raise
except Exception as exc:
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="error",
message=f"SHACL validation error: {exc}",
violations=[],
)
violations = [
ShaclViolation(
node=str(v.focus_node) if v.focus_node is not None else None,
path=str(v.result_path) if v.result_path is not None else None,
severity=str(v.severity or "Violation"),
message=str(
v.message
or v.explanation
or f"SHACL constraint violation ({v.constraint}) on {v.focus_node}"
),
focus_node=str(v.focus_node) if v.focus_node is not None else None,
source_shape=str(v.shape) if v.shape is not None else None,
)
for v in [*report.violations, *report.warnings, *report.infos]
]
_result_counts = []
if report.violations:
_result_counts.append(f"{len(report.violations)} violation(s)")
if report.warnings:
_result_counts.append(f"{len(report.warnings)} warning(s)")
if report.infos:
_result_counts.append(f"{len(report.infos)} info result(s)")
summary_msg = (
f"SHACL validation found {', '.join(_result_counts)}."
if _result_counts
else "Graph conforms to SHACL shapes."
)
return ShaclValidationResponse(
uri=body.uri,
conforms=False,
status="unavailable",
message=(
"Turtle parsed successfully. "
"Live graph validation is not yet wired to a data graph — "
"install semantica[shacl] and connect OntologyEngine.validate_graph() to enable full validation."
),
violations=[],
conforms=report.conforms,
status="success",
message=summary_msg,
violations=violations,
report_text=report.raw_report,
)
@@ -2288,8 +2820,13 @@ async def refresh_ontology(
except Exception as exc:
raise HTTPException(status_code=422, detail=f"Refresh parse error: {exc}") from exc
nodes_added = await asyncio.to_thread(session.add_nodes, nodes)
edges_added = await asyncio.to_thread(session.add_edges, edges)
try:
nodes_added, edges_added = await asyncio.to_thread(
session.add_nodes_and_edges, nodes, edges
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
entry.loaded_at = datetime.now(UTC).isoformat()
return RefreshResponse(uri=ontology_uri, nodes_added=nodes_added, edges_added=edges_added)
+171 -4
View File
@@ -4,6 +4,7 @@ Provenance routes for lineage visualization and exportable reports.
import asyncio
import json
import logging
from typing import Any, Dict, List, Optional
import networkx as nx
@@ -13,7 +14,9 @@ from fastapi.responses import PlainTextResponse, Response
from ..dependencies import get_session
from ..schemas import ProvenanceEdge, ProvenanceNode, ProvenanceResponse
from ..session import GraphSession
from ...provenance.integrity import verify_checksum
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/provenance", tags=["Power User Tools"])
_AGENT_TYPES = {"person", "organization", "system", "agent"}
@@ -29,9 +32,164 @@ def _classify_prov(node_type: str) -> tuple[str, str]:
return "Entity", "group_entity"
def _add_chain_nodes(chain: List[Any], nodes: List[Dict[str, Any]], seen_nodes: set) -> None:
for entry in chain:
if not isinstance(entry, dict):
continue
eid = str(entry.get("entity_id") or "")
if not eid or eid in seen_nodes:
continue
seen_nodes.add(eid)
metadata = entry.get("metadata") if isinstance(entry.get("metadata"), dict) else {}
label = str(metadata.get("title") or metadata.get("label") or eid)
prov_type, parent_id = _classify_prov(str(entry.get("entity_type", "entity")))
nodes.append({
"id": eid,
"label": label,
"prov_type": prov_type,
"parent_id": parent_id,
"source_document": entry.get("source_document") or None,
"source_location": entry.get("source_location") or None,
"source_quote": entry.get("source_quote") or None,
"confidence": entry.get("confidence"),
"checksum": entry.get("checksum") or None,
})
def _add_chain_edges(
chain: List[Any], edges: List[Dict[str, Any]], seen_edges: set, direction: str
) -> None:
for entry in chain:
if not isinstance(entry, dict):
continue
eid = str(entry.get("entity_id") or "")
if not eid:
continue
parents = []
if entry.get("parent_entity_id"):
parents.append(str(entry.get("parent_entity_id")))
used = entry.get("used_entities")
if isinstance(used, list):
for u in used:
if u and str(u) not in parents:
parents.append(str(u))
activity = str(entry.get("activity_id") or "wasDerivedFrom")
for src in parents:
edge_key = (src, eid, direction)
if edge_key in seen_edges:
continue
seen_edges.add(edge_key)
edges.append({
# Includes direction to match the seen_edges uniqueness key
# above: the same (src, eid) pair can legitimately appear in
# both directions (e.g. cycles/overlap between the ancestor
# and descendant chains), and without this the two edges
# would collide on the same id.
"id": f"{src}-{eid}-{direction}",
"source": src,
"target": eid,
"label": activity,
"direction": direction,
})
def _transform_audit_lineage(
lineage: Dict[str, Any],
node_id: str,
descendants: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
# Mapping decision (W3C PROV-O to frontend swim-lanes):
# Every ProvenanceEntry maps to a node classified by _classify_prov(entry["entity_type"]),
# placing documents/chunks/entities in 'group_entity' (prov_type='Entity'), persons/systems
# in 'group_agent', and actions/processes in 'group_activity'.
# For derivation relationships (parent_entity_id and used_entities), we connect
# parent -> child directly with edge label set to activity_id (or 'wasDerivedFrom'),
# keeping the lineage graph scannable without cluttering it with intermediate activity
# nodes when activity_id is an operational label.
#
# Upstream edges come from lineage's ancestor chain (parent_entity_id/
# used_entities, via ProvenanceManager.get_lineage()); downstream edges
# come from ProvenanceManager.get_descendants()'s descendant chain (issue
# #825, Part A item 5). Previously 'direction="downstream"' was dead code
# here since no reverse lookup existed.
nodes: List[Dict[str, Any]] = []
edges: List[Dict[str, Any]] = []
seen_nodes: set = set()
seen_edges: set = set()
ancestor_chain = lineage.get("lineage_chain") or lineage.get("entries") or []
descendant_chain = (
(descendants or {}).get("descendant_chain")
or (descendants or {}).get("entries")
or []
)
_add_chain_nodes(ancestor_chain, nodes, seen_nodes)
_add_chain_nodes(descendant_chain, nodes, seen_nodes)
_add_chain_edges(ancestor_chain, edges, seen_edges, "upstream")
_add_chain_edges(descendant_chain, edges, seen_edges, "downstream")
for edge in edges:
for endpoint in (edge["source"], edge["target"]):
if endpoint not in seen_nodes:
seen_nodes.add(endpoint)
prov_type, parent_id = _classify_prov("entity")
nodes.append({
"id": endpoint,
"label": endpoint,
"prov_type": prov_type,
"parent_id": parent_id,
"source_document": None,
"source_location": None,
"source_quote": None,
"confidence": None,
"checksum": None,
})
return {"nodes": nodes, "edges": edges, "source": "audit"}
def _build_provenance(session: GraphSession, node_id: Optional[str] = None) -> dict:
if not node_id or node_id not in session.graph.nodes:
return {"nodes": [], "edges": []}
"""Build provenance lineage for a node, attempting the audit-grade store first.
Combines ProvenanceManager.get_lineage() (upstream ancestors) with
get_descendants() (downstream issue #825, Part A item 5) so both
directions are populated from the audit-grade store, not just upstream.
"""
if not node_id:
return {"nodes": [], "edges": [], "source": "graph_traversal"}
try:
manager = getattr(session, "provenance_manager", None)
if manager is not None:
lineage = manager.get_lineage(node_id)
if lineage and lineage.get("entity_count", 0) > 0:
integrity_ok = lineage.get("integrity_verified")
if integrity_ok is None:
entries = lineage.get("lineage_chain") or lineage.get("entries") or []
integrity_ok = all(verify_checksum(entry) for entry in entries)
if integrity_ok:
descendants = {}
try:
descendants = manager.get_descendants(node_id) or {}
except Exception as exc:
logger.warning(
f"ProvenanceManager get_descendants failed for {node_id}: {exc}"
)
return _transform_audit_lineage(lineage, node_id, descendants)
logger.warning(
f"Provenance integrity verification failed for {node_id}, falling back to graph traversal"
)
except Exception as exc:
logger.warning(
f"ProvenanceManager get_lineage failed for {node_id}, falling back to graph traversal: {exc}"
)
if node_id not in session.graph.nodes:
return {"nodes": [], "edges": [], "source": "graph_traversal"}
graph = nx.DiGraph()
graph.add_node(node_id)
@@ -81,7 +239,7 @@ def _build_provenance(session: GraphSession, node_id: Optional[str] = None) -> d
}
)
return {"nodes": provenance_nodes, "edges": provenance_edges}
return {"nodes": provenance_nodes, "edges": provenance_edges, "source": "graph_traversal"}
def _build_report(session: GraphSession, node_id: str) -> Dict[str, Any]:
@@ -93,6 +251,7 @@ def _build_report(session: GraphSession, node_id: str) -> Dict[str, Any]:
"type": node.get("type", "entity") if node else "entity",
"properties": node.get("metadata", node.get("properties", {})) if node else {},
"lineage": provenance,
"source": provenance.get("source", "graph_traversal"),
}
@@ -114,7 +273,14 @@ def _render_markdown(report: Dict[str, Any]) -> str:
lines.extend(["", "## Lineage Nodes"])
for node in report.get("lineage", {}).get("nodes", []):
lines.append(f"- `{node['id']}` ({node['prov_type']}): {node['label']}")
line = f"- `{node['id']}` ({node['prov_type']}): {node['label']}"
if node.get("source_document"):
line += f" [source: {node['source_document']}]"
if node.get("confidence") is not None:
line += f" (confidence: {node['confidence']})"
if node.get("checksum"):
line += f" (checksum: {node['checksum'][:8]}...)"
lines.append(line)
edges = report.get("lineage", {}).get("edges", [])
grouped_edges: Dict[str, List] = {"upstream": [], "downstream": [], "lateral": []}
@@ -152,6 +318,7 @@ async def get_provenance_lineage(
return ProvenanceResponse(
nodes=[ProvenanceNode(**node) for node in data["nodes"]],
edges=[ProvenanceEdge(**edge) for edge in data["edges"]],
source=data.get("source", "graph_traversal"),
)
+131 -19
View File
@@ -1,10 +1,20 @@
"""
"""
SPARQL routes backed by an in-memory rdflib projection of the current graph.
Security contract
-----------------
* Only SELECT, ASK, CONSTRUCT, and DESCRIBE are accepted (allowlist enforced
before graph construction so rejected queries never touch the session).
* Multi-statement injections that start with an allowed keyword (e.g.
``SELECT ... ; DROP ALL``) pass the prefix check and reach rdflib, which
rejects non-SELECT/ASK/CONSTRUCT/DESCRIBE update syntax in the parser.
* The in-memory rdflib graph is a read-only projection the live
``GraphSession`` is never mutated by this route.
"""
import asyncio
import re
from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, Optional, Tuple
import rdflib
from fastapi import APIRouter, Depends
@@ -16,14 +26,47 @@ from ..session import GraphSession
router = APIRouter(prefix="/api/sparql", tags=["Power User Tools"])
_ALLOWED_QUERY_TYPES = re.compile(
r"^\s*(SELECT|ASK|CONSTRUCT|DESCRIBE)\b",
r"^(SELECT|ASK|CONSTRUCT|DESCRIBE)\b",
re.IGNORECASE,
)
# SPARQL Update keywords that must never appear in read-only queries.
# These are checked AFTER comment/prefix stripping to prevent bypass via
# comments like: # INSERT DATA { ... }\nSELECT ...
_FORBIDDEN_KEYWORDS = re.compile(
r"\b(INSERT|DELETE|DROP|LOAD|CLEAR|CREATE|COPY|MOVE|ADD)\b",
re.IGNORECASE,
)
# Matches SPARQL single-line comments (# ...) and PREFIX declarations
_COMMENT_LINE = re.compile(r"#[^\n]*", re.MULTILINE)
_PREFIX_DECL = re.compile(r"^\s*(?:PREFIX|BASE)\s+(?:\S+\s+)?<[^>]*>\s*", re.IGNORECASE | re.MULTILINE)
def _is_read_only_query(query: str) -> bool:
"""Return True only for SELECT / ASK / CONSTRUCT / DESCRIBE queries."""
return bool(_ALLOWED_QUERY_TYPES.match(query))
"""Return True only for genuine read-only SPARQL queries.
Strips comments, PREFIX/BASE declarations, and leading whitespace before
checking the first keyword. Also rejects queries containing SPARQL Update
keywords anywhere in the body, preventing injection via embedded strings
or multi-statement tricks.
"""
# 1. Remove PREFIX/BASE declarations (do this first so # in URIs aren't mangled by comment stripping)
cleaned = _PREFIX_DECL.sub("", query)
# 2. Remove single-line comments that could hide the real query type
cleaned = _COMMENT_LINE.sub("", cleaned)
# 3. Strip remaining whitespace
cleaned = cleaned.strip()
# 4. Check that the first keyword is a read-only query type
if not _ALLOWED_QUERY_TYPES.match(cleaned):
return False
# 5. Block any forbidden (mutating) keywords anywhere in the query
if _FORBIDDEN_KEYWORDS.search(cleaned):
return False
return True
class SparqlRequest(BaseModel):
@@ -49,8 +92,27 @@ def _build_rdflib_graph(session: GraphSession) -> rdflib.Graph:
graph.bind("ent", NS)
graph.bind("prop", PROP)
nodes, _ = session.get_nodes(skip=0, limit=999_999)
edges, _ = session.get_edges(skip=0, limit=999_999)
# SECURITY: Cap the number of entities materialized into memory to
# prevent denial-of-service via memory exhaustion. Without this guard
# an attacker can send concurrent SPARQL queries that each load ~1M
# nodes/edges into rdflib Graph objects, consuming gigabytes of RAM.
nodes, total_nodes = session.get_nodes(skip=0, limit=_SPARQL_MAX_GRAPH_NODES + 1)
if len(nodes) > _SPARQL_MAX_GRAPH_NODES:
raise ValueError(
f"Graph has more than {_SPARQL_MAX_GRAPH_NODES:,} nodes. "
f"SPARQL queries are limited to graphs with at most "
f"{_SPARQL_MAX_GRAPH_NODES:,} nodes to prevent excessive "
f"memory usage. Use the REST API for large graph operations."
)
edges, _ = session.get_edges(skip=0, limit=_SPARQL_MAX_GRAPH_NODES + 1)
if len(edges) > _SPARQL_MAX_GRAPH_NODES:
raise ValueError(
f"Graph has more than {_SPARQL_MAX_GRAPH_NODES:,} edges. "
f"SPARQL queries are limited to graphs with at most "
f"{_SPARQL_MAX_GRAPH_NODES:,} edges to prevent excessive "
f"memory usage. Use the REST API for large graph operations."
)
for node in nodes:
subject = NS[str(node.get("id", ""))]
@@ -75,9 +137,13 @@ def _build_rdflib_graph(session: GraphSession) -> rdflib.Graph:
return graph
# ---------------------------------------------------------------------------
# Resource limits (override in tests via patch.object)
# ---------------------------------------------------------------------------
_SPARQL_MAX_ROWS = 5_000 # hard cap on returned rows
_SPARQL_TIMEOUT_S = 30 # seconds before abandoning the await
_SPARQL_MAX_CONCURRENT = 4 # semaphore: max simultaneous executions
_SPARQL_MAX_GRAPH_NODES = 50_000 # cap on graph nodes/edges to prevent OOM
# Semaphore caps how many graph.query calls run concurrently so that
# timed-out threads (which keep running in the pool) cannot crowd out
@@ -85,6 +151,22 @@ _SPARQL_MAX_CONCURRENT = 4 # semaphore: max simultaneous executions
_sparql_semaphore = asyncio.Semaphore(_SPARQL_MAX_CONCURRENT)
def _cap_rows(items: Any, row_builder) -> Tuple[List[Dict[str, Any]], bool]:
"""Materialize up to ``_SPARQL_MAX_ROWS`` items via ``row_builder``, reporting truncation.
Shared by the SELECT and CONSTRUCT/DESCRIBE branches below so the row cap
is enforced identically regardless of query type.
"""
rows: List[Dict[str, Any]] = []
truncated = False
for item in items:
if len(rows) >= _SPARQL_MAX_ROWS:
truncated = True
break
rows.append(row_builder(item))
return rows, truncated
@router.post("", response_model=SparqlResponse)
async def execute_sparql(
req: SparqlRequest,
@@ -98,7 +180,15 @@ async def execute_sparql(
error="Only SELECT, ASK, CONSTRUCT, and DESCRIBE queries are permitted.",
)
graph = await asyncio.to_thread(_build_rdflib_graph, session)
try:
graph = await asyncio.to_thread(_build_rdflib_graph, session)
except ValueError as exc:
return SparqlResponse(
columns=[],
rows=[],
total=0,
error=str(exc),
)
async with _sparql_semaphore:
try:
@@ -126,16 +216,38 @@ async def execute_sparql(
error_column=int(column_match.group(1)) if column_match else None,
)
columns = [str(var) for var in query_results.vars] if query_results.vars else []
rows: List[Dict[str, Any]] = []
for row in query_results:
if len(rows) >= _SPARQL_MAX_ROWS:
break
row_data = {}
for index, column in enumerate(columns):
value = row[index]
row_data[column] = str(value) if value is not None else None
rows.append(row_data)
# ---------------------------------------------------------------------------
# Serialize results into a type-aware tabular representation.
# ASK → single row: {"result": "true"|"false"}
# CONSTRUCT/DESCRIBE → rows of {"subject", "predicate", "object"} triples
# SELECT → rows keyed by projected variable names
# ---------------------------------------------------------------------------
query_type: str = query_results.type # always set by rdflib
if query_type == "ASK":
columns = ["result"]
rows = [{"result": "true" if query_results.askAnswer else "false"}]
truncated = False
elif query_type in ("CONSTRUCT", "DESCRIBE"):
columns = ["subject", "predicate", "object"]
rows, truncated = _cap_rows(
query_results, # rdflib always yields (s, p, o) 3-tuples
lambda triple: {
"subject": str(triple[0]),
"predicate": str(triple[1]),
"object": str(triple[2]),
},
)
else: # SELECT
columns = [str(var) for var in (query_results.vars or [])]
rows, truncated = _cap_rows(
query_results,
lambda row: {
column: (str(row[index]) if row[index] is not None else None)
for index, column in enumerate(columns)
},
)
truncated = len(rows) == _SPARQL_MAX_ROWS
return SparqlResponse(columns=columns, rows=rows, total=len(rows), truncated=truncated)
+3 -3
View File
@@ -1,4 +1,4 @@
"""
"""
Temporal routes for snapshots, diffs, and pattern detection.
"""
@@ -8,7 +8,7 @@ import re
from datetime import datetime, timedelta, timezone, UTC
from typing import List, Optional
from fastapi import APIRouter, Depends, Query
from fastapi import APIRouter, Depends, Query, HTTPException
from pydantic import BaseModel
from ..dependencies import get_session
@@ -117,7 +117,7 @@ async def temporal_patterns(
return TemporalPatternResponse(patterns=[])
except Exception as exc:
logger.warning("temporal_patterns failed: %s", exc, exc_info=True)
return TemporalPatternResponse(patterns=[])
raise HTTPException(status_code=500, detail="Temporal pattern detection failed")
@router.get("/bounds", response_model=TemporalBoundsResponse)
+7 -3
View File
@@ -6,7 +6,7 @@ import asyncio
from collections import defaultdict
from typing import Dict, List, Optional
from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, Request, UploadFile
from fastapi import APIRouter, Depends, File, Form, HTTPException, Query, UploadFile
from ..dependencies import get_session
from ..schemas import ConceptNode, ConceptSummary, VocabularyImportResponse, VocabularyScheme
@@ -224,8 +224,12 @@ async def import_vocabulary(
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
nodes_added = await asyncio.to_thread(session.add_nodes, nodes)
edges_added = await asyncio.to_thread(session.add_edges, edges)
try:
nodes_added, edges_added = await asyncio.to_thread(
session.add_nodes_and_edges, nodes, edges
)
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
return VocabularyImportResponse(
status="success",
+6
View File
@@ -311,6 +311,11 @@ class ProvenanceNode(BaseModel):
label: str
prov_type: str
parent_id: Optional[str] = None
source_document: Optional[str] = None
source_location: Optional[str] = None
source_quote: Optional[str] = None
confidence: Optional[float] = None
checksum: Optional[str] = None
class ProvenanceEdge(BaseModel):
@@ -324,6 +329,7 @@ class ProvenanceEdge(BaseModel):
class ProvenanceResponse(BaseModel):
nodes: List[ProvenanceNode]
edges: List[ProvenanceEdge]
source: Optional[str] = None
# ---------------------------------------------------------------------------
+75 -2
View File
@@ -1,4 +1,4 @@
"""
"""
Semantica Explorer session helpers.
"""
@@ -13,6 +13,7 @@ from typing import Any, Dict, Iterable, List, Optional
from ..context.context_graph import ContextGraph, _resolve_edge_identity
from .search_index import GraphSearchIndex
from ..utils.skos import is_skos_hierarchy_edge, validate_skos_hierarchy
_KG_AVAILABLE = False
try:
@@ -37,8 +38,13 @@ logger = logging.getLogger(__name__)
class GraphSession:
"""Thread-safe session wrapper around a loaded ``ContextGraph``."""
def __init__(self, graph: ContextGraph) -> None:
def __init__(
self,
graph: ContextGraph,
provenance_storage_path: Optional[str] = None,
) -> None:
self.graph = graph
self._provenance_storage_path = provenance_storage_path
self._lock = threading.RLock()
self._search_index = GraphSearchIndex()
@@ -52,6 +58,7 @@ class GraphSession:
self._similarity: Any = None
self._link_predictor: Any = None
self._validator: Any = None
self._provenance_manager: Any = None
self._graph_revision: int = 0
self._cached_embeddings: Optional[Dict[str, List[float]]] = None
@@ -178,6 +185,38 @@ class GraphSession:
self._validator = GraphValidator()
return self._validator
@property
def provenance_manager(self) -> Any:
with self._lock:
if self._provenance_manager is None:
from ..provenance import ProvenanceManager
self._provenance_manager = ProvenanceManager(
storage_path=self._provenance_storage_path
)
return self._provenance_manager
def set_provenance_storage_path(self, storage_path: Optional[str]) -> None:
"""Set or reconfigure the provenance storage path for this session.
Raises a ValueError if a conflicting storage path is already configured or
if the provenance manager has already been constructed with a different path.
"""
with self._lock:
if self._provenance_storage_path == storage_path:
return
if self._provenance_manager is not None:
raise ValueError(
f"Cannot change provenance_storage_path to '{storage_path}': "
f"provenance_manager is already initialized with "
f"'{self._provenance_storage_path}'."
)
if self._provenance_storage_path is not None and storage_path is not None:
raise ValueError(
f"Conflicting provenance_storage_path: session is already configured "
f"with '{self._provenance_storage_path}', cannot overwrite with '{storage_path}'."
)
self._provenance_storage_path = storage_path
def normalize_node(self, node: Dict[str, Any]) -> Dict[str, Any]:
meta: Dict[str, Any] = {}
meta.update(node.get("metadata", {}) or {})
@@ -698,6 +737,7 @@ class GraphSession:
def add_edges(self, edges: List[Dict[str, Any]]) -> int:
with self._lock:
self.validate_skos_hierarchy(edges)
added = self.graph.add_edges(edges)
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
if added and not has_mutation_callback:
@@ -706,6 +746,38 @@ class GraphSession:
self.rebuild_search_index()
return added
def validate_skos_hierarchy(self, edges: List[Dict[str, Any]]) -> None:
"""Validate new SKOS hierarchy edges against the current graph."""
hierarchy_edges = [edge for edge in edges if is_skos_hierarchy_edge(edge)]
if not hierarchy_edges:
return
existing_edges = [
edge for edge in self.graph.find_edges() if is_skos_hierarchy_edge(edge)
]
validate_skos_hierarchy(hierarchy_edges, existing_edges)
def add_nodes_and_edges(
self,
nodes: List[Dict[str, Any]],
edges: List[Dict[str, Any]],
) -> tuple[int, int]:
"""
Validate SKOS hierarchy edges upfront and add nodes and edges under lock.
Note: This provides lock-based mutual exclusion and pre-write validation,
not transactional rollback atomicity.
"""
with self._lock:
self.validate_skos_hierarchy(edges)
nodes_added = self.graph.add_nodes(nodes)
edges_added = self.graph.add_edges(edges)
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
if (nodes_added or edges_added) and not has_mutation_callback:
self._bump_graph_revision_locked()
if (nodes_added or edges_added) and not has_mutation_callback:
self.rebuild_search_index()
return nodes_added, edges_added
def add_node(
self,
node_id: str,
@@ -733,6 +805,7 @@ class GraphSession:
**properties: Any,
) -> bool:
with self._lock:
self.validate_skos_hierarchy([{"source": source_id, "target": target_id, "type": edge_type}])
added = self.graph.add_edge(
source_id,
target_id,
+14 -12
View File
@@ -14,21 +14,23 @@ _HAS_DEFUSEDXML = importlib.util.find_spec("defusedxml") is not None
def _safe_parse_rdf(g: rdflib.Graph, data: bytes, rdf_format: str) -> None:
"""Parse RDF bytes into *g*, guarding against XXE for XML-based formats."""
"""Parse RDF bytes into *g*, guarding against XXE for XML-based formats.
Raises:
ImportError: If ``defusedxml`` is not installed and the format is XML-based.
"""
xml_formats = {"xml", "rdf", "rdf/xml", "application/rdf+xml"}
if rdf_format.lower() in xml_formats:
if _HAS_DEFUSEDXML:
# defusedxml patches xml.etree so rdflib's XML parser inherits the fix
import defusedxml
defusedxml.defuse_stdlib()
else:
# Warn once; best-effort protection via rdflib's own parser
import warnings
warnings.warn(
"defusedxml is not installed. Install it (`pip install defusedxml`) "
"to protect RDF/XML parsing against XXE attacks.",
stacklevel=4,
if not _HAS_DEFUSEDXML:
# Fail closed: refuse to parse untrusted XML without XXE protection.
raise ImportError(
"defusedxml is required to safely parse RDF/XML content but is "
"not installed. Install it with: pip install defusedxml "
"(or install semantica with the explorer extra: "
"pip install 'semantica[explorer]')"
)
import defusedxml
defusedxml.defuse_stdlib()
g.parse(data=data, format=rdf_format)
def _get_best_label(graph: rdflib.Graph, subject: rdflib.URIRef, predicate: rdflib.URIRef) -> str:
+23 -7
View File
@@ -13,36 +13,52 @@ Author: Semantica Contributors
License: MIT
"""
from typing import Any
from typing import Any, Optional
from datetime import datetime
import uuid
class ExporterWithProvenance:
"""Base exporter with provenance tracking."""
def __init__(self, provenance: bool = False, **config):
def __init__(
self,
provenance: bool = False,
agent_id: Optional[str] = None,
is_automated: bool = True,
**config,
):
from .exporter import Exporter
self.provenance = provenance
self._exporter = Exporter(**config)
self._prov_manager = None
self._agent_id = agent_id or self.__class__.__name__
self._is_automated = is_automated
if provenance:
try:
from semantica.provenance import ProvenanceManager
self._prov_manager = ProvenanceManager()
except ImportError:
self.provenance = False
def export(self, data: Any, destination: str, **kwargs):
"""Export data with provenance tracking."""
activity_started_at_time = datetime.utcnow().isoformat()
result = self._exporter.export(data, destination, **kwargs)
activity_ended_at_time = datetime.utcnow().isoformat()
if self.provenance and self._prov_manager:
self._prov_manager.track_entity(
entity_id=f"export_{uuid.uuid4().hex[:8]}",
source="export_operation",
entity_type="export",
agent_id=self._agent_id,
agent_type="software_agent",
is_automated=self._is_automated,
activity_started_at_time=activity_started_at_time,
activity_ended_at_time=activity_ended_at_time,
metadata={
"destination": destination,
"format": kwargs.get('format', 'unknown')
+9 -2
View File
@@ -31,6 +31,12 @@ from ..utils.helpers import ensure_directory
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
# Issue #825, Part B Tier 3 — exporter interlinking. Reuses the same default
# namespace as ProvenanceManager.export_prov() and RDFExporter's
# NamespaceManager "semantica" entry, so ontology URIs, KG instance URIs, and
# PROV-exported URIs co-resolve under one shared namespace by default.
from ..provenance.manager import DEFAULT_BASE_URI
class OWLExporter:
"""
@@ -58,7 +64,7 @@ class OWLExporter:
def __init__(
self,
ontology_uri: str = "https://semantica.dev/ontology/",
ontology_uri: str = DEFAULT_BASE_URI,
version: str = "1.0",
format: str = "owl-xml",
config: Optional[Dict[str, Any]] = None,
@@ -70,7 +76,8 @@ class OWLExporter:
Sets up the exporter with ontology URI, version, and format configuration.
Args:
ontology_uri: Base URI for the ontology (default: "https://semantica.dev/ontology/")
ontology_uri: Base URI for the ontology (default: ProvenanceManager.DEFAULT_BASE_URI,
shared with RDFExporter's NamespaceManager and export_prov() so URIs co-resolve)
version: Ontology version string (default: "1.0")
format: Default export format - 'owl-xml' or 'turtle' (default: 'owl-xml')
config: Optional configuration dictionary (merged with kwargs)
+8 -2
View File
@@ -71,13 +71,19 @@ class NamespaceManager:
"""
self.logger = get_logger("namespace_manager")
# Standard RDF namespaces
# Standard RDF namespaces. "semantica" reuses ProvenanceManager's
# DEFAULT_BASE_URI (issue #825, Part B Tier 3 — exporter
# interlinking) so KG-exported and PROV-exported URIs for the same
# entity_id co-resolve to the same namespace instead of two
# independently-hardcoded placeholder domains.
from ..provenance.manager import DEFAULT_BASE_URI
self.namespaces: Dict[str, str] = {
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"rdfs": "http://www.w3.org/2000/01/rdf-schema#",
"owl": "http://www.w3.org/2002/07/owl#",
"xsd": "http://www.w3.org/2001/XMLSchema#",
"semantica": "https://semantica.dev/ns#",
"semantica": DEFAULT_BASE_URI,
}
self.config = config or {}
+20
View File
@@ -338,6 +338,13 @@ class ApacheAgeStore:
"host=localhost dbname=agedb user=postgres password=postgres",
)
self.graph_name = graph_name or config.get("graph_name", "semantica")
# SECURITY: Sanitize graph_name to prevent SQL injection in cypher() calls.
# The graph_name is interpolated into SQL: cypher('{graph_name}', $$ ... $$)
if not re.match(r"^[A-Za-z_][A-Za-z0-9_]*$", self.graph_name):
raise ValidationError(
f"Invalid graph_name '{self.graph_name}': must contain only "
"alphanumeric characters and underscores."
)
self._conn = None
@@ -445,7 +452,20 @@ class ApacheAgeStore:
Returns:
List of raw row tuples from the cursor.
Raises:
ValidationError: If the query contains ``$$`` which could break
out of the AGE dollar-quoted string delimiter.
"""
# SECURITY: Reject queries containing $$ to prevent breakout from
# AGE's dollar-quoted string delimiter. An attacker who injects $$
# into the Cypher query can terminate the cypher() argument and
# append arbitrary SQL.
if "$$" in cypher:
raise ValidationError(
"Query contains forbidden '$$' sequence. "
"Dollar-quoted delimiters are not allowed in Cypher queries."
)
self._ensure_connection()
sql = (
f"SELECT * FROM cypher('{self.graph_name}', $$ {cypher} $$) "
+2 -2
View File
@@ -294,7 +294,7 @@ class QueryEngine:
import hashlib
key_str = f"{query}:{str(parameters)}"
return hashlib.md5(key_str.encode()).hexdigest()
return hashlib.md5(key_str.encode()).hexdigest() # nosec B324 - cache key, not security-sensitive
def clear_cache(self) -> None:
"""Clear query cache."""
@@ -1056,7 +1056,7 @@ class GraphStore:
if not entity_id and entity_text:
import hashlib
entity_hash = hashlib.md5(
entity_hash = hashlib.md5( # nosec B324 - deterministic entity ID, not security-sensitive
f"{entity_text}_{entity_type}".encode()
).hexdigest()[:12]
entity_id = f"{entity_type.lower()}_{entity_hash}"

Some files were not shown because too many files have changed in this diff Show More