Compare commits

..
50 Commits
Author SHA1 Message Date
KaifAhmad1 2ef6e9f4b1 Release 0.5.0: Distance Intelligence & Ontology Hub Complete 2026-05-11 20:35:16 +05:30
Mohd Kaif 18da322e0d Feature/distance intelligence optimization (#550)
* Implement embedding cache optimization for Distance Intelligence

- Add per-session graph revision-based embedding cache to avoid re-scanning nodes
- Update GraphSession with get_cached_embeddings() and automatic cache invalidation
- Modify distance matrix and semantic neighborhood endpoints to use cached embeddings
- Implement thread-safe caching with proper revision tracking
- Add force refresh capability and automatic invalidation on graph modifications
- Improve performance for repeated distance intelligence queries

Resolves TODO in graph.py: cache embeddings per-session graph revision

* Update changelog with Distance Intelligence embedding cache optimization
2026-05-11 17:38:24 +05:30
Luffy2208andKaifAhmad1 15d58f2b88 Added Parquet ingest support (#234) (#548)
* Added Parquet ingest support (#234)

* docs: Add Parquet ingestion support to CHANGELOG

- Add comprehensive changelog entry for PR #548
- Document ParquetIngestor class and key features
- Include author credit (@Luffy2208) and PR reference
- Follow existing changelog format and structure

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-05-10 12:52:26 +05:30
Mohd Kaif ce3a8c9895 Add Enterprise Support section to README
Added enterprise support section with details on solutions and services.
2026-05-09 17:39:08 +05:30
Mohd Kaif 508e05d367 Update README.md 2026-05-08 17:30:25 +05:30
Mohd Kaif 03f99f016d Update README.md 2026-05-08 17:29:48 +05:30
Mohd Kaif 56e7d9d821 Fix #541: Convert mcp_server to package structure for pipx installation (#544)
- Convert mcp_server.py to package structure (semantica/mcp_server/)
- Add __init__.py and __main__.py for python -m support
- Add semantica-mcp console script entry point in pyproject.toml
- Fix API method calls (extract -> extract_entities/relations/triplets)
- Remove non-existent _result_cache imports
- Update documentation with both usage methods

Resolves pipx installation issue where semantica.mcp_server was not available.
Provides two ways to run: 'semantica-mcp' command or 'python -m semantica.mcp_server'.
2026-05-08 17:17:39 +05:30
Mohd Kaif 6860bdbec3 Merge pull request #540 from Hawksight-AI/conflicts
feat(deduplication): DuplicateDetector result limiting and ranking
2026-05-05 21:18:00 +05:30
Zohaib Hassnain ac5015bc3f fix(deduplication): normalize merged group keys 2026-05-05 20:25:51 +05:00
KaifAhmad1 29c72f59b3 docs(changelog): record Qodo review follow-up fixes for #533 and #534 2026-05-05 19:28:06 +05:30
KaifAhmad1 21c2f190f8 fix: resolve Qodo review bugs and quality issues (DuplicateDetector + ConflictDetector)
- bug_001: top_k_per_entity now uses OR semantics — keep a candidate if
  EITHER entity is under quota, preventing high-quality candidates being
  silently dropped when a popular counterpart saturates its quota
- bug_002: validate max_results and top_k_per_entity at construction;
  negative or non-int values raise ValueError instead of silent empty output
- bug_003: validate min_similarity in [0.0, 1.0] at construction;
  out-of-range values raise ValueError
- bug_004: harden ConflictDetector method='relationship' normalization —
  always produces List[Dict] before calling detect_relationship_conflicts
- quality_001: update detect_duplicates + incremental_detect docstrings to
  reflect configurable sort_by field (not hardcoded 'confidence')
- quality_002: add _normalize_entity_id helper (always str) used in both
  _apply_result_limits and _build_duplicate_groups for consistent ID handling

Backward compatible: callers not using new params see no behavior change.
58 tests pass (0 failures)
2026-05-05 19:25:48 +05:30
KaifAhmad1 8ef67b8bda feat(deduplication): add max_results, top_k_per_entity, min_similarity, sort_by to DuplicateDetector
Fixes #534

- New __init__ params: max_results, top_k_per_entity, min_similarity, sort_by
- _apply_result_limits: drop below min_similarity, sort by sort_by field,
  enforce top_k_per_entity per entity, cap at max_results globally
- Wired into detect_duplicates() and incremental_detect()
- 30 new tests in TestResultLimiting; full suite 42/42 passed
2026-05-05 19:16:58 +05:30
Mohd Kaif bc57837b86 Merge pull request #539 from Hawksight-AI/conflicts
fix(conflicts): consolidate duplicate detect_conflicts into single di…
2026-05-05 18:17:19 +05:30
KaifAhmad1 0439cf884d docs(changelog): record ConflictDetector.detect_conflicts duplicate definition fix (#533) 2026-05-05 18:13:06 +05:30
KaifAhmad1 141bf80394 fix(conflicts): consolidate duplicate detect_conflicts into single dispatcher method
Fixes #533

- Removes duplicate `detect_conflicts` definition that was silently overridden,
  causing AttributeError for callers passing `method=` or `property_name=` kwargs
- Merges dispatcher logic into the surviving method with `method="all"` default
  supporting: "all", "value", "property", "type", "relationship", "temporal",
  "logical", "entity"
- Fixes `method="relationship"` incorrectly defaulting `relationships` to the
  entities list; now defaults to `[]` with dict normalization
- Removes unreachable dead code block after try/except raise in
  `detect_entity_conflicts`
2026-05-05 18:00:48 +05:30
Mohd Kaif 0c4e18e256 fix(deps): remove gpu from [all] extra to fix Windows installation failure (#538)
* fix(deps): remove gpu extra from [all] to fix Windows installation failure

faiss-gpu has no Windows builds, so semantica[all] failed with
'No matching distribution found for faiss-gpu>=1.7.0' on Windows.
Removed gpu from both [all] lines — semantica[gpu] remains available
as an explicit opt-in for Linux GPU environments.

Closes #532

* docs(changelog): record faiss-gpu Windows installation failure fix (#532)
2026-05-05 16:57:33 +05:30
Mohd Kaif 39045d783b Merge pull request #537 from Hawksight-AI/utlis
fix(utils): route all progress tracker stdout writes through _safe_wr…
2026-05-05 16:34:43 +05:30
KaifAhmad1 afa54e1bbf docs(changelog): record progress tracker cp1252 UnicodeEncodeError fix (#531) 2026-05-05 16:19:14 +05:30
KaifAhmad1 a01b3c36fc fix(utils): route all progress tracker stdout writes through _safe_write to prevent UnicodeEncodeError on cp1252 consoles
Closes #531

- Replace 5 direct sys.stdout.write() calls in ConsoleProgressDisplay.update()
  with self._safe_write() so emoji/block characters are encoded safely on
  Windows cp1252 consoles
- Add TestProgressTrackerEncoding regression tests (3 cases) covering
  _safe_write, pipeline header, and auto emoji-disable on cp1252
2026-05-05 16:03:19 +05:30
Mohd Kaif 0dda380580 Merge pull request #536 from Hawksight-AI/fix/semantic-extract-import-cycle
fix(semantic_extract): break extractor import cycle
2026-05-05 14:34:57 +05:30
KaifAhmad1andZohaib Hassan e7c9f6e7f3 fix(tests): restore sys.modules after mock injection in test_retry_logic
test_retry_logic.py injected sys.modules["openai"] = MagicMock() at module
level so providers.py could be imported without the real openai package.
Those mocks were never restored, leaving openai (and spacy, instructor etc.)
as MagicMock objects for the entire test session. This caused
test_pr482_deepseek_openai tests to receive a MagicMock when importing
openai.OpenAI, making MagicMock(spec=OpenAI) raise InvalidSpecError.

Fix: save original sys.modules entries before injection and restore them
immediately after the semantica imports that needed the mocks complete.
The mock objects remain bound inside the already-imported provider module,
so test_retry_logic tests are unaffected; other test modules now see the
real packages again.

Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-05-05 14:21:36 +05:30
KaifAhmad1andZohaib Hassan b828ebfef0 chore: resolve CHANGELOG.md merge conflict with main
main restructured [Unreleased] into ### Added / ### Fixed sections.
Moved PR #536 semantic_extract circular import fix entry into ### Fixed
below the PR #535 ingest lazy-load entry; kept ### Added content from
main intact.

Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-05-05 13:56:48 +05:30
KaifAhmad1andZohaib Hassan 6330627ddb docs(changelog): record semantic_extract circular import fix and Qodo review fix (#536)
Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-05-05 13:54:01 +05:30
KaifAhmad1andZohaib Hassan 24e327d161 fix(tests): add from __future__ import annotations for Py3.8 compatibility
subprocess.CompletedProcess[str] as a return annotation is not subscriptable
at runtime on Python 3.8, causing test collection to abort before any tests
run. Adding PEP 563 deferred evaluation makes all annotations strings at
import time, restoring 3.8 compatibility without changing behaviour on 3.9+.

Co-authored-by: Zohaib Hassan <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
2026-05-05 13:40:11 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> b9f3c59443 Potential fix for pull request finding 'Duplicate key in dict literal'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-05-05 13:28:09 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> 34955ba246 Potential fix for pull request finding 'Explicit export is not defined'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-05-05 13:27:54 +05:30
Mohd KaifandCopilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> 95b46f12b6 Potential fix for pull request finding 'Explicit export is not defined'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
2026-05-05 13:25:07 +05:30
245ff76b99 fix(ingest): lazy-load optional ingestion backends (#535)
* fix(ingest): lazy-load optional ingestion backends

* fix(ingest): address qodo review — use ModuleNotFoundError and guard ConfigurationError

Bug 1: Replace overbroad `except ImportError` with `except ModuleNotFoundError` in
__getattr__ (__init__.py) and all four optional-backend loaders (methods.py). This
prevents internal import errors inside a backend module from being silently rewritten
into a misleading "package not installed" message. Also simplifies _is_missing_dependency
to rely solely on exc.name now that ModuleNotFoundError always sets it.

Bug 2: Add `except ConfigurationError: raise` before the blanket `except Exception`
handlers in ingest_web, ingest_feed, ingest_repository, and ingest_email. Missing
optional dependencies are expected user-config issues and must not be logged as errors.

Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>

* docs(changelog): record lazy ingest backends fix and qodo review fixes (#535)

Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>

* fix(tests): set exc.name on blocker's ModuleNotFoundError to match Python import machinery

OptionalDependencyBlocker was constructing ModuleNotFoundError with only a
message string, leaving .name as None. _is_missing_dependency checks exc.name
directly (the string-scan fallback was removed when switching to
ModuleNotFoundError), so the ConfigurationError conversion never triggered and
the test asserted the wrong exception type.

Python's import machinery always sets .name to the top-level module name when
it raises ModuleNotFoundError; the blocker now does the same.

Co-authored-by: Zohaib Hassan (@ZohaibHassan16) <zohaib179949@gmail.com>
Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>

---------

Co-authored-by: KaifAhmad1 <kaifahmad087@gmail.com>
Co-authored-by: Zohaib Hassan (@ZohaibHassan16) <zohaib179949@gmail.com>
2026-05-05 13:09:14 +05:30
a2b6a481dc chore: resolve CHANGELOG.md merge conflict with main
main restructured [Unreleased] into ### Added / ### Fixed sections.
Moved PR #535 lazy-load fix and Ontology Hub post-review fix entries
into ### Fixed; kept ### Added content from main intact.

Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
2026-05-05 12:56:06 +05:30
c7edba88ea fix(tests): set exc.name on blocker's ModuleNotFoundError to match Python import machinery
OptionalDependencyBlocker was constructing ModuleNotFoundError with only a
message string, leaving .name as None. _is_missing_dependency checks exc.name
directly (the string-scan fallback was removed when switching to
ModuleNotFoundError), so the ConfigurationError conversion never triggered and
the test asserted the wrong exception type.

Python's import machinery always sets .name to the top-level module name when
it raises ModuleNotFoundError; the blocker now does the same.

Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
2026-05-05 12:45:43 +05:30
eb8598e0cf docs(changelog): record lazy ingest backends fix and qodo review fixes (#535)
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
2026-05-05 12:37:17 +05:30
64d4157644 fix(ingest): address qodo review — use ModuleNotFoundError and guard ConfigurationError
Bug 1: Replace overbroad `except ImportError` with `except ModuleNotFoundError` in
__getattr__ (__init__.py) and all four optional-backend loaders (methods.py). This
prevents internal import errors inside a backend module from being silently rewritten
into a misleading "package not installed" message. Also simplifies _is_missing_dependency
to rely solely on exc.name now that ModuleNotFoundError always sets it.

Bug 2: Add `except ConfigurationError: raise` before the blanket `except Exception`
handlers in ingest_web, ingest_feed, ingest_repository, and ingest_email. Missing
optional dependencies are expected user-config issues and must not be logged as errors.

Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
2026-05-05 12:30:48 +05:30
Zohaib Hassnain e1b1e63541 fix(semantic_extract): break extractor import cycle 2026-05-05 02:26:57 +05:00
Zohaib Hassnain 6b0a8e60ce fix(ingest): lazy-load optional ingestion backends 2026-05-05 02:01:27 +05:00
Mohd Kaif 8b22a58b8f Update print statement from 'Hello' to 'Goodbye' 2026-05-04 23:36:03 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> ba286a55ea security(deps): update mkdocs requirement from >=1.5.0 to >=1.6.1 (#507)
Updates the requirements on [mkdocs](https://github.com/mkdocs/mkdocs) to permit the latest version.
- [Release notes](https://github.com/mkdocs/mkdocs/releases)
- [Commits](https://github.com/mkdocs/mkdocs/compare/1.5.0...1.6.1)

---
updated-dependencies:
- dependency-name: mkdocs
  dependency-version: 1.6.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-04 22:34:36 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> f610eff0ed security(deps): update mkdocs-mermaid2-plugin requirement (#508)
Updates the requirements on [mkdocs-mermaid2-plugin](https://github.com/fralau/mkdocs-mermaid2-plugin) to permit the latest version.
- [Release notes](https://github.com/fralau/mkdocs-mermaid2-plugin/releases)
- [Changelog](https://github.com/fralau/mkdocs-mermaid2-plugin/blob/master/CHANGELOG.md)
- [Commits](https://github.com/fralau/mkdocs-mermaid2-plugin/compare/v1.0.1...v1.2.3)

---
updated-dependencies:
- dependency-name: mkdocs-mermaid2-plugin
  dependency-version: 1.2.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-04 21:30:16 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> e9e7278720 security(deps): update mkdocs-jupyter requirement (#509)
Updates the requirements on [mkdocs-jupyter](https://github.com/danielfrg/mkdocs-jupyter) to permit the latest version.
- [Changelog](https://github.com/danielfrg/mkdocs-jupyter/blob/main/CHANGELOG.md)
- [Commits](https://github.com/danielfrg/mkdocs-jupyter/commits)

---
updated-dependencies:
- dependency-name: mkdocs-jupyter
  dependency-version: 0.26.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-03 12:32:57 +05:30
Mohd Kaif 6b2cafd02e Merge pull request #524 from Hawksight-AI/feat/onto-hub-subissue-520
feat(ontology): add alignments, health dashboard, and SHACL studio
2026-05-02 17:21:54 +05:30
90d857a98f docs(changelog): add PR #524 entry — Alignments, Health Dashboard & SHACL Studio
Covers all features, backend endpoints, schemas, helpers, fix-up commits,
and 14 integration tests added during the subissue-3 implementation cycle.

Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@example.com>
2026-05-02 16:53:09 +05:30
1861ca578c chore: resolve merge conflicts with origin/main
- Keep SequenceMatcher + Tuple imports; delegate Literal to typing_extensions
- Preserve both _ALIGNMENT_RELATIONS (subissue-520) and _INGEST_FORMAT_SUFFIXES (main)
- Keep our OntologyAlignment-typed _get_alignment_store; add main's _get_drafts,
  _get_proposals, _get_versions, _alignment_key, _coerce_alignment, _version_field
- Import OntologyEditor + VersionsTab (main) alongside ShaclStudio (subissue-520)
- All 14 subissue-3 tests pass post-resolution

Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@example.com>
2026-05-02 16:45:10 +05:30
63acc7a66e fix(ontology): address Qodo automated review findings from PR #524
Backend (semantica/explorer/routes/ontology.py):
- suggest-alignments: add TF-IDF character-ngram embeddings via sklearn
  (SimilarityCalculator-compatible cosine scoring) so embedding_similarity
  is populated in results; combined score = 0.4*label + 0.6*embedding when
  available, falling back to label-only when sklearn is absent
- suggest-alignments: add token-overlap prefilter before SequenceMatcher so
  zero-Jaccard pairs are skipped without computing full similarity; add
  _MAX_ENTITIES_PER_SIDE=500 per-ontology cap on top of the existing
  _MAX_ANALYSIS_NODES global cap
- suggest-alignments: remove dead try/except OntologyEngine.create_alignment
  block that always failed silently (no TripletStore configured); replace
  with a comment explaining the intentional ephemeral-only storage model
- health: replace O(alignments x entities) any() scans for alignment coverage
  with O(1) set membership checks via assessed_ids
- shacl/validate: run rdflib.Graph().parse(format='turtle') syntax check on
  the submitted Turtle before returning; invalid syntax now raises 422 instead
  of returning a misleading unavailable/success response

Frontend:
- AlignmentsTab: add pairwise alignment matrix section that groups recorded
  alignments by (source_ontology, target_ontology) pair; each cell shows
  color-coded relation badges per RELATION_COLORS; clicking a badge populates
  the create/edit form for quick editing; matrix is shown when at least two
  ontologies are loaded
- ShaclStudio: add selectedShapeId state and fullShacl ref; each shape row in
  the library is now a clickable button that extracts its Turtle block from
  the full SHACL and pre-populates the Monaco editor; a "View all" toggle
  restores the full SHACL; selected shape ID is shown in the editor header
- GraphWorkspace: fix viewMode race in external focus effect — call
  setSelectedNodeId directly instead of going through focusNode(), which
  captured a stale viewMode in its closure; remove focusNode from the
  dependency array since it is no longer called

Tests (14 passing, was 11):
- Add test_suggest_alignments_returns_embedding_similarity: asserts
  embedding_similarity is non-null when sklearn is available
- Add test_shacl_validate_rejects_invalid_turtle_syntax: asserts 422 on
  syntactically invalid Turtle
- Add test_health_alignment_coverage_uses_set_lookup: asserts alignment
  dimension score is non-zero after recording an alignment, verifying the
  O(1) set lookup path works correctly end-to-end

Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
2026-05-02 16:38:09 +05:30
00ceb09960 fix(ontology): address review blockers from PR #524
Backend:
- Replace false conforms=True SHACL stub with status=unavailable always;
  live validation cannot be wired until OntologyEngine.validate_graph is
  connected to a data graph — a stub that returns conforms=True misleads
  users editing shapes
- Cap node/edge fetches in health, suggest-alignments, and SHACL generation
  at _MAX_ANALYSIS_NODES (5 000) with a logger.warning when the graph
  exceeds the limit; unbounded limit=999_999 fetches cause OOM on large graphs
- Set SHACL health dimension score to 0.0 (was 70.0) when status=unavailable;
  exclude unavailable dimensions from the total_score average so they neither
  inflate nor deflate the result
- Allow alignments to reference external/unloaded URIs (e.g. schema.org)
  without raising 404; label falls back to URI fragment or caller-supplied
  source_label/target_label fields added to OntologyAlignmentRequest
- Fix _alignment_id to use uuid.NAMESPACE_OID instead of NAMESPACE_URL;
  the composite key is not a URL
- Fix _summarize_shapes to normalise \r\n before splitting on .\n so shape
  parsing works correctly on Windows line endings

Frontend:
- Wrap handleSave/handleSuggest/handleRemove/handleAcceptSuggestion in
  useCallback in AlignmentsTab for consistency with sibling components
- Add ephemeral-storage banner in AlignmentsTab warning that alignments are
  session-memory-only and not persisted across restarts
- Fix exportReport in HealthTab to append/remove anchor from document before
  clicking and defer URL.revokeObjectURL to avoid Blob URL leak in some browsers
- Derive health dimension grid column count from health.dimensions.length
  instead of the hardcoded repeat(5, ...) that breaks if the backend adds
  or removes a dimension
- Add minimal Monarch tokenizer for the Monaco turtle language registration
  in ShaclStudio so prefix declarations, IRIs, SHACL properties, comments,
  and string literals are syntax-highlighted; previously the editor rendered
  as plain text despite theme rules being defined

Tests (11 passing, was 5):
- Rename test_shacl_validate_has_stable_contract to
  test_shacl_validate_returns_unavailable and assert status == unavailable
- Add test_shacl_validate_rejects_empty_turtle (expects 422)
- Add test_health_returns_404_for_unknown_ontology
- Add test_health_shacl_dimension_is_zero_when_unavailable with total_score check
- Add test_delete_unknown_alignment_returns_404
- Add test_alignment_upsert_is_idempotent (verifies ID stability and created_at
  preservation across updates)
- Add test_alignment_accepts_external_uri (verifies no 404 for schema.org URIs)
- Relax test_alignment_suggestions_are_ranked label assertions to substring
  checks so the test survives similarity algorithm changes

Co-authored-by: KaifAhmad1 <mohammadk78600@gmail.com>
Co-authored-by: ZohaibHassan16 <zohaib@hawksight.ai>
2026-05-02 15:53:47 +05:30
Mohd Kaif 2909e0fa14 Merge pull request #523 from Hawksight-AI/feature/ontology-hub-endpoints
feat: implement Ontology Hub endpoints with Semantica module integration
2026-05-02 11:20:41 +05:30
KaifAhmad1 d2574990fa Merge branch 'feature/ontology-hub-endpoints' of https://github.com/Hawksight-AI/semantica into feature/ontology-hub-endpoints 2026-05-02 11:12:22 +05:30
KaifAhmad1 53d1da87c0 fix: prevent invalid domain/range edges in ontology creation
- Fix domain_uri/range_uri always being truthy strings
- Only create rdfs:domain/rdfs:range edges when domain/range are non-empty strings
- Add proper validation with .strip() to handle whitespace-only values
- Apply fix to both 'data' and 'text' mode ontology creation
- Prevents pollution of graph with invalid edges to namespace root

Fixes issue where empty domain/range values like '' or None would still create
edges pointing to namespace root (e.g., 'https://ex/#/') instead of being
properly omitted.
2026-05-02 11:12:03 +05:30
Zohaib Hassnain 9e916a82b5 fix(ontology): address hub endpoint review blockers 2026-05-02 00:02:06 +05:00
Zohaib Hassnain e8bf0e50d3 feat(ontology): add alignments health and shacl studio 2026-05-01 22:59:15 +05:00
KaifAhmad1 269fdaa9fb feat: implement Ontology Hub endpoints with Semantica module integration
- Add comprehensive ontology API endpoints (27 total)
- Integrate OntologyEngine for validation, SHACL, SKOS, alignments
- Integrate VersionManager for versioning and diffing
- Integrate ChangeLogEntry for audit trails
- Integrate OntologyIngestor for RDF parsing
- Add frontend components: OntologyEditor, ProposalReview, VersionsTab
- Update CHANGELOG with detailed feature documentation
- Add proper error handling and fallback mechanisms
- Fix import issues and dependencies
- All endpoints tested and verified working

Features implemented:
- Draft management with audit trails
- Change proposals with structured diffing
- Version comparison and publishing
- Ontology loading with multiple format support
- SKOS vocabulary management
- Cross-ontology alignments
- Visual ontology editor
- Registry and search functionality
2026-05-01 22:26:21 +05:30
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 2d9bbf08b1 deps(deps): update pytest-benchmark requirement from >=4.0.0 to >=5.2.3 (#522)
Updates the requirements on [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark) to permit the latest version.
- [Changelog](https://github.com/ionelmc/pytest-benchmark/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/ionelmc/pytest-benchmark/compare/v4.0.0...v5.2.3)

---
updated-dependencies:
- dependency-name: pytest-benchmark
  dependency-version: 5.2.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-01 18:05:17 +05:30
47 changed files with 8210 additions and 2721 deletions
+449 -2235
View File
File diff suppressed because it is too large Load Diff
+23
View File
@@ -6,6 +6,8 @@
**A Framework for Building Context Graphs and Decision Intelligence Layers for AI**
🌐 **[Website](https://getsemantica.ai/)** | 📚 **[Documentation](https://docs.getsemantica.ai/)**
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![PyPI](https://img.shields.io/pypi/v/semantica.svg)](https://pypi.org/project/semantica/)
@@ -1029,6 +1031,27 @@ pytest tests/
---
## 🏢 Enterprise Support
**Production-grade Semantica for your organization**
🚀 **[Website](https://getsemantica.ai/)** — Enterprise solutions and custom domain-specific implementations
**Domain-Specific Solutions:**
- 🏥 **Healthcare** — Clinical decision support, patient safety, compliance
- **Finance** — Fraud detection, regulatory compliance, risk management
- ⚖️ **Legal** — Evidence analysis, contract review, case law reasoning
- **Cybersecurity** — Threat attribution, incident response, provenance
- 🏛️ **Government** — Policy tracking, classified information governance
**Custom Solutions Available:**
- Private cloud deployment
- 🛡️ Enterprise security compliance
- Custom analytics and reporting
- 🎯 Professional implementation services
---
## 🤝 Contributing
All contributions welcome — bug fixes, features, tests, and docs.
+186
View File
@@ -0,0 +1,186 @@
# Semantica 0.5.0 Release Notes
## 🎉 Major Release: Distance Intelligence & Ontology Hub Complete
**Release Date:** May 11, 2026
**Version:** 0.5.0
---
## 🚀 **MAJOR HIGHLIGHTS**
### **Distance Intelligence Framework** (PR #502, @KaifAhmad1)
- **Embedding Cache Optimization**: Per-session graph revision-based caching for 10x+ performance improvement
- **Advanced UI Features**: Ego mode, overlays, heatmap, and path inspector
- **Semantic Neighborhood Search**: Context-aware similarity with proximity metrics
- **Distance Matrix API**: N×N semantic distance calculations with caching
### **Complete Ontology Hub Suite** (PR #517, @KaifAhmad1 @ZohaibHassan16)
- **Alignments Tab** (PR #524): Cross-ontology alignment authoring with ML suggestions
- **Health Dashboard** (PR #524): Quality scoring across 5 dimensions with issue tracking
- **SHACL Studio** (PR #524): Interactive shape generation and validation
- **Visual Editor** (PR #519): Canvas-based ontology authoring without hand-coding
- **Registry & Search** (PR #518): Comprehensive ontology management and discovery
### **Security Hardening** (Security Enhancement PR, @KaifAhmad1)
- **12 Critical Vulnerabilities Fixed**: Eval injection, XXE, SQL injection, and more
- **SSRF Protection**: Comprehensive URL validation and hostname resolution
- **Input Validation**: Enhanced file upload restrictions and format detection
- **CORS & Headers**: Proper security headers and WebSocket protection
---
## 📊 **BY THE NUMBERS**
- **12 Major Features** ✅ Tested & Verified
- **16 Ontology Hub API Endpoints** ✅ Production Ready
- **57 New Distance Intelligence Tests** ✅ All Passing
- **32 Parquet Ingestion Tests** ✅ All Passing
- **12 Security Vulnerabilities** ✅ All Patched
- **100% Test Coverage** ✅ Core Features Verified
---
## 🔧 **NEW FEATURES**
### **Performance & Architecture**
- **Distance Intelligence Embedding Cache** (PR #502, @KaifAhmad1): Thread-safe per-session caching with automatic invalidation
- **Parquet File Ingestion** (PR #548, @Luffy2208): PyArrow backend with column selection and partition support
- **Indexed Search** (PR #481, @ZohaibHassan16): O(log n) search for large graphs (118k nodes: 24ms → 0.004ms)
### **Ontology Hub Suite**
- **Cross-ontology Alignments** (PR #524, @KaifAhmad1 @ZohaibHassan16): ML-powered suggestions with confidence scoring
- **Quality Health Dashboard** (PR #524, @KaifAhmad1 @ZohaibHassan16): 5-dimension scoring with actionable issue tracking
- **SHACL Studio** (PR #524, @KaifAhmad1 @ZohaibHassan16): Interactive shape authoring with Monaco editor
- **Visual Ontology Editor** (PR #519, @KaifAhmad1): Drag-and-drop ontology construction
- **16 Backend Endpoints** (PRs #518, #519, #524, @KaifAhmad1 @ZohaibHassan16): Complete CRUD and analysis capabilities
### **UI & User Experience**
- **Distance Intelligence UI** (PR #502, @KaifAhmad1 @ZohaibHassan16): Ego mode, overlays, heatmap, path inspector
- **Explorer Redesign** (PR #516, @ZohaibHassan16): Modern hero section with live metrics
- **Graph Workspace Declutter** (PR #483, @ZohaibHassan16): Improved visualization for dense graphs
- **Bidirectional Path Finding** (PR #469, @KaifAhmad1): Undirected traversal support
### **Platform Compatibility**
- **Windows Installation Fixes** (PR #532, @KaifAhmad1): Removed faiss-gpu from [all], Unicode console support
- **Cross-platform Dependencies** (PR #527, @ZohaibHassan16): Proper optional dependency management
- **MCP Server Package Structure** (PR #541, @KaifAhmad1): Fixed pipx installation issues
### **Algorithm Enhancements**
- **DuplicateDetector Result Limiting** (PR #534, @KaifAhmad1): Ranking, sorting, and incremental detection features
- **ConflictDetector Parameter Handling** (PR #533, @KaifAhmad1): Method parameter validation and error handling
---
## 🛡️ **SECURITY IMPROVEMENTS** (Security Enhancement PR, @KaifAhmad1)
### **Critical Fixes**
- **Eval Injection** (CWE-95): Replaced with `fractions.Fraction` in media parser
- **Pickle Deserialization** (CWE-502): Switched to JSON with migration support
- **SQL Injection** (CWE-89): Parameterized queries and input validation
- **XXE Protection** (CWE-611): `defusedxml` hardening for all RDF parsing
### **Web Security**
- **SSRF Protection**: URL validation with hostname resolution
- **CORS Hardening**: Narrowed origins and WebSocket limits
- **Security Headers**: HSTS, X-Content-Type-Options, X-Frame-Options
- **Path Traversal**: `Path.resolve().relative_to()` protection
### **Input Validation**
- **File Upload Restrictions**: Extension allowlist and size limits
- **SPARQL Limits**: Row caps, timeouts, and concurrency controls
- **ReDoS Prevention**: Eliminated polynomial regex patterns
---
## 🔍 **QUALITY ASSURANCE**
### **Testing Coverage**
- **Distance Intelligence**: 57 new tests, 100% passing
- **Parquet Ingestion**: 32 tests, comprehensive coverage
- **Security Fixes**: 14 vulnerability-specific tests
- **UI Components**: All major features verified
- **Platform Tests**: Windows, Linux compatibility confirmed
### **Performance Benchmarks**
- **Embedding Cache**: 10x+ improvement in repeated requests
- **Search Performance**: 6,000x faster for large graphs
- **Memory Efficiency**: Lazy loading and optional dependencies
- **Concurrent Operations**: Thread-safe caching with locks
---
## 🔄 **BREAKING CHANGES**
### **Dependencies**
- **Windows Users**: `faiss-gpu` removed from `[all]` - install `[gpu]` explicitly if needed
- **Optional Dependencies**: Now lazy-loaded to improve import performance
### **API Changes**
- **ConflictDetector**: Fixed duplicate method definitions with proper parameter handling
- **DuplicateDetector**: New result limiting and ranking options
---
## 📚 **DOCUMENTATION**
- **Comprehensive Changelog**: Detailed feature descriptions and credits
- **API Documentation**: All new endpoints documented
- **Security Advisory**: Complete vulnerability disclosure and fixes
- **Migration Guide**: Breaking changes and upgrade instructions
---
## 🙏 **CREDITS**
**Core Contributors:**
- **@KaifAhmad1** - Distance Intelligence (PR #502), Security Hardening, Ontology Hub (PRs #517, #518, #519, #524), Windows Fixes (PR #532), ConflictDetector (PR #533), Testing & Release Preparation
- **@ZohaibHassan16** - Ontology Hub UI (PRs #516, #518, #519, #524), Graph Explorer (PRs #420, #481, #483, #503), Semantic Extract (PR #536), Lazy Loading (PR #535)
- **@Luffy2208** - Parquet Ingestion Support (PR #548)
- **@liling** - DeepSeek Provider Integration (PR #482)
- **@Sameer6305** - Provenance Traversal Fixes (PR #480), Named Graph Support
**Special Thanks:**
- Security research team for vulnerability disclosures
- Community testers and feedback providers
- Documentation contributors and reviewers
---
## 🚀 **INSTALLATION**
```bash
# Standard installation
pip install semantica==0.5.0
# With all optional dependencies (cross-platform)
pip install "semantica[all]==0.5.0"
# With GPU acceleration (Linux only)
pip install "semantica[gpu]==0.5.0"
# With Parquet support
pip install "semantica[ingest-parquet]==0.5.0"
```
---
## 📈 **WHAT'S NEXT FOR 0.5.0**
The 0.5.0 release establishes Semantica as a production-ready framework for:
- **Enterprise Knowledge Engineering** with comprehensive ontology management
- **Advanced Analytics** through distance intelligence and semantic search
- **Security-First Design** with comprehensive vulnerability protection
- **Cross-Platform Compatibility** supporting diverse deployment environments
**Immediate next steps for 0.5.0:**
- PyPI package publication and distribution
- Docker image updates with new features
- Documentation website deployment with updated guides
- Community outreach and feature announcements
- Integration testing across different deployment scenarios
---
**🎯 Semantica 0.5.0: Production-Ready Knowledge Engineering Platform**
+1 -1
View File
@@ -1,7 +1,7 @@
# Benchmark Tools
pytest>=7.0.0
pytest-benchmark>=4.0.0
pytest-benchmark>=5.2.3
# Core Utils
+50 -7
View File
@@ -12,6 +12,7 @@ The **Ingest Module** is the entry point for loading data into Semantica. It pro
**Data ingestion** is the process of loading data from various sources into Semantica for processing. The ingest module handles:
- **File Systems**: Local files, cloud storage (S3, GCS, Azure)
- **Analytics Files**: Apache Parquet files and partitioned datasets
- **Web Content**: Websites, RSS feeds, APIs
- **Streams**: Real-time data from Kafka, RabbitMQ, etc.
- **Databases**: SQL, NoSQL, and cloud data warehouses including Snowflake
@@ -74,6 +75,12 @@ The **Ingest Module** is the entry point for loading data into Semantica. It pro
Ingest tables and query results from SQL, NoSQL, and cloud data warehouses including Snowflake
- :material-table:{ .lg .middle } **Parquet Datasets**
---
Read Parquet files, schemas, metadata, and Hive-style partitioned directories
</div>
!!! tip "When to Use"
@@ -118,6 +125,19 @@ Handles file systems and object storage.
| `ingest_file(path)` | Process single file |
| `ingest_directory(path)` | Process folder |
### ParquetIngestor
Handles Apache Parquet files and partitioned datasets.
**Methods:**
| Method | Description |
|--------|-------------|
| `ingest_file(path, columns=None, limit=None)` | Read a Parquet file |
| `ingest_directory(path, columns=None, limit=None)` | Read a partitioned Parquet directory |
| `extract_schema(path)` | Extract column names, types, nullability, and schema metadata |
| `extract_metadata(path)` | Extract row counts, row groups, compression, and partition info |
### WebIngestor
Handles web content.
@@ -213,10 +233,33 @@ from semantica.ingest import ingest
# Auto-detect source type
ingest("doc.pdf", source_type="file")
ingest("events.parquet") # Auto-detects Parquet
ingest("https://google.com", source_type="web")
ingest("kafka://topic", source_type="stream")
```
### Parquet Dataset Ingestion
```python
from semantica.ingest import ParquetIngestor, ingest_parquet
ingestor = ParquetIngestor()
# Read selected columns from a local Parquet file
events = ingestor.ingest_file(
"events.parquet",
columns=["event_id", "event_type"],
limit=1000,
)
# Inspect schema and metadata without reading rows
schema = ingestor.extract_schema("events.parquet")
metadata = ingestor.extract_metadata("events.parquet")
# Read a Hive-style partitioned directory such as country=US/year=2026/
partitioned = ingest_parquet("./warehouse/events", method="directory")
```
---
## Configuration
@@ -236,7 +279,7 @@ ingest:
web:
user_agent: "MyBot"
rate_limit: 1.0 # seconds
files:
max_size: 100MB
allowed_extensions: [.pdf, .txt, .md]
@@ -289,12 +332,12 @@ data = ingestor.ingest_snowflake_table("CUSTOMERS")
# 3. Or run custom query
results = ingestor.execute_snowflake_query("""
SELECT
CUSTOMER_ID,
NAME,
EMAIL,
CREATED_AT
FROM CUSTOMERS
SELECT
CUSTOMER_ID,
NAME,
EMAIL,
CREATED_AT
FROM CUSTOMERS
WHERE CREATED_AT > '2024-01-01'
""")
+14 -2
View File
@@ -1237,6 +1237,7 @@ export default function App() {
const [analyzeView, setAnalyzeView] = useState<AnalyzeView>('reasoning');
const [enrichView, setEnrichView] = useState<EnrichView>('import');
const [manageView, setManageView] = useState<ManageView>('lineage');
const [graphFocusRequest, setGraphFocusRequest] = useState<{ nodeId: string; token: number } | null>(null);
const renderWorkspace = () => {
@@ -1284,7 +1285,12 @@ export default function App() {
}
>
<Suspense fallback={<WorkspaceFallback />}>
{exploreView === 'graph' ? <GraphWorkspace /> : <VocabularyWorkspace />}
{exploreView === 'graph' ? (
<GraphWorkspace
externalFocusNodeId={graphFocusRequest?.nodeId}
externalFocusToken={graphFocusRequest?.token}
/>
) : <VocabularyWorkspace />}
</Suspense>
</WorkspaceShell>
);
@@ -1370,7 +1376,13 @@ export default function App() {
compact
>
<Suspense fallback={<WorkspaceFallback />}>
<OntologyWorkspace />
<OntologyWorkspace
onJumpToGraphNode={(nodeId: string) => {
setGraphFocusRequest({ nodeId, token: Date.now() });
setActiveWorkspace('explore');
setExploreView('graph');
}}
/>
</Suspense>
</WorkspaceShell>
);
@@ -1082,7 +1082,12 @@ function collectPluginOverlays(
});
}
export function GraphWorkspace() {
interface GraphWorkspaceProps {
externalFocusNodeId?: string;
externalFocusToken?: number;
}
export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: GraphWorkspaceProps = {}) {
const [selectedNodeId, setSelectedNodeId] = useState("");
const [focusedNodeId, setFocusedNodeId] = useState("");
const [lastGroupedSelectedNodeId, setLastGroupedSelectedNodeId] = useState("");
@@ -1141,6 +1146,7 @@ export function GraphWorkspace() {
const debouncedTime = useDebounce(scrubberTime, 150);
const prevActiveIdsRef = useRef<Set<string>>(new Set());
const sceneRef = useRef<GraphSceneHandle>(null);
const lastExternalFocusTokenRef = useRef<number | undefined>(undefined);
const pluginRuntimeRef = useRef<GraphSceneRuntime | null>(null);
const settlingOverlayTimeoutRef = useRef<number | null>(null);
const pluginInteractionStateRef = useRef<GraphInteractionState>({
@@ -1440,6 +1446,23 @@ export function GraphWorkspace() {
}
}, [viewMode]); // Note: ego/heatmap/distanceMode effects re-run automatically when selectedNodeId changes
useEffect(() => {
if (!externalFocusNodeId || externalFocusToken == null) return;
if (lastExternalFocusTokenRef.current === externalFocusToken) return;
if (!graphReady || !graph.hasNode(externalFocusNodeId)) return;
lastExternalFocusTokenRef.current = externalFocusToken;
// Set state directly instead of going through focusNode(), which captures
// a stale viewMode in its closure. setViewMode is called first so the node
// is visible in the full graph before the scene pans to it.
setViewMode("full");
setSelectedNodeId(externalFocusNodeId);
setSelectedEdgeId("");
window.setTimeout(() => {
sceneRef.current?.focusNode(externalFocusNodeId);
}, 0);
}, [externalFocusNodeId, externalFocusToken, graphReady]);
const handleEdgeSelect = useCallback((edgeId: string) => {
setSelectedEdgeId(edgeId);
}, []);
@@ -0,0 +1,406 @@
import { useCallback, useEffect, useMemo, useState } from "react";
import type { CSSProperties } from "react";
import { GitMerge, Loader2, Sparkles, Trash2 } from "lucide-react";
import {
loadAlignments,
loadOntologyRegistry,
removeAlignment,
saveAlignment,
suggestAlignments,
} from "./api";
import type { AlignmentRelation, AlignmentSuggestion, OntologyAlignment, OntologyEntry } from "./types";
const RELATIONS: AlignmentRelation[] = [
"owl:equivalentClass",
"owl:equivalentProperty",
"skos:exactMatch",
"skos:closeMatch",
"skos:broadMatch",
"skos:narrowMatch",
"skos:relatedMatch",
];
const RELATION_COLORS: Record<AlignmentRelation, string> = {
"owl:equivalentClass": "#7ce7d3",
"owl:equivalentProperty": "#7ce7d3",
"skos:exactMatch": "#9ee8d7",
"skos:closeMatch": "#58a6ff",
"skos:broadMatch": "#f2b66d",
"skos:narrowMatch": "#f2b66d",
"skos:relatedMatch": "#d2a8ff",
};
export function AlignmentsTab() {
const [registry, setRegistry] = useState<OntologyEntry[]>([]);
const [alignments, setAlignments] = useState<OntologyAlignment[]>([]);
const [suggestions, setSuggestions] = useState<AlignmentSuggestion[]>([]);
const [sourceOntology, setSourceOntology] = useState("");
const [targetOntology, setTargetOntology] = useState("");
const [sourceUri, setSourceUri] = useState("");
const [targetUri, setTargetUri] = useState("");
const [relation, setRelation] = useState<AlignmentRelation>("skos:exactMatch");
const [confidence, setConfidence] = useState(0.86);
const [provenance, setProvenance] = useState("");
const [source, setSource] = useState("Ontology Hub");
const [reviewer, setReviewer] = useState("");
const [threshold, setThreshold] = useState(0.68);
const [busy, setBusy] = useState(false);
const [error, setError] = useState("");
const reload = useCallback(async () => {
setError("");
try {
const [registryData, alignmentData] = await Promise.all([
loadOntologyRegistry(),
loadAlignments(),
]);
setRegistry(registryData);
setAlignments(alignmentData);
setSourceOntology((current) => current || registryData[0]?.uri || "");
setTargetOntology((current) => current || registryData[1]?.uri || registryData[0]?.uri || "");
} catch (err) {
setError(err instanceof Error ? err.message : "Could not load ontology alignments.");
}
}, []);
useEffect(() => {
void reload();
}, [reload]);
const relationCounts = useMemo(() => {
const counts = new Map<string, number>();
for (const item of alignments) {
counts.set(item.relation, (counts.get(item.relation) ?? 0) + 1);
}
return counts;
}, [alignments]);
// Pairwise matrix: group alignments by (source_ontology, target_ontology) pair.
const matrix = useMemo(() => {
function ontologyOfUri(uri: string): string {
for (const entry of registry) {
if (uri === entry.uri || uri.startsWith(entry.uri + "#") || uri.startsWith(entry.uri + "/")) {
return entry.uri;
}
}
const hashIdx = uri.lastIndexOf("#");
if (hashIdx > 0) return uri.substring(0, hashIdx);
const slashIdx = uri.lastIndexOf("/");
return slashIdx > 0 ? uri.substring(0, slashIdx) : uri;
}
const cells: Map<string, OntologyAlignment[]> = new Map();
for (const alignment of alignments) {
const key = `${ontologyOfUri(alignment.source_uri)}|||${ontologyOfUri(alignment.target_uri)}`;
const bucket = cells.get(key) ?? [];
bucket.push(alignment);
cells.set(key, bucket);
}
return { ontologies: registry, cells };
}, [registry, alignments]);
const handleSave = useCallback(async () => {
if (!sourceUri.trim() || !targetUri.trim()) {
setError("Provide both source and target entity URIs.");
return;
}
setBusy(true);
setError("");
try {
await saveAlignment({
source_uri: sourceUri.trim(),
target_uri: targetUri.trim(),
relation,
confidence,
provenance: provenance || undefined,
source: source || undefined,
reviewer: reviewer || undefined,
});
setSourceUri("");
setTargetUri("");
await reload();
} catch (err) {
setError(err instanceof Error ? err.message : "Could not save alignment.");
} finally {
setBusy(false);
}
}, [sourceUri, targetUri, relation, confidence, provenance, source, reviewer, reload]);
const handleSuggest = useCallback(async () => {
setBusy(true);
setError("");
try {
const data = await suggestAlignments({
source_ontology_uri: sourceOntology || undefined,
target_ontology_uri: targetOntology || undefined,
threshold,
limit: 40,
});
setSuggestions(data);
} catch (err) {
setError(err instanceof Error ? err.message : "Could not suggest alignments.");
} finally {
setBusy(false);
}
}, [sourceOntology, targetOntology, threshold]);
const handleAcceptSuggestion = useCallback((suggestion: AlignmentSuggestion) => {
setSourceUri(suggestion.source_uri);
setTargetUri(suggestion.target_uri);
setRelation(suggestion.relation);
setConfidence(Math.max(0.1, Math.min(1, suggestion.score)));
setProvenance(suggestion.reason);
}, []);
const handleRemove = useCallback(async (id: string) => {
setBusy(true);
setError("");
try {
await removeAlignment(id);
await reload();
} catch (err) {
setError(err instanceof Error ? err.message : "Could not remove alignment.");
} finally {
setBusy(false);
}
}, [reload]);
return (
<div style={pageStyle}>
<section style={heroStyle}>
<div>
<div style={kickerStyle}><GitMerge size={14} /> Alignment Matrix</div>
<h2 style={titleStyle}>Cross-ontology mappings</h2>
<p style={textStyle}>
Manage equivalence and SKOS match relations with confidence, provenance,
reviewer context, and label-based suggestions.
</p>
</div>
<div style={summaryGridStyle}>
<Metric label="Mappings" value={alignments.length} />
<Metric label="Relations" value={relationCounts.size} />
<Metric label="Suggestions" value={suggestions.length} />
</div>
</section>
{error ? <div style={errorStyle}>{error}</div> : null}
<div style={ephemeralBannerStyle}>
Alignments are stored in server memory and are not persisted across restarts.
Export your graph or ontology to preserve recorded mappings.
</div>
{matrix.ontologies.length >= 2 ? (
<section style={cardStyle}>
<h3 style={sectionTitleStyle}>Pairwise alignment matrix</h3>
<div style={{ overflowX: "auto" }}>
<table style={matrixTableStyle}>
<thead>
<tr>
<th style={matrixCornerStyle} />
{matrix.ontologies.map((col) => (
<th key={col.uri} style={matrixColHeaderStyle}>{col.name}</th>
))}
</tr>
</thead>
<tbody>
{matrix.ontologies.map((row) => (
<tr key={row.uri}>
<td style={matrixRowHeaderStyle}>{row.name}</td>
{matrix.ontologies.map((col) => {
const key = `${row.uri}|||${col.uri}`;
const cellItems = matrix.cells.get(key) ?? [];
const isDiag = row.uri === col.uri;
return (
<td key={col.uri} style={{ ...matrixCellStyle, background: isDiag ? "rgba(255,255,255,0.015)" : undefined }}>
{isDiag ? <span style={mutedStyle}></span> : cellItems.length ? (
<div style={{ display: "flex", flexDirection: "column", gap: 4 }}>
{cellItems.map((item) => (
<span
key={item.id}
style={{ ...relationBadgeStyle, color: RELATION_COLORS[item.relation], borderColor: `${RELATION_COLORS[item.relation]}44`, cursor: "pointer", fontSize: 9 }}
title={`${item.source_label}${item.target_label} (${Math.round(item.confidence * 100)}%)`}
onClick={() => {
setSourceUri(item.source_uri);
setTargetUri(item.target_uri);
setRelation(item.relation);
setConfidence(item.confidence);
setProvenance(item.provenance ?? "");
}}
>
{item.relation.split(":")[1]}
</span>
))}
</div>
) : <span style={{ color: "rgba(127,208,255,0.15)", fontSize: 12 }}>·</span>}
</td>
);
})}
</tr>
))}
</tbody>
</table>
</div>
<p style={{ ...mutedStyle, marginTop: 10 }}>Click a relation badge to load it into the editor below.</p>
</section>
) : null}
<div style={gridStyle}>
<section style={cardStyle}>
<h3 style={sectionTitleStyle}>Create or update alignment</h3>
<label style={labelStyle}>Source entity URI</label>
<input style={inputStyle} value={sourceUri} onChange={(event) => setSourceUri(event.target.value)} />
<label style={labelStyle}>Target entity URI</label>
<input style={inputStyle} value={targetUri} onChange={(event) => setTargetUri(event.target.value)} />
<div style={twoColStyle}>
<div>
<label style={labelStyle}>Relation</label>
<select style={inputStyle} value={relation} onChange={(event) => setRelation(event.target.value as AlignmentRelation)}>
{RELATIONS.map((item) => <option key={item}>{item}</option>)}
</select>
</div>
<div>
<label style={labelStyle}>Confidence {confidence.toFixed(2)}</label>
<input
type="range"
min="0"
max="1"
step="0.01"
value={confidence}
onChange={(event) => setConfidence(Number(event.target.value))}
style={{ width: "100%" }}
/>
</div>
</div>
<label style={labelStyle}>Provenance note</label>
<textarea style={{ ...inputStyle, minHeight: 74, resize: "vertical" }} value={provenance} onChange={(event) => setProvenance(event.target.value)} />
<div style={twoColStyle}>
<div>
<label style={labelStyle}>Source</label>
<input style={inputStyle} value={source} onChange={(event) => setSource(event.target.value)} />
</div>
<div>
<label style={labelStyle}>Reviewer</label>
<input style={inputStyle} value={reviewer} onChange={(event) => setReviewer(event.target.value)} />
</div>
</div>
<button style={primaryButtonStyle} disabled={busy} onClick={handleSave}>
{busy ? <Loader2 size={14} className="spin" /> : <GitMerge size={14} />}
Save alignment
</button>
</section>
<section style={cardStyle}>
<h3 style={sectionTitleStyle}>Suggest alignments</h3>
<div style={twoColStyle}>
<div>
<label style={labelStyle}>Source ontology</label>
<select style={inputStyle} value={sourceOntology} onChange={(event) => setSourceOntology(event.target.value)}>
<option value="">Any ontology</option>
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
</select>
</div>
<div>
<label style={labelStyle}>Target ontology</label>
<select style={inputStyle} value={targetOntology} onChange={(event) => setTargetOntology(event.target.value)}>
<option value="">Any ontology</option>
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
</select>
</div>
</div>
<label style={labelStyle}>Similarity threshold {threshold.toFixed(2)}</label>
<input
type="range"
min="0.25"
max="0.95"
step="0.01"
value={threshold}
onChange={(event) => setThreshold(Number(event.target.value))}
style={{ width: "100%" }}
/>
<button style={secondaryButtonStyle} disabled={busy} onClick={handleSuggest}>
<Sparkles size={14} />
Suggest alignments
</button>
<div style={suggestionListStyle}>
{suggestions.map((item) => (
<button key={`${item.source_uri}-${item.target_uri}-${item.relation}`} style={suggestionStyle} onClick={() => handleAcceptSuggestion(item)}>
<span style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.source_label}</span>
<span style={{ color: RELATION_COLORS[item.relation] }}>{item.relation}</span>
<span style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.target_label}</span>
<span style={{ color: "#8fa8c6" }}>{Math.round(item.score * 100)}%</span>
</button>
))}
{!suggestions.length ? <p style={mutedStyle}>Run suggestions to review ranked candidate mappings.</p> : null}
</div>
</section>
</div>
<section style={cardStyle}>
<h3 style={sectionTitleStyle}>Recorded alignments</h3>
<div style={tableStyle}>
{alignments.map((item) => (
<div key={item.id} style={rowStyle}>
<div>
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.source_label || item.source_uri}</div>
<div style={monoStyle}>{item.source_uri}</div>
</div>
<div style={{ ...relationBadgeStyle, color: RELATION_COLORS[item.relation], borderColor: `${RELATION_COLORS[item.relation]}55` }}>
{item.relation}
</div>
<div>
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.target_label || item.target_uri}</div>
<div style={monoStyle}>{item.target_uri}</div>
</div>
<div style={confidenceStyle}>{Math.round(item.confidence * 100)}%</div>
<button style={iconButtonStyle} disabled={busy} onClick={() => handleRemove(item.id)} title="Remove alignment">
<Trash2 size={14} />
</button>
</div>
))}
{!alignments.length ? <p style={mutedStyle}>No alignments recorded yet.</p> : null}
</div>
</section>
</div>
);
}
function Metric({ label, value }: { label: string; value: number }) {
return (
<div style={metricStyle}>
<span style={{ color: "#9ee8d7", fontSize: 20, fontWeight: 900 }}>{value.toLocaleString()}</span>
<span style={{ color: "#6a7f97", fontSize: 11 }}>{label}</span>
</div>
);
}
const pageStyle: CSSProperties = { height: "100%", overflow: "auto", padding: 22, display: "flex", flexDirection: "column", gap: 16 };
const heroStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 18, padding: 22, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 22, background: "linear-gradient(135deg, rgba(11,25,42,0.94), rgba(7,14,25,0.9))" };
const kickerStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", color: "#9ee8d7", fontSize: 11, fontWeight: 900, letterSpacing: "0.12em", textTransform: "uppercase" };
const titleStyle: CSSProperties = { margin: "8px 0", color: "#ebf3ff", fontSize: 26, letterSpacing: "-0.04em" };
const textStyle: CSSProperties = { margin: 0, color: "#8fa8c6", lineHeight: 1.6, maxWidth: 620 };
const summaryGridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "repeat(3, minmax(100px, 1fr))", gap: 10, minWidth: 320 };
const metricStyle: CSSProperties = { padding: 14, borderRadius: 16, background: "rgba(255,255,255,0.035)", border: "1px solid rgba(127,208,255,0.1)", display: "flex", flexDirection: "column", gap: 4 };
const gridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "minmax(320px, 0.9fr) minmax(360px, 1.1fr)", gap: 16 };
const cardStyle: CSSProperties = { padding: 18, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 20, background: "rgba(9,19,34,0.78)", boxShadow: "inset 0 1px 0 rgba(255,255,255,0.04)" };
const sectionTitleStyle: CSSProperties = { margin: "0 0 14px", color: "#ebf3ff", fontSize: 16 };
const labelStyle: CSSProperties = { display: "block", color: "#6a7f97", fontSize: 11, fontWeight: 800, margin: "10px 0 6px", textTransform: "uppercase", letterSpacing: "0.08em" };
const inputStyle: CSSProperties = { width: "100%", boxSizing: "border-box", border: "1px solid rgba(127,208,255,0.14)", borderRadius: 12, padding: "10px 12px", background: "rgba(3,9,18,0.8)", color: "#ebf3ff" };
const twoColStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr 1fr", gap: 12 };
const primaryButtonStyle: CSSProperties = { marginTop: 14, width: "100%", border: "1px solid rgba(124,231,211,0.35)", borderRadius: 12, padding: "11px 13px", background: "linear-gradient(135deg, rgba(20,151,136,0.55), rgba(74,163,255,0.35))", color: "#ebf3ff", fontWeight: 900, cursor: "pointer", display: "inline-flex", alignItems: "center", justifyContent: "center", gap: 8 };
const secondaryButtonStyle: CSSProperties = { ...primaryButtonStyle, background: "rgba(127,208,255,0.08)", borderColor: "rgba(127,208,255,0.18)" };
const suggestionListStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 8, marginTop: 14, maxHeight: 260, overflow: "auto" };
const suggestionStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr auto 1fr auto", gap: 10, alignItems: "center", textAlign: "left", border: "1px solid rgba(127,208,255,0.1)", borderRadius: 12, background: "rgba(255,255,255,0.03)", padding: 10, cursor: "pointer" };
const tableStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 8 };
const rowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr auto 1fr auto auto", gap: 12, alignItems: "center", padding: 12, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
const monoStyle: CSSProperties = { color: "#6a7f97", fontSize: 11, fontFamily: "JetBrains Mono, monospace", wordBreak: "break-all" };
const relationBadgeStyle: CSSProperties = { padding: "5px 9px", border: "1px solid", borderRadius: 999, fontSize: 10, fontWeight: 900 };
const confidenceStyle: CSSProperties = { color: "#f2b66d", fontWeight: 900 };
const iconButtonStyle: CSSProperties = { width: 34, height: 34, borderRadius: 10, border: "1px solid rgba(255,157,175,0.18)", background: "rgba(255,157,175,0.08)", color: "#ff9daf", cursor: "pointer" };
const mutedStyle: CSSProperties = { margin: 0, color: "#6a7f97", fontSize: 13 };
const errorStyle: CSSProperties = { padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" };
const ephemeralBannerStyle: CSSProperties = { padding: "9px 14px", borderRadius: 12, color: "#f2b66d", background: "rgba(242,182,109,0.08)", border: "1px solid rgba(242,182,109,0.22)", fontSize: 12 };
const matrixTableStyle: CSSProperties = { borderCollapse: "collapse", minWidth: "100%", fontSize: 12 };
const matrixCornerStyle: CSSProperties = { padding: "8px 12px", borderBottom: "1px solid rgba(127,208,255,0.1)" };
const matrixColHeaderStyle: CSSProperties = { padding: "8px 12px", color: "#9ee8d7", fontWeight: 900, borderBottom: "1px solid rgba(127,208,255,0.1)", textAlign: "center", whiteSpace: "nowrap" };
const matrixRowHeaderStyle: CSSProperties = { padding: "8px 12px", color: "#9ee8d7", fontWeight: 900, borderRight: "1px solid rgba(127,208,255,0.1)", whiteSpace: "nowrap" };
const matrixCellStyle: CSSProperties = { padding: "8px 10px", borderBottom: "1px solid rgba(127,208,255,0.06)", borderRight: "1px solid rgba(127,208,255,0.06)", textAlign: "center", verticalAlign: "middle", minWidth: 100 };
@@ -0,0 +1,205 @@
import { useCallback, useEffect, useState } from "react";
import type { CSSProperties } from "react";
import { Download, HeartPulse, Loader2, Wrench } from "lucide-react";
import { loadOntologyHealth, loadOntologyRegistry } from "./api";
import type { OntologyEntry, OntologyHealthResponse, HealthIssue } from "./types";
interface HealthTabProps {
onFixInEditor?: (entityUri: string) => void;
}
export function HealthTab({ onFixInEditor }: HealthTabProps) {
const [registry, setRegistry] = useState<OntologyEntry[]>([]);
const [selectedUri, setSelectedUri] = useState("");
const [health, setHealth] = useState<OntologyHealthResponse | null>(null);
const [loading, setLoading] = useState(false);
const [error, setError] = useState("");
useEffect(() => {
let cancelled = false;
loadOntologyRegistry()
.then((entries) => {
if (cancelled) return;
setRegistry(entries);
setSelectedUri((current) => current || entries[0]?.uri || "");
})
.catch((err) => {
if (!cancelled) setError(err instanceof Error ? err.message : "Could not load ontology registry.");
});
return () => {
cancelled = true;
};
}, []);
const loadHealth = useCallback(async (uri: string) => {
if (!uri) return;
setLoading(true);
setError("");
try {
setHealth(await loadOntologyHealth(uri));
} catch (err) {
setHealth(null);
setError(err instanceof Error ? err.message : "Could not load ontology health.");
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
void loadHealth(selectedUri);
}, [selectedUri, loadHealth]);
const exportReport = useCallback(() => {
if (!health) return;
const blob = new Blob([JSON.stringify(health, null, 2)], { type: "application/json" });
const url = URL.createObjectURL(blob);
const anchor = document.createElement("a");
anchor.href = url;
anchor.download = `${health.name.replace(/[^a-z0-9]+/gi, "-").toLowerCase()}-health.json`;
document.body.appendChild(anchor);
anchor.click();
document.body.removeChild(anchor);
setTimeout(() => URL.revokeObjectURL(url), 100);
}, [health]);
return (
<div style={pageStyle}>
<section style={heroStyle}>
<div>
<div style={kickerStyle}><HeartPulse size={14} /> Ontology Health</div>
<h2 style={titleStyle}>Quality and governance signals</h2>
<p style={textStyle}>
Score completeness, consistency, SHACL readiness, alignment coverage,
and documentation quality for the selected ontology.
</p>
</div>
<div style={selectorShellStyle}>
<label style={labelStyle}>Ontology</label>
<select style={inputStyle} value={selectedUri} onChange={(event) => setSelectedUri(event.target.value)}>
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
</select>
</div>
</section>
{error ? <div style={errorStyle}>{error}</div> : null}
{loading ? (
<div style={loadingStyle}><Loader2 size={18} className="spin" /> Computing health dashboard...</div>
) : health ? (
<>
<section style={{ ...scoreGridStyle, gridTemplateColumns: `220px repeat(${health.dimensions.length}, minmax(180px, 1fr))` }}>
<div style={scoreCardStyle}>
<span style={scoreValueStyle}>{Math.round(health.total_score)}</span>
<span style={mutedStyle}>Total health score</span>
<button style={secondaryButtonStyle} onClick={exportReport}><Download size={14} /> Export report</button>
</div>
{health.dimensions.map((dimension) => (
<div key={dimension.key} style={dimensionCardStyle}>
<div style={dimensionHeadStyle}>
<span style={{ color: "#ebf3ff", fontWeight: 900 }}>{dimension.label}</span>
<span style={statusBadgeStyle(dimension.status)}>{dimension.status}</span>
</div>
<div style={barTrackStyle}>
<div style={{ ...barFillStyle, width: `${dimension.score}%`, background: dimensionColor(dimension.score, dimension.status) }} />
</div>
<div style={dimensionFootStyle}>
<span>{Math.round(dimension.score)} / 100</span>
<span>{dimension.detail}</span>
</div>
</div>
))}
</section>
<section style={cardStyle}>
<h3 style={sectionTitleStyle}>Actionable issues</h3>
<div style={issueListStyle}>
{health.issues.map((issue) => (
<IssueRow key={issue.id} issue={issue} onFixInEditor={onFixInEditor} />
))}
{!health.issues.length ? <p style={mutedStyle}>No actionable issues reported for this ontology.</p> : null}
</div>
</section>
</>
) : (
<div style={emptyStyle}>Select an ontology to compute health signals.</div>
)}
</div>
);
}
function IssueRow({ issue, onFixInEditor }: { issue: HealthIssue; onFixInEditor?: (entityUri: string) => void }) {
return (
<div style={issueRowStyle}>
<div style={severityDotStyle(issue.severity)} />
<div>
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{issue.entity_label || issue.category}</div>
<div style={{ color: "#8fa8c6", fontSize: 13, lineHeight: 1.45 }}>{issue.message}</div>
{issue.entity_uri ? <div style={monoStyle}>{issue.entity_uri}</div> : null}
</div>
<span style={categoryStyle}>{issue.category}</span>
{issue.entity_uri ? (
<button style={smallButtonStyle} onClick={() => onFixInEditor?.(issue.entity_uri || "")}>
<Wrench size={13} />
Fix in Editor
</button>
) : (
<div />
)}
</div>
);
}
function dimensionColor(score: number, status: string) {
if (status === "unavailable") return "#6a7f97";
if (score >= 80) return "#7ce7d3";
if (score >= 55) return "#f2b66d";
return "#ff9daf";
}
function statusBadgeStyle(status: string): CSSProperties {
const color = status === "ok" ? "#7ce7d3" : status === "unavailable" ? "#6a7f97" : "#f2b66d";
return {
color,
background: `${color}18`,
border: `1px solid ${color}30`,
borderRadius: 999,
padding: "2px 7px",
fontSize: 10,
fontWeight: 900,
textTransform: "uppercase",
};
}
function severityDotStyle(severity: string): CSSProperties {
const color = severity === "critical" ? "#ff9daf" : severity === "warning" ? "#f2b66d" : "#58a6ff";
return { width: 10, height: 10, borderRadius: "50%", background: color, boxShadow: `0 0 18px ${color}55`, marginTop: 5 };
}
const pageStyle: CSSProperties = { height: "100%", overflow: "auto", padding: 22, display: "flex", flexDirection: "column", gap: 16 };
const heroStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 18, padding: 22, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 22, background: "linear-gradient(135deg, rgba(11,25,42,0.94), rgba(7,14,25,0.9))" };
const kickerStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", color: "#9ee8d7", fontSize: 11, fontWeight: 900, letterSpacing: "0.12em", textTransform: "uppercase" };
const titleStyle: CSSProperties = { margin: "8px 0", color: "#ebf3ff", fontSize: 26, letterSpacing: "-0.04em" };
const textStyle: CSSProperties = { margin: 0, color: "#8fa8c6", lineHeight: 1.6, maxWidth: 620 };
const selectorShellStyle: CSSProperties = { minWidth: 320 };
const labelStyle: CSSProperties = { display: "block", color: "#6a7f97", fontSize: 11, fontWeight: 800, margin: "0 0 6px", textTransform: "uppercase", letterSpacing: "0.08em" };
const inputStyle: CSSProperties = { width: "100%", boxSizing: "border-box", border: "1px solid rgba(127,208,255,0.14)", borderRadius: 12, padding: "10px 12px", background: "rgba(3,9,18,0.8)", color: "#ebf3ff" };
const scoreGridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "220px repeat(5, minmax(180px, 1fr))", gap: 12 };
const scoreCardStyle: CSSProperties = { padding: 18, borderRadius: 20, background: "rgba(15,35,52,0.88)", border: "1px solid rgba(124,231,211,0.2)", display: "flex", flexDirection: "column", gap: 10 };
const scoreValueStyle: CSSProperties = { color: "#9ee8d7", fontSize: 52, lineHeight: 1, fontWeight: 950, letterSpacing: "-0.06em" };
const dimensionCardStyle: CSSProperties = { padding: 16, borderRadius: 18, background: "rgba(9,19,34,0.78)", border: "1px solid rgba(127,208,255,0.12)" };
const dimensionHeadStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 8, alignItems: "center", marginBottom: 12 };
const barTrackStyle: CSSProperties = { height: 8, borderRadius: 999, background: "rgba(255,255,255,0.06)", overflow: "hidden" };
const barFillStyle: CSSProperties = { height: "100%", borderRadius: 999 };
const dimensionFootStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 6, color: "#8fa8c6", fontSize: 12, marginTop: 10 };
const cardStyle: CSSProperties = { padding: 18, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 20, background: "rgba(9,19,34,0.78)" };
const sectionTitleStyle: CSSProperties = { margin: "0 0 14px", color: "#ebf3ff", fontSize: 16 };
const issueListStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 8 };
const issueRowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "14px 1fr auto auto", gap: 12, alignItems: "start", padding: 12, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
const categoryStyle: CSSProperties = { color: "#9ee8d7", border: "1px solid rgba(158,232,215,0.22)", borderRadius: 999, padding: "4px 8px", fontSize: 10, fontWeight: 900 };
const smallButtonStyle: CSSProperties = { display: "inline-flex", gap: 6, alignItems: "center", border: "1px solid rgba(127,208,255,0.16)", borderRadius: 10, padding: "7px 9px", background: "rgba(127,208,255,0.08)", color: "#ebf3ff", cursor: "pointer", fontWeight: 800 };
const secondaryButtonStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", justifyContent: "center", border: "1px solid rgba(127,208,255,0.16)", borderRadius: 12, padding: "10px 12px", background: "rgba(127,208,255,0.08)", color: "#ebf3ff", cursor: "pointer", fontWeight: 900 };
const monoStyle: CSSProperties = { marginTop: 4, color: "#6a7f97", fontSize: 11, fontFamily: "JetBrains Mono, monospace", wordBreak: "break-all" };
const mutedStyle: CSSProperties = { margin: 0, color: "#6a7f97", fontSize: 13 };
const loadingStyle: CSSProperties = { display: "inline-flex", alignItems: "center", gap: 8, color: "#8fa8c6", padding: 18 };
const emptyStyle: CSSProperties = { color: "#6a7f97", padding: 22 };
const errorStyle: CSSProperties = { display: "flex", alignItems: "center", gap: 8, padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" };
@@ -0,0 +1,498 @@
import { useCallback, useEffect, useState } from "react";
import {
ReactFlow,
Background,
Controls,
MiniMap,
addEdge,
useNodesState,
useEdgesState,
MarkerType,
} from "@xyflow/react";
import type { Connection, Edge, Node } from "@xyflow/react";
import "@xyflow/react/dist/style.css";
import {
Plus,
GitBranch,
User,
Shield,
FileText,
Layout,
Send,
Pencil,
Trash2,
} from "lucide-react";
type OntologyNodeData = {
label?: string;
type?: string;
};
type OntologyNode = Node<OntologyNodeData>;
type OntologyEdge = Edge<Record<string, unknown>>;
const nodeTypes = {
classNode: ({ data }: { data: OntologyNodeData }) => (
<div style={classNodeStyle}>
<div style={classNodeHeader}>{data.label}</div>
<div style={classNodeSub}>{data.type}</div>
</div>
),
};
const classNodeStyle: React.CSSProperties = {
padding: "12px 16px",
borderRadius: "8px",
background: "linear-gradient(135deg, rgba(74, 163, 255, 0.15), rgba(74, 163, 255, 0.05))",
border: "1px solid rgba(127, 208, 255, 0.3)",
color: "#ebf3ff",
fontSize: "13px",
fontWeight: "600",
minWidth: "140px",
textAlign: "center",
boxShadow: "0 4px 12px rgba(0, 0, 0, 0.2)",
};
const classNodeHeader: React.CSSProperties = {
fontSize: "14px",
fontWeight: "700",
marginBottom: "4px",
};
const classNodeSub: React.CSSProperties = {
fontSize: "11px",
color: "#8fa8c6",
fontWeight: "500",
};
interface DraftDiff {
added_classes: string[];
removed_classes: string[];
modified_classes: Record<string, Record<string, any>>;
added_properties: string[];
removed_properties: string[];
modified_properties: Record<string, Record<string, any>>;
added_restrictions: Record<string, any>[];
removed_restrictions: Record<string, any>[];
added_axioms: Record<string, any>[];
removed_axioms: Record<string, any>[];
annotation_changes: Record<string, Record<string, any>>;
}
interface RegistryEntry {
uri: string;
name: string;
}
export function OntologyEditor() {
const [nodes, setNodes, onNodesChange] = useNodesState<OntologyNode>([]);
const [edges, setEdges, onEdgesChange] = useEdgesState<OntologyEdge>([]);
const [selectedElement, setSelectedElement] = useState<OntologyNode | OntologyEdge | null>(null);
const [registry, setRegistry] = useState<RegistryEntry[]>([]);
const [ontologyUri, setOntologyUri] = useState<string>("");
const [draftDiff, setDraftDiff] = useState<DraftDiff>({
added_classes: [],
removed_classes: [],
modified_classes: {},
added_properties: [],
removed_properties: [],
modified_properties: {},
added_restrictions: [],
removed_restrictions: [],
added_axioms: [],
removed_axioms: [],
annotation_changes: {},
});
const [isSaving, setIsSaving] = useState(false);
const [showContext, setShowContext] = useState<{ x: number; y: number; type: string; element: OntologyNode | OntologyEdge } | null>(null);
useEffect(() => {
let cancelled = false;
fetch("/api/ontology/registry")
.then((response) => (response.ok ? response.json() : []))
.then((entries: RegistryEntry[]) => {
if (cancelled) return;
setRegistry(entries);
setOntologyUri((current) => current || entries[0]?.uri || "");
})
.catch((error) => {
console.error("Failed to load ontology registry:", error);
});
return () => {
cancelled = true;
};
}, []);
const onConnect = useCallback(
(params: Connection) => setEdges((eds) => addEdge({ ...params, markerEnd: { type: MarkerType.ArrowClosed } }, eds)),
[setEdges]
);
const addClass = useCallback(() => {
const newId = `class_${Date.now()}`;
const newNode: OntologyNode = {
id: newId,
type: "classNode",
position: { x: Math.random() * 400, y: Math.random() * 300 },
data: { label: "NewClass", type: "owl:Class" },
};
setNodes((nds) => [...nds, newNode]);
setDraftDiff((prev) => ({
...prev,
added_classes: [...prev.added_classes, newId],
}));
}, [setNodes]);
const addProperty = useCallback(() => {
if (nodes.length < 2) {
alert("Add at least two classes before creating a property edge.");
return;
}
const newId = `prop_${Date.now()}`;
const newEdge: OntologyEdge = {
id: newId,
source: nodes[0].id,
target: nodes[1].id,
label: "hasProperty",
type: "smoothstep",
animated: true,
};
setEdges((eds) => [...eds, newEdge]);
setDraftDiff((prev) => ({
...prev,
added_properties: [...prev.added_properties, newId],
}));
}, [nodes, setEdges]);
const addIndividual = useCallback(() => {
const newId = `ind_${Date.now()}`;
const newNode: OntologyNode = {
id: newId,
type: "classNode",
position: { x: Math.random() * 400, y: Math.random() * 300 },
data: { label: "NewIndividual", type: "owl:NamedIndividual" },
};
setNodes((nds) => [...nds, newNode]);
}, [setNodes]);
const addRestriction = useCallback(() => {
setDraftDiff((prev) => ({
...prev,
added_restrictions: [...prev.added_restrictions, { type: "someValuesFrom", value: "" }],
}));
}, []);
const addAxiom = useCallback(() => {
setDraftDiff((prev) => ({
...prev,
added_axioms: [...prev.added_axioms, { type: "subClassOf", value: "" }],
}));
}, []);
const autoLayout = useCallback(() => {
const layoutNodes = nodes.map((node, index) => ({
...node,
position: { x: (index % 4) * 200, y: Math.floor(index / 4) * 150 },
}));
setNodes(layoutNodes);
}, [nodes, setNodes]);
const saveDraft = useCallback(async () => {
if (!ontologyUri) {
alert("Please select an ontology first");
return;
}
setIsSaving(true);
try {
const response = await fetch("/api/ontology/draft", {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
ontology_uri: ontologyUri,
diff: draftDiff,
author: "user",
summary: "Visual editor changes",
}),
});
if (response.ok) {
const data = await response.json();
alert(`Draft saved: ${data.draft_id}`);
}
} catch (error) {
console.error("Failed to save draft:", error);
alert("Failed to save draft");
} finally {
setIsSaving(false);
}
}, [ontologyUri, draftDiff]);
const handleNodeContextMenu = useCallback((event: React.MouseEvent, node: OntologyNode) => {
event.preventDefault();
setSelectedElement(node);
setShowContext({ x: event.clientX, y: event.clientY, type: "node", element: node });
}, []);
const handleEdgeContextMenu = useCallback((event: React.MouseEvent, edge: OntologyEdge) => {
event.preventDefault();
setSelectedElement(edge);
setShowContext({ x: event.clientX, y: event.clientY, type: "edge", element: edge });
}, []);
const deleteSelected = useCallback(() => {
const target = showContext?.element ?? selectedElement;
if (target) {
if ("source" in target) {
setEdges((eds) => eds.filter((e) => e.id !== target.id));
setDraftDiff((prev) => ({
...prev,
removed_properties: [...prev.removed_properties, target.id],
}));
} else {
setNodes((nds) => nds.filter((n) => n.id !== target.id));
setDraftDiff((prev) => ({
...prev,
removed_classes: [...prev.removed_classes, target.id],
}));
}
setSelectedElement(null);
}
setShowContext(null);
}, [selectedElement, setNodes, setEdges, showContext]);
const renameSelected = useCallback(() => {
const target = showContext?.element ?? selectedElement;
if (target && !("source" in target)) {
const newLabel = prompt("Enter new name:", String(target.data.label ?? ""));
if (newLabel) {
setNodes((nds) =>
nds.map((n) => (n.id === target.id ? { ...n, data: { ...n.data, label: newLabel } } : n))
);
setDraftDiff((prev) => ({
...prev,
modified_classes: { ...prev.modified_classes, [target.id]: { label: newLabel } },
}));
}
}
setShowContext(null);
}, [selectedElement, setNodes, showContext]);
useEffect(() => {
const handleClick = () => setShowContext(null);
window.addEventListener("click", handleClick);
return () => window.removeEventListener("click", handleClick);
}, []);
const toolbarStyle: React.CSSProperties = {
display: "flex",
gap: "8px",
padding: "12px 16px",
background: "rgba(3, 9, 18, 0.92)",
borderBottom: "1px solid rgba(140, 192, 255, 0.12)",
flexWrap: "wrap",
};
const toolbarButtonStyle: React.CSSProperties = {
display: "inline-flex",
alignItems: "center",
gap: "6px",
padding: "8px 12px",
borderRadius: "8px",
border: "1px solid rgba(127, 208, 255, 0.18)",
background: "rgba(74, 163, 255, 0.08)",
color: "#ebf3ff",
fontSize: "12px",
fontWeight: "600",
cursor: "pointer",
transition: "160ms ease",
};
const selectStyle: React.CSSProperties = {
padding: "8px 12px",
borderRadius: "8px",
border: "1px solid rgba(127, 208, 255, 0.18)",
background: "rgba(3, 9, 18, 0.88)",
color: "#ebf3ff",
fontSize: "12px",
minWidth: "260px",
};
const contextMenuStyle: React.CSSProperties = {
position: "fixed",
background: "rgba(9, 19, 34, 0.95)",
border: "1px solid rgba(127, 208, 255, 0.3)",
borderRadius: "8px",
padding: "8px 0",
minWidth: "180px",
boxShadow: "0 8px 24px rgba(0, 0, 0, 0.4)",
zIndex: 1000,
};
const contextItemStyle: React.CSSProperties = {
padding: "8px 16px",
display: "flex",
alignItems: "center",
gap: "10px",
color: "#ebf3ff",
fontSize: "13px",
cursor: "pointer",
transition: "160ms ease",
};
const detailPanelStyle: React.CSSProperties = {
position: "absolute",
right: 0,
top: 0,
bottom: 0,
width: "320px",
background: "rgba(9, 19, 34, 0.95)",
borderLeft: "1px solid rgba(140, 192, 255, 0.12)",
padding: "20px",
overflow: "auto",
backdropFilter: "blur(18px)",
};
return (
<div style={{ display: "flex", flexDirection: "column", height: "100%", background: "#07111f" }}>
<div style={toolbarStyle}>
<select
aria-label="Active ontology"
value={ontologyUri}
onChange={(event) => setOntologyUri(event.target.value)}
style={selectStyle}
>
<option value="">Select ontology...</option>
{registry.map((entry) => (
<option key={entry.uri} value={entry.uri}>
{entry.name || entry.uri}
</option>
))}
</select>
<button style={toolbarButtonStyle} onClick={addClass}>
<Plus size={14} />
Add Class
</button>
<button style={toolbarButtonStyle} onClick={addProperty} disabled={nodes.length < 2}>
<GitBranch size={14} />
Add Property
</button>
<button style={toolbarButtonStyle} onClick={addIndividual}>
<User size={14} />
Add Individual
</button>
<button style={toolbarButtonStyle} onClick={addRestriction}>
<Shield size={14} />
Add Restriction
</button>
<button style={toolbarButtonStyle} onClick={addAxiom}>
<FileText size={14} />
Add Axiom
</button>
<button style={toolbarButtonStyle} onClick={autoLayout}>
<Layout size={14} />
Auto Layout
</button>
<div style={{ flex: 1 }} />
<button style={toolbarButtonStyle} onClick={saveDraft} disabled={isSaving}>
<Send size={14} />
{isSaving ? "Saving..." : "Propose"}
</button>
</div>
<div style={{ flex: 1, position: "relative" }}>
<ReactFlow
nodes={nodes}
edges={edges}
onNodesChange={onNodesChange}
onEdgesChange={onEdgesChange}
onConnect={onConnect}
onNodeClick={(_, node) => setSelectedElement(node)}
onEdgeClick={(_, edge) => setSelectedElement(edge)}
onNodeContextMenu={handleNodeContextMenu}
onEdgeContextMenu={handleEdgeContextMenu}
nodeTypes={nodeTypes}
fitView
style={{ background: "#07111f" }}
>
<Background color="#1a2d3d" gap={20} />
<Controls />
<MiniMap nodeColor="#4aa3ff" maskColor="rgba(0,0,0,0.6)" />
</ReactFlow>
{showContext && (
<div style={{ ...contextMenuStyle, left: showContext.x, top: showContext.y }}>
<div style={contextItemStyle} onClick={renameSelected}>
<Pencil size={14} />
Rename
</div>
<div style={contextItemStyle} onClick={deleteSelected}>
<Trash2 size={14} />
Delete
</div>
</div>
)}
{selectedElement && (
<div style={detailPanelStyle}>
<h3 style={{ margin: "0 0 16px", color: "#ebf3ff", fontSize: "16px" }}>
{"source" in selectedElement ? "Property Details" : "Class Details"}
</h3>
<div style={{ marginBottom: "12px" }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
ID
</label>
<div style={{ color: "#ebf3ff", fontSize: "13px", wordBreak: "break-all" }}>
{selectedElement.id}
</div>
</div>
{!("source" in selectedElement) && (
<>
<div style={{ marginBottom: "12px" }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
Label
</label>
<input
type="text"
value={String(selectedElement.data.label ?? "")}
onChange={(e) => {
setNodes((nds) =>
nds.map((n) =>
n.id === selectedElement.id
? { ...n, data: { ...n.data, label: e.target.value } }
: n
)
);
setDraftDiff((prev) => ({
...prev,
modified_classes: {
...prev.modified_classes,
[selectedElement.id]: { label: e.target.value },
},
}));
}}
style={{
width: "100%",
padding: "8px",
borderRadius: "6px",
border: "1px solid rgba(127, 208, 255, 0.2)",
background: "rgba(3, 9, 18, 0.8)",
color: "#ebf3ff",
fontSize: "13px",
}}
/>
</div>
<div style={{ marginBottom: "12px" }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
Type
</label>
<div style={{ color: "#ebf3ff", fontSize: "13px" }}>
{selectedElement.data.type || "owl:Class"}
</div>
</div>
</>
)}
</div>
)}
</div>
</div>
);
}
@@ -0,0 +1,413 @@
import { useCallback, useEffect, useState } from "react";
import {
GitMerge,
CheckCircle,
XCircle,
AlertCircle,
Send,
MessageSquare,
User,
Clock,
Plus,
Minus,
Edit,
} from "lucide-react";
interface Proposal {
proposal_id: string;
draft_id: string;
ontology_uri: string;
summary: string;
author: string;
reviewer: string | null;
state: "draft" | "proposed" | "approved" | "published" | "rejected";
impact_analysis: Record<string, any>;
shacl_validation: Record<string, any>;
created_at: string;
updated_at: string;
comments: Record<string, any>[];
}
interface DiffChange {
type: "added" | "removed" | "modified";
element: string;
details?: Record<string, any>;
}
export function ProposalReview({ proposalId }: { proposalId: string }) {
const [proposal, setProposal] = useState<Proposal | null>(null);
const [diff, setDiff] = useState<DiffChange[]>([]);
const [selectedElement, setSelectedElement] = useState<string | null>(null);
const [commentText, setCommentText] = useState("");
const loadProposal = useCallback(async () => {
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}`);
if (response.ok) {
const data = await response.json();
setProposal(data);
generateDiff(data);
}
} catch (error) {
console.error("Failed to load proposal:", error);
}
}, [proposalId]);
const generateDiff = useCallback((prop: Proposal) => {
const changes: DiffChange[] = [];
// Generate diff from impact analysis
if (prop.impact_analysis) {
if (prop.impact_analysis.class_adds > 0) {
changes.push({ type: "added", element: `Classes (${prop.impact_analysis.class_adds})` });
}
if (prop.impact_analysis.class_removals > 0) {
changes.push({ type: "removed", element: `Classes (${prop.impact_analysis.class_removals})` });
}
if (prop.impact_analysis.property_changes > 0) {
changes.push({ type: "modified", element: `Properties (${prop.impact_analysis.property_changes})` });
}
if (prop.impact_analysis.restriction_changes > 0) {
changes.push({ type: "modified", element: `Restrictions (${prop.impact_analysis.restriction_changes})` });
}
}
setDiff(changes);
}, []);
useEffect(() => {
loadProposal();
}, [loadProposal]);
const addComment = useCallback(async () => {
if (!selectedElement || !commentText || !proposal) return;
try {
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/comment`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
element_uri: selectedElement,
text: commentText,
author: "user",
}),
});
if (response.ok) {
setCommentText("");
loadProposal();
}
} catch (error) {
console.error("Failed to add comment:", error);
alert("Failed to add comment");
}
}, [selectedElement, commentText, proposal, loadProposal]);
const approveProposal = useCallback(async () => {
if (!proposal) return;
try {
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/approve`, {
method: "POST",
});
if (response.ok) {
alert("Proposal approved");
loadProposal();
}
} catch (error) {
console.error("Failed to approve proposal:", error);
alert("Failed to approve proposal");
}
}, [proposal, loadProposal]);
const rejectProposal = useCallback(async () => {
if (!proposal) return;
try {
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/reject`, {
method: "POST",
});
if (response.ok) {
alert("Proposal rejected");
loadProposal();
}
} catch (error) {
console.error("Failed to reject proposal:", error);
alert("Failed to reject proposal");
}
}, [proposal, loadProposal]);
const publishProposal = useCallback(async () => {
if (!proposal) return;
try {
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/publish`, {
method: "POST",
});
if (response.ok) {
alert("Proposal published");
loadProposal();
}
} catch (error) {
console.error("Failed to publish proposal:", error);
alert("Failed to publish proposal");
}
}, [proposal, loadProposal]);
const getChangeIcon = (type: string) => {
switch (type) {
case "added":
return <Plus size={14} color="#4cc38a" />;
case "removed":
return <Minus size={14} color="#ff6b6b" />;
case "modified":
return <Edit size={14} color="#f2b66d" />;
default:
return null;
}
};
const getStateIcon = (state: string) => {
switch (state) {
case "published":
return <CheckCircle size={20} color="#4cc38a" />;
case "approved":
return <CheckCircle size={20} color="#4aa3ff" />;
case "rejected":
return <XCircle size={20} color="#ff6b6b" />;
case "proposed":
return <AlertCircle size={20} color="#f2b66d" />;
default:
return <Clock size={20} color="#8fa8c6" />;
}
};
const containerStyle: React.CSSProperties = {
display: "flex",
flexDirection: "column",
height: "100%",
background: "#07111f",
padding: "20px",
overflow: "auto",
};
const headerStyle: React.CSSProperties = {
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: "20px",
paddingBottom: "16px",
borderBottom: "1px solid rgba(140, 192, 255, 0.12)",
};
const titleStyle: React.CSSProperties = {
margin: 0,
color: "#ebf3ff",
fontSize: "20px",
fontWeight: "700",
};
const contentStyle: React.CSSProperties = {
display: "flex",
gap: "20px",
flex: 1,
minHeight: 0,
};
const diffPanelStyle: React.CSSProperties = {
flex: 1,
background: "rgba(9, 19, 34, 0.8)",
borderRadius: "8px",
border: "1px solid rgba(127, 208, 255, 0.12)",
padding: "16px",
overflow: "auto",
};
const commentsPanelStyle: React.CSSProperties = {
width: "320px",
background: "rgba(9, 19, 34, 0.8)",
borderRadius: "8px",
border: "1px solid rgba(127, 208, 255, 0.12)",
padding: "16px",
display: "flex",
flexDirection: "column",
};
const diffItemStyle: React.CSSProperties = {
display: "flex",
alignItems: "center",
gap: "10px",
padding: "10px 12px",
borderRadius: "6px",
background: "rgba(3, 9, 18, 0.6)",
marginBottom: "8px",
cursor: "pointer",
transition: "160ms ease",
};
const buttonStyle: React.CSSProperties = {
display: "inline-flex",
alignItems: "center",
gap: "6px",
padding: "8px 14px",
borderRadius: "6px",
border: "1px solid rgba(127, 208, 255, 0.2)",
background: "rgba(74, 163, 255, 0.1)",
color: "#ebf3ff",
fontSize: "12px",
fontWeight: "600",
cursor: "pointer",
transition: "160ms ease",
};
const textareaStyle: React.CSSProperties = {
width: "100%",
padding: "10px 12px",
borderRadius: "6px",
border: "1px solid rgba(127, 208, 255, 0.2)",
background: "rgba(3, 9, 18, 0.8)",
color: "#ebf3ff",
fontSize: "13px",
resize: "vertical",
minHeight: "80px",
};
if (!proposal) {
return (
<div style={containerStyle}>
<div style={{ color: "#8fa8c6", fontSize: "14px" }}>Loading proposal...</div>
</div>
);
}
return (
<div style={containerStyle}>
<div style={headerStyle}>
<div style={{ display: "flex", alignItems: "center", gap: "12px" }}>
{getStateIcon(proposal.state)}
<div>
<h1 style={titleStyle}>{proposal.summary}</h1>
<div style={{ color: "#8fa8c6", fontSize: "12px", marginTop: "4px" }}>
{proposal.author} {new Date(proposal.created_at).toLocaleString()}
</div>
</div>
</div>
<div style={{ display: "flex", gap: "8px" }}>
{proposal.state === "proposed" && (
<>
<button style={buttonStyle} onClick={approveProposal}>
<CheckCircle size={12} />
Approve
</button>
<button style={buttonStyle} onClick={rejectProposal}>
<XCircle size={12} />
Reject
</button>
</>
)}
{proposal.state === "approved" && (
<button style={buttonStyle} onClick={publishProposal}>
<Send size={12} />
Publish
</button>
)}
</div>
</div>
<div style={contentStyle}>
<div style={diffPanelStyle}>
<h2 style={{ margin: "0 0 16px", color: "#ebf3ff", fontSize: "14px", fontWeight: "600" }}>
<GitMerge size={16} style={{ marginRight: "8px", verticalAlign: "middle" }} />
Diff Viewer
</h2>
{diff.length === 0 ? (
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No changes detected</div>
) : (
diff.map((change, index) => (
<div
key={index}
style={{
...diffItemStyle,
border: selectedElement === change.element ? "1px solid rgba(74, 163, 255, 0.4)" : "1px solid transparent",
}}
onClick={() => setSelectedElement(change.element)}
>
{getChangeIcon(change.type)}
<div style={{ flex: 1 }}>
<div style={{ color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
{change.element}
</div>
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>
{change.type}
</div>
</div>
</div>
))
)}
<div style={{ marginTop: "20px", paddingTop: "16px", borderTop: "1px solid rgba(140, 192, 255, 0.12)" }}>
<h3 style={{ margin: "0 0 12px", color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
Impact Analysis
</h3>
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
{JSON.stringify(proposal.impact_analysis, null, 2)}
</pre>
</div>
<div style={{ marginTop: "16px" }}>
<h3 style={{ margin: "0 0 12px", color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
SHACL Validation
</h3>
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
{JSON.stringify(proposal.shacl_validation, null, 2)}
</pre>
</div>
</div>
<div style={commentsPanelStyle}>
<h2 style={{ margin: "0 0 16px", color: "#ebf3ff", fontSize: "14px", fontWeight: "600" }}>
<MessageSquare size={16} style={{ marginRight: "8px", verticalAlign: "middle" }} />
Comments ({proposal.comments.length})
</h2>
<div style={{ flex: 1, overflow: "auto", marginBottom: "12px" }}>
{proposal.comments.length === 0 ? (
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No comments yet</div>
) : (
proposal.comments.map((comment) => (
<div
key={comment.id}
style={{
padding: "10px",
background: "rgba(3, 9, 18, 0.6)",
borderRadius: "6px",
marginBottom: "8px",
}}
>
<div style={{ display: "flex", alignItems: "center", gap: "6px", marginBottom: "4px" }}>
<User size={12} color="#8fa8c6" />
<span style={{ color: "#ebf3ff", fontSize: "12px", fontWeight: "600" }}>
{comment.author}
</span>
</div>
<div style={{ color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>{comment.text}</div>
<div style={{ color: "#5a7a9a", fontSize: "10px" }}>
{new Date(comment.created_at).toLocaleString()}
</div>
</div>
))
)}
</div>
{selectedElement && (
<div>
<textarea
placeholder="Add a comment..."
value={commentText}
onChange={(e) => setCommentText(e.target.value)}
style={textareaStyle}
/>
<button style={buttonStyle} onClick={addComment} disabled={!commentText}>
<Send size={12} />
Add Comment
</button>
</div>
)}
</div>
</div>
</div>
);
}
@@ -0,0 +1,346 @@
import { useCallback, useEffect, useMemo, useState } from "react";
import type { CSSProperties } from "react";
import Editor, { type Monaco } from "@monaco-editor/react";
import { FileCode2, Loader2, Play, Shield, Wand2 } from "lucide-react";
import {
generateShacl,
loadOntologyRegistry,
loadShaclShapes,
validateShacl,
} from "./api";
import type { OntologyEntry, ShaclShapeSummary, ShaclValidationResponse } from "./types";
interface ShaclStudioProps {
onJumpToNode?: (nodeId: string) => void;
}
export function ShaclStudio({ onJumpToNode }: ShaclStudioProps) {
const [registry, setRegistry] = useState<OntologyEntry[]>([]);
const [selectedUri, setSelectedUri] = useState("");
const [shacl, setShacl] = useState("");
const [fullShacl, setFullShacl] = useState(""); // preserves complete Turtle across shape selections
const [shapes, setShapes] = useState<ShaclShapeSummary[]>([]);
const [selectedShapeId, setSelectedShapeId] = useState<string | null>(null);
const [validation, setValidation] = useState<ShaclValidationResponse | null>(null);
const [loading, setLoading] = useState(false);
const [error, setError] = useState("");
useEffect(() => {
let cancelled = false;
loadOntologyRegistry()
.then((entries) => {
if (cancelled) return;
setRegistry(entries);
setSelectedUri((current) => current || entries[0]?.uri || "");
})
.catch((err) => {
if (!cancelled) setError(err instanceof Error ? err.message : "Could not load ontology registry.");
});
return () => {
cancelled = true;
};
}, []);
const loadShapes = useCallback(async (uri: string) => {
if (!uri) return;
setLoading(true);
setError("");
try {
const data = await loadShaclShapes(uri);
setShapes(data.shapes);
const turtle = data.shacl_turtle;
setFullShacl(turtle);
setShacl((current) => current || turtle);
setSelectedShapeId(null);
setValidation(null);
} catch (err) {
setError(err instanceof Error ? err.message : "Could not load SHACL shapes.");
} finally {
setLoading(false);
}
}, []);
useEffect(() => {
setShacl("");
setFullShacl("");
setSelectedShapeId(null);
void loadShapes(selectedUri);
}, [selectedUri, loadShapes]);
const handleGenerate = useCallback(async () => {
if (!selectedUri) return;
setLoading(true);
setError("");
try {
const data = await generateShacl(selectedUri, "strict");
setFullShacl(data.shacl_turtle);
setShacl(data.shacl_turtle);
setSelectedShapeId(null);
const shapeData = await loadShaclShapes(selectedUri);
setShapes(shapeData.shapes);
setValidation(null);
} catch (err) {
setError(err instanceof Error ? err.message : "Could not generate SHACL.");
} finally {
setLoading(false);
}
}, [selectedUri]);
const handleSelectShape = useCallback((shapeId: string) => {
setSelectedShapeId(shapeId);
// Extract the Turtle block for this shape from the full SHACL so the editor
// pre-populates with the selected shape's definition.
const src = fullShacl || shacl;
const normalised = src.replace(/\r\n/g, "\n");
// Split on blank lines to isolate statement groups.
const blocks = normalised.split(/\n{2,}/).filter((b) => b.trim());
const match = blocks.find((b) => {
const first = b.trimStart();
return first.startsWith(shapeId + " ") || first.startsWith(shapeId + "\n") || first.startsWith(shapeId + "\t");
});
if (match) {
setShacl(match.trim());
}
}, [fullShacl, shacl]);
const handleShowAllShapes = useCallback(() => {
setSelectedShapeId(null);
setShacl(fullShacl);
}, [fullShacl]);
const handleValidate = async () => {
if (!selectedUri || !shacl.trim()) return;
setLoading(true);
setError("");
try {
setValidation(await validateShacl(selectedUri, shacl));
} catch (err) {
setError(err instanceof Error ? err.message : "Could not validate SHACL.");
} finally {
setLoading(false);
}
};
const groupedShapes = useMemo(() => {
const groups = new Map<string, ShaclShapeSummary[]>();
for (const shape of shapes) {
const key = shape.target_class || "Untargeted shapes";
groups.set(key, [...(groups.get(key) || []), shape]);
}
return Array.from(groups.entries());
}, [shapes]);
const beforeMount = useCallback((monaco: Monaco) => {
if (!monaco.languages.getLanguages().some((language: { id: string }) => language.id === "turtle")) {
monaco.languages.register({ id: "turtle", extensions: [".ttl"], mimetypes: ["text/turtle"] });
monaco.languages.setMonarchTokensProvider("turtle", {
keywords: ["@prefix", "@base", "a"],
tokenizer: {
root: [
[/#[^\n]*/, "comment"],
[/"(?:[^"\\]|\\.)*"(?:@[a-zA-Z-]+|\\^\\^[^\s,;.]+)?/, "string"],
[/'(?:[^'\\]|\\.)*'(?:@[a-zA-Z-]+|\\^\\^[^\s,;.]+)?/, "string"],
[/"""[\s\S]*?"""/, "string"],
[/<[^>]*>/, "type.identifier"],
[/\b(?:@prefix|@base|a)\b/, "keyword"],
[/\b(?:sh|xsd|owl|rdf|rdfs|skos):[\w]+/, "variable"],
[/[a-zA-Z_][\w-]*:[\w]+/, "namespace"],
[/[;,.]/, "delimiter"],
[/\d+(?:\.\d+)?/, "number"],
],
},
});
}
monaco.editor.defineTheme("shacl-dark", {
base: "vs-dark",
inherit: true,
rules: [
{ token: "keyword", foreground: "9ee8d7" },
{ token: "string", foreground: "f2b66d" },
{ token: "comment", foreground: "4a6070", fontStyle: "italic" },
{ token: "type.identifier", foreground: "7ce7d3" },
{ token: "variable", foreground: "d2a8ff" },
{ token: "namespace", foreground: "a5d6ff" },
{ token: "number", foreground: "79c0ff" },
{ token: "delimiter", foreground: "8fa8c6" },
],
colors: {
"editor.background": "#050b13",
"editor.foreground": "#d7e7f8",
"editorLineNumber.foreground": "#41536b",
},
});
}, []);
return (
<div style={pageStyle}>
<section style={heroStyle}>
<div>
<div style={kickerStyle}><Shield size={14} /> SHACL Studio</div>
<h2 style={titleStyle}>Generate, edit, and validate shapes</h2>
<p style={textStyle}>
Create strict SHACL Turtle from ontology structure, inspect shape targets,
run validation, and jump from violations back into the graph.
</p>
</div>
<div style={selectorShellStyle}>
<label style={labelStyle}>Ontology</label>
<select style={inputStyle} value={selectedUri} onChange={(event) => setSelectedUri(event.target.value)}>
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
</select>
</div>
</section>
{error ? <div style={errorStyle}>{error}</div> : null}
<div style={gridStyle}>
<section style={cardStyle}>
<div style={panelHeaderStyle}>
<h3 style={sectionTitleStyle}>Shape library</h3>
<div style={{ display: "flex", gap: 8, alignItems: "center" }}>
<span style={countBadgeStyle}>{shapes.length} shapes</span>
{selectedShapeId ? (
<button style={smallButtonStyle} onClick={handleShowAllShapes}>View all</button>
) : null}
</div>
</div>
<div style={shapeListStyle}>
{groupedShapes.map(([target, items]) => (
<div key={target} style={shapeGroupStyle}>
<div style={shapeTargetStyle}>{target}</div>
{items.map((shape) => {
const isSelected = selectedShapeId === shape.id;
return (
<button
key={shape.id}
style={{
...shapeRowStyle,
cursor: "pointer",
background: isSelected ? "rgba(124,231,211,0.1)" : "rgba(255,255,255,0.03)",
border: isSelected ? "1px solid rgba(124,231,211,0.35)" : "1px solid rgba(127,208,255,0.08)",
textAlign: "left",
width: "100%",
}}
onClick={() => handleSelectShape(shape.id)}
title="Click to load this shape into the editor"
>
<FileCode2 size={14} color={isSelected ? "#7ce7d3" : "#9ee8d7"} />
<div>
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{shape.id}</div>
<div style={mutedStyle}>
{shape.constraint_count} constraints
{shape.constraints.length ? ` · ${shape.constraints.join(", ")}` : ""}
</div>
</div>
<span style={violationBadgeStyle}>{shape.violation_count}</span>
</button>
);
})}
</div>
))}
{!shapes.length ? <p style={mutedStyle}>No shapes generated yet.</p> : null}
</div>
</section>
<section style={editorShellStyle}>
<div style={panelHeaderStyle}>
<h3 style={sectionTitleStyle}>
{selectedShapeId ? selectedShapeId : "Turtle shape editor"}
</h3>
<div style={{ display: "flex", gap: 8 }}>
<button style={secondaryButtonStyle} disabled={loading} onClick={handleGenerate}><Wand2 size={14} /> Generate strict</button>
<button style={primaryButtonStyle} disabled={loading || !shacl.trim()} onClick={handleValidate}>
{loading ? <Loader2 size={14} className="spin" /> : <Play size={14} />}
Validate
</button>
</div>
</div>
<div style={editorFrameStyle}>
<Editor
height="100%"
language="turtle"
theme="shacl-dark"
beforeMount={beforeMount}
value={shacl}
onChange={(value) => setShacl(value || "")}
options={{
minimap: { enabled: false },
fontSize: 13,
fontFamily: "JetBrains Mono, monospace",
wordWrap: "on",
scrollBeyondLastLine: false,
}}
/>
</div>
</section>
</div>
<section style={cardStyle}>
<div style={panelHeaderStyle}>
<h3 style={sectionTitleStyle}>Validation report</h3>
{validation ? <span style={validationBadgeStyle(validation.status, validation.conforms)}>{validation.status}{validation.conforms ? " · conforms" : ""}</span> : null}
</div>
{validation ? (
<>
<p style={textStyle}>{validation.message}</p>
<div style={shapeListStyle}>
{validation.violations.map((violation, index) => {
const nodeId = violation.focus_node || violation.node;
return (
<div key={`${violation.node}-${violation.path}-${index}`} style={violationRowStyle}>
<div>
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{violation.message}</div>
<div style={mutedStyle}>{violation.severity} {violation.path ? `· ${violation.path}` : ""}</div>
{nodeId ? <div style={monoStyle}>{nodeId}</div> : null}
</div>
{nodeId ? (
<button style={smallButtonStyle} onClick={() => onJumpToNode?.(nodeId)}>
Jump to Node
</button>
) : null}
</div>
);
})}
{!validation.violations.length ? <p style={mutedStyle}>No validation violations returned.</p> : null}
</div>
</>
) : (
<p style={mutedStyle}>Generate or edit SHACL Turtle, then run validation.</p>
)}
</section>
</div>
);
}
function validationBadgeStyle(status: string, conforms: boolean): CSSProperties {
const color = status === "unavailable" ? "#f2b66d" : conforms ? "#7ce7d3" : "#ff9daf";
return { color, border: `1px solid ${color}35`, background: `${color}14`, borderRadius: 999, padding: "4px 9px", fontSize: 11, fontWeight: 900, textTransform: "uppercase" };
}
const pageStyle: CSSProperties = { height: "100%", overflow: "auto", padding: 22, display: "flex", flexDirection: "column", gap: 16 };
const heroStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 18, padding: 22, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 22, background: "linear-gradient(135deg, rgba(11,25,42,0.94), rgba(7,14,25,0.9))" };
const kickerStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", color: "#9ee8d7", fontSize: 11, fontWeight: 900, letterSpacing: "0.12em", textTransform: "uppercase" };
const titleStyle: CSSProperties = { margin: "8px 0", color: "#ebf3ff", fontSize: 26, letterSpacing: "-0.04em" };
const textStyle: CSSProperties = { margin: 0, color: "#8fa8c6", lineHeight: 1.6, maxWidth: 680 };
const selectorShellStyle: CSSProperties = { minWidth: 320 };
const labelStyle: CSSProperties = { display: "block", color: "#6a7f97", fontSize: 11, fontWeight: 800, margin: "0 0 6px", textTransform: "uppercase", letterSpacing: "0.08em" };
const inputStyle: CSSProperties = { width: "100%", boxSizing: "border-box", border: "1px solid rgba(127,208,255,0.14)", borderRadius: 12, padding: "10px 12px", background: "rgba(3,9,18,0.8)", color: "#ebf3ff" };
const gridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "360px minmax(0, 1fr)", gap: 16, minHeight: 560 };
const cardStyle: CSSProperties = { padding: 18, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 20, background: "rgba(9,19,34,0.78)" };
const editorShellStyle: CSSProperties = { ...cardStyle, display: "flex", flexDirection: "column", minHeight: 560 };
const panelHeaderStyle: CSSProperties = { display: "flex", alignItems: "center", justifyContent: "space-between", gap: 12, marginBottom: 12 };
const sectionTitleStyle: CSSProperties = { margin: 0, color: "#ebf3ff", fontSize: 16 };
const countBadgeStyle: CSSProperties = { color: "#9ee8d7", border: "1px solid rgba(158,232,215,0.2)", borderRadius: 999, padding: "4px 9px", fontSize: 11, fontWeight: 900 };
const shapeListStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 10 };
const shapeGroupStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 6 };
const shapeTargetStyle: CSSProperties = { color: "#6a7f97", fontSize: 11, fontWeight: 900, textTransform: "uppercase", letterSpacing: "0.08em" };
const shapeRowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "18px 1fr auto", gap: 10, padding: 10, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
const violationBadgeStyle: CSSProperties = { color: "#f2b66d", fontWeight: 900 };
const editorFrameStyle: CSSProperties = { flex: 1, minHeight: 0, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 16, overflow: "hidden" };
const primaryButtonStyle: CSSProperties = { border: "1px solid rgba(124,231,211,0.35)", borderRadius: 12, padding: "9px 11px", background: "linear-gradient(135deg, rgba(20,151,136,0.55), rgba(74,163,255,0.35))", color: "#ebf3ff", fontWeight: 900, cursor: "pointer", display: "inline-flex", alignItems: "center", gap: 8 };
const secondaryButtonStyle: CSSProperties = { ...primaryButtonStyle, background: "rgba(127,208,255,0.08)", borderColor: "rgba(127,208,255,0.18)" };
const violationRowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr auto", gap: 12, padding: 12, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
const smallButtonStyle: CSSProperties = { border: "1px solid rgba(127,208,255,0.16)", borderRadius: 10, padding: "7px 9px", background: "rgba(127,208,255,0.08)", color: "#ebf3ff", cursor: "pointer", fontWeight: 800 };
const monoStyle: CSSProperties = { marginTop: 4, color: "#6a7f97", fontSize: 11, fontFamily: "JetBrains Mono, monospace", wordBreak: "break-all" };
const mutedStyle: CSSProperties = { margin: 0, color: "#6a7f97", fontSize: 12, lineHeight: 1.5 };
const errorStyle: CSSProperties = { padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" };
@@ -0,0 +1,475 @@
import { useCallback, useEffect, useState } from "react";
import {
Layers,
GitMerge,
Clock,
FileText,
CheckCircle,
XCircle,
AlertCircle,
Send,
X,
ArrowRight,
Scale,
} from "lucide-react";
interface VersionEntry {
version_id: string;
ontology_uri: string;
state: "draft" | "published";
author: string;
date: string;
diff_summary: Record<string, any>;
}
interface Proposal {
proposal_id: string;
draft_id: string;
ontology_uri: string;
summary: string;
author: string;
reviewer: string | null;
state: "draft" | "proposed" | "approved" | "published" | "rejected";
impact_analysis: Record<string, any>;
shacl_validation: Record<string, any>;
created_at: string;
updated_at: string;
comments: Record<string, any>[];
}
export function VersionsTab() {
const [ontologyUri, setOntologyUri] = useState<string>("");
const [versions, setVersions] = useState<VersionEntry[]>([]);
const [proposals, setProposals] = useState<Proposal[]>([]);
const [selectedProposal, setSelectedProposal] = useState<Proposal | null>(null);
const [showProposalModal, setShowProposalModal] = useState(false);
const [showCompareModal, setShowCompareModal] = useState(false);
const [comparePair, setComparePair] = useState<{ v1: string; v2: string } | null>(null);
const [compareResult, setCompareResult] = useState<Record<string, any> | null>(null);
const [isLoading, setIsLoading] = useState(false);
const loadVersions = useCallback(async () => {
if (!ontologyUri) return;
try {
const response = await fetch(`/api/ontology/versions/${encodeURIComponent(ontologyUri)}`);
if (response.ok) {
const data = await response.json();
setVersions(data);
}
} catch (error) {
console.error("Failed to load versions:", error);
}
}, [ontologyUri]);
const loadProposals = useCallback(async () => {
try {
const response = await fetch("/api/ontology/proposals");
if (response.ok) {
const data = await response.json();
setProposals(data);
}
} catch (error) {
console.error("Failed to load proposals:", error);
}
}, []);
useEffect(() => {
loadVersions();
loadProposals();
}, [loadVersions, loadProposals]);
const approveProposal = useCallback(async (proposalId: string) => {
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}/approve`, {
method: "POST",
});
if (response.ok) {
alert("Proposal approved");
loadProposals();
}
} catch (error) {
console.error("Failed to approve proposal:", error);
alert("Failed to approve proposal");
}
}, [loadProposals]);
const rejectProposal = useCallback(async (proposalId: string) => {
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}/reject`, {
method: "POST",
});
if (response.ok) {
alert("Proposal rejected");
loadProposals();
}
} catch (error) {
console.error("Failed to reject proposal:", error);
alert("Failed to reject proposal");
}
}, [loadProposals]);
const publishProposal = useCallback(async (proposalId: string) => {
try {
const response = await fetch(`/api/ontology/proposals/${proposalId}/publish`, {
method: "POST",
});
if (response.ok) {
alert("Proposal published");
loadProposals();
loadVersions();
}
} catch (error) {
console.error("Failed to publish proposal:", error);
alert("Failed to publish proposal");
}
}, [loadProposals, loadVersions]);
const runVersionComparison = useCallback(async () => {
if (!comparePair || !ontologyUri) return;
setIsLoading(true);
try {
const response = await fetch(`/api/ontology/versions/${encodeURIComponent(ontologyUri)}/compare`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
version1: comparePair.v1,
version2: comparePair.v2,
}),
});
if (response.ok) {
const data = await response.json();
setCompareResult(data);
}
} catch (error) {
console.error("Failed to compare versions:", error);
alert("Failed to compare versions");
} finally {
setIsLoading(false);
}
}, [comparePair, ontologyUri]);
const getStateIcon = (state: string) => {
switch (state) {
case "published":
return <CheckCircle size={16} color="#4cc38a" />;
case "approved":
return <CheckCircle size={16} color="#4aa3ff" />;
case "rejected":
return <XCircle size={16} color="#ff6b6b" />;
case "proposed":
return <AlertCircle size={16} color="#f2b66d" />;
default:
return <Clock size={16} color="#8fa8c6" />;
}
};
const containerStyle: React.CSSProperties = {
display: "flex",
flexDirection: "column",
height: "100%",
background: "#07111f",
padding: "20px",
overflow: "auto",
};
const headerStyle: React.CSSProperties = {
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: "20px",
};
const titleStyle: React.CSSProperties = {
margin: 0,
color: "#ebf3ff",
fontSize: "20px",
fontWeight: "700",
};
const sectionStyle: React.CSSProperties = {
marginBottom: "24px",
};
const sectionTitleStyle: React.CSSProperties = {
margin: "0 0 12px",
color: "#ebf3ff",
fontSize: "14px",
fontWeight: "600",
display: "flex",
alignItems: "center",
gap: "8px",
};
const listStyle: React.CSSProperties = {
display: "flex",
flexDirection: "column",
gap: "8px",
};
const itemStyle: React.CSSProperties = {
display: "flex",
alignItems: "center",
gap: "12px",
padding: "12px 16px",
borderRadius: "8px",
background: "rgba(9, 19, 34, 0.8)",
border: "1px solid rgba(127, 208, 255, 0.12)",
transition: "160ms ease",
};
const modalOverlayStyle: React.CSSProperties = {
position: "fixed",
inset: 0,
background: "rgba(0, 0, 0, 0.7)",
display: "flex",
alignItems: "center",
justifyContent: "center",
zIndex: 1000,
};
const modalStyle: React.CSSProperties = {
background: "rgba(9, 19, 34, 0.95)",
border: "1px solid rgba(127, 208, 255, 0.2)",
borderRadius: "12px",
padding: "24px",
minWidth: "480px",
maxWidth: "640px",
maxHeight: "80vh",
overflow: "auto",
backdropFilter: "blur(18px)",
};
const buttonStyle: React.CSSProperties = {
display: "inline-flex",
alignItems: "center",
gap: "6px",
padding: "8px 14px",
borderRadius: "6px",
border: "1px solid rgba(127, 208, 255, 0.2)",
background: "rgba(74, 163, 255, 0.1)",
color: "#ebf3ff",
fontSize: "12px",
fontWeight: "600",
cursor: "pointer",
transition: "160ms ease",
};
const inputStyle: React.CSSProperties = {
width: "100%",
padding: "10px 12px",
borderRadius: "6px",
border: "1px solid rgba(127, 208, 255, 0.2)",
background: "rgba(3, 9, 18, 0.8)",
color: "#ebf3ff",
fontSize: "13px",
marginBottom: "12px",
};
return (
<div style={containerStyle}>
<div style={headerStyle}>
<h1 style={titleStyle}>Versions & Change Proposals</h1>
<input
type="text"
placeholder="Ontology URI"
value={ontologyUri}
onChange={(e) => setOntologyUri(e.target.value)}
style={{ ...inputStyle, width: "300px", marginBottom: 0 }}
/>
</div>
<div style={sectionStyle}>
<h2 style={sectionTitleStyle}>
<Layers size={16} />
Version History
</h2>
<div style={listStyle}>
{versions.length === 0 ? (
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No versions found</div>
) : (
versions.map((version) => (
<div key={version.version_id} style={itemStyle}>
{getStateIcon(version.state)}
<div style={{ flex: 1 }}>
<div style={{ color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
{version.version_id}
</div>
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>
{version.author} {new Date(version.date).toLocaleDateString()}
</div>
</div>
<button
style={buttonStyle}
onClick={() => {
setComparePair({ v1: version.version_id, v2: versions[0]?.version_id || "" });
setShowCompareModal(true);
}}
>
<ArrowRight size={12} />
Compare
</button>
</div>
))
)}
</div>
</div>
<div style={sectionStyle}>
<h2 style={sectionTitleStyle}>
<GitMerge size={16} />
Change Proposals
</h2>
<div style={listStyle}>
{proposals.length === 0 ? (
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No proposals found</div>
) : (
proposals.map((proposal) => (
<div key={proposal.proposal_id} style={itemStyle}>
{getStateIcon(proposal.state)}
<div style={{ flex: 1 }}>
<div style={{ color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
{proposal.summary}
</div>
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>
{proposal.author} {new Date(proposal.created_at).toLocaleDateString()}
</div>
</div>
{proposal.state === "proposed" && (
<div style={{ display: "flex", gap: "6px" }}>
<button style={buttonStyle} onClick={() => approveProposal(proposal.proposal_id)}>
<CheckCircle size={12} />
Approve
</button>
<button style={buttonStyle} onClick={() => rejectProposal(proposal.proposal_id)}>
<XCircle size={12} />
Reject
</button>
</div>
)}
{proposal.state === "approved" && (
<button style={buttonStyle} onClick={() => publishProposal(proposal.proposal_id)}>
<Send size={12} />
Publish
</button>
)}
<button
style={buttonStyle}
onClick={() => {
setSelectedProposal(proposal);
setShowProposalModal(true);
}}
>
<FileText size={12} />
Details
</button>
</div>
))
)}
</div>
</div>
{showProposalModal && selectedProposal && (
<div style={modalOverlayStyle} onClick={() => setShowProposalModal(false)}>
<div style={modalStyle} onClick={(e) => e.stopPropagation()}>
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: "16px" }}>
<h3 style={{ margin: 0, color: "#ebf3ff", fontSize: "16px" }}>Proposal Details</h3>
<button onClick={() => setShowProposalModal(false)} style={{ background: "none", border: "none", color: "#8fa8c6", cursor: "pointer" }}>
<X size={18} />
</button>
</div>
<div style={{ marginBottom: "12px" }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
Summary
</label>
<div style={{ color: "#ebf3ff", fontSize: "13px" }}>{selectedProposal.summary}</div>
</div>
<div style={{ marginBottom: "12px" }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
State
</label>
<div style={{ display: "flex", alignItems: "center", gap: "6px", color: "#ebf3ff", fontSize: "13px" }}>
{getStateIcon(selectedProposal.state)}
{selectedProposal.state}
</div>
</div>
<div style={{ marginBottom: "12px" }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
Impact Analysis
</label>
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
{JSON.stringify(selectedProposal.impact_analysis, null, 2)}
</pre>
</div>
<div style={{ marginBottom: "12px" }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
SHACL Validation
</label>
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
{JSON.stringify(selectedProposal.shacl_validation, null, 2)}
</pre>
</div>
<div>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
Comments ({selectedProposal.comments.length})
</label>
<div style={{ maxHeight: "120px", overflow: "auto" }}>
{selectedProposal.comments.map((comment) => (
<div key={comment.id} style={{ padding: "8px", background: "rgba(3, 9, 18, 0.6)", borderRadius: "4px", marginBottom: "6px" }}>
<div style={{ color: "#ebf3ff", fontSize: "12px", fontWeight: "600" }}>{comment.author}</div>
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>{comment.text}</div>
</div>
))}
</div>
</div>
</div>
</div>
)}
{showCompareModal && comparePair && (
<div style={modalOverlayStyle} onClick={() => setShowCompareModal(false)}>
<div style={modalStyle} onClick={(e) => e.stopPropagation()}>
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: "16px" }}>
<h3 style={{ margin: 0, color: "#ebf3ff", fontSize: "16px" }}>Compare Versions</h3>
<button onClick={() => setShowCompareModal(false)} style={{ background: "none", border: "none", color: "#8fa8c6", cursor: "pointer" }}>
<X size={18} />
</button>
</div>
<div style={{ display: "flex", gap: "12px", marginBottom: "16px" }}>
<div style={{ flex: 1 }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
Version 1
</label>
<input
type="text"
value={comparePair.v1}
onChange={(e) => setComparePair({ ...comparePair, v1: e.target.value })}
style={inputStyle}
/>
</div>
<div style={{ flex: 1 }}>
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
Version 2
</label>
<input
type="text"
value={comparePair.v2}
onChange={(e) => setComparePair({ ...comparePair, v2: e.target.value })}
style={inputStyle}
/>
</div>
</div>
<button style={buttonStyle} onClick={runVersionComparison} disabled={isLoading}>
<Scale size={12} />
{isLoading ? "Comparing..." : "Compare"}
</button>
{compareResult && (
<pre style={{ marginTop: "16px", background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
{JSON.stringify(compareResult, null, 2)}
</pre>
)}
</div>
</div>
)}
</div>
);
}
@@ -0,0 +1,104 @@
import type {
AlignmentRelation,
AlignmentSuggestion,
OntologyAlignment,
OntologyEntry,
OntologyHealthResponse,
ShaclGenerateResponse,
ShaclShapesResponse,
ShaclValidationResponse,
} from "./types";
async function parseResponse<T>(response: Response): Promise<T> {
if (!response.ok) {
let detail = `Request failed with status ${response.status}`;
try {
const body = await response.json();
detail = body.detail || detail;
} catch {
// Keep the generic HTTP detail.
}
throw new Error(detail);
}
return response.json() as Promise<T>;
}
export async function loadOntologyRegistry(): Promise<OntologyEntry[]> {
return parseResponse<OntologyEntry[]>(await fetch("/api/ontology/registry"));
}
export async function loadAlignments(uri?: string): Promise<OntologyAlignment[]> {
const query = uri ? `?uri=${encodeURIComponent(uri)}` : "";
return parseResponse<OntologyAlignment[]>(await fetch(`/api/ontology/alignments${query}`));
}
export async function saveAlignment(payload: {
source_uri: string;
target_uri: string;
relation: AlignmentRelation;
confidence: number;
provenance?: string;
source?: string;
reviewer?: string;
}): Promise<OntologyAlignment> {
return parseResponse<OntologyAlignment>(
await fetch("/api/ontology/alignments", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(payload),
}),
);
}
export async function removeAlignment(id: string): Promise<void> {
await parseResponse<{ status: string }>(
await fetch(`/api/ontology/alignments?id=${encodeURIComponent(id)}`, { method: "DELETE" }),
);
}
export async function suggestAlignments(payload: {
source_ontology_uri?: string;
target_ontology_uri?: string;
threshold: number;
limit: number;
}): Promise<AlignmentSuggestion[]> {
return parseResponse<AlignmentSuggestion[]>(
await fetch("/api/ontology/suggest-alignments", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(payload),
}),
);
}
export async function loadOntologyHealth(uri: string): Promise<OntologyHealthResponse> {
return parseResponse<OntologyHealthResponse>(
await fetch(`/api/ontology/health?uri=${encodeURIComponent(uri)}`),
);
}
export async function generateShacl(uri: string, qualityTier: "standard" | "strict" = "strict"): Promise<ShaclGenerateResponse> {
return parseResponse<ShaclGenerateResponse>(
await fetch("/api/ontology/shacl/generate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ uri, quality_tier: qualityTier }),
}),
);
}
export async function loadShaclShapes(uri: string): Promise<ShaclShapesResponse> {
return parseResponse<ShaclShapesResponse>(
await fetch(`/api/ontology/shacl/shapes?uri=${encodeURIComponent(uri)}`),
);
}
export async function validateShacl(uri: string, shaclTurtle: string): Promise<ShaclValidationResponse> {
return parseResponse<ShaclValidationResponse>(
await fetch("/api/ontology/shacl/validate", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ uri, shacl_turtle: shaclTurtle }),
}),
);
}
@@ -7,7 +7,12 @@ import {
Shield,
Sliders,
} from "lucide-react";
import { AlignmentsTab } from "./AlignmentsTab";
import { HealthTab } from "./HealthTab";
import { OntologyManager } from "./OntologyManager";
import { OntologyEditor } from "./OntologyEditor";
import { ShaclStudio } from "./ShaclStudio";
import { VersionsTab } from "./VersionsTab";
export type OntologyHubTab =
| "registry"
@@ -76,7 +81,11 @@ function ComingSoonStub({
);
}
export function OntologyWorkspace() {
interface OntologyWorkspaceProps {
onJumpToGraphNode?: (nodeId: string) => void;
}
export function OntologyWorkspace({ onJumpToGraphNode }: OntologyWorkspaceProps) {
const [activeTab, setActiveTab] = useState<OntologyHubTab>(readTabParam);
useEffect(() => {
@@ -87,55 +96,28 @@ export function OntologyWorkspace() {
setActiveTab(tab);
}, []);
const handleFixInEditor = useCallback((entityUri: string) => {
const params = new URLSearchParams(window.location.search);
params.set(TAB_PARAM, "editor");
params.set("ontologyEntity", entityUri);
window.history.replaceState(null, "", `?${params.toString()}`);
setActiveTab("editor");
}, []);
const renderTab = () => {
switch (activeTab) {
case "registry":
return <OntologyManager />;
case "editor":
return (
<ComingSoonStub
icon={Sliders}
title="Visual Ontology Editor"
description="Visually edit classes, properties, individuals, restrictions, axioms, and SKOS metadata. Create and propose schema changes through a governed draft workflow."
badge="Subissue 2"
/>
);
return <OntologyEditor />;
case "versions":
return (
<ComingSoonStub
icon={Layers}
title="Versions & Change Proposals"
description="View version history, compare schema diffs, submit change proposals, and manage the review-to-publish lifecycle."
badge="Subissue 2"
/>
);
return <VersionsTab />;
case "alignments":
return (
<ComingSoonStub
icon={GitMerge}
title="Cross-Ontology Alignments"
description="Manage mappings between ontologies, review suggested alignments from embedding-assisted similarity, and publish alignment sets."
badge="Subissue 3"
/>
);
return <AlignmentsTab />;
case "health":
return (
<ComingSoonStub
icon={HeartPulse}
title="Ontology Health Dashboard"
description="Score completeness, consistency, SHACL conformance, alignment coverage, and documentation quality across all loaded ontologies."
badge="Subissue 3"
/>
);
return <HealthTab onFixInEditor={handleFixInEditor} />;
case "shacl":
return (
<ComingSoonStub
icon={Shield}
title="SHACL Studio"
description="Generate, edit, and validate SHACL shapes. Preview constraint violations against the active graph before publishing."
badge="Subissue 3"
/>
);
return <ShaclStudio onJumpToNode={onJumpToGraphNode} />;
}
};
@@ -0,0 +1,119 @@
export interface OntologyEntry {
uri: string;
name: string;
description?: string;
format: string;
status: "published" | "draft" | "external";
source_url?: string;
version?: string;
class_count: number;
concept_count: number;
property_count: number;
loaded_at: string;
enabled: boolean;
tags: string[];
}
export type AlignmentRelation =
| "owl:equivalentClass"
| "owl:equivalentProperty"
| "skos:exactMatch"
| "skos:closeMatch"
| "skos:broadMatch"
| "skos:narrowMatch"
| "skos:relatedMatch";
export interface OntologyAlignment {
id: string;
source_uri: string;
source_label: string;
target_uri: string;
target_label: string;
relation: AlignmentRelation;
predicate_uri: string;
confidence: number;
provenance?: string;
source?: string;
reviewer?: string;
created_at: string;
updated_at: string;
}
export interface AlignmentSuggestion {
source_uri: string;
source_label: string;
target_uri: string;
target_label: string;
relation: AlignmentRelation;
score: number;
label_similarity: number;
embedding_similarity?: number | null;
reason: string;
}
export interface HealthDimension {
key: string;
label: string;
score: number;
status: "ok" | "warning" | "critical" | "unavailable";
detail: string;
}
export interface HealthIssue {
id: string;
severity: "info" | "warning" | "critical";
category: string;
entity_uri?: string;
entity_label?: string;
message: string;
action?: string;
}
export interface OntologyHealthResponse {
uri: string;
name: string;
total_score: number;
dimensions: HealthDimension[];
issues: HealthIssue[];
generated_at: string;
}
export interface ShaclShapeSummary {
id: string;
target_class?: string;
constraint_count: number;
constraints: string[];
violation_count: number;
}
export interface ShaclViolation {
node?: string;
path?: string;
severity: string;
message: string;
focus_node?: string;
source_shape?: string;
}
export interface ShaclGenerateResponse {
uri: string;
shacl_turtle: string;
shape_count: number;
generated_at: string;
}
export interface ShaclShapesResponse {
uri: string;
shapes: ShaclShapeSummary[];
shacl_turtle: string;
generated_at: string;
}
export interface ShaclValidationResponse {
uri?: string;
conforms: boolean;
status: "success" | "unavailable" | "error";
message: string;
violations: ShaclViolation[];
report_text?: string;
}
+6 -4
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "semantica"
version = "0.4.0"
version = "0.5.0"
description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering"
readme = "README.md"
license = { text = "MIT" }
@@ -104,6 +104,7 @@ parse-docling = ["docling>=1.0.0"]
# ---- Database Connectors ----
db-snowflake = ["snowflake-connector-python>=3.0.0", "cryptography>=3.4.0"]
db-arrow = ["pyarrow>=10.0.0"]
ingest-parquet = ["pyarrow>=10.0.0"]
db-all = [
"semantica[db-snowflake,db-arrow]"
@@ -211,10 +212,10 @@ explorer-lite = [
"streamlit-agraph>=0.0.45"
]
# Everything
# Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
all = [
"semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,explorer]",
"semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,agno]"
"semantica[dev,viz,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,explorer]",
"semantica[dev,viz,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,agno]"
]
# ---------------- ENTRYPOINTS ----------------
@@ -223,6 +224,7 @@ semantica = "semantica.cli:main"
semantica-server = "semantica.server:main"
semantica-worker = "semantica.worker:main"
semantica-explorer = "semantica.explorer:main"
semantica-mcp = "semantica.mcp_server:main"
# ---------------- TOOLING ----------------
[tool.setuptools.packages.find]
+3 -3
View File
@@ -1,9 +1,9 @@
mkdocs>=1.5.0
mkdocs>=1.6.1
mkdocs-material>=9.7.6
mkdocs-minify-plugin>=0.7.0
mkdocs-mermaid2-plugin>=1.0.0
mkdocs-mermaid2-plugin>=1.2.3
pymdown-extensions>=10.21.2
mkdocstrings[python]>=0.24.0
mkdocs-jupyter>=0.24.0
mkdocs-jupyter>=0.26.3
+1 -1
View File
@@ -10,7 +10,7 @@ Main exports:
- Config: Configuration management
"""
__version__ = "0.4.0"
__version__ = "0.5.0"
__author__ = "Semantica Contributors"
__license__ = "MIT"
+39 -53
View File
@@ -133,48 +133,6 @@ class ConflictDetector:
self.detected_conflicts: Dict[str, Conflict] = {}
def detect_conflicts(
self,
entities: Union[List[Dict[str, Any]], Dict[str, Any]],
method: str = "entity",
property_name: Optional[str] = None,
entity_type: Optional[str] = None,
**kwargs,
) -> List[Conflict]:
"""
Detect conflicts using the specified method (convenience method).
Args:
entities: Entities to check (List of dicts or a KG dict)
method: Detection method ("entity", "value", "type", "relationship", "temporal", "logical")
property_name: Property name for "value" method
entity_type: Optional entity type filter
**kwargs: Additional arguments
Returns:
List of detected conflicts
"""
# If passed a KG dict, extract entities
if isinstance(entities, dict) and "entities" in entities:
entities = entities["entities"]
if method == "value":
if not property_name:
raise ValueError("property_name is required for value conflict detection")
return self.detect_value_conflicts(entities, property_name, entity_type)
elif method == "type":
return self.detect_type_conflicts(entities)
elif method == "relationship":
relationships = kwargs.get("relationships", [])
return self.detect_relationship_conflicts(relationships)
elif method == "temporal":
return self.detect_temporal_conflicts(entities)
elif method == "logical":
return self.detect_logical_conflicts(entities)
else:
# Default to entity-wide detection
return self.detect_entity_conflicts(entities, entity_type)
def detect_value_conflicts(
self,
entities: Union[List[Dict[str, Any]], Dict[str, Any]],
@@ -596,11 +554,6 @@ class ConflictDetector:
tracking_id, status="failed", message=str(e)
)
raise
for field_name in fields_to_check:
conflicts = self.detect_value_conflicts(entities, field_name, entity_type)
all_conflicts.extend(conflicts)
return all_conflicts
def _calculate_conflict_confidence(
self, values: List[Any], sources: List[Dict[str, Any]]
@@ -1252,20 +1205,25 @@ class ConflictDetector:
def detect_conflicts(
self,
entities: Union[List[Dict[str, Any]], Dict[str, Any]],
method: str = "all",
property_name: Optional[str] = None,
entity_type: Optional[str] = None,
**kwargs,
) -> List[Conflict]:
"""
Detect all conflicts for entities (general method).
This method detects all types of conflicts: value, type, relationship,
temporal, and logical conflicts.
Detect conflicts using the specified method.
Args:
entities: List of entity dictionaries or Graph dictionary (containing "entities" key)
entities: List of entity dictionaries or Graph dictionary
method: Detection method "all" (default), "value", "property", "type",
"relationship", "temporal", "logical", or "entity"
property_name: Property name required for ``method="value"`` and ``method="property"``
entity_type: Optional entity type filter
**kwargs: Extra arguments forwarded to the underlying method
(e.g. ``relationships=`` for ``method="relationship"``)
Returns:
List of all detected conflicts
List of detected conflicts
"""
# Handle graph dictionary input
if isinstance(entities, dict):
@@ -1275,6 +1233,34 @@ class ConflictDetector:
# If it's a single entity dict, wrap in list
entities = [entities]
# Dispatch to a specific sub-method when one is requested
if method == "value":
if not property_name:
raise ValueError("property_name is required for method='value'")
return self.detect_value_conflicts(entities, property_name, entity_type)
elif method == "property":
if not property_name:
raise ValueError("property_name is required for method='property'")
return self.detect_property_conflicts(entities, property_name)
elif method == "type":
return self.detect_type_conflicts(entities)
elif method == "relationship":
relationships = kwargs.get("relationships", [])
if isinstance(relationships, dict):
inner = relationships.get("relationships", relationships)
relationships = inner if isinstance(inner, list) else [inner]
if not isinstance(relationships, list):
relationships = [relationships]
return self.detect_relationship_conflicts(relationships)
elif method == "temporal":
return self.detect_temporal_conflicts(entities)
elif method == "logical":
return self.detect_logical_conflicts(entities)
elif method == "entity":
return self.detect_entity_conflicts(entities, entity_type)
elif method != "all":
raise ValueError(f"Unknown conflict detection method: {method!r}")
tracking_id = self.progress_tracker.start_tracking(
file=None,
module="conflicts",
+109 -17
View File
@@ -100,6 +100,10 @@ class DuplicateDetector:
confidence_threshold: float = 0.6,
use_clustering: bool = True,
config: Optional[Dict[str, Any]] = None,
max_results: Optional[int] = None,
top_k_per_entity: Optional[int] = None,
min_similarity: Optional[float] = None,
sort_by: str = "confidence",
**kwargs,
):
"""
@@ -115,6 +119,16 @@ class DuplicateDetector:
(0.0 to 1.0, default: 0.6)
use_clustering: Whether to use clustering for group formation (default: True)
config: Configuration dictionary (merged with kwargs)
max_results: Hard cap on total candidates returned across all entities.
Applied after sorting. ``None`` means no limit.
top_k_per_entity: Keep at most this many candidates per entity (by the
sort field). ``None`` means no per-entity limit.
min_similarity: Additional similarity floor applied on top of
``similarity_threshold``. Candidates whose
``similarity_score`` is below this value are dropped
before ranking. ``None`` means no extra floor.
sort_by: Field used for ranking before limits are applied.
``"confidence"`` (default) or ``"similarity_score"``.
**kwargs: Additional configuration options:
- similarity: Configuration for SimilarityCalculator
"""
@@ -133,6 +147,30 @@ class DuplicateDetector:
self.confidence_threshold = confidence_threshold
self.use_clustering = use_clustering
# Result limiting / ranking options — validate at construction time
if max_results is not None and (not isinstance(max_results, int) or max_results < 0):
raise ValueError(
f"max_results must be None or a non-negative int, got {max_results!r}"
)
if top_k_per_entity is not None and (
not isinstance(top_k_per_entity, int) or top_k_per_entity < 0
):
raise ValueError(
f"top_k_per_entity must be None or a non-negative int, got {top_k_per_entity!r}"
)
if min_similarity is not None and not (0.0 <= min_similarity <= 1.0):
raise ValueError(
f"min_similarity must be None or a float in [0.0, 1.0], got {min_similarity!r}"
)
if sort_by not in ("confidence", "similarity_score"):
raise ValueError(
f"sort_by must be 'confidence' or 'similarity_score', got {sort_by!r}"
)
self.max_results = max_results
self.top_k_per_entity = top_k_per_entity
self.min_similarity = min_similarity
self.sort_by = sort_by
# Initialize progress tracker and ensure it's enabled
self.progress_tracker = get_progress_tracker()
if not self.progress_tracker.enabled:
@@ -140,7 +178,9 @@ class DuplicateDetector:
self.logger.debug(
f"Duplicate detector initialized: similarity_threshold={similarity_threshold}, "
f"confidence_threshold={confidence_threshold}"
f"confidence_threshold={confidence_threshold}, max_results={max_results}, "
f"top_k_per_entity={top_k_per_entity}, min_similarity={min_similarity}, "
f"sort_by={sort_by!r}"
)
def detect_duplicates(
@@ -153,8 +193,10 @@ class DuplicateDetector:
Detect duplicate entities from a list.
This method compares all pairs of entities and identifies duplicates based
on similarity scores and confidence thresholds. Returns candidates sorted
by confidence (highest first).
on similarity scores and confidence thresholds. Results are filtered and
ranked by ``_apply_result_limits`` using the instance ``sort_by`` field
(default: ``"confidence"``), then capped by ``top_k_per_entity`` and
``max_results``.
Args:
entities: List of entity dictionaries to check for duplicates.
@@ -163,8 +205,9 @@ class DuplicateDetector:
**options: Additional detection options passed to similarity calculator
Returns:
List of DuplicateCandidate objects, sorted by confidence (highest first).
Each candidate contains:
List of DuplicateCandidate objects sorted by the ``sort_by`` field
(highest first, default ``"confidence"``), capped by ``top_k_per_entity``
and ``max_results``. Each candidate contains:
- entity1, entity2: The duplicate entity pair
- similarity_score: Similarity score (0.0 to 1.0)
- confidence: Confidence score (0.0 to 1.0)
@@ -255,8 +298,8 @@ class DuplicateDetector:
message=f"Creating duplicate candidates... {i + 1}/{total_similarities} (remaining: {remaining})",
)
# Sort by confidence (highest first)
candidates.sort(key=lambda c: c.confidence, reverse=True)
# Sort, filter, and cap results
candidates = self._apply_result_limits(candidates)
self.logger.info(
f"Detected {len(candidates)} duplicate candidate(s) "
@@ -521,7 +564,9 @@ class DuplicateDetector:
Returns:
List of DuplicateCandidate objects representing duplicates between
new and existing entities, sorted by confidence (highest first).
new and existing entities, sorted by the ``sort_by`` field (highest
first, default ``"confidence"``), capped by ``top_k_per_entity`` and
``max_results``.
Example:
>>> new_entities = [{"id": "3", "name": "Apple Corp"}]
@@ -600,8 +645,8 @@ class DuplicateDetector:
message=f"Comparing entities... {processed}/{total_comparisons} (remaining: {remaining})",
)
# Sort by confidence (highest first)
candidates.sort(key=lambda c: c.confidence, reverse=True)
# Sort, filter, and cap results
candidates = self._apply_result_limits(candidates)
self.logger.info(
f"Incremental detection found {len(candidates)} duplicate candidate(s)"
@@ -620,6 +665,57 @@ class DuplicateDetector:
)
raise
def _apply_result_limits(
self, candidates: List[DuplicateCandidate]
) -> List[DuplicateCandidate]:
"""
Apply min_similarity filter, sort, top_k_per_entity, and max_results cap.
Order of operations:
1. Drop candidates below ``min_similarity`` (if set).
2. Sort by ``sort_by`` field descending.
3. Apply ``top_k_per_entity``: for each entity id keep only the top-k
candidates in which it appears.
4. Apply ``max_results`` global cap.
"""
# 1. min_similarity filter
if self.min_similarity is not None:
candidates = [
c for c in candidates if c.similarity_score >= self.min_similarity
]
# 2. Sort descending by the chosen field
candidates.sort(key=lambda c: getattr(c, self.sort_by), reverse=True)
# 3. top_k_per_entity — keep a candidate if *either* entity is still under
# quota; once an entity reaches k it no longer sponsors new candidates.
if self.top_k_per_entity is not None:
entity_counts: Dict[str, int] = {}
kept: List[DuplicateCandidate] = []
for c in candidates:
nid1 = self._normalize_entity_id(c.entity1)
nid2 = self._normalize_entity_id(c.entity2)
count1 = entity_counts.get(nid1, 0)
count2 = entity_counts.get(nid2, 0)
if count1 < self.top_k_per_entity or count2 < self.top_k_per_entity:
kept.append(c)
entity_counts[nid1] = count1 + 1
entity_counts[nid2] = count2 + 1
candidates = kept
# 4. max_results global cap
if self.max_results is not None:
candidates = candidates[: self.max_results]
return candidates
def _normalize_entity_id(self, entity: Any) -> str:
"""Return a stable string key for an entity, used as a dict key throughout the class."""
raw = self._get_entity_value(entity, "id")
if raw is None:
raw = id(entity)
return str(raw)
def _get_entity_value(self, entity: Any, key: str, default: Any = None) -> Any:
"""Get value from entity dictionary or object safely."""
if hasattr(entity, "__dict__"):
@@ -715,12 +811,8 @@ class DuplicateDetector:
groups = []
for candidate in candidates:
entity1_id = self._get_entity_value(candidate.entity1, "id") or id(
candidate.entity1
)
entity2_id = self._get_entity_value(candidate.entity2, "id") or id(
candidate.entity2
)
entity1_id = self._normalize_entity_id(candidate.entity1)
entity2_id = self._normalize_entity_id(candidate.entity2)
group1 = entity_to_group.get(entity1_id)
group2 = entity_to_group.get(entity2_id)
@@ -764,7 +856,7 @@ class DuplicateDetector:
# Update references
for entity in group2.entities:
entity_id = self._get_entity_value(entity, "id") or id(entity)
entity_id = self._normalize_entity_id(entity)
entity_to_group[entity_id] = group1
if group2 in groups:
+23 -8
View File
@@ -102,10 +102,10 @@ def _coerce_embedding_vector(value: object) -> Optional[List[float]]:
def _extract_node_embeddings(graph_dict: dict) -> dict[str, List[float]]:
"""Extract embeddings from graph dictionary."""
# Top-level keys to probe on each entity (and its metadata/properties dicts).
# Priority: generic names first, then KG-extras-specific names.
# Must stay aligned with the inner probe list in _coerce_embedding_vector.
# TODO: cache this per-session graph revision to avoid re-scanning all nodes on every request.
embedding_keys = (
"embedding",
"embeddings",
@@ -138,6 +138,11 @@ def _extract_node_embeddings(graph_dict: dict) -> dict[str, List[float]]:
return embeddings
def _get_cached_embeddings(session: GraphSession) -> dict[str, List[float]]:
"""Get embeddings from session cache for optimal performance."""
return session.get_cached_embeddings()
def _node_response(node: dict) -> NodeResponse:
return NodeResponse(**node)
@@ -480,7 +485,6 @@ async def distance_matrix(
)
started = time.perf_counter()
graph_dict = await asyncio.to_thread(session.build_graph_dict)
path_finder = session.path_finder
n = len(body.node_ids)
@@ -493,10 +497,21 @@ async def distance_matrix(
src, tgt = body.node_ids[i], body.node_ids[j]
try:
if body.metric == "semantic" and session.similarity is not None:
sim = await asyncio.to_thread(
session.similarity.cosine_similarity, graph_dict, src, tgt
)
val = 1.0 - float(sim) if isinstance(sim, (int, float)) else None
# Use cached embeddings for semantic distance calculation
embeddings = _get_cached_embeddings(session)
src_embedding = embeddings.get(src)
tgt_embedding = embeddings.get(tgt)
if src_embedding is None or tgt_embedding is None:
val = None
else:
# Calculate cosine similarity directly from cached embeddings
import numpy as np
src_vec = np.array(src_embedding)
tgt_vec = np.array(tgt_embedding)
sim = np.dot(src_vec, tgt_vec) / (np.linalg.norm(src_vec) * np.linalg.norm(tgt_vec))
val = 1.0 - float(sim) if isinstance(sim, (int, float)) else None
matrix[i][j] = val
matrix[j][i] = val
elif path_finder is not None:
@@ -505,6 +520,7 @@ async def distance_matrix(
if body.metric == "weighted"
else path_finder.bfs_shortest_path
)
graph_dict = await asyncio.to_thread(session.build_graph_dict)
result = await asyncio.to_thread(path_fn, graph_dict, src, tgt)
path_nodes = result.get("path", []) if isinstance(result, dict) else (result or [])
if path_nodes:
@@ -550,8 +566,7 @@ async def _semantic_neighborhood_impl(
detail="Semantic similarity is unavailable for this graph session.",
)
graph_dict = await asyncio.to_thread(session.build_graph_dict)
embeddings = _extract_node_embeddings(graph_dict)
embeddings = _get_cached_embeddings(session)
query_embedding = embeddings.get(node_id)
if not embeddings or query_embedding is None:
raise HTTPException(
File diff suppressed because it is too large Load Diff
+103
View File
@@ -52,6 +52,10 @@ class GraphSession:
self._similarity: Any = None
self._link_predictor: Any = None
self._validator: Any = None
self._graph_revision: int = 0
self._cached_embeddings: Optional[Dict[str, List[float]]] = None
self._cached_graph_revision: int = -1
self.rebuild_search_index()
@classmethod
@@ -408,6 +412,20 @@ class GraphSession:
def handle_graph_mutation(self, event_type: str, entity_id: str, payload: Dict[str, Any]) -> None:
normalized_event = str(event_type or "").upper()
if normalized_event in {
"ADD_NODE",
"UPDATE_NODE",
"REMOVE_NODE",
"DELETE_NODE",
"ADD_EDGE",
"UPDATE_EDGE",
"REMOVE_EDGE",
"DELETE_EDGE",
"RELOAD_GRAPH",
"RESET_GRAPH",
}:
with self._lock:
self._bump_graph_revision_locked()
if normalized_event in {"ADD_NODE", "UPDATE_NODE"}:
normalized_node = self.normalize_node(payload or {})
if normalized_node.get("id"):
@@ -529,6 +547,83 @@ class GraphSession:
with self._lock:
return self.annotations.pop(annotation_id, None) is not None
def _bump_graph_revision_locked(self) -> None:
self._graph_revision += 1
self._cached_embeddings = None
self._cached_graph_revision = -1
@staticmethod
def _coerce_embedding_vector(value: Any) -> Optional[List[float]]:
if isinstance(value, dict):
for key in ("embedding", "embeddings", "vector", "values", "node2vec", "semantic"):
nested = GraphSession._coerce_embedding_vector(value.get(key))
if nested is not None:
return nested
return None
if not isinstance(value, (list, tuple)):
return None
vector: List[float] = []
for item in value:
try:
vector.append(float(item))
except (TypeError, ValueError):
return None
return vector if vector else None
def get_cached_embeddings(self, force_refresh: bool = False) -> Dict[str, List[float]]:
with self._lock:
current_revision = self._graph_revision
if (
not force_refresh
and self._cached_embeddings is not None
and self._cached_graph_revision == current_revision
):
return self._cached_embeddings
raw_nodes = [
node.to_dict() if hasattr(node, "to_dict") else node
for node in self.graph.nodes.values()
if node is not None
]
embedding_keys = (
"embedding",
"embeddings",
"vector",
"node_embedding",
"node2vec_embedding",
"semantic_embedding",
"reasoning_embedding",
)
embeddings: Dict[str, List[float]] = {}
for raw in raw_nodes:
if not isinstance(raw, dict):
continue
normalized = self.normalize_node(raw)
node_id = normalized.get("id")
if not node_id:
continue
properties = normalized.get("properties") if isinstance(normalized.get("properties"), dict) else {}
for key in embedding_keys:
vector = self._coerce_embedding_vector(normalized.get(key, properties.get(key)))
if vector is not None:
embeddings[str(node_id)] = vector
break
with self._lock:
if self._graph_revision == current_revision:
self._cached_embeddings = embeddings
self._cached_graph_revision = current_revision
return embeddings
def invalidate_embedding_cache(self) -> None:
with self._lock:
self._cached_embeddings = None
self._cached_graph_revision = -1
def build_graph_dict(self, node_ids: Optional[list] = None) -> dict:
nodes, _ = self.get_nodes(skip=0, limit=999_999)
edges, _ = self.get_edges(skip=0, limit=999_999)
@@ -595,6 +690,8 @@ class GraphSession:
with self._lock:
added = self.graph.add_nodes(nodes)
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
if added and not has_mutation_callback:
self._bump_graph_revision_locked()
if added and not has_mutation_callback:
self.rebuild_search_index()
return added
@@ -603,6 +700,8 @@ class GraphSession:
with self._lock:
added = self.graph.add_edges(edges)
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
if added and not has_mutation_callback:
self._bump_graph_revision_locked()
if added and not has_mutation_callback:
self.rebuild_search_index()
return added
@@ -617,6 +716,8 @@ class GraphSession:
with self._lock:
added = self.graph.add_node(node_id, node_type, content=content, **properties)
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
if added and not has_mutation_callback:
self._bump_graph_revision_locked()
if added and not has_mutation_callback:
normalized = self.get_node(node_id)
if normalized is not None:
@@ -640,4 +741,6 @@ class GraphSession:
**properties,
)
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
if added and not has_mutation_callback:
self._bump_graph_revision_locked()
return added
+123 -42
View File
@@ -7,12 +7,15 @@ including files, web content, feeds, streams, repositories, emails, and database
Algorithms Used:
File Ingestion:
- File Type Detection: Multi-method detection (extension-based, MIME type, magic number analysis)
- File Type Detection: Multi-method detection using extension,
MIME type, and magic number analysis
- Directory Scanning: Recursive directory traversal with filtering
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob Storage API integration
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob
Storage API integration
- File Validation: Size limits, format validation, content verification
- Batch Processing: Parallel file processing with progress tracking
- Magic Number Analysis: Binary file signature detection for accurate type identification
- Magic Number Analysis: Binary file signature detection for accurate
type identification
Web Ingestion:
- HTTP Request Handling: GET/POST requests with retry logic and error handling
@@ -46,9 +49,11 @@ Repository Ingestion:
- Git Operations: Repository cloning, branch checking, commit traversal
- Code Extraction: File content extraction with language detection
- Commit Analysis: Git log parsing, diff analysis, statistics calculation
- Language Detection: File extension and content-based programming language identification
- Language Detection: File extension and content-based programming
language identification
- Code Structure Analysis: AST parsing for classes, functions, imports extraction
- Dependency Analysis: Package manager file parsing (requirements.txt, package.json, etc.)
- Dependency Analysis: Package manager file parsing
(requirements.txt, package.json, etc.)
- Documentation Extraction: README, docstring, and comment extraction
Email Ingestion:
@@ -61,7 +66,8 @@ Email Ingestion:
- Link Extraction: URL extraction from email HTML content
Database Ingestion:
- Database Connection: SQLAlchemy-based connection management with connection pooling
- Database Connection: SQLAlchemy-based connection management with
connection pooling
- SQL Query Execution: Parameterized query execution with result set processing
- Schema Introspection: Database schema analysis and table/column discovery
- Data Type Conversion: Database-specific type to Python type conversion
@@ -87,6 +93,7 @@ Main Classes:
- EmailIngestor: Email protocol handling
- DBIngestor: Database export handling
- OntologyIngestor: Ontology file processing
- ParquetIngestor: Apache Parquet file and partitioned dataset processing
- MethodRegistry: Registry for custom ingestion methods
- IngestConfig: Configuration manager for ingest module
@@ -100,6 +107,7 @@ Convenience Functions:
- ingest_email: Email ingestion wrapper
- ingest_database: Database ingestion wrapper
- ingest_ontology: Ontology ingestion wrapper
- ingest_parquet: Parquet ingestion wrapper
Example Usage:
@@ -112,22 +120,18 @@ Example Usage:
>>> content = ingest_web("https://example.com", method="url")
"""
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from __future__ import annotations
import importlib
from typing import Any, Dict, Tuple
from .config import IngestConfig, ingest_config
from .db_ingestor import DatabaseConnector, DataExporter, DBIngestor, TableData
from .email_ingestor import AttachmentProcessor, EmailData, EmailIngestor
from .email_ingestor import EmailParser as EmailIngestorParser
from .feed_ingestor import FeedData, FeedIngestor, FeedItem, FeedMonitor, FeedParser
from .file_ingestor import (
CloudStorageIngestor,
FileIngestor,
FileObject,
FileTypeDetector,
)
from .mcp_client import MCPClient, MCPResource, MCPTool
from .mcp_ingestor import MCPData, MCPIngestor
from .methods import (
get_ingest_method,
ingest,
@@ -137,40 +141,114 @@ from .methods import (
ingest_file,
ingest_mcp,
ingest_ontology,
ingest_parquet,
ingest_repository,
ingest_stream,
ingest_web,
list_available_methods,
)
from .registry import MethodRegistry, method_registry
from .repo_ingestor import (
CodeExtractor,
CodeFile,
CommitInfo,
GitAnalyzer,
RepoIngestor,
)
from .stream_ingestor import (
KafkaProcessor,
KinesisProcessor,
PulsarProcessor,
RabbitMQProcessor,
StreamIngestor,
StreamMessage,
StreamMonitor,
StreamProcessor,
)
from .web_ingestor import (
ContentExtractor,
RateLimiter,
RobotsChecker,
SitemapCrawler,
WebContent,
WebIngestor,
)
from .ontology_ingestor import OntologyData, OntologyIngestor
from .snowflake_ingestor import SnowflakeConnector, SnowflakeData, SnowflakeIngestor
_LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
# Web ingestion
"WebIngestor": (".web_ingestor", "WebIngestor"),
"WebContent": (".web_ingestor", "WebContent"),
"RateLimiter": (".web_ingestor", "RateLimiter"),
"RobotsChecker": (".web_ingestor", "RobotsChecker"),
"ContentExtractor": (".web_ingestor", "ContentExtractor"),
"SitemapCrawler": (".web_ingestor", "SitemapCrawler"),
# Feed ingestion
"FeedIngestor": (".feed_ingestor", "FeedIngestor"),
"FeedItem": (".feed_ingestor", "FeedItem"),
"FeedData": (".feed_ingestor", "FeedData"),
"FeedParser": (".feed_ingestor", "FeedParser"),
"FeedMonitor": (".feed_ingestor", "FeedMonitor"),
# Stream ingestion
"StreamIngestor": (".stream_ingestor", "StreamIngestor"),
"StreamMessage": (".stream_ingestor", "StreamMessage"),
"StreamProcessor": (".stream_ingestor", "StreamProcessor"),
"KafkaProcessor": (".stream_ingestor", "KafkaProcessor"),
"RabbitMQProcessor": (".stream_ingestor", "RabbitMQProcessor"),
"KinesisProcessor": (".stream_ingestor", "KinesisProcessor"),
"PulsarProcessor": (".stream_ingestor", "PulsarProcessor"),
"StreamMonitor": (".stream_ingestor", "StreamMonitor"),
# Repository ingestion
"RepoIngestor": (".repo_ingestor", "RepoIngestor"),
"CodeFile": (".repo_ingestor", "CodeFile"),
"CommitInfo": (".repo_ingestor", "CommitInfo"),
"CodeExtractor": (".repo_ingestor", "CodeExtractor"),
"GitAnalyzer": (".repo_ingestor", "GitAnalyzer"),
# Email ingestion
"EmailIngestor": (".email_ingestor", "EmailIngestor"),
"EmailData": (".email_ingestor", "EmailData"),
"AttachmentProcessor": (".email_ingestor", "AttachmentProcessor"),
"EmailIngestorParser": (".email_ingestor", "EmailParser"),
# Database ingestion
"DBIngestor": (".db_ingestor", "DBIngestor"),
"TableData": (".db_ingestor", "TableData"),
"DatabaseConnector": (".db_ingestor", "DatabaseConnector"),
"DataExporter": (".db_ingestor", "DataExporter"),
# MCP ingestion
"MCPIngestor": (".mcp_ingestor", "MCPIngestor"),
"MCPData": (".mcp_ingestor", "MCPData"),
"MCPClient": (".mcp_client", "MCPClient"),
"MCPResource": (".mcp_client", "MCPResource"),
"MCPTool": (".mcp_client", "MCPTool"),
# Ontology ingestion
"OntologyIngestor": (".ontology_ingestor", "OntologyIngestor"),
"OntologyData": (".ontology_ingestor", "OntologyData"),
# Snowflake ingestion
"SnowflakeIngestor": (".snowflake_ingestor", "SnowflakeIngestor"),
"SnowflakeData": (".snowflake_ingestor", "SnowflakeData"),
"SnowflakeConnector": (".snowflake_ingestor", "SnowflakeConnector"),
# Parquet ingestion
"ParquetIngestor": (".parquet_ingestor", "ParquetIngestor"),
"ParquetData": (".parquet_ingestor", "ParquetData"),
}
_OPTIONAL_DEPENDENCY_MESSAGES = {
".repo_ingestor": (
"Repository ingestion requires optional dependency 'GitPython'. "
"Install it before importing RepoIngestor or using ingest_repository()."
),
".web_ingestor": (
"Web ingestion requires optional dependency 'beautifulsoup4'. "
"Install it before importing WebIngestor or using ingest_web()."
),
".feed_ingestor": (
"Feed ingestion requires optional dependency 'beautifulsoup4'. "
"Install it before importing FeedIngestor or using ingest_feed()."
),
".email_ingestor": (
"Email ingestion requires optional dependency 'beautifulsoup4'. "
"Install it before importing EmailIngestor or using ingest_email()."
),
".parquet_ingestor": (
"Parquet ingestion requires optional dependency 'pyarrow'. "
"Install it before importing ParquetIngestor or using ingest_parquet()."
),
}
def __getattr__(name: str) -> Any:
"""Load optional ingestion backends only when callers request them."""
if name not in _LAZY_EXPORTS:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
module_name, attr_name = _LAZY_EXPORTS[name]
try:
module = importlib.import_module(module_name, __name__)
except ModuleNotFoundError as exc:
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
missing_name = getattr(exc, "name", None)
if message and missing_name in {"git", "bs4", "pyarrow"}:
raise ImportError(message) from exc
raise
value = getattr(module, attr_name)
globals()[name] = value
return value
__all__ = [
# File ingestion
@@ -229,6 +307,9 @@ __all__ = [
"SnowflakeIngestor",
"SnowflakeData",
"SnowflakeConnector",
# Parquet ingestion
"ParquetIngestor",
"ParquetData",
# Registry and Methods
"MethodRegistry",
"method_registry",
@@ -241,6 +322,7 @@ __all__ = [
"ingest_email",
"ingest_database",
"ingest_ontology",
"ingest_parquet",
"ingest_mcp",
"get_ingest_method",
"list_available_methods",
@@ -248,4 +330,3 @@ __all__ = [
"IngestConfig",
"ingest_config",
]
+10 -6
View File
@@ -23,7 +23,6 @@ License: MIT
"""
import mimetypes
import os
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
@@ -112,7 +111,8 @@ class FileTypeDetector:
mimetypes.init()
self.logger.debug(
f"File type detector initialized with {len(self.supported_formats)} supported formats"
"File type detector initialized with "
f"{len(self.supported_formats)} supported formats"
)
def detect_type(
@@ -190,11 +190,12 @@ class FileTypeDetector:
magic_numbers = {
b"\x25\x50\x44\x46": "pdf", # PDF (binary)
b"%PDF": "pdf", # PDF (text header)
b"\x50\x4B\x03\x04": "zip", # ZIP, DOCX, XLSX, PPTX (Office Open XML)
b"\x89\x50\x4E\x47": "png", # PNG image
b"\xFF\xD8\xFF": "jpg", # JPEG image
b"\x50\x4b\x03\x04": "zip", # ZIP, DOCX, XLSX, PPTX (Office Open XML)
b"\x89\x50\x4e\x47": "png", # PNG image
b"\xff\xd8\xff": "jpg", # JPEG image
b"\x47\x49\x46\x38": "gif", # GIF image
b"PK\x03\x04": "zip", # ZIP (alternative)
b"PAR1": "parquet", # Apache Parquet
}
# Check if content starts with any known magic number
@@ -516,7 +517,10 @@ class FileIngestor:
tracking_id,
processed=idx,
total=total_files,
message=f"Processing file {idx}/{total_files}: {Path(file_info['path']).name}"
message=(
f"Processing file {idx}/{total_files}: "
f"{Path(file_info['path']).name}"
),
)
# Track progress via callback if provided
+86 -18
View File
@@ -6,18 +6,19 @@ This guide demonstrates how to use the ingest module for ingesting data from var
1. [Basic Usage](#basic-usage)
2. [File Ingestion](#file-ingestion)
3. [Web Ingestion](#web-ingestion)
4. [Feed Ingestion](#feed-ingestion)
5. [Stream Ingestion](#stream-ingestion)
6. [Repository Ingestion](#repository-ingestion)
7. [Email Ingestion](#email-ingestion)
8. [Database Ingestion](#database-ingestion)
9. [MCP Server Ingestion](#mcp-server-ingestion)
10. [Unified Ingestion](#unified-ingestion)
11. [Using Methods](#using-methods)
12. [Using Registry](#using-registry)
13. [Configuration](#configuration)
14. [Advanced Examples](#advanced-examples)
3. [Parquet Ingestion](#parquet-ingestion)
4. [Web Ingestion](#web-ingestion)
5. [Feed Ingestion](#feed-ingestion)
6. [Stream Ingestion](#stream-ingestion)
7. [Repository Ingestion](#repository-ingestion)
8. [Email Ingestion](#email-ingestion)
9. [Database Ingestion](#database-ingestion)
10. [MCP Server Ingestion](#mcp-server-ingestion)
11. [Unified Ingestion](#unified-ingestion)
12. [Using Methods](#using-methods)
13. [Using Registry](#using-registry)
14. [Configuration](#configuration)
15. [Advanced Examples](#advanced-examples)
## Basic Usage
@@ -29,6 +30,9 @@ from semantica.ingest import ingest
# Ingest a file (auto-detects source type)
result = ingest("document.pdf", source_type="file")
# Ingest a Parquet file
result = ingest("events.parquet")
# Ingest from web URL
result = ingest("https://example.com", source_type="web")
@@ -142,6 +146,66 @@ with open("document.pdf", "rb") as f:
file_type = detector.detect_type("document.pdf", content=content)
```
## Parquet Ingestion
Parquet ingestion requires PyArrow:
```bash
pip install pyarrow
```
### Single Parquet File
```python
from semantica.ingest import ParquetIngestor, ingest_parquet
# Using convenience function
data = ingest_parquet(
"events.parquet",
columns=["event_id", "event_type"],
limit=1000,
)
# Using class directly
ingestor = ParquetIngestor()
data = ingestor.ingest_file("events.parquet")
print(f"Rows returned: {data.row_count}")
print(f"Columns: {data.columns}")
print(f"Total rows in file: {data.metadata['total_rows']}")
```
### Schema and Metadata Extraction
```python
from semantica.ingest import ParquetIngestor
ingestor = ParquetIngestor()
schema = ingestor.extract_schema("events.parquet")
metadata = ingestor.extract_metadata("events.parquet")
print(schema["columns"])
print(metadata["compression_codecs"])
print(metadata["row_groups"])
```
### Partitioned Parquet Directories
```python
from semantica.ingest import ingest_parquet
# Reads Hive-style directories such as country=US/year=2026/part-0.parquet
data = ingest_parquet(
"./warehouse/events",
method="directory",
columns=["event_id", "event_type", "country", "year"],
)
print(data.metadata["partition_columns"])
print(data.metadata["partition_values"])
```
## Web Ingestion
### Single URL Ingestion
@@ -936,6 +1000,7 @@ from semantica.ingest import ingest
# Auto-detect source type from source
result = ingest("document.pdf") # Auto-detects file
result = ingest("events.parquet") # Auto-detects Parquet
result = ingest("https://example.com") # Auto-detects web
result = ingest("https://example.com/feed.xml") # Auto-detects feed
result = ingest("postgresql://user:pass@localhost/db") # Auto-detects database
@@ -982,7 +1047,8 @@ from semantica.ingest.methods import (
ingest_repository,
ingest_email,
ingest_database,
ingest_mcp
ingest_mcp,
ingest_parquet
)
# File ingestion
@@ -1006,6 +1072,9 @@ emails = ingest_email({"host": "imap.example.com", "username": "user", "password
# Database ingestion
data = ingest_database("postgresql://user:pass@localhost/db", table="users")
# Parquet ingestion
events = ingest_parquet("events.parquet", columns=["event_id"], limit=1000)
# MCP server ingestion via URL
data = ingest_mcp("http://localhost:8000/mcp", method="resources")
```
@@ -1189,16 +1258,16 @@ from semantica.ingest.methods import ingest_file
def custom_pdf_ingestion(source, **kwargs):
"""Custom PDF ingestion with special processing."""
from semantica.ingest import FileIngestor
ingestor = FileIngestor()
file_obj = ingestor.ingest_file(source, **kwargs)
# Custom processing
if file_obj.file_type == "pdf":
# Add custom metadata
file_obj.metadata["processed"] = True
file_obj.metadata["custom_field"] = "custom_value"
return file_obj
# Register custom method
@@ -1286,7 +1355,7 @@ for source_type, source_list in sources.items():
1. **Parallel Processing**: Use parallel processing for multiple sources
```python
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor() as executor:
executor.submit(ingest_file, "./documents1")
executor.submit(ingest_file, "./documents2")
@@ -1329,4 +1398,3 @@ for source_type, source_list in sources.items():
for batch in process_in_batches(large_dataset, batch_size=1000):
result = ingest(batch)
```
+242 -35
View File
@@ -13,6 +13,12 @@ File Ingestion:
- "directory": Directory ingestion with recursive scanning
- "cloud": Cloud storage ingestion (S3, GCS, Azure)
Parquet Ingestion:
- "file": Single Parquet file ingestion
- "directory": Partitioned Parquet directory ingestion
- "schema": Parquet schema extraction
- "metadata": Parquet file or directory metadata extraction
Web Ingestion:
- "url": Single URL ingestion
- "sitemap": Sitemap-based crawling
@@ -48,12 +54,15 @@ Database Ingestion:
Algorithms Used:
File Ingestion:
- File Type Detection: Multi-method detection (extension-based, MIME type, magic number analysis)
- File Type Detection: Multi-method detection using extension,
MIME type, and magic number analysis
- Directory Scanning: Recursive directory traversal with filtering
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob Storage API integration
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob
Storage API integration
- File Validation: Size limits, format validation, content verification
- Batch Processing: Parallel file processing with progress tracking
- Magic Number Analysis: Binary file signature detection for accurate type identification
- Magic Number Analysis: Binary file signature detection for accurate
type identification
Web Ingestion:
- HTTP Request Handling: GET/POST requests with retry logic and error handling
@@ -87,9 +96,11 @@ Repository Ingestion:
- Git Operations: Repository cloning, branch checking, commit traversal
- Code Extraction: File content extraction with language detection
- Commit Analysis: Git log parsing, diff analysis, statistics calculation
- Language Detection: File extension and content-based programming language identification
- Language Detection: File extension and content-based programming
language identification
- Code Structure Analysis: AST parsing for classes, functions, imports extraction
- Dependency Analysis: Package manager file parsing (requirements.txt, package.json, etc.)
- Dependency Analysis: Package manager file parsing
(requirements.txt, package.json, etc.)
- Documentation Extraction: README, docstring, and comment extraction
Email Ingestion:
@@ -102,7 +113,8 @@ Email Ingestion:
- Link Extraction: URL extraction from email HTML content
Database Ingestion:
- Database Connection: SQLAlchemy-based connection management with connection pooling
- Database Connection: SQLAlchemy-based connection management with
connection pooling
- SQL Query Execution: Parameterized query execution with result set processing
- Schema Introspection: Database schema analysis and table/column discovery
- Data Type Conversion: Database-specific type to Python type conversion
@@ -125,6 +137,7 @@ Main Functions:
- ingest_repository: Repository ingestion wrapper
- ingest_email: Email ingestion wrapper
- ingest_database: Database ingestion wrapper
- ingest_parquet: Parquet ingestion wrapper
- ingest: Unified ingestion function with source type dispatch
- get_ingest_method: Get ingestion method by name
- list_available_methods: List registered methods
@@ -139,26 +152,42 @@ Example Usage:
>>> content = ingest_web("https://example.com", method="url")
"""
from __future__ import annotations
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional, Union
from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union
from ..utils.exceptions import ConfigurationError, ProcessingError
from ..utils.logging import get_logger
from .config import ingest_config
from .db_ingestor import DBIngestor, TableData
from .email_ingestor import EmailData, EmailIngestor
from .feed_ingestor import FeedData, FeedIngestor
from .file_ingestor import FileIngestor, FileObject
from .mcp_ingestor import MCPData, MCPIngestor
from .ontology_ingestor import OntologyData, OntologyIngestor
from .registry import method_registry
from .repo_ingestor import RepoIngestor
from .stream_ingestor import StreamIngestor, StreamProcessor
from .web_ingestor import WebContent, WebIngestor
if TYPE_CHECKING:
from .db_ingestor import TableData
from .email_ingestor import EmailData
from .feed_ingestor import FeedData
from .mcp_ingestor import MCPData
from .ontology_ingestor import OntologyData
from .parquet_ingestor import ParquetData
from .stream_ingestor import StreamProcessor
from .web_ingestor import WebContent
logger = get_logger("ingest_methods")
def _missing_optional_dependency(feature: str, package: str) -> ConfigurationError:
return ConfigurationError(
f"{feature} requires optional dependency '{package}'. "
f"Install it before using this ingestion backend."
)
def _is_missing_dependency(exc: ModuleNotFoundError, *dependency_names: str) -> bool:
missing_name = getattr(exc, "name", None)
return missing_name in dependency_names
def ingest_file(
source: Union[str, Path, List[Union[str, Path]]], method: str = "file", **kwargs
) -> Union[FileObject, List[FileObject], Dict[str, Any]]:
@@ -224,6 +253,86 @@ def ingest_file(
raise
def ingest_parquet(
source: Union[str, Path, List[Union[str, Path]]],
method: str = "file",
**kwargs,
) -> Union[ParquetData, List[ParquetData], Dict[str, Any]]:
"""
Ingest Apache Parquet files or partitioned directories.
Args:
source: Parquet file path, directory path, or list of paths
method: Ingestion method:
- "file": Single Parquet file ingestion
- "directory": Parquet directory or partitioned dataset ingestion
- "schema": Extract schema without reading data
- "metadata": Extract file/directory metadata without reading data
**kwargs: Additional options passed to ParquetIngestor
Returns:
ParquetData, list of ParquetData, or metadata/schema dictionary
Examples:
>>> from semantica.ingest.methods import ingest_parquet
>>> data = ingest_parquet("events.parquet", columns=["id"], limit=100)
>>> schema = ingest_parquet("events.parquet", method="schema")
>>> dataset = ingest_parquet("./events_by_date", method="directory")
"""
custom_method = method_registry.get("parquet", method)
if custom_method and custom_method != ingest_parquet:
try:
return custom_method(source, **kwargs)
except Exception as e:
logger.warning(
f"Custom method {method} failed: {e}, falling back to default"
)
try:
try:
from .parquet_ingestor import ParquetIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "pyarrow"):
raise _missing_optional_dependency(
"Parquet ingestion",
"pyarrow",
) from exc
raise
config = ingest_config.get_method_config("parquet")
config.update(kwargs)
try:
ingestor = ParquetIngestor(**config)
except ImportError as exc:
raise _missing_optional_dependency(
"Parquet ingestion",
"pyarrow",
) from exc
def _run_single(path: Union[str, Path]) -> Union[ParquetData, Dict[str, Any]]:
source_path = Path(path)
if method == "schema":
return ingestor.extract_schema(source_path, **kwargs)
if method == "metadata":
return ingestor.extract_metadata(source_path, **kwargs)
if method == "directory" or source_path.is_dir():
return ingestor.ingest_directory(source_path, **kwargs)
return ingestor.ingest_file(source_path, **kwargs)
if isinstance(source, list):
return [_run_single(path) for path in source]
return _run_single(source)
except ConfigurationError:
raise
except Exception as e:
logger.error(f"Failed to ingest Parquet: {e}")
raise
def ingest_web(
source: Union[str, List[str]], method: str = "url", **kwargs
) -> Union[WebContent, List[WebContent], Dict[str, Any]]:
@@ -259,6 +368,16 @@ def ingest_web(
)
try:
try:
from .web_ingestor import WebIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "bs4"):
raise _missing_optional_dependency(
"Web ingestion",
"beautifulsoup4",
) from exc
raise
# Get config
config = ingest_config.get_method_config("web")
config.update(kwargs)
@@ -278,6 +397,8 @@ def ingest_web(
# Default: try as URL
return ingestor.ingest_url(source, **kwargs)
except ConfigurationError:
raise
except Exception as e:
logger.error(f"Failed to ingest web: {e}")
raise
@@ -318,6 +439,16 @@ def ingest_feed(
)
try:
try:
from .feed_ingestor import FeedIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "bs4"):
raise _missing_optional_dependency(
"Feed ingestion",
"beautifulsoup4",
) from exc
raise
# Get config
config = ingest_config.get_method_config("feed")
config.update(kwargs)
@@ -332,6 +463,8 @@ def ingest_feed(
else:
return ingestor.ingest_feed(source, **kwargs)
except ConfigurationError:
raise
except Exception as e:
logger.error(f"Failed to ingest feed: {e}")
raise
@@ -375,6 +508,8 @@ def ingest_stream(
)
try:
from .stream_ingestor import StreamIngestor
# Get config
config = ingest_config.get_method_config("stream")
config.update(kwargs)
@@ -418,7 +553,8 @@ def ingest_repository(
"""
Ingest repository from source (convenience function).
This is a user-friendly wrapper that ingests repositories using the specified method.
This is a user-friendly wrapper that ingests repositories using the
specified method.
Args:
source: Repository URL or local path
@@ -433,7 +569,9 @@ def ingest_repository(
Examples:
>>> from semantica.ingest.methods import ingest_repository
>>> repo_data = ingest_repository("https://github.com/user/repo.git", method="git")
>>> repo_data = ingest_repository(
... "https://github.com/user/repo.git", method="git"
... )
>>> analysis = ingest_repository("./repo", method="analyze")
"""
# Check for custom method in registry
@@ -447,6 +585,15 @@ def ingest_repository(
)
try:
try:
from .repo_ingestor import RepoIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "git"):
raise _missing_optional_dependency(
"Repository ingestion", "GitPython"
) from exc
raise
# Get config
config = ingest_config.get_method_config("repo")
config.update(kwargs)
@@ -464,6 +611,8 @@ def ingest_repository(
# Default: ingest repository
return ingestor.ingest_repository(source, **kwargs)
except ConfigurationError:
raise
except Exception as e:
logger.error(f"Failed to ingest repository: {e}")
raise
@@ -505,6 +654,16 @@ def ingest_email(
)
try:
try:
from .email_ingestor import EmailIngestor
except ModuleNotFoundError as exc:
if _is_missing_dependency(exc, "bs4"):
raise _missing_optional_dependency(
"Email ingestion",
"beautifulsoup4",
) from exc
raise
# Get config
config = ingest_config.get_method_config("email")
config.update(kwargs)
@@ -533,6 +692,8 @@ def ingest_email(
else:
raise ProcessingError(f"Unknown email method: {method}")
except ConfigurationError:
raise
except Exception as e:
logger.error(f"Failed to ingest email: {e}")
raise
@@ -572,6 +733,8 @@ def ingest_ontology(
)
try:
from .ontology_ingestor import OntologyIngestor
# Get config
config = ingest_config.get_method_config("ontology")
config.update(kwargs)
@@ -579,19 +742,19 @@ def ingest_ontology(
ingestor = OntologyIngestor(**config)
source_path = str(source) if isinstance(source, (str, Path)) else None
if method == "file" and source_path:
if isinstance(source, list):
return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
return ingestor.ingest_ontology(source_path, **kwargs)
elif method == "directory" and source_path:
recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
else:
# Default: try as file
if isinstance(source, list):
# Default: try as file
if isinstance(source, list):
return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
return ingestor.ingest_ontology(str(source), **kwargs)
return ingestor.ingest_ontology(str(source), **kwargs)
except Exception as e:
logger.error(f"Failed to ingest ontology: {e}")
@@ -636,6 +799,8 @@ def ingest_database(
)
try:
from .db_ingestor import DBIngestor
# Get config
config = ingest_config.get_method_config("db")
config.update(kwargs)
@@ -686,7 +851,8 @@ def ingest_mcp(
the specified method. Works with Python and FastMCP MCP servers.
Args:
source: MCP server URL (str) or configuration dict with "url" key, or server name (str) if already connected
source: MCP server URL, configuration dict with "url" key, or server
name if already connected
- URL string: "http://localhost:8000/mcp"
- Dict: {"url": "http://localhost:8000/mcp", "headers": {...}}
method: Ingestion method (default: "resources")
@@ -709,13 +875,25 @@ def ingest_mcp(
>>> data = ingest_mcp("http://localhost:8000/mcp", method="resources")
>>> # Connect via URL dict and ingest all resources
>>> data = ingest_mcp(
... {"url": "https://api.example.com/mcp", "headers": {"Authorization": "Bearer token"}},
... {
... "url": "https://api.example.com/mcp",
... "headers": {"Authorization": "Bearer token"},
... },
... method="all"
... )
>>> # Ingest from already connected server
>>> data = ingest_mcp("server1", method="resources", resource_uris=["resource://example"])
>>> data = ingest_mcp(
... "server1",
... method="resources",
... resource_uris=["resource://example"],
... )
>>> # Call tool
>>> result = ingest_mcp("server1", method="tools", tool_name="get_data", tool_arguments={})
>>> result = ingest_mcp(
... "server1",
... method="tools",
... tool_name="get_data",
... tool_arguments={},
... )
"""
# Check for custom method in registry
custom_method = method_registry.get("mcp", method)
@@ -728,6 +906,8 @@ def ingest_mcp(
)
try:
from .mcp_ingestor import MCPIngestor
# Get config
config = ingest_config.get_method_config("mcp")
config.update(kwargs)
@@ -779,7 +959,8 @@ def ingest_mcp(
)
else:
raise ProcessingError(
"Source must be MCP server URL (str), configuration dict with 'url' key, "
"Source must be MCP server URL (str), configuration dict "
"with 'url' key, "
"or server name (str) if already connected"
)
@@ -831,6 +1012,7 @@ def ingest(
- "email": Email ingestion
- "db": Database ingestion
- "ontology": Ontology ingestion
- "parquet": Apache Parquet file or directory ingestion
method: Optional specific ingestion method
**kwargs: Additional options passed to ingestor
@@ -850,24 +1032,40 @@ def ingest(
if not source_type:
if isinstance(sources, (str, Path)):
source_str = str(sources)
if source_str.startswith(("http://", "https://")):
source_str_lower = source_str.lower()
if source_str_lower.startswith(("http://", "https://")):
# Check if it's a feed URL
if any(ext in source_str for ext in [".xml", "/feed", "/rss", "/atom"]):
if any(
ext in source_str_lower
for ext in [".xml", "/feed", "/rss", "/atom"]
):
source_type = "feed"
else:
source_type = "web"
elif source_str.startswith(
elif source_str_lower.startswith(
("postgresql://", "mysql://", "sqlite://", "oracle://", "mssql://")
):
source_type = "db"
elif source_str.startswith(
("git@", "https://github.com", "https://gitlab.com")
elif source_str.startswith("git@") or source_str_lower.startswith(
("https://github.com", "https://gitlab.com")
):
source_type = "repo"
elif source_str.endswith((".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")):
elif source_str_lower.endswith(
(".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")
):
source_type = "ontology"
elif source_str_lower.endswith((".parquet", ".pq")):
source_type = "parquet"
else:
source_type = "file"
elif (
isinstance(sources, list)
and sources
and all(
str(source).lower().endswith((".parquet", ".pq")) for source in sources
)
):
source_type = "parquet"
else:
source_type = "file"
@@ -894,6 +1092,8 @@ def ingest(
raise ProcessingError("Email ingestion requires configuration dictionary")
elif source_type == "db":
return {"data": ingest_database(sources, method=method, **kwargs)}
elif source_type == "parquet":
return {"data": ingest_parquet(sources, method=method or "file", **kwargs)}
elif source_type == "ontology":
return {"ontology": ingest_ontology(sources, method=method or "file", **kwargs)}
elif source_type == "mcp":
@@ -907,7 +1107,8 @@ def get_ingest_method(task: str, name: str) -> Optional[Callable]:
Get a registered ingestion method.
Args:
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
task: Task type ("file", "web", "feed", "stream", "repo", "email",
"db", "mcp", "ingest")
name: Method name
Returns:
@@ -971,6 +1172,12 @@ method_registry.register("db", "mysql", ingest_database)
method_registry.register("db", "sqlite", ingest_database)
method_registry.register("db", "oracle", ingest_database)
method_registry.register("db", "mssql", ingest_database)
method_registry.register("parquet", "default", ingest_parquet)
method_registry.register("parquet", "file", ingest_parquet)
method_registry.register("parquet", "directory", ingest_parquet)
method_registry.register("parquet", "schema", ingest_parquet)
method_registry.register("parquet", "metadata", ingest_parquet)
method_registry.register("file", "parquet", ingest_parquet)
method_registry.register("mcp", "default", ingest_mcp)
method_registry.register("mcp", "resources", ingest_mcp)
method_registry.register("mcp", "tools", ingest_mcp)
+766
View File
@@ -0,0 +1,766 @@
"""
Apache Parquet Ingestion Module
This module provides dedicated Parquet ingestion for local files and partitioned
directories. It uses PyArrow when available so callers can read selected
columns, inspect schemas and file metadata, and ingest Hive-style partitioned
datasets without database credentials.
Example Usage:
>>> from semantica.ingest import ParquetIngestor
>>> ingestor = ParquetIngestor()
>>> data = ingestor.ingest_file("events.parquet", columns=["id", "event_type"])
>>> schema = ingestor.extract_schema("events.parquet")
>>> partitioned = ingestor.ingest_directory("./events_by_date")
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Sequence, Union
try:
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
PARQUET_AVAILABLE = True
except (ImportError, OSError):
pa = None
ds = None
pq = None
PARQUET_AVAILABLE = False
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
@dataclass
class ParquetData:
"""Parquet ingestion result."""
data: List[Dict[str, Any]]
row_count: int
columns: List[str]
schema: Dict[str, Any]
source: str
metadata: Dict[str, Any] = field(default_factory=dict)
ingested_at: datetime = field(default_factory=datetime.now)
class ParquetIngestor:
"""
Dedicated Parquet ingestion handler.
Features:
- Single Parquet file ingestion
- Partitioned directory ingestion with Hive partition discovery
- Selective column reads
- Schema and file metadata extraction
- Optional row limits for sampling large files
"""
def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
"""
Initialize Parquet ingestor.
Args:
config: Optional configuration dictionary
**kwargs: Additional configuration options
Raises:
ImportError: If pyarrow is not installed
"""
if not PARQUET_AVAILABLE:
raise ImportError(
"pyarrow is required for ParquetIngestor. "
"Install it with: pip install pyarrow"
)
self.logger = get_logger("parquet_ingestor")
self.config = config or {}
self.config.update(kwargs)
self.progress_tracker = get_progress_tracker()
if not self.progress_tracker.enabled:
self.progress_tracker.enabled = True
self.logger.debug("Parquet ingestor initialized")
def ingest(
self,
source: Union[str, Path],
columns: Optional[Union[str, Sequence[str]]] = None,
limit: Optional[int] = None,
filters: Any = None,
include_data: bool = True,
**options,
) -> ParquetData:
"""
Ingest a Parquet file or partitioned Parquet directory.
Args:
source: Parquet file or directory path
columns: Optional column name or names to read
limit: Optional maximum number of rows to return
filters: Optional PyArrow filter expression or tuple filters
include_data: If False, return schema and metadata without rows
**options: Additional options
Returns:
ParquetData: Ingested data and metadata
"""
source_path = Path(source)
if source_path.is_dir():
return self.ingest_directory(
source_path,
columns=columns,
limit=limit,
filters=filters,
include_data=include_data,
**options,
)
return self.ingest_file(
source_path,
columns=columns,
limit=limit,
filters=filters,
include_data=include_data,
**options,
)
def ingest_file(
self,
file_path: Union[str, Path],
columns: Optional[Union[str, Sequence[str]]] = None,
limit: Optional[int] = None,
filters: Any = None,
include_data: bool = True,
batch_size: Optional[int] = None,
**options,
) -> ParquetData:
"""
Ingest a single Parquet file.
Args:
file_path: Path to Parquet file
columns: Optional column name or names to read
limit: Optional maximum number of rows to return
filters: Optional PyArrow-compatible filters
include_data: If False, skip reading row data
batch_size: Batch size used when sampling with limit
**options: Additional options
Returns:
ParquetData: Ingested data, schema, and metadata
"""
file_path = Path(file_path)
self._validate_file(file_path)
tracking_id = self.progress_tracker.start_tracking(
file=str(file_path),
module="ingest",
submodule="ParquetIngestor",
message=f"Ingesting Parquet: {file_path.name}",
)
try:
parquet_file = pq.ParquetFile(str(file_path))
selected_columns = self._normalize_columns(
columns,
[field.name for field in parquet_file.schema_arrow],
)
metadata = self._file_metadata(file_path, parquet_file)
if include_data:
table = self._read_file_table(
file_path=file_path,
parquet_file=parquet_file,
columns=selected_columns,
limit=limit,
filters=filters,
batch_size=batch_size,
)
data = table.to_pylist()
schema = self._schema_to_dict(table.schema)
result_columns = list(table.column_names)
else:
selected_schema = self._select_schema(
parquet_file.schema_arrow, selected_columns
)
data = []
schema = self._schema_to_dict(selected_schema)
result_columns = [field.name for field in selected_schema]
metadata.update(
{
"returned_rows": len(data),
"selected_columns": result_columns,
"filters_applied": filters is not None,
"limit": limit,
"include_data": include_data,
}
)
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Ingested Parquet: {len(data)} rows",
)
self.logger.info(
f"Parquet ingestion completed: {len(data)} row(s) from {file_path}"
)
return ParquetData(
data=data,
row_count=len(data),
columns=result_columns,
schema=schema,
source=str(file_path),
metadata=metadata,
)
except (ValidationError, ProcessingError):
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message="Parquet ingestion failed"
)
raise
except Exception as e:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e)
)
self.logger.error(f"Failed to ingest Parquet {file_path}: {e}")
raise ProcessingError(f"Failed to ingest Parquet: {e}") from e
def ingest_directory(
self,
directory_path: Union[str, Path],
columns: Optional[Union[str, Sequence[str]]] = None,
limit: Optional[int] = None,
filters: Any = None,
include_data: bool = True,
partitioning: Optional[Union[str, Any]] = "hive",
**options,
) -> ParquetData:
"""
Ingest a directory containing Parquet files.
Hive-style partitions such as ``country=US/year=2026`` are discovered
by default and included as partition columns in the returned schema/data.
Args:
directory_path: Directory containing Parquet files
columns: Optional column name or names to read
limit: Optional maximum number of rows to return
filters: Optional PyArrow filter expression or tuple filters
include_data: If False, return only schema and metadata
partitioning: PyArrow partitioning mode, defaults to "hive"
**options: Additional options
Returns:
ParquetData: Ingested dataset data and metadata
"""
directory_path = Path(directory_path)
parquet_files = self._validate_directory(directory_path)
if limit is not None and limit < 0:
raise ValidationError("limit must be greater than or equal to 0")
tracking_id = self.progress_tracker.start_tracking(
file=str(directory_path),
module="ingest",
submodule="ParquetIngestor",
message=f"Ingesting Parquet directory: {directory_path.name}",
)
try:
dataset = ds.dataset(
str(directory_path),
format="parquet",
partitioning=partitioning,
)
selected_columns = self._normalize_columns(
columns,
[field.name for field in dataset.schema],
)
filter_expression = self._dataset_filter(filters)
metadata = self._directory_metadata(
directory_path,
parquet_files,
partitioning=partitioning,
)
if include_data:
if limit is not None:
table = dataset.head(
limit,
columns=selected_columns,
filter=filter_expression,
)
else:
table = dataset.to_table(
columns=selected_columns,
filter=filter_expression,
)
data = table.to_pylist()
schema = self._schema_to_dict(table.schema)
result_columns = list(table.column_names)
else:
selected_schema = self._select_schema(dataset.schema, selected_columns)
data = []
schema = self._schema_to_dict(selected_schema)
result_columns = [field.name for field in selected_schema]
metadata.update(
{
"returned_rows": len(data),
"selected_columns": result_columns,
"filters_applied": filters is not None,
"limit": limit,
"include_data": include_data,
}
)
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Ingested Parquet directory: {len(data)} rows",
)
self.logger.info(
"Parquet directory ingestion completed: "
f"{len(data)} row(s) from {directory_path}"
)
return ParquetData(
data=data,
row_count=len(data),
columns=result_columns,
schema=schema,
source=str(directory_path),
metadata=metadata,
)
except (ValidationError, ProcessingError):
self.progress_tracker.stop_tracking(
tracking_id,
status="failed",
message="Parquet directory ingestion failed",
)
raise
except Exception as e:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e)
)
self.logger.error(
f"Failed to ingest Parquet directory {directory_path}: {e}"
)
raise ProcessingError(f"Failed to ingest Parquet directory: {e}") from e
def read_columns(
self,
source: Union[str, Path],
columns: Union[str, Sequence[str]],
**options,
) -> ParquetData:
"""
Read selected columns from a Parquet file or directory.
Args:
source: Parquet file or directory path
columns: Column name or names to read
**options: Additional ingestion options
Returns:
ParquetData: Ingested data containing only selected columns
"""
return self.ingest(source, columns=columns, **options)
def extract_schema(self, source: Union[str, Path], **options) -> Dict[str, Any]:
"""
Extract schema from a Parquet file or directory.
Args:
source: Parquet file or directory path
**options: Additional options
Returns:
dict: Schema with column names, types, nullability, and metadata
"""
source_path = Path(source)
if source_path.is_dir():
self._validate_directory(source_path)
dataset = ds.dataset(
str(source_path),
format="parquet",
partitioning=options.get("partitioning", "hive"),
)
return self._schema_to_dict(dataset.schema)
self._validate_file(source_path)
parquet_file = pq.ParquetFile(str(source_path))
return self._schema_to_dict(parquet_file.schema_arrow)
def extract_metadata(self, source: Union[str, Path], **options) -> Dict[str, Any]:
"""
Extract Parquet file or directory metadata without reading row data.
Args:
source: Parquet file or directory path
**options: Additional options
Returns:
dict: Row counts, row groups, compression, partitions, and file info
"""
source_path = Path(source)
if source_path.is_dir():
parquet_files = self._validate_directory(source_path)
return self._directory_metadata(
source_path,
parquet_files,
partitioning=options.get("partitioning", "hive"),
)
self._validate_file(source_path)
parquet_file = pq.ParquetFile(str(source_path))
return self._file_metadata(source_path, parquet_file)
def _read_file_table(
self,
file_path: Path,
parquet_file: Any,
columns: Optional[List[str]],
limit: Optional[int],
filters: Any,
batch_size: Optional[int],
) -> Any:
"""Read a Parquet file, using batches when a simple limit is requested."""
if limit is not None and limit < 0:
raise ValidationError("limit must be greater than or equal to 0")
if limit == 0:
return pa.Table.from_batches(
[],
schema=self._select_schema(parquet_file.schema_arrow, columns),
)
if limit is not None and filters is None:
return self._read_file_limited(parquet_file, columns, limit, batch_size)
table = pq.read_table(str(file_path), columns=columns, filters=filters)
if limit is not None:
table = table.slice(0, limit)
return table
def _read_file_limited(
self,
parquet_file: Any,
columns: Optional[List[str]],
limit: int,
batch_size: Optional[int],
) -> Any:
"""Read at most ``limit`` rows from a file without loading the full file."""
batches = []
remaining = limit
effective_batch_size = batch_size or min(max(limit, 1), 65_536)
for batch in parquet_file.iter_batches(
batch_size=effective_batch_size,
columns=columns,
):
if batch.num_rows > remaining:
batch = batch.slice(0, remaining)
batches.append(batch)
remaining -= batch.num_rows
if remaining <= 0:
break
return pa.Table.from_batches(
batches,
schema=self._select_schema(parquet_file.schema_arrow, columns),
)
def _validate_file(self, file_path: Path) -> None:
"""Validate a local Parquet file path."""
if not file_path.exists():
raise ValidationError(f"Parquet file not found: {file_path}")
if not file_path.is_file():
raise ValidationError(f"Path is not a file: {file_path}")
if file_path.suffix.lower() not in {".parquet", ".pq"}:
raise ValidationError(f"File is not a Parquet file: {file_path}")
def _validate_directory(self, directory_path: Path) -> List[Path]:
"""Validate a Parquet directory and return contained Parquet files."""
if not directory_path.exists():
raise ValidationError(f"Parquet directory not found: {directory_path}")
if not directory_path.is_dir():
raise ValidationError(f"Path is not a directory: {directory_path}")
parquet_files = self._parquet_files(directory_path)
if not parquet_files:
raise ValidationError(
f"No Parquet files found in directory: {directory_path}"
)
return parquet_files
def _parquet_files(self, directory_path: Path) -> List[Path]:
"""Return Parquet files under a directory."""
return sorted(
path
for path in directory_path.rglob("*")
if path.is_file() and path.suffix.lower() in {".parquet", ".pq"}
)
def _normalize_columns(
self,
columns: Optional[Union[str, Sequence[str]]],
available_columns: Sequence[str],
) -> Optional[List[str]]:
"""Normalize and validate optional selected columns."""
if columns is None:
configured_columns = self.config.get("columns")
if configured_columns is None:
return None
columns = configured_columns
if isinstance(columns, str):
normalized = [columns]
else:
normalized = list(columns)
missing = [column for column in normalized if column not in available_columns]
if missing:
raise ValidationError(
"Column(s) not found in Parquet schema: "
f"{', '.join(missing)}. Available columns: "
f"{', '.join(available_columns)}"
)
return normalized
def _select_schema(self, schema: Any, columns: Optional[List[str]]) -> Any:
"""Return schema limited to selected columns when provided."""
if columns is None:
return schema
fields = [schema.field(column) for column in columns]
return pa.schema(fields, metadata=schema.metadata)
def _schema_to_dict(self, schema: Any) -> Dict[str, Any]:
"""Convert PyArrow schema to serializable metadata."""
fields = []
for schema_field in schema:
fields.append(
{
"name": schema_field.name,
"type": str(schema_field.type),
"nullable": schema_field.nullable,
"metadata": self._decode_metadata_map(schema_field.metadata),
}
)
return {
"columns": [field_info["name"] for field_info in fields],
"fields": fields,
"metadata": self._decode_metadata_map(schema.metadata),
}
def _file_metadata(self, file_path: Path, parquet_file: Any) -> Dict[str, Any]:
"""Extract metadata for a single Parquet file."""
metadata = parquet_file.metadata
compression_by_column = self._compression_by_column(metadata)
return {
"format": "parquet",
"source_type": "file",
"file": str(file_path),
"file_size": file_path.stat().st_size,
"total_rows": metadata.num_rows,
"row_groups": metadata.num_row_groups,
"created_by": metadata.created_by,
"format_version": getattr(metadata, "format_version", None),
"serialized_size": getattr(metadata, "serialized_size", None),
"schema_metadata": self._decode_metadata_map(metadata.metadata),
"compression": {
column: sorted(codecs)
for column, codecs in compression_by_column.items()
},
"compression_codecs": sorted(
{codec for codecs in compression_by_column.values() for codec in codecs}
),
}
def _directory_metadata(
self,
directory_path: Path,
parquet_files: Sequence[Path],
partitioning: Optional[Union[str, Any]],
) -> Dict[str, Any]:
"""Extract aggregate metadata for a Parquet directory."""
file_entries = []
total_rows = 0
total_row_groups = 0
compression_by_column: Dict[str, set] = {}
partition_columns = set()
partition_values: Dict[str, set] = {}
for parquet_path in parquet_files:
parquet_file = pq.ParquetFile(str(parquet_path))
file_metadata = self._file_metadata(parquet_path, parquet_file)
partitions = self._partition_values(directory_path, parquet_path)
total_rows += file_metadata["total_rows"]
total_row_groups += file_metadata["row_groups"]
for column, codecs in file_metadata["compression"].items():
compression_by_column.setdefault(column, set()).update(codecs)
for key, value in partitions.items():
partition_columns.add(key)
partition_values.setdefault(key, set()).add(value)
file_entries.append(
{
"path": str(parquet_path),
"relative_path": str(parquet_path.relative_to(directory_path)),
"rows": file_metadata["total_rows"],
"row_groups": file_metadata["row_groups"],
"file_size": file_metadata["file_size"],
"partitions": partitions,
}
)
return {
"format": "parquet",
"source_type": "directory",
"directory": str(directory_path),
"file_count": len(parquet_files),
"files": file_entries,
"total_rows": total_rows,
"row_groups": total_row_groups,
"partitioning": partitioning,
"partition_columns": sorted(partition_columns),
"partition_values": {
key: sorted(values) for key, values in partition_values.items()
},
"compression": {
column: sorted(codecs)
for column, codecs in compression_by_column.items()
},
"compression_codecs": sorted(
{codec for codecs in compression_by_column.values() for codec in codecs}
),
}
def _compression_by_column(self, metadata: Any) -> Dict[str, set]:
"""Return compression codecs used for each column across row groups."""
compression_by_column: Dict[str, set] = {}
for row_group_index in range(metadata.num_row_groups):
row_group = metadata.row_group(row_group_index)
for column_index in range(row_group.num_columns):
column_chunk = row_group.column(column_index)
column_name = column_chunk.path_in_schema
compression = str(column_chunk.compression)
compression_by_column.setdefault(column_name, set()).add(compression)
return compression_by_column
def _partition_values(self, root: Path, parquet_path: Path) -> Dict[str, str]:
"""Extract Hive-style partition key/value pairs from a file path."""
partitions = {}
relative_parent = parquet_path.parent.relative_to(root)
for part in relative_parent.parts:
if "=" not in part:
continue
key, value = part.split("=", 1)
if key:
partitions[key] = value
return partitions
def _decode_metadata_map(
self, metadata: Optional[Dict[Any, Any]]
) -> Dict[str, str]:
"""Decode PyArrow metadata bytes to strings."""
if not metadata:
return {}
decoded = {}
for key, value in metadata.items():
decoded[self._decode_metadata_value(key)] = self._decode_metadata_value(
value
)
return decoded
def _decode_metadata_value(self, value: Any) -> str:
"""Decode a metadata key or value."""
if isinstance(value, bytes):
return value.decode("utf-8", errors="replace")
return str(value)
def _dataset_filter(self, filters: Any) -> Any:
"""Convert simple tuple filters to a PyArrow dataset expression."""
if filters is None:
return None
if self._is_filter_tuple(filters):
return self._comparison_expression(*filters)
if isinstance(filters, list):
if all(self._is_filter_tuple(item) for item in filters):
return self._and_expressions(
self._comparison_expression(*item) for item in filters
)
if all(isinstance(group, list) for group in filters):
return self._or_expressions(
self._and_expressions(
self._comparison_expression(*item) for item in group
)
for group in filters
)
return filters
def _is_filter_tuple(self, value: Any) -> bool:
"""Return whether value is a simple (column, operator, value) filter."""
return (
isinstance(value, tuple)
and len(value) == 3
and isinstance(value[0], str)
and isinstance(value[1], str)
)
def _comparison_expression(self, column: str, operator: str, value: Any) -> Any:
"""Create a PyArrow dataset comparison expression."""
field = ds.field(column)
if operator in {"=", "=="}:
return field == value
if operator == "!=":
return field != value
if operator == ">":
return field > value
if operator == ">=":
return field >= value
if operator == "<":
return field < value
if operator == "<=":
return field <= value
if operator.lower() == "in":
return field.isin(value)
if operator.lower() in {"not in", "not_in"}:
return ~field.isin(value)
raise ValidationError(f"Unsupported Parquet filter operator: {operator}")
def _and_expressions(self, expressions: Iterable[Any]) -> Any:
"""Combine expressions with AND."""
expression_list = list(expressions)
if not expression_list:
return None
combined = expression_list[0]
for expression in expression_list[1:]:
combined = combined & expression
return combined
def _or_expressions(self, expressions: Iterable[Any]) -> Any:
"""Combine expressions with OR."""
expression_list = [expr for expr in expressions if expr is not None]
if not expression_list:
return None
combined = expression_list[0]
for expression in expression_list[1:]:
combined = combined | expression
return combined
+13 -6
View File
@@ -13,6 +13,7 @@ Supported Registration Types:
* "repo": Repository ingestion methods
* "email": Email ingestion methods
* "db": Database ingestion methods
* "parquet": Parquet file and dataset ingestion methods
* "ingest": General ingestion methods
Algorithms Used:
@@ -24,7 +25,7 @@ Algorithms Used:
Key Features:
- Method registry for custom ingestion methods
- Task-based method organization (file, web, feed, stream, repo, email, db, ingest)
- Task-based method organization by source category
- Dynamic registration and unregistration
- Easy discovery of available methods
- Support for community-contributed extensions
@@ -37,11 +38,13 @@ Global Instances:
Example Usage:
>>> from semantica.ingest.registry import method_registry
>>> method_registry.register("file", "custom_method", custom_file_ingestion_function)
>>> method_registry.register(
... "file", "custom_method", custom_file_ingestion_function
... )
>>> available = method_registry.list_all("file")
"""
from typing import Any, Callable, Dict, List, Optional
from typing import Callable, Dict, List, Optional
class MethodRegistry:
@@ -56,6 +59,7 @@ class MethodRegistry:
"email": {},
"db": {},
"mcp": {},
"parquet": {},
"ingest": {},
}
@@ -65,7 +69,8 @@ class MethodRegistry:
Register a custom ingestion method.
Args:
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
task: Task type such as "file", "web", "feed", "stream",
"repo", "email", "db", "mcp", "parquet", or "ingest"
name: Method name
method_func: Method function
"""
@@ -79,7 +84,8 @@ class MethodRegistry:
Get method by task and name.
Args:
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
task: Task type such as "file", "web", "feed", "stream",
"repo", "email", "db", "mcp", "parquet", or "ingest"
name: Method name
Returns:
@@ -108,7 +114,8 @@ class MethodRegistry:
Unregister a method.
Args:
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
task: Task type such as "file", "web", "feed", "stream",
"repo", "email", "db", "mcp", "parquet", or "ingest"
name: Method name
"""
if task in cls._methods and name in cls._methods[task]:
@@ -11,6 +11,15 @@ Usage
Configure in your tool's MCP settings:
Claude Desktop / Windsurf / Cline / Continue / VS Code:
{
"mcpServers": {
"semantica": {
"command": "semantica-mcp"
}
}
}
Or using python -m:
{
"mcpServers": {
"semantica": {
@@ -20,7 +29,9 @@ Configure in your tool's MCP settings:
}
}
Run directly:
Run directly for testing:
semantica-mcp
# or
python -m semantica.mcp_server
Environment variables:
@@ -71,9 +82,7 @@ def _tool_extract_entities(args: dict) -> dict:
if not text:
return {"error": "text is required"}
from semantica.semantic_extract import NamedEntityRecognizer
from semantica.semantic_extract.cache import _result_cache
_result_cache.clear()
entities = NamedEntityRecognizer().extract(text)
entities = NamedEntityRecognizer().extract_entities(text)
return {
"entities": [
{"label": getattr(e, "label", str(e)),
@@ -91,10 +100,8 @@ def _tool_extract_relations(args: dict) -> dict:
if not text:
return {"error": "text is required"}
from semantica.semantic_extract import RelationExtractor, TripletExtractor
from semantica.semantic_extract.cache import _result_cache
_result_cache.clear()
relations = RelationExtractor().extract(text)
triplets = TripletExtractor().extract(text)
relations = RelationExtractor().extract_relations(text)
triplets = TripletExtractor().extract_triplets(text)
return {
"relations": [
{"source": getattr(r, "source", None),
@@ -600,7 +607,3 @@ def _run_stdio():
def main():
_run_stdio()
if __name__ == "__main__":
main()
+8
View File
@@ -0,0 +1,8 @@
"""
Entry point for python -m semantica.mcp_server
"""
from semantica.mcp_server import main
if __name__ == "__main__":
main()
+90 -66
View File
@@ -47,73 +47,97 @@ Author: Semantica Contributors
License: MIT
"""
from typing import Any, Dict, List, Optional, Union
from __future__ import annotations
from .config import Config, config
from .coreference_resolver import (
CoreferenceChain,
CoreferenceChainBuilder,
CoreferenceResolver,
EntityCoreferenceDetector,
Mention,
PronounResolver,
)
from .event_detector import (
Event,
EventClassifier,
EventDetector,
EventRelationshipExtractor,
TemporalEventProcessor,
)
from .extraction_validator import ExtractionValidator, ValidationResult
from .llm_extraction import LLMEnhancer, LLMExtraction, LLMResponse
from .methods import get_entity_method, get_relation_method, get_triplet_method
from .named_entity_recognizer import (
CustomEntityDetector,
EntityClassifier,
EntityConfidenceScorer,
NamedEntityRecognizer,
)
from .ner_extractor import Entity, NERExtractor
from .providers import (
AnthropicProvider,
BaseProvider,
GeminiProvider,
GroqProvider,
HuggingFaceLLMProvider,
HuggingFaceModelLoader,
OllamaProvider,
OpenAIProvider,
create_provider,
)
from .registry import (
MethodRegistry,
ProviderRegistry,
method_registry,
provider_registry,
)
from .relation_extractor import Relation, RelationExtractor
from .semantic_analyzer import (
RoleLabeler,
SemanticAnalyzer,
SemanticCluster,
SemanticClusterer,
SemanticRole,
SimilarityAnalyzer,
)
from .semantic_network_extractor import (
SemanticEdge,
SemanticNetwork,
SemanticNetworkExtractor,
SemanticNode,
)
from .triplet_extractor import (
RDFSerializer,
Triplet,
TripletExtractor,
TripletQualityChecker,
TripletValidator,
)
import importlib
from typing import Any, Dict, Tuple
_LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
# Named Entity Recognition
"NamedEntityRecognizer": (".named_entity_recognizer", "NamedEntityRecognizer"),
"EntityClassifier": (".named_entity_recognizer", "EntityClassifier"),
"EntityConfidenceScorer": (".named_entity_recognizer", "EntityConfidenceScorer"),
"CustomEntityDetector": (".named_entity_recognizer", "CustomEntityDetector"),
"NERExtractor": (".ner_extractor", "NERExtractor"),
"Entity": (".types", "Entity"),
# Relation Extraction
"RelationExtractor": (".relation_extractor", "RelationExtractor"),
"Relation": (".types", "Relation"),
# Event Detection
"EventDetector": (".event_detector", "EventDetector"),
"Event": (".event_detector", "Event"),
"EventClassifier": (".event_detector", "EventClassifier"),
"TemporalEventProcessor": (".event_detector", "TemporalEventProcessor"),
"EventRelationshipExtractor": (".event_detector", "EventRelationshipExtractor"),
# Coreference Resolution
"CoreferenceResolver": (".coreference_resolver", "CoreferenceResolver"),
"Mention": (".coreference_resolver", "Mention"),
"CoreferenceChain": (".coreference_resolver", "CoreferenceChain"),
"PronounResolver": (".coreference_resolver", "PronounResolver"),
"EntityCoreferenceDetector": (".coreference_resolver", "EntityCoreferenceDetector"),
"CoreferenceChainBuilder": (".coreference_resolver", "CoreferenceChainBuilder"),
# Triplet Extraction
"TripletExtractor": (".triplet_extractor", "TripletExtractor"),
"TripleExtractor": (".triplet_extractor", "TripletExtractor"),
"Triplet": (".types", "Triplet"),
"TripletValidator": (".triplet_extractor", "TripletValidator"),
"RDFSerializer": (".triplet_extractor", "RDFSerializer"),
"TripletQualityChecker": (".triplet_extractor", "TripletQualityChecker"),
# Semantic Analysis
"SemanticAnalyzer": (".semantic_analyzer", "SemanticAnalyzer"),
"SemanticRole": (".semantic_analyzer", "SemanticRole"),
"SemanticCluster": (".semantic_analyzer", "SemanticCluster"),
"SimilarityAnalyzer": (".semantic_analyzer", "SimilarityAnalyzer"),
"RoleLabeler": (".semantic_analyzer", "RoleLabeler"),
"SemanticClusterer": (".semantic_analyzer", "SemanticClusterer"),
# Semantic Network
"SemanticNetworkExtractor": (".semantic_network_extractor", "SemanticNetworkExtractor"),
"SemanticNode": (".semantic_network_extractor", "SemanticNode"),
"SemanticEdge": (".semantic_network_extractor", "SemanticEdge"),
"SemanticNetwork": (".semantic_network_extractor", "SemanticNetwork"),
# LLM Enhancement
"LLMExtraction": (".llm_extraction", "LLMExtraction"),
"LLMEnhancer": (".llm_extraction", "LLMEnhancer"),
"LLMResponse": (".llm_extraction", "LLMResponse"),
# Validation
"ExtractionValidator": (".extraction_validator", "ExtractionValidator"),
"ValidationResult": (".extraction_validator", "ValidationResult"),
# Providers
"BaseProvider": (".providers", "BaseProvider"),
"OpenAIProvider": (".providers", "OpenAIProvider"),
"GeminiProvider": (".providers", "GeminiProvider"),
"GroqProvider": (".providers", "GroqProvider"),
"AnthropicProvider": (".providers", "AnthropicProvider"),
"OllamaProvider": (".providers", "OllamaProvider"),
"HuggingFaceLLMProvider": (".providers", "HuggingFaceLLMProvider"),
"HuggingFaceModelLoader": (".providers", "HuggingFaceModelLoader"),
"create_provider": (".providers", "create_provider"),
# Registry
"ProviderRegistry": (".registry", "ProviderRegistry"),
"MethodRegistry": (".registry", "MethodRegistry"),
"provider_registry": (".registry", "provider_registry"),
"method_registry": (".registry", "method_registry"),
# Config
"Config": (".config", "Config"),
"config": (".config", "config"),
# Methods
"get_entity_method": (".methods", "get_entity_method"),
"get_relation_method": (".methods", "get_relation_method"),
"get_triplet_method": (".methods", "get_triplet_method"),
}
def __getattr__(name: str) -> Any:
"""Lazily load semantic extraction exports to avoid import cycles and optional deps."""
if name not in _LAZY_EXPORTS:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
module_name, attr_name = _LAZY_EXPORTS[name]
module = importlib.import_module(module_name, __name__)
value = getattr(module, attr_name)
globals()[name] = value
return value
__all__ = [
# Named Entity Recognition
+1 -3
View File
@@ -113,13 +113,11 @@ from typing import Any, Dict, List, Optional, Tuple, Union
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
from .ner_extractor import Entity
from .providers import HuggingFaceModelLoader, create_provider
from .registry import method_registry
from .relation_extractor import Relation
from .triplet_extractor import Triplet
from .cache import ExtractionCache
from .config import config
from .types import Entity, Relation, Triplet
try:
from .schemas import (
+1 -13
View File
@@ -70,29 +70,17 @@ Author: Semantica Contributors
License: MIT
"""
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Tuple, Union
from ..utils.exceptions import ProcessingError
from ..utils.helpers import safe_import
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .types import Entity
spacy, SPACY_AVAILABLE = safe_import("spacy")
@dataclass
class Entity:
"""Entity representation."""
text: str
label: str
start_char: int
end_char: int
confidence: float = 1.0
metadata: Dict[str, Any] = field(default_factory=dict)
class NERExtractor:
"""Named Entity Recognition extractor."""
@@ -70,25 +70,12 @@ License: MIT
"""
import re
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Tuple, Union
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .ner_extractor import Entity
@dataclass
class Relation:
"""Relation representation."""
subject: Entity
predicate: str
object: Entity
confidence: float = 1.0
context: str = ""
metadata: Dict[str, Any] = field(default_factory=dict)
from .types import Entity, Relation
class RelationExtractor:
@@ -70,34 +70,13 @@ Author: Semantica Contributors
License: MIT
"""
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Union
from urllib.parse import quote
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
from .ner_extractor import Entity
from .relation_extractor import Relation
@dataclass
class Triplet:
"""RDF triplet representation."""
subject: str
predicate: str
object: str
confidence: float = 1.0
metadata: Dict[str, Any] = field(default_factory=dict)
def get(self, key: str, default: Any = None) -> Any:
"""Get attribute value like a dictionary."""
return getattr(self, key, default)
def __getitem__(self, key: str) -> Any:
"""Get item like a dictionary."""
return getattr(self, key)
from .types import Entity, Relation, Triplet
class TripletExtractor:
+52
View File
@@ -0,0 +1,52 @@
"""
Shared semantic extraction data types.
These lightweight dataclasses live outside the extractor implementations so
method dispatchers and extractors can share result models without import cycles.
"""
from dataclasses import dataclass, field
from typing import Any, Dict
@dataclass
class Entity:
"""Entity representation."""
text: str
label: str
start_char: int
end_char: int
confidence: float = 1.0
metadata: Dict[str, Any] = field(default_factory=dict)
@dataclass
class Relation:
"""Relation representation."""
subject: Entity
predicate: str
object: Entity
confidence: float = 1.0
context: str = ""
metadata: Dict[str, Any] = field(default_factory=dict)
@dataclass
class Triplet:
"""RDF triplet representation."""
subject: str
predicate: str
object: str
confidence: float = 1.0
metadata: Dict[str, Any] = field(default_factory=dict)
def get(self, key: str, default: Any = None) -> Any:
"""Get attribute value like a dictionary."""
return getattr(self, key, default)
def __getitem__(self, key: str) -> Any:
"""Get item like a dictionary."""
return getattr(self, key)
+7 -5
View File
@@ -152,10 +152,11 @@ if EXPLORER_AVAILABLE:
enrich,
export_import,
graph,
ontology,
temporal,
vocabulary,
provenance,
sparql
provenance,
sparql
)
app.include_router(analytics.router)
@@ -164,12 +165,13 @@ if EXPLORER_AVAILABLE:
app.include_router(enrich.router)
app.include_router(export_import.router)
app.include_router(graph.router)
app.include_router(ontology.router)
app.include_router(temporal.router)
app.include_router(vocabulary.router)
app.include_router(provenance.router)
app.include_router(sparql.router)
app.include_router(provenance.router)
app.include_router(sparql.router)
logging.info("Explorer, Vocabulary, SPARQL, and Provenance API routes successfully mounted.")
logging.info("Explorer, Vocabulary, SPARQL, Provenance, and Ontology API routes successfully mounted.")
except Exception as exc:
logging.error(f"Failed to mount explorer routes: {exc}")
else:
+4 -2
View File
@@ -33,10 +33,10 @@ Example Usage:
>>> from semantica.utils import SUPPORTED_DOCUMENT_FORMATS, DEFAULT_CONFIG
>>> if file_extension in SUPPORTED_DOCUMENT_FORMATS:
... process_document(file_path)
>>>
>>>
>>> config = DEFAULT_CONFIG.copy()
>>> config["processing"]["batch_size"] = 200
>>>
>>>
>>> from semantica.utils import ERROR_CODES, PERFORMANCE_THRESHOLDS
>>> error_code = ERROR_CODES["VALIDATION_ERROR"]
>>> max_time = PERFORMANCE_THRESHOLDS["max_processing_time"]
@@ -57,6 +57,8 @@ SUPPORTED_DOCUMENT_FORMATS = [
"csv",
"xlsx",
"pptx",
"parquet",
"pq",
]
SUPPORTED_IMAGE_FORMATS = ["jpg", "jpeg", "png", "gif", "bmp", "tiff", "webp", "svg"]
+7 -7
View File
@@ -253,21 +253,21 @@ class ConsoleProgressDisplay(ProgressDisplay):
# If we have pipeline items, show all of them
if pipeline_items:
# Clear and show all pipeline items
sys.stdout.write("\r" + " " * 150 + "\r")
self._safe_write("\r" + " " * 150 + "\r")
# Show header if first time
if not hasattr(self, '_pipeline_header_shown'):
if self.use_emoji:
sys.stdout.write("🧠 Semantica - 📊 Current Progress\n")
self._safe_write("🧠 Semantica - 📊 Current Progress\n")
else:
sys.stdout.write("Semantica - Current Progress\n")
sys.stdout.write("=" * 150 + "\n")
self._safe_write("Semantica - Current Progress\n")
self._safe_write("=" * 150 + "\n")
self._pipeline_header_shown = True
# Display all pipeline items
for pipeline_item in pipeline_items:
self._display_item_line(pipeline_item)
sys.stdout.write("\n")
self._safe_write("\n")
sys.stdout.flush()
else:
+403 -1
View File
@@ -1,13 +1,18 @@
import sys
import unittest
from typing import Dict, Any, List
from semantica.deduplication.similarity_calculator import SimilarityCalculator
from semantica.deduplication.duplicate_detector import DuplicateDetector
from semantica.deduplication.duplicate_detector import (
DuplicateCandidate,
DuplicateDetector,
)
from semantica.deduplication.entity_merger import EntityMerger
from semantica.deduplication.merge_strategy import MergeStrategy
from semantica.deduplication.cluster_builder import ClusterBuilder
from semantica.deduplication.registry import MethodRegistry
from semantica.deduplication.config import DeduplicationConfig
from semantica.deduplication.methods import get_deduplication_method
from semantica.utils.progress_tracker import ConsoleProgressDisplay
class TestDeduplication(unittest.TestCase):
@@ -203,5 +208,402 @@ class TestDeduplication(unittest.TestCase):
self.assertIsNone(invalid)
class TestProgressTrackerEncoding(unittest.TestCase):
"""Regression tests for issue #531 — Unicode crash on cp1252 Windows consoles."""
def _make_cp1252_stdout(self):
"""Return a stdout-like object that raises UnicodeEncodeError for non-cp1252 chars."""
class CP1252Writer:
encoding = "cp1252"
def write(self, text):
text.encode("cp1252") # raises on emoji / block chars
def flush(self):
pass
return CP1252Writer()
def test_safe_write_does_not_crash_on_cp1252(self):
"""_safe_write must not raise UnicodeEncodeError on a cp1252 console."""
display = ConsoleProgressDisplay()
orig = sys.stdout
sys.stdout = self._make_cp1252_stdout()
try:
display._safe_write("🧠 Semantica - 📊 Current Progress\n")
except UnicodeEncodeError:
self.fail("_safe_write raised UnicodeEncodeError on cp1252 stdout")
finally:
sys.stdout = orig
def test_update_pipeline_header_does_not_crash_on_cp1252(self):
"""update() pipeline header write must not crash on a cp1252 console (issue #531)."""
from semantica.utils.progress_tracker import ProgressItem
display = ConsoleProgressDisplay()
display.use_emoji = True # force emoji path to exercise the fixed branch
orig = sys.stdout
sys.stdout = self._make_cp1252_stdout()
try:
display._safe_write("🧠 Semantica - 📊 Current Progress\n")
display._safe_write("=" * 150 + "\n")
except UnicodeEncodeError:
self.fail("Pipeline header write raised UnicodeEncodeError on cp1252 stdout")
finally:
sys.stdout = orig
def test_emoji_detection_disables_on_cp1252(self):
"""ConsoleProgressDisplay should auto-disable emoji when stdout is cp1252."""
orig = sys.stdout
sys.stdout = self._make_cp1252_stdout()
try:
display = ConsoleProgressDisplay()
self.assertFalse(display.use_emoji, "use_emoji should be False on cp1252 stdout")
finally:
sys.stdout = orig
class TestResultLimiting(unittest.TestCase):
"""Tests for issue #534 — max_results, top_k_per_entity, min_similarity, sort_by."""
def setUp(self):
# Six entities: three near-duplicate Apple variants + two Microsoft variants + one Google.
# Lower thresholds so all intra-brand pairs clear the bar.
self.entities = [
{"id": "a1", "name": "Apple Inc.", "type": "Company",
"properties": {"industry": "Technology"}},
{"id": "a2", "name": "Apple", "type": "Company",
"properties": {"industry": "Tech"}},
{"id": "a3", "name": "Apple Corp", "type": "Company",
"properties": {"industry": "Technology"}},
{"id": "b1", "name": "Microsoft Corporation", "type": "Company",
"properties": {"industry": "Software"}},
{"id": "b2", "name": "Microsoft Corp", "type": "Company",
"properties": {"industry": "Software"}},
{"id": "c1", "name": "Google LLC", "type": "Company",
"properties": {"industry": "Internet"}},
]
self.threshold = 0.3
def _base_detector(self, **kwargs):
return DuplicateDetector(
similarity_threshold=self.threshold,
confidence_threshold=self.threshold,
**kwargs,
)
# ------------------------------------------------------------------
# max_results
# ------------------------------------------------------------------
def test_max_results_caps_output(self):
detector = self._base_detector(max_results=1)
results = detector.detect_duplicates(self.entities)
self.assertLessEqual(len(results), 1)
def test_max_results_two(self):
detector = self._base_detector(max_results=2)
results = detector.detect_duplicates(self.entities)
self.assertLessEqual(len(results), 2)
def test_max_results_none_no_cap(self):
uncapped = self._base_detector()
large_cap = self._base_detector(max_results=999)
self.assertEqual(
len(uncapped.detect_duplicates(self.entities)),
len(large_cap.detect_duplicates(self.entities)),
)
def test_max_results_zero_returns_empty(self):
detector = self._base_detector(max_results=0)
self.assertEqual(detector.detect_duplicates(self.entities), [])
def test_max_results_returns_highest_confidence_first(self):
"""When capped, the kept candidates must be the highest-confidence ones."""
n = 2
all_results = self._base_detector().detect_duplicates(self.entities)
capped = self._base_detector(max_results=n).detect_duplicates(self.entities)
if len(all_results) >= n:
expected_ids = {
(c.entity1["id"], c.entity2["id"]) for c in all_results[:n]
}
actual_ids = {
(c.entity1["id"], c.entity2["id"]) for c in capped
}
self.assertEqual(expected_ids, actual_ids)
def test_max_results_empty_input(self):
detector = self._base_detector(max_results=5)
self.assertEqual(detector.detect_duplicates([]), [])
# ------------------------------------------------------------------
# top_k_per_entity
# ------------------------------------------------------------------
def test_top_k_per_entity_k1(self):
# OR semantics: keep if EITHER entity is under quota.
# A popular entity can appear > k times (each new partner brings it back).
# Invariant: no (entity1, entity2) pair is returned more than once.
k = 1
results = self._base_detector(top_k_per_entity=k).detect_duplicates(self.entities)
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
self.assertEqual(len(pairs), len(set(pairs)), "No duplicate pairs should appear")
# OR gives >= results than AND — at least 1 result when matches exist
and_results = self._base_detector().detect_duplicates(self.entities)
if and_results:
self.assertGreater(len(results), 0)
def test_top_k_per_entity_k2(self):
# Same OR semantics: no pair appears twice; result is bounded below by k=1 count
k = 2
results = self._base_detector(top_k_per_entity=k).detect_duplicates(self.entities)
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
self.assertEqual(len(pairs), len(set(pairs)), "No duplicate pairs should appear")
k1_results = self._base_detector(top_k_per_entity=1).detect_duplicates(self.entities)
self.assertGreaterEqual(len(results), len(k1_results))
def test_top_k_per_entity_large_k_same_as_none(self):
uncapped = self._base_detector().detect_duplicates(self.entities)
large_k = self._base_detector(top_k_per_entity=999).detect_duplicates(self.entities)
self.assertEqual(len(uncapped), len(large_k))
def test_top_k_per_entity_empty_input(self):
detector = self._base_detector(top_k_per_entity=2)
self.assertEqual(detector.detect_duplicates([]), [])
# ------------------------------------------------------------------
# min_similarity
# ------------------------------------------------------------------
def test_min_similarity_all_results_above_floor(self):
floor = 0.6
results = self._base_detector(min_similarity=floor).detect_duplicates(self.entities)
for c in results:
self.assertGreaterEqual(
c.similarity_score, floor,
f"Candidate score {c.similarity_score} is below min_similarity={floor}",
)
def test_min_similarity_very_high_returns_only_exact(self):
results = self._base_detector(min_similarity=1.0).detect_duplicates(self.entities)
for c in results:
self.assertEqual(c.similarity_score, 1.0)
def test_min_similarity_zero_does_not_over_filter(self):
no_floor = self._base_detector().detect_duplicates(self.entities)
zero_floor = self._base_detector(min_similarity=0.0).detect_duplicates(self.entities)
self.assertEqual(len(no_floor), len(zero_floor))
def test_min_similarity_stricter_than_threshold_reduces_results(self):
"""A min_similarity above similarity_threshold must not increase the result count."""
base = self._base_detector().detect_duplicates(self.entities)
stricter = self._base_detector(min_similarity=0.8).detect_duplicates(self.entities)
self.assertLessEqual(len(stricter), len(base))
def test_min_similarity_empty_input(self):
detector = self._base_detector(min_similarity=0.5)
self.assertEqual(detector.detect_duplicates([]), [])
# ------------------------------------------------------------------
# sort_by
# ------------------------------------------------------------------
def test_sort_by_confidence_descending(self):
results = self._base_detector(sort_by="confidence").detect_duplicates(self.entities)
scores = [c.confidence for c in results]
self.assertEqual(scores, sorted(scores, reverse=True))
def test_sort_by_similarity_score_descending(self):
results = self._base_detector(sort_by="similarity_score").detect_duplicates(self.entities)
scores = [c.similarity_score for c in results]
self.assertEqual(scores, sorted(scores, reverse=True))
def test_sort_by_default_is_confidence(self):
default = self._base_detector().detect_duplicates(self.entities)
explicit = self._base_detector(sort_by="confidence").detect_duplicates(self.entities)
self.assertEqual(
[(c.entity1["id"], c.entity2["id"]) for c in default],
[(c.entity1["id"], c.entity2["id"]) for c in explicit],
)
def test_sort_by_invalid_raises_at_construction(self):
with self.assertRaises(ValueError):
self._base_detector(sort_by="bogus_field")
def test_sort_by_invalid_message_contains_field_name(self):
with self.assertRaises(ValueError, msg="bogus_field") as ctx:
self._base_detector(sort_by="bogus_field")
self.assertIn("bogus_field", str(ctx.exception))
# ------------------------------------------------------------------
# Input validation (bug_002 / bug_003)
# ------------------------------------------------------------------
def test_max_results_negative_raises(self):
with self.assertRaises(ValueError):
self._base_detector(max_results=-1)
def test_max_results_float_raises(self):
with self.assertRaises(ValueError):
self._base_detector(max_results=1.5)
def test_top_k_per_entity_negative_raises(self):
with self.assertRaises(ValueError):
self._base_detector(top_k_per_entity=-1)
def test_top_k_per_entity_float_raises(self):
with self.assertRaises(ValueError):
self._base_detector(top_k_per_entity=2.5)
def test_min_similarity_above_1_raises(self):
with self.assertRaises(ValueError):
self._base_detector(min_similarity=1.1)
def test_min_similarity_below_0_raises(self):
with self.assertRaises(ValueError):
self._base_detector(min_similarity=-0.1)
def test_max_results_zero_is_valid(self):
# 0 is a non-negative int — must not raise
detector = self._base_detector(max_results=0)
self.assertEqual(detector.detect_duplicates(self.entities), [])
def test_top_k_per_entity_zero_is_valid(self):
detector = self._base_detector(top_k_per_entity=0)
self.assertEqual(detector.detect_duplicates(self.entities), [])
def test_min_similarity_boundary_0_valid(self):
self._base_detector(min_similarity=0.0) # must not raise
def test_min_similarity_boundary_1_valid(self):
self._base_detector(min_similarity=1.0) # must not raise
# ------------------------------------------------------------------
# top_k_per_entity OR semantics (bug_001)
# ------------------------------------------------------------------
def test_top_k_or_semantics_keeps_candidate_if_either_under_quota(self):
"""A high-ranked candidate must survive even if one of its entities hit k,
as long as the other entity is still under quota."""
# With k=1 and OR semantics, every entity can appear in AT LEAST one
# candidate. Verify that more candidates survive than would under AND.
k = 1
or_results = self._base_detector(top_k_per_entity=k).detect_duplicates(self.entities)
# Each entity should appear at least once — no entity completely starved
seen_ids: set = set()
for c in or_results:
seen_ids.add(c.entity1["id"])
seen_ids.add(c.entity2["id"])
# Entities that have at least one match above threshold must appear
all_ids_in_any_pair: set = set()
uncapped = self._base_detector().detect_duplicates(self.entities)
for c in uncapped:
all_ids_in_any_pair.add(c.entity1["id"])
all_ids_in_any_pair.add(c.entity2["id"])
self.assertEqual(seen_ids, all_ids_in_any_pair)
def test_group_merge_updates_normalized_entity_keys_for_int_ids(self):
"""Merged groups must update the normalized string lookup keys.
Regression for stale raw int keys: after a bridge candidate merges two
groups, later candidates involving the moved int-ID entities must still
attach to the returned group rather than an orphaned removed group.
"""
entities = [
{"id": 1, "name": "Alpha"},
{"id": 2, "name": "Alpha duplicate"},
{"id": 3, "name": "Alpha bridge"},
{"id": 4, "name": "Alpha merged"},
{"id": 5, "name": "Alpha later"},
]
candidates = [
DuplicateCandidate(entities[0], entities[1], 0.95, 0.95),
DuplicateCandidate(entities[2], entities[3], 0.94, 0.94),
DuplicateCandidate(entities[1], entities[2], 0.93, 0.93),
DuplicateCandidate(entities[2], entities[4], 0.92, 0.92),
]
groups = self._base_detector()._build_duplicate_groups(candidates)
self.assertEqual(len(groups), 1)
self.assertEqual(
{entity["id"] for entity in groups[0].entities},
{1, 2, 3, 4, 5},
)
# ------------------------------------------------------------------
# Combined options
# ------------------------------------------------------------------
def test_max_results_and_sort_by_similarity(self):
n = 2
results = self._base_detector(max_results=n, sort_by="similarity_score").detect_duplicates(self.entities)
self.assertLessEqual(len(results), n)
if len(results) == 2:
self.assertGreaterEqual(results[0].similarity_score, results[1].similarity_score)
def test_min_similarity_and_top_k_combined(self):
floor, k = 0.5, 1
results = self._base_detector(min_similarity=floor, top_k_per_entity=k).detect_duplicates(self.entities)
# min_similarity floor still applies
for c in results:
self.assertGreaterEqual(c.similarity_score, floor)
# OR semantics: no pair duplicated
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
self.assertEqual(len(pairs), len(set(pairs)))
def test_all_four_options_combined(self):
results = self._base_detector(
max_results=3,
top_k_per_entity=1,
min_similarity=0.3,
sort_by="similarity_score",
).detect_duplicates(self.entities)
self.assertLessEqual(len(results), 3)
scores = [c.similarity_score for c in results]
self.assertEqual(scores, sorted(scores, reverse=True))
for c in results:
self.assertGreaterEqual(c.similarity_score, 0.3)
def test_max_results_applied_after_top_k(self):
"""max_results must slice the already-top-k-filtered list, not pre-empt it."""
top_k_only = self._base_detector(top_k_per_entity=1).detect_duplicates(self.entities)
both = self._base_detector(top_k_per_entity=1, max_results=1).detect_duplicates(self.entities)
self.assertLessEqual(len(both), min(1, len(top_k_only)))
# ------------------------------------------------------------------
# incremental_detect
# ------------------------------------------------------------------
def test_incremental_detect_max_results(self):
new_e, existing = self.entities[:3], self.entities[3:]
results = self._base_detector(max_results=1).incremental_detect(new_e, existing)
self.assertLessEqual(len(results), 1)
def test_incremental_detect_min_similarity(self):
new_e, existing = self.entities[:3], self.entities[3:]
results = self._base_detector(min_similarity=0.99).incremental_detect(new_e, existing)
for c in results:
self.assertGreaterEqual(c.similarity_score, 0.99)
def test_incremental_detect_sort_by_similarity(self):
new_e, existing = self.entities[:3], self.entities[3:]
results = self._base_detector(sort_by="similarity_score").incremental_detect(new_e, existing)
scores = [c.similarity_score for c in results]
self.assertEqual(scores, sorted(scores, reverse=True))
def test_incremental_detect_top_k_per_entity(self):
new_e, existing = self.entities[:3], self.entities[3:]
k = 1
results = self._base_detector(top_k_per_entity=k).incremental_detect(new_e, existing)
# OR semantics: no pair appears twice
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
self.assertEqual(len(pairs), len(set(pairs)))
def test_incremental_detect_empty_new_entities(self):
detector = self._base_detector(max_results=5)
self.assertEqual(detector.incremental_detect([], self.entities), [])
def test_incremental_detect_empty_existing_entities(self):
detector = self._base_detector(max_results=5)
self.assertEqual(detector.incremental_detect(self.entities, []), [])
if __name__ == "__main__":
unittest.main()
+263
View File
@@ -0,0 +1,263 @@
"""Tests for Ontology Hub subissue 3 APIs."""
import pytest
from semantica.context.context_graph import ContextGraph
from semantica.explorer.app import create_app
from semantica.explorer.session import GraphSession
try:
from starlette.testclient import TestClient
except ImportError:
pytest.skip(
"starlette TestClient is required for explorer tests. Install semantica[explorer].",
allow_module_level=True,
)
def _build_ontology_graph() -> ContextGraph:
graph = ContextGraph(advanced_analytics=False)
onto_a = "http://example.org/onto-a"
onto_b = "http://example.org/onto-b"
person_a = "http://example.org/onto-a#Person"
person_b = "http://example.org/onto-b#PersonRecord"
name_a = "http://example.org/onto-a#name"
graph.add_node(
onto_a,
node_type="owl:Ontology",
content="Ontology A",
**{"rdfs:label": "Ontology A", "rdfs:comment": "Primary ontology", "version": "1.0.0"},
)
graph.add_node(
onto_b,
node_type="owl:Ontology",
content="Ontology B",
**{"rdfs:label": "Ontology B", "rdfs:comment": "Partner ontology", "version": "1.0.0"},
)
graph.add_node(
person_a,
node_type="owl:Class",
content="Person",
scheme_uri=onto_a,
**{"rdfs:label": "Person", "rdfs:comment": "A person", "skos:definition": "Human actor"},
)
graph.add_node(
name_a,
node_type="owl:DatatypeProperty",
content="name",
scheme_uri=onto_a,
**{"rdfs:label": "name", "rdfs:comment": "Display name"},
)
graph.add_node(
person_b,
node_type="owl:Class",
content="Person Record",
scheme_uri=onto_b,
**{"rdfs:label": "Person Record", "rdfs:comment": "A person profile"},
)
graph.add_edge(name_a, person_a, edge_type="rdfs:domain")
return graph
@pytest.fixture()
def client():
app = create_app(session=GraphSession(_build_ontology_graph()))
with TestClient(app) as test_client:
yield test_client
def test_alignment_round_trip(client):
payload = {
"source_uri": "http://example.org/onto-a#Person",
"target_uri": "http://example.org/onto-b#PersonRecord",
"relation": "owl:equivalentClass",
"confidence": 0.91,
"provenance": "Reviewed from source mapping table",
"source": "test",
"reviewer": "qa",
}
created = client.post("/api/ontology/alignments", json=payload)
assert created.status_code == 200
alignment = created.json()
assert alignment["confidence"] == 0.91
assert alignment["provenance"] == "Reviewed from source mapping table"
listed = client.get("/api/ontology/alignments")
assert listed.status_code == 200
assert [item["id"] for item in listed.json()] == [alignment["id"]]
removed = client.delete(f"/api/ontology/alignments?id={alignment['id']}")
assert removed.status_code == 200
assert client.get("/api/ontology/alignments").json() == []
def test_alignment_suggestions_are_ranked(client):
response = client.post(
"/api/ontology/suggest-alignments",
json={
"source_ontology_uri": "http://example.org/onto-a",
"target_ontology_uri": "http://example.org/onto-b",
"threshold": 0.35,
"limit": 5,
},
)
assert response.status_code == 200
suggestions = response.json()
assert suggestions
# Top suggestion should be the Person→PersonRecord pair (highest label similarity).
top = suggestions[0]
assert "Person" in top["source_label"]
assert "Person" in top["target_label"]
# Results must be sorted descending by score.
assert suggestions == sorted(suggestions, key=lambda item: item["score"], reverse=True)
def test_health_returns_dimensions_and_issues(client):
response = client.get("/api/ontology/health?uri=http%3A%2F%2Fexample.org%2Fonto-a")
assert response.status_code == 200
payload = response.json()
assert payload["total_score"] >= 0
assert {dimension["key"] for dimension in payload["dimensions"]} == {
"completeness",
"consistency",
"shacl",
"alignment",
"documentation",
}
assert isinstance(payload["issues"], list)
def test_shacl_generate_and_shapes(client):
response = client.post(
"/api/ontology/shacl/generate",
json={"uri": "http://example.org/onto-a", "quality_tier": "strict"},
)
assert response.status_code == 200
payload = response.json()
assert "sh:NodeShape" in payload["shacl_turtle"]
assert payload["shape_count"] >= 1
shapes = client.get("/api/ontology/shacl/shapes?uri=http%3A%2F%2Fexample.org%2Fonto-a")
assert shapes.status_code == 200
assert shapes.json()["shapes"]
def test_shacl_validate_returns_unavailable(client):
response = client.post(
"/api/ontology/shacl/validate",
json={
"uri": "http://example.org/onto-a",
"shacl_turtle": "@prefix sh: <http://www.w3.org/ns/shacl#> .",
},
)
assert response.status_code == 200
payload = response.json()
assert payload["status"] == "unavailable", "stub must not report conforms=True before validation is wired"
assert payload["conforms"] is False
assert isinstance(payload["violations"], list)
def test_shacl_validate_rejects_empty_turtle(client):
response = client.post(
"/api/ontology/shacl/validate",
json={"uri": "http://example.org/onto-a", "shacl_turtle": " "},
)
assert response.status_code == 422
def test_health_returns_404_for_unknown_ontology(client):
response = client.get("/api/ontology/health?uri=http%3A%2F%2Fnot-loaded.example%2Fonto")
assert response.status_code == 404
def test_health_shacl_dimension_is_zero_when_unavailable(client):
payload = client.get("/api/ontology/health?uri=http%3A%2F%2Fexample.org%2Fonto-a").json()
shacl_dim = next(d for d in payload["dimensions"] if d["key"] == "shacl")
assert shacl_dim["status"] == "unavailable"
assert shacl_dim["score"] == 0.0
# Total score must NOT include the unavailable dimension in its average.
scoreable = [d for d in payload["dimensions"] if d["status"] != "unavailable"]
expected_total = round(sum(d["score"] for d in scoreable) / len(scoreable), 1)
assert payload["total_score"] == expected_total
def test_delete_unknown_alignment_returns_404(client):
response = client.delete("/api/ontology/alignments?id=does-not-exist")
assert response.status_code == 404
def test_alignment_upsert_is_idempotent(client):
payload = {
"source_uri": "http://example.org/onto-a#Person",
"target_uri": "http://example.org/onto-b#PersonRecord",
"relation": "owl:equivalentClass",
"confidence": 0.80,
}
first = client.post("/api/ontology/alignments", json=payload).json()
updated_payload = {**payload, "confidence": 0.95}
second = client.post("/api/ontology/alignments", json=updated_payload).json()
assert first["id"] == second["id"], "upsert must reuse the same deterministic ID"
assert second["confidence"] == 0.95
assert second["created_at"] == first["created_at"], "created_at must not change on update"
listed = client.get("/api/ontology/alignments").json()
assert len(listed) == 1
def test_alignment_accepts_external_uri(client):
payload = {
"source_uri": "http://example.org/onto-a#Person",
"target_uri": "http://schema.org/Person", # not in local graph
"relation": "owl:equivalentClass",
"confidence": 0.75,
}
response = client.post("/api/ontology/alignments", json=payload)
assert response.status_code == 200
alignment = response.json()
assert alignment["target_label"] == "Person" # derived from URI fragment
def test_suggest_alignments_returns_embedding_similarity(client):
response = client.post(
"/api/ontology/suggest-alignments",
json={
"source_ontology_uri": "http://example.org/onto-a",
"target_ontology_uri": "http://example.org/onto-b",
"threshold": 0.20,
"limit": 10,
},
)
assert response.status_code == 200
suggestions = response.json()
assert suggestions
# When sklearn is available, embedding_similarity should be populated.
top = suggestions[0]
assert top["embedding_similarity"] is not None, (
"TF-IDF embedding similarity must be returned when sklearn is installed"
)
# Combined score must be a weighted blend, not purely the label score.
assert top["score"] != top["label_similarity"] or top["embedding_similarity"] == top["label_similarity"]
def test_shacl_validate_rejects_invalid_turtle_syntax(client):
response = client.post(
"/api/ontology/shacl/validate",
json={
"uri": "http://example.org/onto-a",
"shacl_turtle": "this is not valid turtle !!!",
},
)
assert response.status_code == 422
def test_health_alignment_coverage_uses_set_lookup(client):
# Create an alignment first so coverage score can be non-zero.
client.post("/api/ontology/alignments", json={
"source_uri": "http://example.org/onto-a#Person",
"target_uri": "http://example.org/onto-b#PersonRecord",
"relation": "owl:equivalentClass",
"confidence": 0.9,
})
payload = client.get("/api/ontology/health?uri=http%3A%2F%2Fexample.org%2Fonto-a").json()
alignment_dim = next(d for d in payload["dimensions"] if d["key"] == "alignment")
assert alignment_dim["score"] > 0.0, "alignment coverage must be non-zero after recording an alignment"
+98
View File
@@ -0,0 +1,98 @@
import os
import subprocess
import sys
import textwrap
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
def _run_python_with_blocked_modules(
code: str,
blocked_modules: tuple[str, ...],
) -> subprocess.CompletedProcess[str]:
blocker = f"""
import importlib.abc
import sys
BLOCKED_MODULES = {blocked_modules!r}
class OptionalDependencyBlocker(importlib.abc.MetaPathFinder):
def find_spec(self, fullname, path=None, target=None):
root_name = fullname.split(".", 1)[0]
if root_name in BLOCKED_MODULES:
err = ModuleNotFoundError(f"No module named '{{root_name}}'")
err.name = root_name
raise err
return None
sys.meta_path.insert(0, OptionalDependencyBlocker())
"""
env = os.environ.copy()
env["PYTHONPATH"] = str(REPO_ROOT)
env["PYTHONDONTWRITEBYTECODE"] = "1"
return subprocess.run(
[sys.executable, "-c", textwrap.dedent(blocker + "\n" + code)],
cwd=REPO_ROOT,
env=env,
text=True,
capture_output=True,
check=False,
)
def test_file_ingestion_imports_without_optional_backends() -> None:
result = _run_python_with_blocked_modules(
"""
from semantica.ingest import FileIngestor, ingest_file
print(FileIngestor.__name__, callable(ingest_file))
""",
("git", "bs4", "pyarrow"),
)
assert result.returncode == 0, result.stderr
assert "FileIngestor True" in result.stdout
def test_repository_ingestion_reports_missing_gitpython_when_used() -> None:
result = _run_python_with_blocked_modules(
"""
from semantica.ingest import ingest_repository
try:
ingest_repository("https://example.com/repo.git")
except Exception as exc:
print(type(exc).__name__, exc)
else:
raise SystemExit("expected repository ingestion to fail without GitPython")
""",
("git",),
)
assert result.returncode == 0, result.stderr
assert "ConfigurationError" in result.stdout
assert "Repository ingestion" in result.stdout
assert "GitPython" in result.stdout
def test_parquet_ingestion_reports_missing_pyarrow_when_used() -> None:
result = _run_python_with_blocked_modules(
"""
from semantica.ingest import ingest_parquet
try:
ingest_parquet("events.parquet")
except Exception as exc:
print(type(exc).__name__, exc)
else:
raise SystemExit("expected parquet ingestion to fail without pyarrow")
""",
("pyarrow",),
)
assert result.returncode == 0, result.stderr
assert "ConfigurationError" in result.stdout
assert "Parquet ingestion" in result.stdout
assert "pyarrow" in result.stdout
+150
View File
@@ -0,0 +1,150 @@
from pathlib import Path
import pytest
pa = pytest.importorskip("pyarrow")
pq = pytest.importorskip("pyarrow.parquet")
from semantica.ingest import ( # noqa: E402
ParquetData,
ParquetIngestor,
ingest,
ingest_file,
ingest_parquet,
list_available_methods,
)
from semantica.ingest.file_ingestor import FileTypeDetector # noqa: E402
from semantica.utils.exceptions import ValidationError # noqa: E402
@pytest.fixture
def sample_parquet(tmp_path: Path) -> Path:
path = tmp_path / "events.parquet"
table = pa.table(
{
"id": [1, 2, 3],
"name": ["alpha", "beta", "gamma"],
"score": [0.7, 0.8, 0.9],
"city": ["Pune", "Delhi", "Mumbai"],
}
)
pq.write_table(table, path, compression="snappy")
return path
@pytest.fixture
def partitioned_parquet(tmp_path: Path) -> Path:
root = tmp_path / "events_partitioned"
us_2025 = root / "country=US" / "year=2025"
us_2025.mkdir(parents=True)
pq.write_table(
pa.table({"id": [1, 2], "value": ["a", "b"]}),
us_2025 / "part-0.parquet",
compression="gzip",
)
ca_2026 = root / "country=CA" / "year=2026"
ca_2026.mkdir(parents=True)
pq.write_table(
pa.table({"id": [3], "value": ["c"]}),
ca_2026 / "part-1.parquet",
compression="gzip",
)
return root
def test_parquet_file_ingestion_reads_data_schema_and_metadata(
sample_parquet: Path,
) -> None:
ingestor = ParquetIngestor()
result = ingestor.ingest_file(sample_parquet)
assert isinstance(result, ParquetData)
assert result.row_count == 3
assert result.columns == ["id", "name", "score", "city"]
assert result.data[0]["name"] == "alpha"
assert result.schema["columns"] == ["id", "name", "score", "city"]
assert result.schema["fields"][0]["type"] == "int64"
assert result.metadata["total_rows"] == 3
assert result.metadata["row_groups"] == 1
assert result.metadata["compression_codecs"] == ["SNAPPY"]
def test_parquet_selective_column_reading_with_limit(sample_parquet: Path) -> None:
ingestor = ParquetIngestor()
result = ingestor.ingest_file(sample_parquet, columns=["id", "name"], limit=2)
assert result.row_count == 2
assert result.columns == ["id", "name"]
assert result.data == [{"id": 1, "name": "alpha"}, {"id": 2, "name": "beta"}]
assert result.metadata["selected_columns"] == ["id", "name"]
assert result.metadata["limit"] == 2
def test_parquet_schema_and_metadata_can_be_extracted_without_rows(
sample_parquet: Path,
) -> None:
ingestor = ParquetIngestor()
schema = ingestor.extract_schema(sample_parquet)
metadata = ingestor.extract_metadata(sample_parquet)
result = ingestor.ingest_file(sample_parquet, include_data=False)
assert schema["columns"] == ["id", "name", "score", "city"]
assert metadata["total_rows"] == 3
assert metadata["format"] == "parquet"
assert result.row_count == 0
assert result.data == []
assert result.metadata["include_data"] is False
def test_partitioned_parquet_directory_ingestion(partitioned_parquet: Path) -> None:
ingestor = ParquetIngestor()
result = ingestor.ingest_directory(partitioned_parquet)
assert result.row_count == 3
assert set(result.columns) == {"id", "value", "country", "year"}
assert {row["country"] for row in result.data} == {"US", "CA"}
assert result.metadata["file_count"] == 2
assert result.metadata["total_rows"] == 3
assert result.metadata["partition_columns"] == ["country", "year"]
assert result.metadata["partition_values"] == {
"country": ["CA", "US"],
"year": ["2025", "2026"],
}
assert result.metadata["compression_codecs"] == ["GZIP"]
def test_parquet_convenience_methods_and_unified_dispatch(sample_parquet: Path) -> None:
direct = ingest_parquet(sample_parquet, columns=["name"])
via_file_method = ingest_file(sample_parquet, method="parquet", limit=1)
unified = ingest(sample_parquet)
unified_batch = ingest([sample_parquet])
methods = list_available_methods("parquet")
assert isinstance(direct, ParquetData)
assert direct.columns == ["name"]
assert isinstance(via_file_method, ParquetData)
assert via_file_method.row_count == 1
assert isinstance(unified["data"], ParquetData)
assert isinstance(unified_batch["data"][0], ParquetData)
assert "metadata" in methods["parquet"]
def test_file_type_detector_recognizes_parquet_magic_number() -> None:
detector = FileTypeDetector()
assert detector.detect_type("dataset", content=b"PAR1payload") == "parquet"
assert detector.is_supported("parquet")
def test_parquet_ingestion_rejects_negative_limit(sample_parquet: Path) -> None:
ingestor = ParquetIngestor()
with pytest.raises(ValidationError):
ingestor.ingest_file(sample_parquet, limit=-1)
+120
View File
@@ -0,0 +1,120 @@
from __future__ import annotations
import os
import subprocess
import sys
import textwrap
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
def _run_python(code: str) -> subprocess.CompletedProcess[str]:
env = os.environ.copy()
env["PYTHONPATH"] = str(REPO_ROOT)
env["PYTHONDONTWRITEBYTECODE"] = "1"
return subprocess.run(
[sys.executable, "-c", textwrap.dedent(code)],
cwd=REPO_ROOT,
env=env,
text=True,
capture_output=True,
check=False,
)
def test_core_extractors_import_from_package() -> None:
from semantica.semantic_extract import (
NERExtractor,
RelationExtractor,
TripletExtractor,
)
assert NERExtractor.__name__ == "NERExtractor"
assert RelationExtractor.__name__ == "RelationExtractor"
assert TripletExtractor.__name__ == "TripletExtractor"
def test_triple_extractor_alias_imports() -> None:
from semantica.semantic_extract import TripleExtractor, TripletExtractor
assert TripleExtractor is TripletExtractor
def test_method_dispatchers_import_without_extractor_cycle() -> None:
from semantica.semantic_extract.methods import (
get_entity_method,
get_relation_method,
get_triplet_method,
)
assert callable(get_entity_method("pattern"))
assert callable(get_relation_method("pattern"))
assert callable(get_triplet_method("pattern"))
def test_import_order_methods_before_extractors() -> None:
result = _run_python(
"""
from semantica.semantic_extract.methods import get_entity_method
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
print(get_entity_method("pattern").__name__, NERExtractor.__name__, RelationExtractor.__name__, TripletExtractor.__name__)
"""
)
assert result.returncode == 0, result.stderr
assert "extract_entities_pattern NERExtractor RelationExtractor TripletExtractor" in result.stdout
def test_import_order_extractors_before_methods() -> None:
result = _run_python(
"""
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
from semantica.semantic_extract.methods import get_entity_method
print(NERExtractor.__name__, RelationExtractor.__name__, TripletExtractor.__name__, get_entity_method("pattern").__name__)
"""
)
assert result.returncode == 0, result.stderr
assert "NERExtractor RelationExtractor TripletExtractor extract_entities_pattern" in result.stdout
def test_legacy_type_imports_still_work() -> None:
from semantica.semantic_extract.ner_extractor import Entity
from semantica.semantic_extract.relation_extractor import Relation
from semantica.semantic_extract.triplet_extractor import Triplet
from semantica.semantic_extract.types import (
Entity as SharedEntity,
Relation as SharedRelation,
Triplet as SharedTriplet,
)
assert Entity is SharedEntity
assert Relation is SharedRelation
assert Triplet is SharedTriplet
def test_core_package_imports_do_not_require_yaml() -> None:
result = _run_python(
"""
import importlib.abc
import sys
class OptionalDependencyBlocker(importlib.abc.MetaPathFinder):
def find_spec(self, fullname, path=None, target=None):
if fullname.split(".", 1)[0] == "yaml":
raise ModuleNotFoundError("No module named 'yaml'")
return None
sys.meta_path.insert(0, OptionalDependencyBlocker())
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
print(NERExtractor.__name__, RelationExtractor.__name__, TripletExtractor.__name__)
"""
)
assert result.returncode == 0, result.stderr
assert "NERExtractor RelationExtractor TripletExtractor" in result.stdout
@@ -10,6 +10,10 @@ import importlib.util
# Add project root to path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
# Save originals before injecting mocks so they can be restored after import.
_MOCKED_MODULES = ["spacy", "instructor", "groq", "openai", "sentence_transformers", "transformers"]
_original_modules = {k: sys.modules.get(k) for k in _MOCKED_MODULES}
# Mock external dependencies with __spec__ for importlib checks
mock_spacy = MagicMock()
mock_spacy.__spec__ = MagicMock()
@@ -33,6 +37,16 @@ from semantica.semantic_extract import NERExtractor
from semantica.semantic_extract.methods import extract_entities_llm, _extract_entities_chunked, extract_relations_llm, extract_triplets_llm
from semantica.semantic_extract.providers import BaseProvider
# Restore real sys.modules entries now that the mocks have served their purpose
# for the imports above. Leaving them in place would poison other test modules
# (e.g. test_pr482_deepseek_openai) that need the real packages at test run time.
for _key, _original in _original_modules.items():
if _original is None:
sys.modules.pop(_key, None)
else:
sys.modules[_key] = _original
class EntitiesResponse(BaseModel):
entities: List[dict]