mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-30 04:40:16 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2ef6e9f4b1 | ||
|
|
18da322e0d | ||
|
|
15d58f2b88 | ||
|
|
ce3a8c9895 | ||
|
|
508e05d367 | ||
|
|
03f99f016d | ||
|
|
56e7d9d821 | ||
|
|
6860bdbec3 | ||
|
|
ac5015bc3f | ||
|
|
29c72f59b3 | ||
|
|
21c2f190f8 | ||
|
|
8ef67b8bda | ||
|
|
bc57837b86 | ||
|
|
0439cf884d | ||
|
|
141bf80394 | ||
|
|
0c4e18e256 | ||
|
|
39045d783b | ||
|
|
afa54e1bbf | ||
|
|
a01b3c36fc | ||
|
|
0dda380580 | ||
|
|
e7c9f6e7f3 | ||
|
|
b828ebfef0 | ||
|
|
6330627ddb | ||
|
|
24e327d161 | ||
|
|
b9f3c59443 | ||
|
|
34955ba246 | ||
|
|
95b46f12b6 | ||
|
|
245ff76b99 | ||
|
|
a2b6a481dc | ||
|
|
c7edba88ea | ||
|
|
eb8598e0cf | ||
|
|
64d4157644 | ||
|
|
e1b1e63541 | ||
|
|
6b0a8e60ce | ||
|
|
8b22a58b8f | ||
|
|
ba286a55ea | ||
|
|
f610eff0ed | ||
|
|
e9e7278720 | ||
|
|
6b2cafd02e | ||
|
|
90d857a98f | ||
|
|
1861ca578c | ||
|
|
63acc7a66e | ||
|
|
00ceb09960 | ||
|
|
2909e0fa14 | ||
|
|
d2574990fa | ||
|
|
53d1da87c0 | ||
|
|
9e916a82b5 | ||
|
|
e8bf0e50d3 | ||
|
|
269fdaa9fb | ||
|
|
2d9bbf08b1 |
+449
-2235
File diff suppressed because it is too large
Load Diff
@@ -6,6 +6,8 @@
|
||||
|
||||
**A Framework for Building Context Graphs and Decision Intelligence Layers for AI**
|
||||
|
||||
🌐 **[Website](https://getsemantica.ai/)** | 📚 **[Documentation](https://docs.getsemantica.ai/)**
|
||||
|
||||
[](https://www.python.org/)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://pypi.org/project/semantica/)
|
||||
@@ -1029,6 +1031,27 @@ pytest tests/
|
||||
|
||||
---
|
||||
|
||||
## 🏢 Enterprise Support
|
||||
|
||||
**Production-grade Semantica for your organization**
|
||||
|
||||
🚀 **[Website](https://getsemantica.ai/)** — Enterprise solutions and custom domain-specific implementations
|
||||
|
||||
**Domain-Specific Solutions:**
|
||||
- 🏥 **Healthcare** — Clinical decision support, patient safety, compliance
|
||||
- � **Finance** — Fraud detection, regulatory compliance, risk management
|
||||
- ⚖️ **Legal** — Evidence analysis, contract review, case law reasoning
|
||||
- � **Cybersecurity** — Threat attribution, incident response, provenance
|
||||
- 🏛️ **Government** — Policy tracking, classified information governance
|
||||
|
||||
**Custom Solutions Available:**
|
||||
- � Private cloud deployment
|
||||
- 🛡️ Enterprise security compliance
|
||||
- � Custom analytics and reporting
|
||||
- 🎯 Professional implementation services
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
All contributions welcome — bug fixes, features, tests, and docs.
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
# Semantica 0.5.0 Release Notes
|
||||
|
||||
## 🎉 Major Release: Distance Intelligence & Ontology Hub Complete
|
||||
|
||||
**Release Date:** May 11, 2026
|
||||
**Version:** 0.5.0
|
||||
|
||||
---
|
||||
|
||||
## 🚀 **MAJOR HIGHLIGHTS**
|
||||
|
||||
### **Distance Intelligence Framework** (PR #502, @KaifAhmad1)
|
||||
- **Embedding Cache Optimization**: Per-session graph revision-based caching for 10x+ performance improvement
|
||||
- **Advanced UI Features**: Ego mode, overlays, heatmap, and path inspector
|
||||
- **Semantic Neighborhood Search**: Context-aware similarity with proximity metrics
|
||||
- **Distance Matrix API**: N×N semantic distance calculations with caching
|
||||
|
||||
### **Complete Ontology Hub Suite** (PR #517, @KaifAhmad1 @ZohaibHassan16)
|
||||
- **Alignments Tab** (PR #524): Cross-ontology alignment authoring with ML suggestions
|
||||
- **Health Dashboard** (PR #524): Quality scoring across 5 dimensions with issue tracking
|
||||
- **SHACL Studio** (PR #524): Interactive shape generation and validation
|
||||
- **Visual Editor** (PR #519): Canvas-based ontology authoring without hand-coding
|
||||
- **Registry & Search** (PR #518): Comprehensive ontology management and discovery
|
||||
|
||||
### **Security Hardening** (Security Enhancement PR, @KaifAhmad1)
|
||||
- **12 Critical Vulnerabilities Fixed**: Eval injection, XXE, SQL injection, and more
|
||||
- **SSRF Protection**: Comprehensive URL validation and hostname resolution
|
||||
- **Input Validation**: Enhanced file upload restrictions and format detection
|
||||
- **CORS & Headers**: Proper security headers and WebSocket protection
|
||||
|
||||
---
|
||||
|
||||
## 📊 **BY THE NUMBERS**
|
||||
|
||||
- **12 Major Features** ✅ Tested & Verified
|
||||
- **16 Ontology Hub API Endpoints** ✅ Production Ready
|
||||
- **57 New Distance Intelligence Tests** ✅ All Passing
|
||||
- **32 Parquet Ingestion Tests** ✅ All Passing
|
||||
- **12 Security Vulnerabilities** ✅ All Patched
|
||||
- **100% Test Coverage** ✅ Core Features Verified
|
||||
|
||||
---
|
||||
|
||||
## 🔧 **NEW FEATURES**
|
||||
|
||||
### **Performance & Architecture**
|
||||
- **Distance Intelligence Embedding Cache** (PR #502, @KaifAhmad1): Thread-safe per-session caching with automatic invalidation
|
||||
- **Parquet File Ingestion** (PR #548, @Luffy2208): PyArrow backend with column selection and partition support
|
||||
- **Indexed Search** (PR #481, @ZohaibHassan16): O(log n) search for large graphs (118k nodes: 24ms → 0.004ms)
|
||||
|
||||
### **Ontology Hub Suite**
|
||||
- **Cross-ontology Alignments** (PR #524, @KaifAhmad1 @ZohaibHassan16): ML-powered suggestions with confidence scoring
|
||||
- **Quality Health Dashboard** (PR #524, @KaifAhmad1 @ZohaibHassan16): 5-dimension scoring with actionable issue tracking
|
||||
- **SHACL Studio** (PR #524, @KaifAhmad1 @ZohaibHassan16): Interactive shape authoring with Monaco editor
|
||||
- **Visual Ontology Editor** (PR #519, @KaifAhmad1): Drag-and-drop ontology construction
|
||||
- **16 Backend Endpoints** (PRs #518, #519, #524, @KaifAhmad1 @ZohaibHassan16): Complete CRUD and analysis capabilities
|
||||
|
||||
### **UI & User Experience**
|
||||
- **Distance Intelligence UI** (PR #502, @KaifAhmad1 @ZohaibHassan16): Ego mode, overlays, heatmap, path inspector
|
||||
- **Explorer Redesign** (PR #516, @ZohaibHassan16): Modern hero section with live metrics
|
||||
- **Graph Workspace Declutter** (PR #483, @ZohaibHassan16): Improved visualization for dense graphs
|
||||
- **Bidirectional Path Finding** (PR #469, @KaifAhmad1): Undirected traversal support
|
||||
|
||||
### **Platform Compatibility**
|
||||
- **Windows Installation Fixes** (PR #532, @KaifAhmad1): Removed faiss-gpu from [all], Unicode console support
|
||||
- **Cross-platform Dependencies** (PR #527, @ZohaibHassan16): Proper optional dependency management
|
||||
- **MCP Server Package Structure** (PR #541, @KaifAhmad1): Fixed pipx installation issues
|
||||
|
||||
### **Algorithm Enhancements**
|
||||
- **DuplicateDetector Result Limiting** (PR #534, @KaifAhmad1): Ranking, sorting, and incremental detection features
|
||||
- **ConflictDetector Parameter Handling** (PR #533, @KaifAhmad1): Method parameter validation and error handling
|
||||
|
||||
---
|
||||
|
||||
## 🛡️ **SECURITY IMPROVEMENTS** (Security Enhancement PR, @KaifAhmad1)
|
||||
|
||||
### **Critical Fixes**
|
||||
- **Eval Injection** (CWE-95): Replaced with `fractions.Fraction` in media parser
|
||||
- **Pickle Deserialization** (CWE-502): Switched to JSON with migration support
|
||||
- **SQL Injection** (CWE-89): Parameterized queries and input validation
|
||||
- **XXE Protection** (CWE-611): `defusedxml` hardening for all RDF parsing
|
||||
|
||||
### **Web Security**
|
||||
- **SSRF Protection**: URL validation with hostname resolution
|
||||
- **CORS Hardening**: Narrowed origins and WebSocket limits
|
||||
- **Security Headers**: HSTS, X-Content-Type-Options, X-Frame-Options
|
||||
- **Path Traversal**: `Path.resolve().relative_to()` protection
|
||||
|
||||
### **Input Validation**
|
||||
- **File Upload Restrictions**: Extension allowlist and size limits
|
||||
- **SPARQL Limits**: Row caps, timeouts, and concurrency controls
|
||||
- **ReDoS Prevention**: Eliminated polynomial regex patterns
|
||||
|
||||
---
|
||||
|
||||
## 🔍 **QUALITY ASSURANCE**
|
||||
|
||||
### **Testing Coverage**
|
||||
- **Distance Intelligence**: 57 new tests, 100% passing
|
||||
- **Parquet Ingestion**: 32 tests, comprehensive coverage
|
||||
- **Security Fixes**: 14 vulnerability-specific tests
|
||||
- **UI Components**: All major features verified
|
||||
- **Platform Tests**: Windows, Linux compatibility confirmed
|
||||
|
||||
### **Performance Benchmarks**
|
||||
- **Embedding Cache**: 10x+ improvement in repeated requests
|
||||
- **Search Performance**: 6,000x faster for large graphs
|
||||
- **Memory Efficiency**: Lazy loading and optional dependencies
|
||||
- **Concurrent Operations**: Thread-safe caching with locks
|
||||
|
||||
---
|
||||
|
||||
## 🔄 **BREAKING CHANGES**
|
||||
|
||||
### **Dependencies**
|
||||
- **Windows Users**: `faiss-gpu` removed from `[all]` - install `[gpu]` explicitly if needed
|
||||
- **Optional Dependencies**: Now lazy-loaded to improve import performance
|
||||
|
||||
### **API Changes**
|
||||
- **ConflictDetector**: Fixed duplicate method definitions with proper parameter handling
|
||||
- **DuplicateDetector**: New result limiting and ranking options
|
||||
|
||||
---
|
||||
|
||||
## 📚 **DOCUMENTATION**
|
||||
|
||||
- **Comprehensive Changelog**: Detailed feature descriptions and credits
|
||||
- **API Documentation**: All new endpoints documented
|
||||
- **Security Advisory**: Complete vulnerability disclosure and fixes
|
||||
- **Migration Guide**: Breaking changes and upgrade instructions
|
||||
|
||||
---
|
||||
|
||||
## 🙏 **CREDITS**
|
||||
|
||||
**Core Contributors:**
|
||||
- **@KaifAhmad1** - Distance Intelligence (PR #502), Security Hardening, Ontology Hub (PRs #517, #518, #519, #524), Windows Fixes (PR #532), ConflictDetector (PR #533), Testing & Release Preparation
|
||||
- **@ZohaibHassan16** - Ontology Hub UI (PRs #516, #518, #519, #524), Graph Explorer (PRs #420, #481, #483, #503), Semantic Extract (PR #536), Lazy Loading (PR #535)
|
||||
- **@Luffy2208** - Parquet Ingestion Support (PR #548)
|
||||
- **@liling** - DeepSeek Provider Integration (PR #482)
|
||||
- **@Sameer6305** - Provenance Traversal Fixes (PR #480), Named Graph Support
|
||||
|
||||
**Special Thanks:**
|
||||
- Security research team for vulnerability disclosures
|
||||
- Community testers and feedback providers
|
||||
- Documentation contributors and reviewers
|
||||
|
||||
---
|
||||
|
||||
## 🚀 **INSTALLATION**
|
||||
|
||||
```bash
|
||||
# Standard installation
|
||||
pip install semantica==0.5.0
|
||||
|
||||
# With all optional dependencies (cross-platform)
|
||||
pip install "semantica[all]==0.5.0"
|
||||
|
||||
# With GPU acceleration (Linux only)
|
||||
pip install "semantica[gpu]==0.5.0"
|
||||
|
||||
# With Parquet support
|
||||
pip install "semantica[ingest-parquet]==0.5.0"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 **WHAT'S NEXT FOR 0.5.0**
|
||||
|
||||
The 0.5.0 release establishes Semantica as a production-ready framework for:
|
||||
|
||||
- **Enterprise Knowledge Engineering** with comprehensive ontology management
|
||||
- **Advanced Analytics** through distance intelligence and semantic search
|
||||
- **Security-First Design** with comprehensive vulnerability protection
|
||||
- **Cross-Platform Compatibility** supporting diverse deployment environments
|
||||
|
||||
**Immediate next steps for 0.5.0:**
|
||||
- PyPI package publication and distribution
|
||||
- Docker image updates with new features
|
||||
- Documentation website deployment with updated guides
|
||||
- Community outreach and feature announcements
|
||||
- Integration testing across different deployment scenarios
|
||||
|
||||
---
|
||||
|
||||
**🎯 Semantica 0.5.0: Production-Ready Knowledge Engineering Platform**
|
||||
@@ -1,7 +1,7 @@
|
||||
# Benchmark Tools
|
||||
|
||||
pytest>=7.0.0
|
||||
pytest-benchmark>=4.0.0
|
||||
pytest-benchmark>=5.2.3
|
||||
|
||||
# Core Utils
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ The **Ingest Module** is the entry point for loading data into Semantica. It pro
|
||||
|
||||
**Data ingestion** is the process of loading data from various sources into Semantica for processing. The ingest module handles:
|
||||
- **File Systems**: Local files, cloud storage (S3, GCS, Azure)
|
||||
- **Analytics Files**: Apache Parquet files and partitioned datasets
|
||||
- **Web Content**: Websites, RSS feeds, APIs
|
||||
- **Streams**: Real-time data from Kafka, RabbitMQ, etc.
|
||||
- **Databases**: SQL, NoSQL, and cloud data warehouses including Snowflake
|
||||
@@ -74,6 +75,12 @@ The **Ingest Module** is the entry point for loading data into Semantica. It pro
|
||||
|
||||
Ingest tables and query results from SQL, NoSQL, and cloud data warehouses including Snowflake
|
||||
|
||||
- :material-table:{ .lg .middle } **Parquet Datasets**
|
||||
|
||||
---
|
||||
|
||||
Read Parquet files, schemas, metadata, and Hive-style partitioned directories
|
||||
|
||||
</div>
|
||||
|
||||
!!! tip "When to Use"
|
||||
@@ -118,6 +125,19 @@ Handles file systems and object storage.
|
||||
| `ingest_file(path)` | Process single file |
|
||||
| `ingest_directory(path)` | Process folder |
|
||||
|
||||
### ParquetIngestor
|
||||
|
||||
Handles Apache Parquet files and partitioned datasets.
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `ingest_file(path, columns=None, limit=None)` | Read a Parquet file |
|
||||
| `ingest_directory(path, columns=None, limit=None)` | Read a partitioned Parquet directory |
|
||||
| `extract_schema(path)` | Extract column names, types, nullability, and schema metadata |
|
||||
| `extract_metadata(path)` | Extract row counts, row groups, compression, and partition info |
|
||||
|
||||
### WebIngestor
|
||||
|
||||
Handles web content.
|
||||
@@ -213,10 +233,33 @@ from semantica.ingest import ingest
|
||||
|
||||
# Auto-detect source type
|
||||
ingest("doc.pdf", source_type="file")
|
||||
ingest("events.parquet") # Auto-detects Parquet
|
||||
ingest("https://google.com", source_type="web")
|
||||
ingest("kafka://topic", source_type="stream")
|
||||
```
|
||||
|
||||
### Parquet Dataset Ingestion
|
||||
|
||||
```python
|
||||
from semantica.ingest import ParquetIngestor, ingest_parquet
|
||||
|
||||
ingestor = ParquetIngestor()
|
||||
|
||||
# Read selected columns from a local Parquet file
|
||||
events = ingestor.ingest_file(
|
||||
"events.parquet",
|
||||
columns=["event_id", "event_type"],
|
||||
limit=1000,
|
||||
)
|
||||
|
||||
# Inspect schema and metadata without reading rows
|
||||
schema = ingestor.extract_schema("events.parquet")
|
||||
metadata = ingestor.extract_metadata("events.parquet")
|
||||
|
||||
# Read a Hive-style partitioned directory such as country=US/year=2026/
|
||||
partitioned = ingest_parquet("./warehouse/events", method="directory")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
@@ -236,7 +279,7 @@ ingest:
|
||||
web:
|
||||
user_agent: "MyBot"
|
||||
rate_limit: 1.0 # seconds
|
||||
|
||||
|
||||
files:
|
||||
max_size: 100MB
|
||||
allowed_extensions: [.pdf, .txt, .md]
|
||||
@@ -289,12 +332,12 @@ data = ingestor.ingest_snowflake_table("CUSTOMERS")
|
||||
|
||||
# 3. Or run custom query
|
||||
results = ingestor.execute_snowflake_query("""
|
||||
SELECT
|
||||
CUSTOMER_ID,
|
||||
NAME,
|
||||
EMAIL,
|
||||
CREATED_AT
|
||||
FROM CUSTOMERS
|
||||
SELECT
|
||||
CUSTOMER_ID,
|
||||
NAME,
|
||||
EMAIL,
|
||||
CREATED_AT
|
||||
FROM CUSTOMERS
|
||||
WHERE CREATED_AT > '2024-01-01'
|
||||
""")
|
||||
|
||||
|
||||
+14
-2
@@ -1237,6 +1237,7 @@ export default function App() {
|
||||
const [analyzeView, setAnalyzeView] = useState<AnalyzeView>('reasoning');
|
||||
const [enrichView, setEnrichView] = useState<EnrichView>('import');
|
||||
const [manageView, setManageView] = useState<ManageView>('lineage');
|
||||
const [graphFocusRequest, setGraphFocusRequest] = useState<{ nodeId: string; token: number } | null>(null);
|
||||
|
||||
|
||||
const renderWorkspace = () => {
|
||||
@@ -1284,7 +1285,12 @@ export default function App() {
|
||||
}
|
||||
>
|
||||
<Suspense fallback={<WorkspaceFallback />}>
|
||||
{exploreView === 'graph' ? <GraphWorkspace /> : <VocabularyWorkspace />}
|
||||
{exploreView === 'graph' ? (
|
||||
<GraphWorkspace
|
||||
externalFocusNodeId={graphFocusRequest?.nodeId}
|
||||
externalFocusToken={graphFocusRequest?.token}
|
||||
/>
|
||||
) : <VocabularyWorkspace />}
|
||||
</Suspense>
|
||||
</WorkspaceShell>
|
||||
);
|
||||
@@ -1370,7 +1376,13 @@ export default function App() {
|
||||
compact
|
||||
>
|
||||
<Suspense fallback={<WorkspaceFallback />}>
|
||||
<OntologyWorkspace />
|
||||
<OntologyWorkspace
|
||||
onJumpToGraphNode={(nodeId: string) => {
|
||||
setGraphFocusRequest({ nodeId, token: Date.now() });
|
||||
setActiveWorkspace('explore');
|
||||
setExploreView('graph');
|
||||
}}
|
||||
/>
|
||||
</Suspense>
|
||||
</WorkspaceShell>
|
||||
);
|
||||
|
||||
@@ -1082,7 +1082,12 @@ function collectPluginOverlays(
|
||||
});
|
||||
}
|
||||
|
||||
export function GraphWorkspace() {
|
||||
interface GraphWorkspaceProps {
|
||||
externalFocusNodeId?: string;
|
||||
externalFocusToken?: number;
|
||||
}
|
||||
|
||||
export function GraphWorkspace({ externalFocusNodeId, externalFocusToken }: GraphWorkspaceProps = {}) {
|
||||
const [selectedNodeId, setSelectedNodeId] = useState("");
|
||||
const [focusedNodeId, setFocusedNodeId] = useState("");
|
||||
const [lastGroupedSelectedNodeId, setLastGroupedSelectedNodeId] = useState("");
|
||||
@@ -1141,6 +1146,7 @@ export function GraphWorkspace() {
|
||||
const debouncedTime = useDebounce(scrubberTime, 150);
|
||||
const prevActiveIdsRef = useRef<Set<string>>(new Set());
|
||||
const sceneRef = useRef<GraphSceneHandle>(null);
|
||||
const lastExternalFocusTokenRef = useRef<number | undefined>(undefined);
|
||||
const pluginRuntimeRef = useRef<GraphSceneRuntime | null>(null);
|
||||
const settlingOverlayTimeoutRef = useRef<number | null>(null);
|
||||
const pluginInteractionStateRef = useRef<GraphInteractionState>({
|
||||
@@ -1440,6 +1446,23 @@ export function GraphWorkspace() {
|
||||
}
|
||||
}, [viewMode]); // Note: ego/heatmap/distanceMode effects re-run automatically when selectedNodeId changes
|
||||
|
||||
useEffect(() => {
|
||||
if (!externalFocusNodeId || externalFocusToken == null) return;
|
||||
if (lastExternalFocusTokenRef.current === externalFocusToken) return;
|
||||
if (!graphReady || !graph.hasNode(externalFocusNodeId)) return;
|
||||
|
||||
lastExternalFocusTokenRef.current = externalFocusToken;
|
||||
// Set state directly instead of going through focusNode(), which captures
|
||||
// a stale viewMode in its closure. setViewMode is called first so the node
|
||||
// is visible in the full graph before the scene pans to it.
|
||||
setViewMode("full");
|
||||
setSelectedNodeId(externalFocusNodeId);
|
||||
setSelectedEdgeId("");
|
||||
window.setTimeout(() => {
|
||||
sceneRef.current?.focusNode(externalFocusNodeId);
|
||||
}, 0);
|
||||
}, [externalFocusNodeId, externalFocusToken, graphReady]);
|
||||
|
||||
const handleEdgeSelect = useCallback((edgeId: string) => {
|
||||
setSelectedEdgeId(edgeId);
|
||||
}, []);
|
||||
|
||||
@@ -0,0 +1,406 @@
|
||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||
import type { CSSProperties } from "react";
|
||||
import { GitMerge, Loader2, Sparkles, Trash2 } from "lucide-react";
|
||||
import {
|
||||
loadAlignments,
|
||||
loadOntologyRegistry,
|
||||
removeAlignment,
|
||||
saveAlignment,
|
||||
suggestAlignments,
|
||||
} from "./api";
|
||||
import type { AlignmentRelation, AlignmentSuggestion, OntologyAlignment, OntologyEntry } from "./types";
|
||||
|
||||
const RELATIONS: AlignmentRelation[] = [
|
||||
"owl:equivalentClass",
|
||||
"owl:equivalentProperty",
|
||||
"skos:exactMatch",
|
||||
"skos:closeMatch",
|
||||
"skos:broadMatch",
|
||||
"skos:narrowMatch",
|
||||
"skos:relatedMatch",
|
||||
];
|
||||
|
||||
const RELATION_COLORS: Record<AlignmentRelation, string> = {
|
||||
"owl:equivalentClass": "#7ce7d3",
|
||||
"owl:equivalentProperty": "#7ce7d3",
|
||||
"skos:exactMatch": "#9ee8d7",
|
||||
"skos:closeMatch": "#58a6ff",
|
||||
"skos:broadMatch": "#f2b66d",
|
||||
"skos:narrowMatch": "#f2b66d",
|
||||
"skos:relatedMatch": "#d2a8ff",
|
||||
};
|
||||
|
||||
export function AlignmentsTab() {
|
||||
const [registry, setRegistry] = useState<OntologyEntry[]>([]);
|
||||
const [alignments, setAlignments] = useState<OntologyAlignment[]>([]);
|
||||
const [suggestions, setSuggestions] = useState<AlignmentSuggestion[]>([]);
|
||||
const [sourceOntology, setSourceOntology] = useState("");
|
||||
const [targetOntology, setTargetOntology] = useState("");
|
||||
const [sourceUri, setSourceUri] = useState("");
|
||||
const [targetUri, setTargetUri] = useState("");
|
||||
const [relation, setRelation] = useState<AlignmentRelation>("skos:exactMatch");
|
||||
const [confidence, setConfidence] = useState(0.86);
|
||||
const [provenance, setProvenance] = useState("");
|
||||
const [source, setSource] = useState("Ontology Hub");
|
||||
const [reviewer, setReviewer] = useState("");
|
||||
const [threshold, setThreshold] = useState(0.68);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
|
||||
const reload = useCallback(async () => {
|
||||
setError("");
|
||||
try {
|
||||
const [registryData, alignmentData] = await Promise.all([
|
||||
loadOntologyRegistry(),
|
||||
loadAlignments(),
|
||||
]);
|
||||
setRegistry(registryData);
|
||||
setAlignments(alignmentData);
|
||||
setSourceOntology((current) => current || registryData[0]?.uri || "");
|
||||
setTargetOntology((current) => current || registryData[1]?.uri || registryData[0]?.uri || "");
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Could not load ontology alignments.");
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
void reload();
|
||||
}, [reload]);
|
||||
|
||||
const relationCounts = useMemo(() => {
|
||||
const counts = new Map<string, number>();
|
||||
for (const item of alignments) {
|
||||
counts.set(item.relation, (counts.get(item.relation) ?? 0) + 1);
|
||||
}
|
||||
return counts;
|
||||
}, [alignments]);
|
||||
|
||||
// Pairwise matrix: group alignments by (source_ontology, target_ontology) pair.
|
||||
const matrix = useMemo(() => {
|
||||
function ontologyOfUri(uri: string): string {
|
||||
for (const entry of registry) {
|
||||
if (uri === entry.uri || uri.startsWith(entry.uri + "#") || uri.startsWith(entry.uri + "/")) {
|
||||
return entry.uri;
|
||||
}
|
||||
}
|
||||
const hashIdx = uri.lastIndexOf("#");
|
||||
if (hashIdx > 0) return uri.substring(0, hashIdx);
|
||||
const slashIdx = uri.lastIndexOf("/");
|
||||
return slashIdx > 0 ? uri.substring(0, slashIdx) : uri;
|
||||
}
|
||||
const cells: Map<string, OntologyAlignment[]> = new Map();
|
||||
for (const alignment of alignments) {
|
||||
const key = `${ontologyOfUri(alignment.source_uri)}|||${ontologyOfUri(alignment.target_uri)}`;
|
||||
const bucket = cells.get(key) ?? [];
|
||||
bucket.push(alignment);
|
||||
cells.set(key, bucket);
|
||||
}
|
||||
return { ontologies: registry, cells };
|
||||
}, [registry, alignments]);
|
||||
|
||||
const handleSave = useCallback(async () => {
|
||||
if (!sourceUri.trim() || !targetUri.trim()) {
|
||||
setError("Provide both source and target entity URIs.");
|
||||
return;
|
||||
}
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
await saveAlignment({
|
||||
source_uri: sourceUri.trim(),
|
||||
target_uri: targetUri.trim(),
|
||||
relation,
|
||||
confidence,
|
||||
provenance: provenance || undefined,
|
||||
source: source || undefined,
|
||||
reviewer: reviewer || undefined,
|
||||
});
|
||||
setSourceUri("");
|
||||
setTargetUri("");
|
||||
await reload();
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Could not save alignment.");
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}, [sourceUri, targetUri, relation, confidence, provenance, source, reviewer, reload]);
|
||||
|
||||
const handleSuggest = useCallback(async () => {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
const data = await suggestAlignments({
|
||||
source_ontology_uri: sourceOntology || undefined,
|
||||
target_ontology_uri: targetOntology || undefined,
|
||||
threshold,
|
||||
limit: 40,
|
||||
});
|
||||
setSuggestions(data);
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Could not suggest alignments.");
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}, [sourceOntology, targetOntology, threshold]);
|
||||
|
||||
const handleAcceptSuggestion = useCallback((suggestion: AlignmentSuggestion) => {
|
||||
setSourceUri(suggestion.source_uri);
|
||||
setTargetUri(suggestion.target_uri);
|
||||
setRelation(suggestion.relation);
|
||||
setConfidence(Math.max(0.1, Math.min(1, suggestion.score)));
|
||||
setProvenance(suggestion.reason);
|
||||
}, []);
|
||||
|
||||
const handleRemove = useCallback(async (id: string) => {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
await removeAlignment(id);
|
||||
await reload();
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Could not remove alignment.");
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}, [reload]);
|
||||
|
||||
return (
|
||||
<div style={pageStyle}>
|
||||
<section style={heroStyle}>
|
||||
<div>
|
||||
<div style={kickerStyle}><GitMerge size={14} /> Alignment Matrix</div>
|
||||
<h2 style={titleStyle}>Cross-ontology mappings</h2>
|
||||
<p style={textStyle}>
|
||||
Manage equivalence and SKOS match relations with confidence, provenance,
|
||||
reviewer context, and label-based suggestions.
|
||||
</p>
|
||||
</div>
|
||||
<div style={summaryGridStyle}>
|
||||
<Metric label="Mappings" value={alignments.length} />
|
||||
<Metric label="Relations" value={relationCounts.size} />
|
||||
<Metric label="Suggestions" value={suggestions.length} />
|
||||
</div>
|
||||
</section>
|
||||
|
||||
{error ? <div style={errorStyle}>{error}</div> : null}
|
||||
|
||||
<div style={ephemeralBannerStyle}>
|
||||
Alignments are stored in server memory and are not persisted across restarts.
|
||||
Export your graph or ontology to preserve recorded mappings.
|
||||
</div>
|
||||
|
||||
{matrix.ontologies.length >= 2 ? (
|
||||
<section style={cardStyle}>
|
||||
<h3 style={sectionTitleStyle}>Pairwise alignment matrix</h3>
|
||||
<div style={{ overflowX: "auto" }}>
|
||||
<table style={matrixTableStyle}>
|
||||
<thead>
|
||||
<tr>
|
||||
<th style={matrixCornerStyle} />
|
||||
{matrix.ontologies.map((col) => (
|
||||
<th key={col.uri} style={matrixColHeaderStyle}>{col.name}</th>
|
||||
))}
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{matrix.ontologies.map((row) => (
|
||||
<tr key={row.uri}>
|
||||
<td style={matrixRowHeaderStyle}>{row.name}</td>
|
||||
{matrix.ontologies.map((col) => {
|
||||
const key = `${row.uri}|||${col.uri}`;
|
||||
const cellItems = matrix.cells.get(key) ?? [];
|
||||
const isDiag = row.uri === col.uri;
|
||||
return (
|
||||
<td key={col.uri} style={{ ...matrixCellStyle, background: isDiag ? "rgba(255,255,255,0.015)" : undefined }}>
|
||||
{isDiag ? <span style={mutedStyle}>—</span> : cellItems.length ? (
|
||||
<div style={{ display: "flex", flexDirection: "column", gap: 4 }}>
|
||||
{cellItems.map((item) => (
|
||||
<span
|
||||
key={item.id}
|
||||
style={{ ...relationBadgeStyle, color: RELATION_COLORS[item.relation], borderColor: `${RELATION_COLORS[item.relation]}44`, cursor: "pointer", fontSize: 9 }}
|
||||
title={`${item.source_label} → ${item.target_label} (${Math.round(item.confidence * 100)}%)`}
|
||||
onClick={() => {
|
||||
setSourceUri(item.source_uri);
|
||||
setTargetUri(item.target_uri);
|
||||
setRelation(item.relation);
|
||||
setConfidence(item.confidence);
|
||||
setProvenance(item.provenance ?? "");
|
||||
}}
|
||||
>
|
||||
{item.relation.split(":")[1]}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
) : <span style={{ color: "rgba(127,208,255,0.15)", fontSize: 12 }}>·</span>}
|
||||
</td>
|
||||
);
|
||||
})}
|
||||
</tr>
|
||||
))}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
<p style={{ ...mutedStyle, marginTop: 10 }}>Click a relation badge to load it into the editor below.</p>
|
||||
</section>
|
||||
) : null}
|
||||
|
||||
<div style={gridStyle}>
|
||||
<section style={cardStyle}>
|
||||
<h3 style={sectionTitleStyle}>Create or update alignment</h3>
|
||||
<label style={labelStyle}>Source entity URI</label>
|
||||
<input style={inputStyle} value={sourceUri} onChange={(event) => setSourceUri(event.target.value)} />
|
||||
<label style={labelStyle}>Target entity URI</label>
|
||||
<input style={inputStyle} value={targetUri} onChange={(event) => setTargetUri(event.target.value)} />
|
||||
<div style={twoColStyle}>
|
||||
<div>
|
||||
<label style={labelStyle}>Relation</label>
|
||||
<select style={inputStyle} value={relation} onChange={(event) => setRelation(event.target.value as AlignmentRelation)}>
|
||||
{RELATIONS.map((item) => <option key={item}>{item}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
<label style={labelStyle}>Confidence {confidence.toFixed(2)}</label>
|
||||
<input
|
||||
type="range"
|
||||
min="0"
|
||||
max="1"
|
||||
step="0.01"
|
||||
value={confidence}
|
||||
onChange={(event) => setConfidence(Number(event.target.value))}
|
||||
style={{ width: "100%" }}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<label style={labelStyle}>Provenance note</label>
|
||||
<textarea style={{ ...inputStyle, minHeight: 74, resize: "vertical" }} value={provenance} onChange={(event) => setProvenance(event.target.value)} />
|
||||
<div style={twoColStyle}>
|
||||
<div>
|
||||
<label style={labelStyle}>Source</label>
|
||||
<input style={inputStyle} value={source} onChange={(event) => setSource(event.target.value)} />
|
||||
</div>
|
||||
<div>
|
||||
<label style={labelStyle}>Reviewer</label>
|
||||
<input style={inputStyle} value={reviewer} onChange={(event) => setReviewer(event.target.value)} />
|
||||
</div>
|
||||
</div>
|
||||
<button style={primaryButtonStyle} disabled={busy} onClick={handleSave}>
|
||||
{busy ? <Loader2 size={14} className="spin" /> : <GitMerge size={14} />}
|
||||
Save alignment
|
||||
</button>
|
||||
</section>
|
||||
|
||||
<section style={cardStyle}>
|
||||
<h3 style={sectionTitleStyle}>Suggest alignments</h3>
|
||||
<div style={twoColStyle}>
|
||||
<div>
|
||||
<label style={labelStyle}>Source ontology</label>
|
||||
<select style={inputStyle} value={sourceOntology} onChange={(event) => setSourceOntology(event.target.value)}>
|
||||
<option value="">Any ontology</option>
|
||||
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
<label style={labelStyle}>Target ontology</label>
|
||||
<select style={inputStyle} value={targetOntology} onChange={(event) => setTargetOntology(event.target.value)}>
|
||||
<option value="">Any ontology</option>
|
||||
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
<label style={labelStyle}>Similarity threshold {threshold.toFixed(2)}</label>
|
||||
<input
|
||||
type="range"
|
||||
min="0.25"
|
||||
max="0.95"
|
||||
step="0.01"
|
||||
value={threshold}
|
||||
onChange={(event) => setThreshold(Number(event.target.value))}
|
||||
style={{ width: "100%" }}
|
||||
/>
|
||||
<button style={secondaryButtonStyle} disabled={busy} onClick={handleSuggest}>
|
||||
<Sparkles size={14} />
|
||||
Suggest alignments
|
||||
</button>
|
||||
<div style={suggestionListStyle}>
|
||||
{suggestions.map((item) => (
|
||||
<button key={`${item.source_uri}-${item.target_uri}-${item.relation}`} style={suggestionStyle} onClick={() => handleAcceptSuggestion(item)}>
|
||||
<span style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.source_label}</span>
|
||||
<span style={{ color: RELATION_COLORS[item.relation] }}>{item.relation}</span>
|
||||
<span style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.target_label}</span>
|
||||
<span style={{ color: "#8fa8c6" }}>{Math.round(item.score * 100)}%</span>
|
||||
</button>
|
||||
))}
|
||||
{!suggestions.length ? <p style={mutedStyle}>Run suggestions to review ranked candidate mappings.</p> : null}
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<section style={cardStyle}>
|
||||
<h3 style={sectionTitleStyle}>Recorded alignments</h3>
|
||||
<div style={tableStyle}>
|
||||
{alignments.map((item) => (
|
||||
<div key={item.id} style={rowStyle}>
|
||||
<div>
|
||||
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.source_label || item.source_uri}</div>
|
||||
<div style={monoStyle}>{item.source_uri}</div>
|
||||
</div>
|
||||
<div style={{ ...relationBadgeStyle, color: RELATION_COLORS[item.relation], borderColor: `${RELATION_COLORS[item.relation]}55` }}>
|
||||
{item.relation}
|
||||
</div>
|
||||
<div>
|
||||
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{item.target_label || item.target_uri}</div>
|
||||
<div style={monoStyle}>{item.target_uri}</div>
|
||||
</div>
|
||||
<div style={confidenceStyle}>{Math.round(item.confidence * 100)}%</div>
|
||||
<button style={iconButtonStyle} disabled={busy} onClick={() => handleRemove(item.id)} title="Remove alignment">
|
||||
<Trash2 size={14} />
|
||||
</button>
|
||||
</div>
|
||||
))}
|
||||
{!alignments.length ? <p style={mutedStyle}>No alignments recorded yet.</p> : null}
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function Metric({ label, value }: { label: string; value: number }) {
|
||||
return (
|
||||
<div style={metricStyle}>
|
||||
<span style={{ color: "#9ee8d7", fontSize: 20, fontWeight: 900 }}>{value.toLocaleString()}</span>
|
||||
<span style={{ color: "#6a7f97", fontSize: 11 }}>{label}</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
const pageStyle: CSSProperties = { height: "100%", overflow: "auto", padding: 22, display: "flex", flexDirection: "column", gap: 16 };
|
||||
const heroStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 18, padding: 22, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 22, background: "linear-gradient(135deg, rgba(11,25,42,0.94), rgba(7,14,25,0.9))" };
|
||||
const kickerStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", color: "#9ee8d7", fontSize: 11, fontWeight: 900, letterSpacing: "0.12em", textTransform: "uppercase" };
|
||||
const titleStyle: CSSProperties = { margin: "8px 0", color: "#ebf3ff", fontSize: 26, letterSpacing: "-0.04em" };
|
||||
const textStyle: CSSProperties = { margin: 0, color: "#8fa8c6", lineHeight: 1.6, maxWidth: 620 };
|
||||
const summaryGridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "repeat(3, minmax(100px, 1fr))", gap: 10, minWidth: 320 };
|
||||
const metricStyle: CSSProperties = { padding: 14, borderRadius: 16, background: "rgba(255,255,255,0.035)", border: "1px solid rgba(127,208,255,0.1)", display: "flex", flexDirection: "column", gap: 4 };
|
||||
const gridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "minmax(320px, 0.9fr) minmax(360px, 1.1fr)", gap: 16 };
|
||||
const cardStyle: CSSProperties = { padding: 18, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 20, background: "rgba(9,19,34,0.78)", boxShadow: "inset 0 1px 0 rgba(255,255,255,0.04)" };
|
||||
const sectionTitleStyle: CSSProperties = { margin: "0 0 14px", color: "#ebf3ff", fontSize: 16 };
|
||||
const labelStyle: CSSProperties = { display: "block", color: "#6a7f97", fontSize: 11, fontWeight: 800, margin: "10px 0 6px", textTransform: "uppercase", letterSpacing: "0.08em" };
|
||||
const inputStyle: CSSProperties = { width: "100%", boxSizing: "border-box", border: "1px solid rgba(127,208,255,0.14)", borderRadius: 12, padding: "10px 12px", background: "rgba(3,9,18,0.8)", color: "#ebf3ff" };
|
||||
const twoColStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr 1fr", gap: 12 };
|
||||
const primaryButtonStyle: CSSProperties = { marginTop: 14, width: "100%", border: "1px solid rgba(124,231,211,0.35)", borderRadius: 12, padding: "11px 13px", background: "linear-gradient(135deg, rgba(20,151,136,0.55), rgba(74,163,255,0.35))", color: "#ebf3ff", fontWeight: 900, cursor: "pointer", display: "inline-flex", alignItems: "center", justifyContent: "center", gap: 8 };
|
||||
const secondaryButtonStyle: CSSProperties = { ...primaryButtonStyle, background: "rgba(127,208,255,0.08)", borderColor: "rgba(127,208,255,0.18)" };
|
||||
const suggestionListStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 8, marginTop: 14, maxHeight: 260, overflow: "auto" };
|
||||
const suggestionStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr auto 1fr auto", gap: 10, alignItems: "center", textAlign: "left", border: "1px solid rgba(127,208,255,0.1)", borderRadius: 12, background: "rgba(255,255,255,0.03)", padding: 10, cursor: "pointer" };
|
||||
const tableStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 8 };
|
||||
const rowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr auto 1fr auto auto", gap: 12, alignItems: "center", padding: 12, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
|
||||
const monoStyle: CSSProperties = { color: "#6a7f97", fontSize: 11, fontFamily: "JetBrains Mono, monospace", wordBreak: "break-all" };
|
||||
const relationBadgeStyle: CSSProperties = { padding: "5px 9px", border: "1px solid", borderRadius: 999, fontSize: 10, fontWeight: 900 };
|
||||
const confidenceStyle: CSSProperties = { color: "#f2b66d", fontWeight: 900 };
|
||||
const iconButtonStyle: CSSProperties = { width: 34, height: 34, borderRadius: 10, border: "1px solid rgba(255,157,175,0.18)", background: "rgba(255,157,175,0.08)", color: "#ff9daf", cursor: "pointer" };
|
||||
const mutedStyle: CSSProperties = { margin: 0, color: "#6a7f97", fontSize: 13 };
|
||||
const errorStyle: CSSProperties = { padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" };
|
||||
const ephemeralBannerStyle: CSSProperties = { padding: "9px 14px", borderRadius: 12, color: "#f2b66d", background: "rgba(242,182,109,0.08)", border: "1px solid rgba(242,182,109,0.22)", fontSize: 12 };
|
||||
const matrixTableStyle: CSSProperties = { borderCollapse: "collapse", minWidth: "100%", fontSize: 12 };
|
||||
const matrixCornerStyle: CSSProperties = { padding: "8px 12px", borderBottom: "1px solid rgba(127,208,255,0.1)" };
|
||||
const matrixColHeaderStyle: CSSProperties = { padding: "8px 12px", color: "#9ee8d7", fontWeight: 900, borderBottom: "1px solid rgba(127,208,255,0.1)", textAlign: "center", whiteSpace: "nowrap" };
|
||||
const matrixRowHeaderStyle: CSSProperties = { padding: "8px 12px", color: "#9ee8d7", fontWeight: 900, borderRight: "1px solid rgba(127,208,255,0.1)", whiteSpace: "nowrap" };
|
||||
const matrixCellStyle: CSSProperties = { padding: "8px 10px", borderBottom: "1px solid rgba(127,208,255,0.06)", borderRight: "1px solid rgba(127,208,255,0.06)", textAlign: "center", verticalAlign: "middle", minWidth: 100 };
|
||||
@@ -0,0 +1,205 @@
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
import type { CSSProperties } from "react";
|
||||
import { Download, HeartPulse, Loader2, Wrench } from "lucide-react";
|
||||
import { loadOntologyHealth, loadOntologyRegistry } from "./api";
|
||||
import type { OntologyEntry, OntologyHealthResponse, HealthIssue } from "./types";
|
||||
|
||||
interface HealthTabProps {
|
||||
onFixInEditor?: (entityUri: string) => void;
|
||||
}
|
||||
|
||||
export function HealthTab({ onFixInEditor }: HealthTabProps) {
|
||||
const [registry, setRegistry] = useState<OntologyEntry[]>([]);
|
||||
const [selectedUri, setSelectedUri] = useState("");
|
||||
const [health, setHealth] = useState<OntologyHealthResponse | null>(null);
|
||||
const [loading, setLoading] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
loadOntologyRegistry()
|
||||
.then((entries) => {
|
||||
if (cancelled) return;
|
||||
setRegistry(entries);
|
||||
setSelectedUri((current) => current || entries[0]?.uri || "");
|
||||
})
|
||||
.catch((err) => {
|
||||
if (!cancelled) setError(err instanceof Error ? err.message : "Could not load ontology registry.");
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, []);
|
||||
|
||||
const loadHealth = useCallback(async (uri: string) => {
|
||||
if (!uri) return;
|
||||
setLoading(true);
|
||||
setError("");
|
||||
try {
|
||||
setHealth(await loadOntologyHealth(uri));
|
||||
} catch (err) {
|
||||
setHealth(null);
|
||||
setError(err instanceof Error ? err.message : "Could not load ontology health.");
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
void loadHealth(selectedUri);
|
||||
}, [selectedUri, loadHealth]);
|
||||
|
||||
const exportReport = useCallback(() => {
|
||||
if (!health) return;
|
||||
const blob = new Blob([JSON.stringify(health, null, 2)], { type: "application/json" });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const anchor = document.createElement("a");
|
||||
anchor.href = url;
|
||||
anchor.download = `${health.name.replace(/[^a-z0-9]+/gi, "-").toLowerCase()}-health.json`;
|
||||
document.body.appendChild(anchor);
|
||||
anchor.click();
|
||||
document.body.removeChild(anchor);
|
||||
setTimeout(() => URL.revokeObjectURL(url), 100);
|
||||
}, [health]);
|
||||
|
||||
return (
|
||||
<div style={pageStyle}>
|
||||
<section style={heroStyle}>
|
||||
<div>
|
||||
<div style={kickerStyle}><HeartPulse size={14} /> Ontology Health</div>
|
||||
<h2 style={titleStyle}>Quality and governance signals</h2>
|
||||
<p style={textStyle}>
|
||||
Score completeness, consistency, SHACL readiness, alignment coverage,
|
||||
and documentation quality for the selected ontology.
|
||||
</p>
|
||||
</div>
|
||||
<div style={selectorShellStyle}>
|
||||
<label style={labelStyle}>Ontology</label>
|
||||
<select style={inputStyle} value={selectedUri} onChange={(event) => setSelectedUri(event.target.value)}>
|
||||
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
{error ? <div style={errorStyle}>{error}</div> : null}
|
||||
|
||||
{loading ? (
|
||||
<div style={loadingStyle}><Loader2 size={18} className="spin" /> Computing health dashboard...</div>
|
||||
) : health ? (
|
||||
<>
|
||||
<section style={{ ...scoreGridStyle, gridTemplateColumns: `220px repeat(${health.dimensions.length}, minmax(180px, 1fr))` }}>
|
||||
<div style={scoreCardStyle}>
|
||||
<span style={scoreValueStyle}>{Math.round(health.total_score)}</span>
|
||||
<span style={mutedStyle}>Total health score</span>
|
||||
<button style={secondaryButtonStyle} onClick={exportReport}><Download size={14} /> Export report</button>
|
||||
</div>
|
||||
{health.dimensions.map((dimension) => (
|
||||
<div key={dimension.key} style={dimensionCardStyle}>
|
||||
<div style={dimensionHeadStyle}>
|
||||
<span style={{ color: "#ebf3ff", fontWeight: 900 }}>{dimension.label}</span>
|
||||
<span style={statusBadgeStyle(dimension.status)}>{dimension.status}</span>
|
||||
</div>
|
||||
<div style={barTrackStyle}>
|
||||
<div style={{ ...barFillStyle, width: `${dimension.score}%`, background: dimensionColor(dimension.score, dimension.status) }} />
|
||||
</div>
|
||||
<div style={dimensionFootStyle}>
|
||||
<span>{Math.round(dimension.score)} / 100</span>
|
||||
<span>{dimension.detail}</span>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</section>
|
||||
|
||||
<section style={cardStyle}>
|
||||
<h3 style={sectionTitleStyle}>Actionable issues</h3>
|
||||
<div style={issueListStyle}>
|
||||
{health.issues.map((issue) => (
|
||||
<IssueRow key={issue.id} issue={issue} onFixInEditor={onFixInEditor} />
|
||||
))}
|
||||
{!health.issues.length ? <p style={mutedStyle}>No actionable issues reported for this ontology.</p> : null}
|
||||
</div>
|
||||
</section>
|
||||
</>
|
||||
) : (
|
||||
<div style={emptyStyle}>Select an ontology to compute health signals.</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function IssueRow({ issue, onFixInEditor }: { issue: HealthIssue; onFixInEditor?: (entityUri: string) => void }) {
|
||||
return (
|
||||
<div style={issueRowStyle}>
|
||||
<div style={severityDotStyle(issue.severity)} />
|
||||
<div>
|
||||
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{issue.entity_label || issue.category}</div>
|
||||
<div style={{ color: "#8fa8c6", fontSize: 13, lineHeight: 1.45 }}>{issue.message}</div>
|
||||
{issue.entity_uri ? <div style={monoStyle}>{issue.entity_uri}</div> : null}
|
||||
</div>
|
||||
<span style={categoryStyle}>{issue.category}</span>
|
||||
{issue.entity_uri ? (
|
||||
<button style={smallButtonStyle} onClick={() => onFixInEditor?.(issue.entity_uri || "")}>
|
||||
<Wrench size={13} />
|
||||
Fix in Editor
|
||||
</button>
|
||||
) : (
|
||||
<div />
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function dimensionColor(score: number, status: string) {
|
||||
if (status === "unavailable") return "#6a7f97";
|
||||
if (score >= 80) return "#7ce7d3";
|
||||
if (score >= 55) return "#f2b66d";
|
||||
return "#ff9daf";
|
||||
}
|
||||
|
||||
function statusBadgeStyle(status: string): CSSProperties {
|
||||
const color = status === "ok" ? "#7ce7d3" : status === "unavailable" ? "#6a7f97" : "#f2b66d";
|
||||
return {
|
||||
color,
|
||||
background: `${color}18`,
|
||||
border: `1px solid ${color}30`,
|
||||
borderRadius: 999,
|
||||
padding: "2px 7px",
|
||||
fontSize: 10,
|
||||
fontWeight: 900,
|
||||
textTransform: "uppercase",
|
||||
};
|
||||
}
|
||||
|
||||
function severityDotStyle(severity: string): CSSProperties {
|
||||
const color = severity === "critical" ? "#ff9daf" : severity === "warning" ? "#f2b66d" : "#58a6ff";
|
||||
return { width: 10, height: 10, borderRadius: "50%", background: color, boxShadow: `0 0 18px ${color}55`, marginTop: 5 };
|
||||
}
|
||||
|
||||
const pageStyle: CSSProperties = { height: "100%", overflow: "auto", padding: 22, display: "flex", flexDirection: "column", gap: 16 };
|
||||
const heroStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 18, padding: 22, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 22, background: "linear-gradient(135deg, rgba(11,25,42,0.94), rgba(7,14,25,0.9))" };
|
||||
const kickerStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", color: "#9ee8d7", fontSize: 11, fontWeight: 900, letterSpacing: "0.12em", textTransform: "uppercase" };
|
||||
const titleStyle: CSSProperties = { margin: "8px 0", color: "#ebf3ff", fontSize: 26, letterSpacing: "-0.04em" };
|
||||
const textStyle: CSSProperties = { margin: 0, color: "#8fa8c6", lineHeight: 1.6, maxWidth: 620 };
|
||||
const selectorShellStyle: CSSProperties = { minWidth: 320 };
|
||||
const labelStyle: CSSProperties = { display: "block", color: "#6a7f97", fontSize: 11, fontWeight: 800, margin: "0 0 6px", textTransform: "uppercase", letterSpacing: "0.08em" };
|
||||
const inputStyle: CSSProperties = { width: "100%", boxSizing: "border-box", border: "1px solid rgba(127,208,255,0.14)", borderRadius: 12, padding: "10px 12px", background: "rgba(3,9,18,0.8)", color: "#ebf3ff" };
|
||||
const scoreGridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "220px repeat(5, minmax(180px, 1fr))", gap: 12 };
|
||||
const scoreCardStyle: CSSProperties = { padding: 18, borderRadius: 20, background: "rgba(15,35,52,0.88)", border: "1px solid rgba(124,231,211,0.2)", display: "flex", flexDirection: "column", gap: 10 };
|
||||
const scoreValueStyle: CSSProperties = { color: "#9ee8d7", fontSize: 52, lineHeight: 1, fontWeight: 950, letterSpacing: "-0.06em" };
|
||||
const dimensionCardStyle: CSSProperties = { padding: 16, borderRadius: 18, background: "rgba(9,19,34,0.78)", border: "1px solid rgba(127,208,255,0.12)" };
|
||||
const dimensionHeadStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 8, alignItems: "center", marginBottom: 12 };
|
||||
const barTrackStyle: CSSProperties = { height: 8, borderRadius: 999, background: "rgba(255,255,255,0.06)", overflow: "hidden" };
|
||||
const barFillStyle: CSSProperties = { height: "100%", borderRadius: 999 };
|
||||
const dimensionFootStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 6, color: "#8fa8c6", fontSize: 12, marginTop: 10 };
|
||||
const cardStyle: CSSProperties = { padding: 18, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 20, background: "rgba(9,19,34,0.78)" };
|
||||
const sectionTitleStyle: CSSProperties = { margin: "0 0 14px", color: "#ebf3ff", fontSize: 16 };
|
||||
const issueListStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 8 };
|
||||
const issueRowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "14px 1fr auto auto", gap: 12, alignItems: "start", padding: 12, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
|
||||
const categoryStyle: CSSProperties = { color: "#9ee8d7", border: "1px solid rgba(158,232,215,0.22)", borderRadius: 999, padding: "4px 8px", fontSize: 10, fontWeight: 900 };
|
||||
const smallButtonStyle: CSSProperties = { display: "inline-flex", gap: 6, alignItems: "center", border: "1px solid rgba(127,208,255,0.16)", borderRadius: 10, padding: "7px 9px", background: "rgba(127,208,255,0.08)", color: "#ebf3ff", cursor: "pointer", fontWeight: 800 };
|
||||
const secondaryButtonStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", justifyContent: "center", border: "1px solid rgba(127,208,255,0.16)", borderRadius: 12, padding: "10px 12px", background: "rgba(127,208,255,0.08)", color: "#ebf3ff", cursor: "pointer", fontWeight: 900 };
|
||||
const monoStyle: CSSProperties = { marginTop: 4, color: "#6a7f97", fontSize: 11, fontFamily: "JetBrains Mono, monospace", wordBreak: "break-all" };
|
||||
const mutedStyle: CSSProperties = { margin: 0, color: "#6a7f97", fontSize: 13 };
|
||||
const loadingStyle: CSSProperties = { display: "inline-flex", alignItems: "center", gap: 8, color: "#8fa8c6", padding: 18 };
|
||||
const emptyStyle: CSSProperties = { color: "#6a7f97", padding: 22 };
|
||||
const errorStyle: CSSProperties = { display: "flex", alignItems: "center", gap: 8, padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" };
|
||||
@@ -0,0 +1,498 @@
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
import {
|
||||
ReactFlow,
|
||||
Background,
|
||||
Controls,
|
||||
MiniMap,
|
||||
addEdge,
|
||||
useNodesState,
|
||||
useEdgesState,
|
||||
MarkerType,
|
||||
} from "@xyflow/react";
|
||||
import type { Connection, Edge, Node } from "@xyflow/react";
|
||||
import "@xyflow/react/dist/style.css";
|
||||
import {
|
||||
Plus,
|
||||
GitBranch,
|
||||
User,
|
||||
Shield,
|
||||
FileText,
|
||||
Layout,
|
||||
Send,
|
||||
Pencil,
|
||||
Trash2,
|
||||
} from "lucide-react";
|
||||
|
||||
type OntologyNodeData = {
|
||||
label?: string;
|
||||
type?: string;
|
||||
};
|
||||
|
||||
type OntologyNode = Node<OntologyNodeData>;
|
||||
type OntologyEdge = Edge<Record<string, unknown>>;
|
||||
|
||||
const nodeTypes = {
|
||||
classNode: ({ data }: { data: OntologyNodeData }) => (
|
||||
<div style={classNodeStyle}>
|
||||
<div style={classNodeHeader}>{data.label}</div>
|
||||
<div style={classNodeSub}>{data.type}</div>
|
||||
</div>
|
||||
),
|
||||
};
|
||||
|
||||
const classNodeStyle: React.CSSProperties = {
|
||||
padding: "12px 16px",
|
||||
borderRadius: "8px",
|
||||
background: "linear-gradient(135deg, rgba(74, 163, 255, 0.15), rgba(74, 163, 255, 0.05))",
|
||||
border: "1px solid rgba(127, 208, 255, 0.3)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "13px",
|
||||
fontWeight: "600",
|
||||
minWidth: "140px",
|
||||
textAlign: "center",
|
||||
boxShadow: "0 4px 12px rgba(0, 0, 0, 0.2)",
|
||||
};
|
||||
|
||||
const classNodeHeader: React.CSSProperties = {
|
||||
fontSize: "14px",
|
||||
fontWeight: "700",
|
||||
marginBottom: "4px",
|
||||
};
|
||||
|
||||
const classNodeSub: React.CSSProperties = {
|
||||
fontSize: "11px",
|
||||
color: "#8fa8c6",
|
||||
fontWeight: "500",
|
||||
};
|
||||
|
||||
interface DraftDiff {
|
||||
added_classes: string[];
|
||||
removed_classes: string[];
|
||||
modified_classes: Record<string, Record<string, any>>;
|
||||
added_properties: string[];
|
||||
removed_properties: string[];
|
||||
modified_properties: Record<string, Record<string, any>>;
|
||||
added_restrictions: Record<string, any>[];
|
||||
removed_restrictions: Record<string, any>[];
|
||||
added_axioms: Record<string, any>[];
|
||||
removed_axioms: Record<string, any>[];
|
||||
annotation_changes: Record<string, Record<string, any>>;
|
||||
}
|
||||
|
||||
interface RegistryEntry {
|
||||
uri: string;
|
||||
name: string;
|
||||
}
|
||||
|
||||
export function OntologyEditor() {
|
||||
const [nodes, setNodes, onNodesChange] = useNodesState<OntologyNode>([]);
|
||||
const [edges, setEdges, onEdgesChange] = useEdgesState<OntologyEdge>([]);
|
||||
const [selectedElement, setSelectedElement] = useState<OntologyNode | OntologyEdge | null>(null);
|
||||
const [registry, setRegistry] = useState<RegistryEntry[]>([]);
|
||||
const [ontologyUri, setOntologyUri] = useState<string>("");
|
||||
const [draftDiff, setDraftDiff] = useState<DraftDiff>({
|
||||
added_classes: [],
|
||||
removed_classes: [],
|
||||
modified_classes: {},
|
||||
added_properties: [],
|
||||
removed_properties: [],
|
||||
modified_properties: {},
|
||||
added_restrictions: [],
|
||||
removed_restrictions: [],
|
||||
added_axioms: [],
|
||||
removed_axioms: [],
|
||||
annotation_changes: {},
|
||||
});
|
||||
const [isSaving, setIsSaving] = useState(false);
|
||||
const [showContext, setShowContext] = useState<{ x: number; y: number; type: string; element: OntologyNode | OntologyEdge } | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
fetch("/api/ontology/registry")
|
||||
.then((response) => (response.ok ? response.json() : []))
|
||||
.then((entries: RegistryEntry[]) => {
|
||||
if (cancelled) return;
|
||||
setRegistry(entries);
|
||||
setOntologyUri((current) => current || entries[0]?.uri || "");
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error("Failed to load ontology registry:", error);
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, []);
|
||||
|
||||
const onConnect = useCallback(
|
||||
(params: Connection) => setEdges((eds) => addEdge({ ...params, markerEnd: { type: MarkerType.ArrowClosed } }, eds)),
|
||||
[setEdges]
|
||||
);
|
||||
|
||||
const addClass = useCallback(() => {
|
||||
const newId = `class_${Date.now()}`;
|
||||
const newNode: OntologyNode = {
|
||||
id: newId,
|
||||
type: "classNode",
|
||||
position: { x: Math.random() * 400, y: Math.random() * 300 },
|
||||
data: { label: "NewClass", type: "owl:Class" },
|
||||
};
|
||||
setNodes((nds) => [...nds, newNode]);
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
added_classes: [...prev.added_classes, newId],
|
||||
}));
|
||||
}, [setNodes]);
|
||||
|
||||
const addProperty = useCallback(() => {
|
||||
if (nodes.length < 2) {
|
||||
alert("Add at least two classes before creating a property edge.");
|
||||
return;
|
||||
}
|
||||
const newId = `prop_${Date.now()}`;
|
||||
const newEdge: OntologyEdge = {
|
||||
id: newId,
|
||||
source: nodes[0].id,
|
||||
target: nodes[1].id,
|
||||
label: "hasProperty",
|
||||
type: "smoothstep",
|
||||
animated: true,
|
||||
};
|
||||
setEdges((eds) => [...eds, newEdge]);
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
added_properties: [...prev.added_properties, newId],
|
||||
}));
|
||||
}, [nodes, setEdges]);
|
||||
|
||||
const addIndividual = useCallback(() => {
|
||||
const newId = `ind_${Date.now()}`;
|
||||
const newNode: OntologyNode = {
|
||||
id: newId,
|
||||
type: "classNode",
|
||||
position: { x: Math.random() * 400, y: Math.random() * 300 },
|
||||
data: { label: "NewIndividual", type: "owl:NamedIndividual" },
|
||||
};
|
||||
setNodes((nds) => [...nds, newNode]);
|
||||
}, [setNodes]);
|
||||
|
||||
const addRestriction = useCallback(() => {
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
added_restrictions: [...prev.added_restrictions, { type: "someValuesFrom", value: "" }],
|
||||
}));
|
||||
}, []);
|
||||
|
||||
const addAxiom = useCallback(() => {
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
added_axioms: [...prev.added_axioms, { type: "subClassOf", value: "" }],
|
||||
}));
|
||||
}, []);
|
||||
|
||||
const autoLayout = useCallback(() => {
|
||||
const layoutNodes = nodes.map((node, index) => ({
|
||||
...node,
|
||||
position: { x: (index % 4) * 200, y: Math.floor(index / 4) * 150 },
|
||||
}));
|
||||
setNodes(layoutNodes);
|
||||
}, [nodes, setNodes]);
|
||||
|
||||
const saveDraft = useCallback(async () => {
|
||||
if (!ontologyUri) {
|
||||
alert("Please select an ontology first");
|
||||
return;
|
||||
}
|
||||
setIsSaving(true);
|
||||
try {
|
||||
const response = await fetch("/api/ontology/draft", {
|
||||
method: "PATCH",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
ontology_uri: ontologyUri,
|
||||
diff: draftDiff,
|
||||
author: "user",
|
||||
summary: "Visual editor changes",
|
||||
}),
|
||||
});
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
alert(`Draft saved: ${data.draft_id}`);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to save draft:", error);
|
||||
alert("Failed to save draft");
|
||||
} finally {
|
||||
setIsSaving(false);
|
||||
}
|
||||
}, [ontologyUri, draftDiff]);
|
||||
|
||||
const handleNodeContextMenu = useCallback((event: React.MouseEvent, node: OntologyNode) => {
|
||||
event.preventDefault();
|
||||
setSelectedElement(node);
|
||||
setShowContext({ x: event.clientX, y: event.clientY, type: "node", element: node });
|
||||
}, []);
|
||||
|
||||
const handleEdgeContextMenu = useCallback((event: React.MouseEvent, edge: OntologyEdge) => {
|
||||
event.preventDefault();
|
||||
setSelectedElement(edge);
|
||||
setShowContext({ x: event.clientX, y: event.clientY, type: "edge", element: edge });
|
||||
}, []);
|
||||
|
||||
const deleteSelected = useCallback(() => {
|
||||
const target = showContext?.element ?? selectedElement;
|
||||
if (target) {
|
||||
if ("source" in target) {
|
||||
setEdges((eds) => eds.filter((e) => e.id !== target.id));
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
removed_properties: [...prev.removed_properties, target.id],
|
||||
}));
|
||||
} else {
|
||||
setNodes((nds) => nds.filter((n) => n.id !== target.id));
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
removed_classes: [...prev.removed_classes, target.id],
|
||||
}));
|
||||
}
|
||||
setSelectedElement(null);
|
||||
}
|
||||
setShowContext(null);
|
||||
}, [selectedElement, setNodes, setEdges, showContext]);
|
||||
|
||||
const renameSelected = useCallback(() => {
|
||||
const target = showContext?.element ?? selectedElement;
|
||||
if (target && !("source" in target)) {
|
||||
const newLabel = prompt("Enter new name:", String(target.data.label ?? ""));
|
||||
if (newLabel) {
|
||||
setNodes((nds) =>
|
||||
nds.map((n) => (n.id === target.id ? { ...n, data: { ...n.data, label: newLabel } } : n))
|
||||
);
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
modified_classes: { ...prev.modified_classes, [target.id]: { label: newLabel } },
|
||||
}));
|
||||
}
|
||||
}
|
||||
setShowContext(null);
|
||||
}, [selectedElement, setNodes, showContext]);
|
||||
|
||||
useEffect(() => {
|
||||
const handleClick = () => setShowContext(null);
|
||||
window.addEventListener("click", handleClick);
|
||||
return () => window.removeEventListener("click", handleClick);
|
||||
}, []);
|
||||
|
||||
const toolbarStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
gap: "8px",
|
||||
padding: "12px 16px",
|
||||
background: "rgba(3, 9, 18, 0.92)",
|
||||
borderBottom: "1px solid rgba(140, 192, 255, 0.12)",
|
||||
flexWrap: "wrap",
|
||||
};
|
||||
|
||||
const toolbarButtonStyle: React.CSSProperties = {
|
||||
display: "inline-flex",
|
||||
alignItems: "center",
|
||||
gap: "6px",
|
||||
padding: "8px 12px",
|
||||
borderRadius: "8px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.18)",
|
||||
background: "rgba(74, 163, 255, 0.08)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "12px",
|
||||
fontWeight: "600",
|
||||
cursor: "pointer",
|
||||
transition: "160ms ease",
|
||||
};
|
||||
|
||||
const selectStyle: React.CSSProperties = {
|
||||
padding: "8px 12px",
|
||||
borderRadius: "8px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.18)",
|
||||
background: "rgba(3, 9, 18, 0.88)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "12px",
|
||||
minWidth: "260px",
|
||||
};
|
||||
|
||||
const contextMenuStyle: React.CSSProperties = {
|
||||
position: "fixed",
|
||||
background: "rgba(9, 19, 34, 0.95)",
|
||||
border: "1px solid rgba(127, 208, 255, 0.3)",
|
||||
borderRadius: "8px",
|
||||
padding: "8px 0",
|
||||
minWidth: "180px",
|
||||
boxShadow: "0 8px 24px rgba(0, 0, 0, 0.4)",
|
||||
zIndex: 1000,
|
||||
};
|
||||
|
||||
const contextItemStyle: React.CSSProperties = {
|
||||
padding: "8px 16px",
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "10px",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "13px",
|
||||
cursor: "pointer",
|
||||
transition: "160ms ease",
|
||||
};
|
||||
|
||||
const detailPanelStyle: React.CSSProperties = {
|
||||
position: "absolute",
|
||||
right: 0,
|
||||
top: 0,
|
||||
bottom: 0,
|
||||
width: "320px",
|
||||
background: "rgba(9, 19, 34, 0.95)",
|
||||
borderLeft: "1px solid rgba(140, 192, 255, 0.12)",
|
||||
padding: "20px",
|
||||
overflow: "auto",
|
||||
backdropFilter: "blur(18px)",
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={{ display: "flex", flexDirection: "column", height: "100%", background: "#07111f" }}>
|
||||
<div style={toolbarStyle}>
|
||||
<select
|
||||
aria-label="Active ontology"
|
||||
value={ontologyUri}
|
||||
onChange={(event) => setOntologyUri(event.target.value)}
|
||||
style={selectStyle}
|
||||
>
|
||||
<option value="">Select ontology...</option>
|
||||
{registry.map((entry) => (
|
||||
<option key={entry.uri} value={entry.uri}>
|
||||
{entry.name || entry.uri}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
<button style={toolbarButtonStyle} onClick={addClass}>
|
||||
<Plus size={14} />
|
||||
Add Class
|
||||
</button>
|
||||
<button style={toolbarButtonStyle} onClick={addProperty} disabled={nodes.length < 2}>
|
||||
<GitBranch size={14} />
|
||||
Add Property
|
||||
</button>
|
||||
<button style={toolbarButtonStyle} onClick={addIndividual}>
|
||||
<User size={14} />
|
||||
Add Individual
|
||||
</button>
|
||||
<button style={toolbarButtonStyle} onClick={addRestriction}>
|
||||
<Shield size={14} />
|
||||
Add Restriction
|
||||
</button>
|
||||
<button style={toolbarButtonStyle} onClick={addAxiom}>
|
||||
<FileText size={14} />
|
||||
Add Axiom
|
||||
</button>
|
||||
<button style={toolbarButtonStyle} onClick={autoLayout}>
|
||||
<Layout size={14} />
|
||||
Auto Layout
|
||||
</button>
|
||||
<div style={{ flex: 1 }} />
|
||||
<button style={toolbarButtonStyle} onClick={saveDraft} disabled={isSaving}>
|
||||
<Send size={14} />
|
||||
{isSaving ? "Saving..." : "Propose"}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<div style={{ flex: 1, position: "relative" }}>
|
||||
<ReactFlow
|
||||
nodes={nodes}
|
||||
edges={edges}
|
||||
onNodesChange={onNodesChange}
|
||||
onEdgesChange={onEdgesChange}
|
||||
onConnect={onConnect}
|
||||
onNodeClick={(_, node) => setSelectedElement(node)}
|
||||
onEdgeClick={(_, edge) => setSelectedElement(edge)}
|
||||
onNodeContextMenu={handleNodeContextMenu}
|
||||
onEdgeContextMenu={handleEdgeContextMenu}
|
||||
nodeTypes={nodeTypes}
|
||||
fitView
|
||||
style={{ background: "#07111f" }}
|
||||
>
|
||||
<Background color="#1a2d3d" gap={20} />
|
||||
<Controls />
|
||||
<MiniMap nodeColor="#4aa3ff" maskColor="rgba(0,0,0,0.6)" />
|
||||
</ReactFlow>
|
||||
|
||||
{showContext && (
|
||||
<div style={{ ...contextMenuStyle, left: showContext.x, top: showContext.y }}>
|
||||
<div style={contextItemStyle} onClick={renameSelected}>
|
||||
<Pencil size={14} />
|
||||
Rename
|
||||
</div>
|
||||
<div style={contextItemStyle} onClick={deleteSelected}>
|
||||
<Trash2 size={14} />
|
||||
Delete
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{selectedElement && (
|
||||
<div style={detailPanelStyle}>
|
||||
<h3 style={{ margin: "0 0 16px", color: "#ebf3ff", fontSize: "16px" }}>
|
||||
{"source" in selectedElement ? "Property Details" : "Class Details"}
|
||||
</h3>
|
||||
<div style={{ marginBottom: "12px" }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
ID
|
||||
</label>
|
||||
<div style={{ color: "#ebf3ff", fontSize: "13px", wordBreak: "break-all" }}>
|
||||
{selectedElement.id}
|
||||
</div>
|
||||
</div>
|
||||
{!("source" in selectedElement) && (
|
||||
<>
|
||||
<div style={{ marginBottom: "12px" }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
Label
|
||||
</label>
|
||||
<input
|
||||
type="text"
|
||||
value={String(selectedElement.data.label ?? "")}
|
||||
onChange={(e) => {
|
||||
setNodes((nds) =>
|
||||
nds.map((n) =>
|
||||
n.id === selectedElement.id
|
||||
? { ...n, data: { ...n.data, label: e.target.value } }
|
||||
: n
|
||||
)
|
||||
);
|
||||
setDraftDiff((prev) => ({
|
||||
...prev,
|
||||
modified_classes: {
|
||||
...prev.modified_classes,
|
||||
[selectedElement.id]: { label: e.target.value },
|
||||
},
|
||||
}));
|
||||
}}
|
||||
style={{
|
||||
width: "100%",
|
||||
padding: "8px",
|
||||
borderRadius: "6px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.2)",
|
||||
background: "rgba(3, 9, 18, 0.8)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "13px",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
<div style={{ marginBottom: "12px" }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
Type
|
||||
</label>
|
||||
<div style={{ color: "#ebf3ff", fontSize: "13px" }}>
|
||||
{selectedElement.data.type || "owl:Class"}
|
||||
</div>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,413 @@
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
import {
|
||||
GitMerge,
|
||||
CheckCircle,
|
||||
XCircle,
|
||||
AlertCircle,
|
||||
Send,
|
||||
MessageSquare,
|
||||
User,
|
||||
Clock,
|
||||
Plus,
|
||||
Minus,
|
||||
Edit,
|
||||
} from "lucide-react";
|
||||
|
||||
interface Proposal {
|
||||
proposal_id: string;
|
||||
draft_id: string;
|
||||
ontology_uri: string;
|
||||
summary: string;
|
||||
author: string;
|
||||
reviewer: string | null;
|
||||
state: "draft" | "proposed" | "approved" | "published" | "rejected";
|
||||
impact_analysis: Record<string, any>;
|
||||
shacl_validation: Record<string, any>;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
comments: Record<string, any>[];
|
||||
}
|
||||
|
||||
interface DiffChange {
|
||||
type: "added" | "removed" | "modified";
|
||||
element: string;
|
||||
details?: Record<string, any>;
|
||||
}
|
||||
|
||||
export function ProposalReview({ proposalId }: { proposalId: string }) {
|
||||
const [proposal, setProposal] = useState<Proposal | null>(null);
|
||||
const [diff, setDiff] = useState<DiffChange[]>([]);
|
||||
const [selectedElement, setSelectedElement] = useState<string | null>(null);
|
||||
const [commentText, setCommentText] = useState("");
|
||||
|
||||
const loadProposal = useCallback(async () => {
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposalId}`);
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
setProposal(data);
|
||||
generateDiff(data);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to load proposal:", error);
|
||||
}
|
||||
}, [proposalId]);
|
||||
|
||||
const generateDiff = useCallback((prop: Proposal) => {
|
||||
const changes: DiffChange[] = [];
|
||||
|
||||
// Generate diff from impact analysis
|
||||
if (prop.impact_analysis) {
|
||||
if (prop.impact_analysis.class_adds > 0) {
|
||||
changes.push({ type: "added", element: `Classes (${prop.impact_analysis.class_adds})` });
|
||||
}
|
||||
if (prop.impact_analysis.class_removals > 0) {
|
||||
changes.push({ type: "removed", element: `Classes (${prop.impact_analysis.class_removals})` });
|
||||
}
|
||||
if (prop.impact_analysis.property_changes > 0) {
|
||||
changes.push({ type: "modified", element: `Properties (${prop.impact_analysis.property_changes})` });
|
||||
}
|
||||
if (prop.impact_analysis.restriction_changes > 0) {
|
||||
changes.push({ type: "modified", element: `Restrictions (${prop.impact_analysis.restriction_changes})` });
|
||||
}
|
||||
}
|
||||
|
||||
setDiff(changes);
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
loadProposal();
|
||||
}, [loadProposal]);
|
||||
|
||||
const addComment = useCallback(async () => {
|
||||
if (!selectedElement || !commentText || !proposal) return;
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/comment`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
element_uri: selectedElement,
|
||||
text: commentText,
|
||||
author: "user",
|
||||
}),
|
||||
});
|
||||
if (response.ok) {
|
||||
setCommentText("");
|
||||
loadProposal();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to add comment:", error);
|
||||
alert("Failed to add comment");
|
||||
}
|
||||
}, [selectedElement, commentText, proposal, loadProposal]);
|
||||
|
||||
const approveProposal = useCallback(async () => {
|
||||
if (!proposal) return;
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/approve`, {
|
||||
method: "POST",
|
||||
});
|
||||
if (response.ok) {
|
||||
alert("Proposal approved");
|
||||
loadProposal();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to approve proposal:", error);
|
||||
alert("Failed to approve proposal");
|
||||
}
|
||||
}, [proposal, loadProposal]);
|
||||
|
||||
const rejectProposal = useCallback(async () => {
|
||||
if (!proposal) return;
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/reject`, {
|
||||
method: "POST",
|
||||
});
|
||||
if (response.ok) {
|
||||
alert("Proposal rejected");
|
||||
loadProposal();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to reject proposal:", error);
|
||||
alert("Failed to reject proposal");
|
||||
}
|
||||
}, [proposal, loadProposal]);
|
||||
|
||||
const publishProposal = useCallback(async () => {
|
||||
if (!proposal) return;
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposal.proposal_id}/publish`, {
|
||||
method: "POST",
|
||||
});
|
||||
if (response.ok) {
|
||||
alert("Proposal published");
|
||||
loadProposal();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to publish proposal:", error);
|
||||
alert("Failed to publish proposal");
|
||||
}
|
||||
}, [proposal, loadProposal]);
|
||||
|
||||
const getChangeIcon = (type: string) => {
|
||||
switch (type) {
|
||||
case "added":
|
||||
return <Plus size={14} color="#4cc38a" />;
|
||||
case "removed":
|
||||
return <Minus size={14} color="#ff6b6b" />;
|
||||
case "modified":
|
||||
return <Edit size={14} color="#f2b66d" />;
|
||||
default:
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
const getStateIcon = (state: string) => {
|
||||
switch (state) {
|
||||
case "published":
|
||||
return <CheckCircle size={20} color="#4cc38a" />;
|
||||
case "approved":
|
||||
return <CheckCircle size={20} color="#4aa3ff" />;
|
||||
case "rejected":
|
||||
return <XCircle size={20} color="#ff6b6b" />;
|
||||
case "proposed":
|
||||
return <AlertCircle size={20} color="#f2b66d" />;
|
||||
default:
|
||||
return <Clock size={20} color="#8fa8c6" />;
|
||||
}
|
||||
};
|
||||
|
||||
const containerStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
height: "100%",
|
||||
background: "#07111f",
|
||||
padding: "20px",
|
||||
overflow: "auto",
|
||||
};
|
||||
|
||||
const headerStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
marginBottom: "20px",
|
||||
paddingBottom: "16px",
|
||||
borderBottom: "1px solid rgba(140, 192, 255, 0.12)",
|
||||
};
|
||||
|
||||
const titleStyle: React.CSSProperties = {
|
||||
margin: 0,
|
||||
color: "#ebf3ff",
|
||||
fontSize: "20px",
|
||||
fontWeight: "700",
|
||||
};
|
||||
|
||||
const contentStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
gap: "20px",
|
||||
flex: 1,
|
||||
minHeight: 0,
|
||||
};
|
||||
|
||||
const diffPanelStyle: React.CSSProperties = {
|
||||
flex: 1,
|
||||
background: "rgba(9, 19, 34, 0.8)",
|
||||
borderRadius: "8px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.12)",
|
||||
padding: "16px",
|
||||
overflow: "auto",
|
||||
};
|
||||
|
||||
const commentsPanelStyle: React.CSSProperties = {
|
||||
width: "320px",
|
||||
background: "rgba(9, 19, 34, 0.8)",
|
||||
borderRadius: "8px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.12)",
|
||||
padding: "16px",
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
};
|
||||
|
||||
const diffItemStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "10px",
|
||||
padding: "10px 12px",
|
||||
borderRadius: "6px",
|
||||
background: "rgba(3, 9, 18, 0.6)",
|
||||
marginBottom: "8px",
|
||||
cursor: "pointer",
|
||||
transition: "160ms ease",
|
||||
};
|
||||
|
||||
const buttonStyle: React.CSSProperties = {
|
||||
display: "inline-flex",
|
||||
alignItems: "center",
|
||||
gap: "6px",
|
||||
padding: "8px 14px",
|
||||
borderRadius: "6px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.2)",
|
||||
background: "rgba(74, 163, 255, 0.1)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "12px",
|
||||
fontWeight: "600",
|
||||
cursor: "pointer",
|
||||
transition: "160ms ease",
|
||||
};
|
||||
|
||||
const textareaStyle: React.CSSProperties = {
|
||||
width: "100%",
|
||||
padding: "10px 12px",
|
||||
borderRadius: "6px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.2)",
|
||||
background: "rgba(3, 9, 18, 0.8)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "13px",
|
||||
resize: "vertical",
|
||||
minHeight: "80px",
|
||||
};
|
||||
|
||||
if (!proposal) {
|
||||
return (
|
||||
<div style={containerStyle}>
|
||||
<div style={{ color: "#8fa8c6", fontSize: "14px" }}>Loading proposal...</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div style={containerStyle}>
|
||||
<div style={headerStyle}>
|
||||
<div style={{ display: "flex", alignItems: "center", gap: "12px" }}>
|
||||
{getStateIcon(proposal.state)}
|
||||
<div>
|
||||
<h1 style={titleStyle}>{proposal.summary}</h1>
|
||||
<div style={{ color: "#8fa8c6", fontSize: "12px", marginTop: "4px" }}>
|
||||
{proposal.author} • {new Date(proposal.created_at).toLocaleString()}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div style={{ display: "flex", gap: "8px" }}>
|
||||
{proposal.state === "proposed" && (
|
||||
<>
|
||||
<button style={buttonStyle} onClick={approveProposal}>
|
||||
<CheckCircle size={12} />
|
||||
Approve
|
||||
</button>
|
||||
<button style={buttonStyle} onClick={rejectProposal}>
|
||||
<XCircle size={12} />
|
||||
Reject
|
||||
</button>
|
||||
</>
|
||||
)}
|
||||
{proposal.state === "approved" && (
|
||||
<button style={buttonStyle} onClick={publishProposal}>
|
||||
<Send size={12} />
|
||||
Publish
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style={contentStyle}>
|
||||
<div style={diffPanelStyle}>
|
||||
<h2 style={{ margin: "0 0 16px", color: "#ebf3ff", fontSize: "14px", fontWeight: "600" }}>
|
||||
<GitMerge size={16} style={{ marginRight: "8px", verticalAlign: "middle" }} />
|
||||
Diff Viewer
|
||||
</h2>
|
||||
{diff.length === 0 ? (
|
||||
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No changes detected</div>
|
||||
) : (
|
||||
diff.map((change, index) => (
|
||||
<div
|
||||
key={index}
|
||||
style={{
|
||||
...diffItemStyle,
|
||||
border: selectedElement === change.element ? "1px solid rgba(74, 163, 255, 0.4)" : "1px solid transparent",
|
||||
}}
|
||||
onClick={() => setSelectedElement(change.element)}
|
||||
>
|
||||
{getChangeIcon(change.type)}
|
||||
<div style={{ flex: 1 }}>
|
||||
<div style={{ color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
|
||||
{change.element}
|
||||
</div>
|
||||
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>
|
||||
{change.type}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
))
|
||||
)}
|
||||
|
||||
<div style={{ marginTop: "20px", paddingTop: "16px", borderTop: "1px solid rgba(140, 192, 255, 0.12)" }}>
|
||||
<h3 style={{ margin: "0 0 12px", color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
|
||||
Impact Analysis
|
||||
</h3>
|
||||
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
|
||||
{JSON.stringify(proposal.impact_analysis, null, 2)}
|
||||
</pre>
|
||||
</div>
|
||||
|
||||
<div style={{ marginTop: "16px" }}>
|
||||
<h3 style={{ margin: "0 0 12px", color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
|
||||
SHACL Validation
|
||||
</h3>
|
||||
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
|
||||
{JSON.stringify(proposal.shacl_validation, null, 2)}
|
||||
</pre>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style={commentsPanelStyle}>
|
||||
<h2 style={{ margin: "0 0 16px", color: "#ebf3ff", fontSize: "14px", fontWeight: "600" }}>
|
||||
<MessageSquare size={16} style={{ marginRight: "8px", verticalAlign: "middle" }} />
|
||||
Comments ({proposal.comments.length})
|
||||
</h2>
|
||||
<div style={{ flex: 1, overflow: "auto", marginBottom: "12px" }}>
|
||||
{proposal.comments.length === 0 ? (
|
||||
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No comments yet</div>
|
||||
) : (
|
||||
proposal.comments.map((comment) => (
|
||||
<div
|
||||
key={comment.id}
|
||||
style={{
|
||||
padding: "10px",
|
||||
background: "rgba(3, 9, 18, 0.6)",
|
||||
borderRadius: "6px",
|
||||
marginBottom: "8px",
|
||||
}}
|
||||
>
|
||||
<div style={{ display: "flex", alignItems: "center", gap: "6px", marginBottom: "4px" }}>
|
||||
<User size={12} color="#8fa8c6" />
|
||||
<span style={{ color: "#ebf3ff", fontSize: "12px", fontWeight: "600" }}>
|
||||
{comment.author}
|
||||
</span>
|
||||
</div>
|
||||
<div style={{ color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>{comment.text}</div>
|
||||
<div style={{ color: "#5a7a9a", fontSize: "10px" }}>
|
||||
{new Date(comment.created_at).toLocaleString()}
|
||||
</div>
|
||||
</div>
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
{selectedElement && (
|
||||
<div>
|
||||
<textarea
|
||||
placeholder="Add a comment..."
|
||||
value={commentText}
|
||||
onChange={(e) => setCommentText(e.target.value)}
|
||||
style={textareaStyle}
|
||||
/>
|
||||
<button style={buttonStyle} onClick={addComment} disabled={!commentText}>
|
||||
<Send size={12} />
|
||||
Add Comment
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,346 @@
|
||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||
import type { CSSProperties } from "react";
|
||||
import Editor, { type Monaco } from "@monaco-editor/react";
|
||||
import { FileCode2, Loader2, Play, Shield, Wand2 } from "lucide-react";
|
||||
import {
|
||||
generateShacl,
|
||||
loadOntologyRegistry,
|
||||
loadShaclShapes,
|
||||
validateShacl,
|
||||
} from "./api";
|
||||
import type { OntologyEntry, ShaclShapeSummary, ShaclValidationResponse } from "./types";
|
||||
|
||||
interface ShaclStudioProps {
|
||||
onJumpToNode?: (nodeId: string) => void;
|
||||
}
|
||||
|
||||
export function ShaclStudio({ onJumpToNode }: ShaclStudioProps) {
|
||||
const [registry, setRegistry] = useState<OntologyEntry[]>([]);
|
||||
const [selectedUri, setSelectedUri] = useState("");
|
||||
const [shacl, setShacl] = useState("");
|
||||
const [fullShacl, setFullShacl] = useState(""); // preserves complete Turtle across shape selections
|
||||
const [shapes, setShapes] = useState<ShaclShapeSummary[]>([]);
|
||||
const [selectedShapeId, setSelectedShapeId] = useState<string | null>(null);
|
||||
const [validation, setValidation] = useState<ShaclValidationResponse | null>(null);
|
||||
const [loading, setLoading] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
loadOntologyRegistry()
|
||||
.then((entries) => {
|
||||
if (cancelled) return;
|
||||
setRegistry(entries);
|
||||
setSelectedUri((current) => current || entries[0]?.uri || "");
|
||||
})
|
||||
.catch((err) => {
|
||||
if (!cancelled) setError(err instanceof Error ? err.message : "Could not load ontology registry.");
|
||||
});
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, []);
|
||||
|
||||
const loadShapes = useCallback(async (uri: string) => {
|
||||
if (!uri) return;
|
||||
setLoading(true);
|
||||
setError("");
|
||||
try {
|
||||
const data = await loadShaclShapes(uri);
|
||||
setShapes(data.shapes);
|
||||
const turtle = data.shacl_turtle;
|
||||
setFullShacl(turtle);
|
||||
setShacl((current) => current || turtle);
|
||||
setSelectedShapeId(null);
|
||||
setValidation(null);
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Could not load SHACL shapes.");
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
setShacl("");
|
||||
setFullShacl("");
|
||||
setSelectedShapeId(null);
|
||||
void loadShapes(selectedUri);
|
||||
}, [selectedUri, loadShapes]);
|
||||
|
||||
const handleGenerate = useCallback(async () => {
|
||||
if (!selectedUri) return;
|
||||
setLoading(true);
|
||||
setError("");
|
||||
try {
|
||||
const data = await generateShacl(selectedUri, "strict");
|
||||
setFullShacl(data.shacl_turtle);
|
||||
setShacl(data.shacl_turtle);
|
||||
setSelectedShapeId(null);
|
||||
const shapeData = await loadShaclShapes(selectedUri);
|
||||
setShapes(shapeData.shapes);
|
||||
setValidation(null);
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Could not generate SHACL.");
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, [selectedUri]);
|
||||
|
||||
const handleSelectShape = useCallback((shapeId: string) => {
|
||||
setSelectedShapeId(shapeId);
|
||||
// Extract the Turtle block for this shape from the full SHACL so the editor
|
||||
// pre-populates with the selected shape's definition.
|
||||
const src = fullShacl || shacl;
|
||||
const normalised = src.replace(/\r\n/g, "\n");
|
||||
// Split on blank lines to isolate statement groups.
|
||||
const blocks = normalised.split(/\n{2,}/).filter((b) => b.trim());
|
||||
const match = blocks.find((b) => {
|
||||
const first = b.trimStart();
|
||||
return first.startsWith(shapeId + " ") || first.startsWith(shapeId + "\n") || first.startsWith(shapeId + "\t");
|
||||
});
|
||||
if (match) {
|
||||
setShacl(match.trim());
|
||||
}
|
||||
}, [fullShacl, shacl]);
|
||||
|
||||
const handleShowAllShapes = useCallback(() => {
|
||||
setSelectedShapeId(null);
|
||||
setShacl(fullShacl);
|
||||
}, [fullShacl]);
|
||||
|
||||
const handleValidate = async () => {
|
||||
if (!selectedUri || !shacl.trim()) return;
|
||||
setLoading(true);
|
||||
setError("");
|
||||
try {
|
||||
setValidation(await validateShacl(selectedUri, shacl));
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : "Could not validate SHACL.");
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
};
|
||||
|
||||
const groupedShapes = useMemo(() => {
|
||||
const groups = new Map<string, ShaclShapeSummary[]>();
|
||||
for (const shape of shapes) {
|
||||
const key = shape.target_class || "Untargeted shapes";
|
||||
groups.set(key, [...(groups.get(key) || []), shape]);
|
||||
}
|
||||
return Array.from(groups.entries());
|
||||
}, [shapes]);
|
||||
|
||||
const beforeMount = useCallback((monaco: Monaco) => {
|
||||
if (!monaco.languages.getLanguages().some((language: { id: string }) => language.id === "turtle")) {
|
||||
monaco.languages.register({ id: "turtle", extensions: [".ttl"], mimetypes: ["text/turtle"] });
|
||||
monaco.languages.setMonarchTokensProvider("turtle", {
|
||||
keywords: ["@prefix", "@base", "a"],
|
||||
tokenizer: {
|
||||
root: [
|
||||
[/#[^\n]*/, "comment"],
|
||||
[/"(?:[^"\\]|\\.)*"(?:@[a-zA-Z-]+|\\^\\^[^\s,;.]+)?/, "string"],
|
||||
[/'(?:[^'\\]|\\.)*'(?:@[a-zA-Z-]+|\\^\\^[^\s,;.]+)?/, "string"],
|
||||
[/"""[\s\S]*?"""/, "string"],
|
||||
[/<[^>]*>/, "type.identifier"],
|
||||
[/\b(?:@prefix|@base|a)\b/, "keyword"],
|
||||
[/\b(?:sh|xsd|owl|rdf|rdfs|skos):[\w]+/, "variable"],
|
||||
[/[a-zA-Z_][\w-]*:[\w]+/, "namespace"],
|
||||
[/[;,.]/, "delimiter"],
|
||||
[/\d+(?:\.\d+)?/, "number"],
|
||||
],
|
||||
},
|
||||
});
|
||||
}
|
||||
monaco.editor.defineTheme("shacl-dark", {
|
||||
base: "vs-dark",
|
||||
inherit: true,
|
||||
rules: [
|
||||
{ token: "keyword", foreground: "9ee8d7" },
|
||||
{ token: "string", foreground: "f2b66d" },
|
||||
{ token: "comment", foreground: "4a6070", fontStyle: "italic" },
|
||||
{ token: "type.identifier", foreground: "7ce7d3" },
|
||||
{ token: "variable", foreground: "d2a8ff" },
|
||||
{ token: "namespace", foreground: "a5d6ff" },
|
||||
{ token: "number", foreground: "79c0ff" },
|
||||
{ token: "delimiter", foreground: "8fa8c6" },
|
||||
],
|
||||
colors: {
|
||||
"editor.background": "#050b13",
|
||||
"editor.foreground": "#d7e7f8",
|
||||
"editorLineNumber.foreground": "#41536b",
|
||||
},
|
||||
});
|
||||
}, []);
|
||||
|
||||
return (
|
||||
<div style={pageStyle}>
|
||||
<section style={heroStyle}>
|
||||
<div>
|
||||
<div style={kickerStyle}><Shield size={14} /> SHACL Studio</div>
|
||||
<h2 style={titleStyle}>Generate, edit, and validate shapes</h2>
|
||||
<p style={textStyle}>
|
||||
Create strict SHACL Turtle from ontology structure, inspect shape targets,
|
||||
run validation, and jump from violations back into the graph.
|
||||
</p>
|
||||
</div>
|
||||
<div style={selectorShellStyle}>
|
||||
<label style={labelStyle}>Ontology</label>
|
||||
<select style={inputStyle} value={selectedUri} onChange={(event) => setSelectedUri(event.target.value)}>
|
||||
{registry.map((entry) => <option key={entry.uri} value={entry.uri}>{entry.name}</option>)}
|
||||
</select>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
{error ? <div style={errorStyle}>{error}</div> : null}
|
||||
|
||||
<div style={gridStyle}>
|
||||
<section style={cardStyle}>
|
||||
<div style={panelHeaderStyle}>
|
||||
<h3 style={sectionTitleStyle}>Shape library</h3>
|
||||
<div style={{ display: "flex", gap: 8, alignItems: "center" }}>
|
||||
<span style={countBadgeStyle}>{shapes.length} shapes</span>
|
||||
{selectedShapeId ? (
|
||||
<button style={smallButtonStyle} onClick={handleShowAllShapes}>View all</button>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
<div style={shapeListStyle}>
|
||||
{groupedShapes.map(([target, items]) => (
|
||||
<div key={target} style={shapeGroupStyle}>
|
||||
<div style={shapeTargetStyle}>{target}</div>
|
||||
{items.map((shape) => {
|
||||
const isSelected = selectedShapeId === shape.id;
|
||||
return (
|
||||
<button
|
||||
key={shape.id}
|
||||
style={{
|
||||
...shapeRowStyle,
|
||||
cursor: "pointer",
|
||||
background: isSelected ? "rgba(124,231,211,0.1)" : "rgba(255,255,255,0.03)",
|
||||
border: isSelected ? "1px solid rgba(124,231,211,0.35)" : "1px solid rgba(127,208,255,0.08)",
|
||||
textAlign: "left",
|
||||
width: "100%",
|
||||
}}
|
||||
onClick={() => handleSelectShape(shape.id)}
|
||||
title="Click to load this shape into the editor"
|
||||
>
|
||||
<FileCode2 size={14} color={isSelected ? "#7ce7d3" : "#9ee8d7"} />
|
||||
<div>
|
||||
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{shape.id}</div>
|
||||
<div style={mutedStyle}>
|
||||
{shape.constraint_count} constraints
|
||||
{shape.constraints.length ? ` · ${shape.constraints.join(", ")}` : ""}
|
||||
</div>
|
||||
</div>
|
||||
<span style={violationBadgeStyle}>{shape.violation_count}</span>
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
))}
|
||||
{!shapes.length ? <p style={mutedStyle}>No shapes generated yet.</p> : null}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section style={editorShellStyle}>
|
||||
<div style={panelHeaderStyle}>
|
||||
<h3 style={sectionTitleStyle}>
|
||||
{selectedShapeId ? selectedShapeId : "Turtle shape editor"}
|
||||
</h3>
|
||||
<div style={{ display: "flex", gap: 8 }}>
|
||||
<button style={secondaryButtonStyle} disabled={loading} onClick={handleGenerate}><Wand2 size={14} /> Generate strict</button>
|
||||
<button style={primaryButtonStyle} disabled={loading || !shacl.trim()} onClick={handleValidate}>
|
||||
{loading ? <Loader2 size={14} className="spin" /> : <Play size={14} />}
|
||||
Validate
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<div style={editorFrameStyle}>
|
||||
<Editor
|
||||
height="100%"
|
||||
language="turtle"
|
||||
theme="shacl-dark"
|
||||
beforeMount={beforeMount}
|
||||
value={shacl}
|
||||
onChange={(value) => setShacl(value || "")}
|
||||
options={{
|
||||
minimap: { enabled: false },
|
||||
fontSize: 13,
|
||||
fontFamily: "JetBrains Mono, monospace",
|
||||
wordWrap: "on",
|
||||
scrollBeyondLastLine: false,
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<section style={cardStyle}>
|
||||
<div style={panelHeaderStyle}>
|
||||
<h3 style={sectionTitleStyle}>Validation report</h3>
|
||||
{validation ? <span style={validationBadgeStyle(validation.status, validation.conforms)}>{validation.status}{validation.conforms ? " · conforms" : ""}</span> : null}
|
||||
</div>
|
||||
{validation ? (
|
||||
<>
|
||||
<p style={textStyle}>{validation.message}</p>
|
||||
<div style={shapeListStyle}>
|
||||
{validation.violations.map((violation, index) => {
|
||||
const nodeId = violation.focus_node || violation.node;
|
||||
return (
|
||||
<div key={`${violation.node}-${violation.path}-${index}`} style={violationRowStyle}>
|
||||
<div>
|
||||
<div style={{ color: "#ebf3ff", fontWeight: 800 }}>{violation.message}</div>
|
||||
<div style={mutedStyle}>{violation.severity} {violation.path ? `· ${violation.path}` : ""}</div>
|
||||
{nodeId ? <div style={monoStyle}>{nodeId}</div> : null}
|
||||
</div>
|
||||
{nodeId ? (
|
||||
<button style={smallButtonStyle} onClick={() => onJumpToNode?.(nodeId)}>
|
||||
Jump to Node
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
{!validation.violations.length ? <p style={mutedStyle}>No validation violations returned.</p> : null}
|
||||
</div>
|
||||
</>
|
||||
) : (
|
||||
<p style={mutedStyle}>Generate or edit SHACL Turtle, then run validation.</p>
|
||||
)}
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function validationBadgeStyle(status: string, conforms: boolean): CSSProperties {
|
||||
const color = status === "unavailable" ? "#f2b66d" : conforms ? "#7ce7d3" : "#ff9daf";
|
||||
return { color, border: `1px solid ${color}35`, background: `${color}14`, borderRadius: 999, padding: "4px 9px", fontSize: 11, fontWeight: 900, textTransform: "uppercase" };
|
||||
}
|
||||
|
||||
const pageStyle: CSSProperties = { height: "100%", overflow: "auto", padding: 22, display: "flex", flexDirection: "column", gap: 16 };
|
||||
const heroStyle: CSSProperties = { display: "flex", justifyContent: "space-between", gap: 18, padding: 22, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 22, background: "linear-gradient(135deg, rgba(11,25,42,0.94), rgba(7,14,25,0.9))" };
|
||||
const kickerStyle: CSSProperties = { display: "inline-flex", gap: 8, alignItems: "center", color: "#9ee8d7", fontSize: 11, fontWeight: 900, letterSpacing: "0.12em", textTransform: "uppercase" };
|
||||
const titleStyle: CSSProperties = { margin: "8px 0", color: "#ebf3ff", fontSize: 26, letterSpacing: "-0.04em" };
|
||||
const textStyle: CSSProperties = { margin: 0, color: "#8fa8c6", lineHeight: 1.6, maxWidth: 680 };
|
||||
const selectorShellStyle: CSSProperties = { minWidth: 320 };
|
||||
const labelStyle: CSSProperties = { display: "block", color: "#6a7f97", fontSize: 11, fontWeight: 800, margin: "0 0 6px", textTransform: "uppercase", letterSpacing: "0.08em" };
|
||||
const inputStyle: CSSProperties = { width: "100%", boxSizing: "border-box", border: "1px solid rgba(127,208,255,0.14)", borderRadius: 12, padding: "10px 12px", background: "rgba(3,9,18,0.8)", color: "#ebf3ff" };
|
||||
const gridStyle: CSSProperties = { display: "grid", gridTemplateColumns: "360px minmax(0, 1fr)", gap: 16, minHeight: 560 };
|
||||
const cardStyle: CSSProperties = { padding: 18, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 20, background: "rgba(9,19,34,0.78)" };
|
||||
const editorShellStyle: CSSProperties = { ...cardStyle, display: "flex", flexDirection: "column", minHeight: 560 };
|
||||
const panelHeaderStyle: CSSProperties = { display: "flex", alignItems: "center", justifyContent: "space-between", gap: 12, marginBottom: 12 };
|
||||
const sectionTitleStyle: CSSProperties = { margin: 0, color: "#ebf3ff", fontSize: 16 };
|
||||
const countBadgeStyle: CSSProperties = { color: "#9ee8d7", border: "1px solid rgba(158,232,215,0.2)", borderRadius: 999, padding: "4px 9px", fontSize: 11, fontWeight: 900 };
|
||||
const shapeListStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 10 };
|
||||
const shapeGroupStyle: CSSProperties = { display: "flex", flexDirection: "column", gap: 6 };
|
||||
const shapeTargetStyle: CSSProperties = { color: "#6a7f97", fontSize: 11, fontWeight: 900, textTransform: "uppercase", letterSpacing: "0.08em" };
|
||||
const shapeRowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "18px 1fr auto", gap: 10, padding: 10, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
|
||||
const violationBadgeStyle: CSSProperties = { color: "#f2b66d", fontWeight: 900 };
|
||||
const editorFrameStyle: CSSProperties = { flex: 1, minHeight: 0, border: "1px solid rgba(127,208,255,0.12)", borderRadius: 16, overflow: "hidden" };
|
||||
const primaryButtonStyle: CSSProperties = { border: "1px solid rgba(124,231,211,0.35)", borderRadius: 12, padding: "9px 11px", background: "linear-gradient(135deg, rgba(20,151,136,0.55), rgba(74,163,255,0.35))", color: "#ebf3ff", fontWeight: 900, cursor: "pointer", display: "inline-flex", alignItems: "center", gap: 8 };
|
||||
const secondaryButtonStyle: CSSProperties = { ...primaryButtonStyle, background: "rgba(127,208,255,0.08)", borderColor: "rgba(127,208,255,0.18)" };
|
||||
const violationRowStyle: CSSProperties = { display: "grid", gridTemplateColumns: "1fr auto", gap: 12, padding: 12, borderRadius: 14, background: "rgba(255,255,255,0.03)", border: "1px solid rgba(127,208,255,0.08)" };
|
||||
const smallButtonStyle: CSSProperties = { border: "1px solid rgba(127,208,255,0.16)", borderRadius: 10, padding: "7px 9px", background: "rgba(127,208,255,0.08)", color: "#ebf3ff", cursor: "pointer", fontWeight: 800 };
|
||||
const monoStyle: CSSProperties = { marginTop: 4, color: "#6a7f97", fontSize: 11, fontFamily: "JetBrains Mono, monospace", wordBreak: "break-all" };
|
||||
const mutedStyle: CSSProperties = { margin: 0, color: "#6a7f97", fontSize: 12, lineHeight: 1.5 };
|
||||
const errorStyle: CSSProperties = { padding: 12, borderRadius: 14, color: "#ffb4c2", background: "rgba(255,157,175,0.1)", border: "1px solid rgba(255,157,175,0.18)" };
|
||||
@@ -0,0 +1,475 @@
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
import {
|
||||
Layers,
|
||||
GitMerge,
|
||||
Clock,
|
||||
FileText,
|
||||
CheckCircle,
|
||||
XCircle,
|
||||
AlertCircle,
|
||||
Send,
|
||||
X,
|
||||
ArrowRight,
|
||||
Scale,
|
||||
} from "lucide-react";
|
||||
|
||||
interface VersionEntry {
|
||||
version_id: string;
|
||||
ontology_uri: string;
|
||||
state: "draft" | "published";
|
||||
author: string;
|
||||
date: string;
|
||||
diff_summary: Record<string, any>;
|
||||
}
|
||||
|
||||
interface Proposal {
|
||||
proposal_id: string;
|
||||
draft_id: string;
|
||||
ontology_uri: string;
|
||||
summary: string;
|
||||
author: string;
|
||||
reviewer: string | null;
|
||||
state: "draft" | "proposed" | "approved" | "published" | "rejected";
|
||||
impact_analysis: Record<string, any>;
|
||||
shacl_validation: Record<string, any>;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
comments: Record<string, any>[];
|
||||
}
|
||||
|
||||
export function VersionsTab() {
|
||||
const [ontologyUri, setOntologyUri] = useState<string>("");
|
||||
const [versions, setVersions] = useState<VersionEntry[]>([]);
|
||||
const [proposals, setProposals] = useState<Proposal[]>([]);
|
||||
const [selectedProposal, setSelectedProposal] = useState<Proposal | null>(null);
|
||||
const [showProposalModal, setShowProposalModal] = useState(false);
|
||||
const [showCompareModal, setShowCompareModal] = useState(false);
|
||||
const [comparePair, setComparePair] = useState<{ v1: string; v2: string } | null>(null);
|
||||
const [compareResult, setCompareResult] = useState<Record<string, any> | null>(null);
|
||||
const [isLoading, setIsLoading] = useState(false);
|
||||
|
||||
const loadVersions = useCallback(async () => {
|
||||
if (!ontologyUri) return;
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/versions/${encodeURIComponent(ontologyUri)}`);
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
setVersions(data);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to load versions:", error);
|
||||
}
|
||||
}, [ontologyUri]);
|
||||
|
||||
const loadProposals = useCallback(async () => {
|
||||
try {
|
||||
const response = await fetch("/api/ontology/proposals");
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
setProposals(data);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to load proposals:", error);
|
||||
}
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
loadVersions();
|
||||
loadProposals();
|
||||
}, [loadVersions, loadProposals]);
|
||||
|
||||
const approveProposal = useCallback(async (proposalId: string) => {
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposalId}/approve`, {
|
||||
method: "POST",
|
||||
});
|
||||
if (response.ok) {
|
||||
alert("Proposal approved");
|
||||
loadProposals();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to approve proposal:", error);
|
||||
alert("Failed to approve proposal");
|
||||
}
|
||||
}, [loadProposals]);
|
||||
|
||||
const rejectProposal = useCallback(async (proposalId: string) => {
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposalId}/reject`, {
|
||||
method: "POST",
|
||||
});
|
||||
if (response.ok) {
|
||||
alert("Proposal rejected");
|
||||
loadProposals();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to reject proposal:", error);
|
||||
alert("Failed to reject proposal");
|
||||
}
|
||||
}, [loadProposals]);
|
||||
|
||||
const publishProposal = useCallback(async (proposalId: string) => {
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/proposals/${proposalId}/publish`, {
|
||||
method: "POST",
|
||||
});
|
||||
if (response.ok) {
|
||||
alert("Proposal published");
|
||||
loadProposals();
|
||||
loadVersions();
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to publish proposal:", error);
|
||||
alert("Failed to publish proposal");
|
||||
}
|
||||
}, [loadProposals, loadVersions]);
|
||||
|
||||
const runVersionComparison = useCallback(async () => {
|
||||
if (!comparePair || !ontologyUri) return;
|
||||
setIsLoading(true);
|
||||
try {
|
||||
const response = await fetch(`/api/ontology/versions/${encodeURIComponent(ontologyUri)}/compare`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
version1: comparePair.v1,
|
||||
version2: comparePair.v2,
|
||||
}),
|
||||
});
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
setCompareResult(data);
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to compare versions:", error);
|
||||
alert("Failed to compare versions");
|
||||
} finally {
|
||||
setIsLoading(false);
|
||||
}
|
||||
}, [comparePair, ontologyUri]);
|
||||
|
||||
const getStateIcon = (state: string) => {
|
||||
switch (state) {
|
||||
case "published":
|
||||
return <CheckCircle size={16} color="#4cc38a" />;
|
||||
case "approved":
|
||||
return <CheckCircle size={16} color="#4aa3ff" />;
|
||||
case "rejected":
|
||||
return <XCircle size={16} color="#ff6b6b" />;
|
||||
case "proposed":
|
||||
return <AlertCircle size={16} color="#f2b66d" />;
|
||||
default:
|
||||
return <Clock size={16} color="#8fa8c6" />;
|
||||
}
|
||||
};
|
||||
|
||||
const containerStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
height: "100%",
|
||||
background: "#07111f",
|
||||
padding: "20px",
|
||||
overflow: "auto",
|
||||
};
|
||||
|
||||
const headerStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
marginBottom: "20px",
|
||||
};
|
||||
|
||||
const titleStyle: React.CSSProperties = {
|
||||
margin: 0,
|
||||
color: "#ebf3ff",
|
||||
fontSize: "20px",
|
||||
fontWeight: "700",
|
||||
};
|
||||
|
||||
const sectionStyle: React.CSSProperties = {
|
||||
marginBottom: "24px",
|
||||
};
|
||||
|
||||
const sectionTitleStyle: React.CSSProperties = {
|
||||
margin: "0 0 12px",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "14px",
|
||||
fontWeight: "600",
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "8px",
|
||||
};
|
||||
|
||||
const listStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
gap: "8px",
|
||||
};
|
||||
|
||||
const itemStyle: React.CSSProperties = {
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "12px",
|
||||
padding: "12px 16px",
|
||||
borderRadius: "8px",
|
||||
background: "rgba(9, 19, 34, 0.8)",
|
||||
border: "1px solid rgba(127, 208, 255, 0.12)",
|
||||
transition: "160ms ease",
|
||||
};
|
||||
|
||||
const modalOverlayStyle: React.CSSProperties = {
|
||||
position: "fixed",
|
||||
inset: 0,
|
||||
background: "rgba(0, 0, 0, 0.7)",
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
zIndex: 1000,
|
||||
};
|
||||
|
||||
const modalStyle: React.CSSProperties = {
|
||||
background: "rgba(9, 19, 34, 0.95)",
|
||||
border: "1px solid rgba(127, 208, 255, 0.2)",
|
||||
borderRadius: "12px",
|
||||
padding: "24px",
|
||||
minWidth: "480px",
|
||||
maxWidth: "640px",
|
||||
maxHeight: "80vh",
|
||||
overflow: "auto",
|
||||
backdropFilter: "blur(18px)",
|
||||
};
|
||||
|
||||
const buttonStyle: React.CSSProperties = {
|
||||
display: "inline-flex",
|
||||
alignItems: "center",
|
||||
gap: "6px",
|
||||
padding: "8px 14px",
|
||||
borderRadius: "6px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.2)",
|
||||
background: "rgba(74, 163, 255, 0.1)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "12px",
|
||||
fontWeight: "600",
|
||||
cursor: "pointer",
|
||||
transition: "160ms ease",
|
||||
};
|
||||
|
||||
const inputStyle: React.CSSProperties = {
|
||||
width: "100%",
|
||||
padding: "10px 12px",
|
||||
borderRadius: "6px",
|
||||
border: "1px solid rgba(127, 208, 255, 0.2)",
|
||||
background: "rgba(3, 9, 18, 0.8)",
|
||||
color: "#ebf3ff",
|
||||
fontSize: "13px",
|
||||
marginBottom: "12px",
|
||||
};
|
||||
|
||||
return (
|
||||
<div style={containerStyle}>
|
||||
<div style={headerStyle}>
|
||||
<h1 style={titleStyle}>Versions & Change Proposals</h1>
|
||||
<input
|
||||
type="text"
|
||||
placeholder="Ontology URI"
|
||||
value={ontologyUri}
|
||||
onChange={(e) => setOntologyUri(e.target.value)}
|
||||
style={{ ...inputStyle, width: "300px", marginBottom: 0 }}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div style={sectionStyle}>
|
||||
<h2 style={sectionTitleStyle}>
|
||||
<Layers size={16} />
|
||||
Version History
|
||||
</h2>
|
||||
<div style={listStyle}>
|
||||
{versions.length === 0 ? (
|
||||
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No versions found</div>
|
||||
) : (
|
||||
versions.map((version) => (
|
||||
<div key={version.version_id} style={itemStyle}>
|
||||
{getStateIcon(version.state)}
|
||||
<div style={{ flex: 1 }}>
|
||||
<div style={{ color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
|
||||
{version.version_id}
|
||||
</div>
|
||||
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>
|
||||
{version.author} • {new Date(version.date).toLocaleDateString()}
|
||||
</div>
|
||||
</div>
|
||||
<button
|
||||
style={buttonStyle}
|
||||
onClick={() => {
|
||||
setComparePair({ v1: version.version_id, v2: versions[0]?.version_id || "" });
|
||||
setShowCompareModal(true);
|
||||
}}
|
||||
>
|
||||
<ArrowRight size={12} />
|
||||
Compare
|
||||
</button>
|
||||
</div>
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style={sectionStyle}>
|
||||
<h2 style={sectionTitleStyle}>
|
||||
<GitMerge size={16} />
|
||||
Change Proposals
|
||||
</h2>
|
||||
<div style={listStyle}>
|
||||
{proposals.length === 0 ? (
|
||||
<div style={{ color: "#8fa8c6", fontSize: "13px" }}>No proposals found</div>
|
||||
) : (
|
||||
proposals.map((proposal) => (
|
||||
<div key={proposal.proposal_id} style={itemStyle}>
|
||||
{getStateIcon(proposal.state)}
|
||||
<div style={{ flex: 1 }}>
|
||||
<div style={{ color: "#ebf3ff", fontSize: "13px", fontWeight: "600" }}>
|
||||
{proposal.summary}
|
||||
</div>
|
||||
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>
|
||||
{proposal.author} • {new Date(proposal.created_at).toLocaleDateString()}
|
||||
</div>
|
||||
</div>
|
||||
{proposal.state === "proposed" && (
|
||||
<div style={{ display: "flex", gap: "6px" }}>
|
||||
<button style={buttonStyle} onClick={() => approveProposal(proposal.proposal_id)}>
|
||||
<CheckCircle size={12} />
|
||||
Approve
|
||||
</button>
|
||||
<button style={buttonStyle} onClick={() => rejectProposal(proposal.proposal_id)}>
|
||||
<XCircle size={12} />
|
||||
Reject
|
||||
</button>
|
||||
</div>
|
||||
)}
|
||||
{proposal.state === "approved" && (
|
||||
<button style={buttonStyle} onClick={() => publishProposal(proposal.proposal_id)}>
|
||||
<Send size={12} />
|
||||
Publish
|
||||
</button>
|
||||
)}
|
||||
<button
|
||||
style={buttonStyle}
|
||||
onClick={() => {
|
||||
setSelectedProposal(proposal);
|
||||
setShowProposalModal(true);
|
||||
}}
|
||||
>
|
||||
<FileText size={12} />
|
||||
Details
|
||||
</button>
|
||||
</div>
|
||||
))
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{showProposalModal && selectedProposal && (
|
||||
<div style={modalOverlayStyle} onClick={() => setShowProposalModal(false)}>
|
||||
<div style={modalStyle} onClick={(e) => e.stopPropagation()}>
|
||||
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: "16px" }}>
|
||||
<h3 style={{ margin: 0, color: "#ebf3ff", fontSize: "16px" }}>Proposal Details</h3>
|
||||
<button onClick={() => setShowProposalModal(false)} style={{ background: "none", border: "none", color: "#8fa8c6", cursor: "pointer" }}>
|
||||
<X size={18} />
|
||||
</button>
|
||||
</div>
|
||||
<div style={{ marginBottom: "12px" }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
Summary
|
||||
</label>
|
||||
<div style={{ color: "#ebf3ff", fontSize: "13px" }}>{selectedProposal.summary}</div>
|
||||
</div>
|
||||
<div style={{ marginBottom: "12px" }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
State
|
||||
</label>
|
||||
<div style={{ display: "flex", alignItems: "center", gap: "6px", color: "#ebf3ff", fontSize: "13px" }}>
|
||||
{getStateIcon(selectedProposal.state)}
|
||||
{selectedProposal.state}
|
||||
</div>
|
||||
</div>
|
||||
<div style={{ marginBottom: "12px" }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
Impact Analysis
|
||||
</label>
|
||||
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
|
||||
{JSON.stringify(selectedProposal.impact_analysis, null, 2)}
|
||||
</pre>
|
||||
</div>
|
||||
<div style={{ marginBottom: "12px" }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
SHACL Validation
|
||||
</label>
|
||||
<pre style={{ background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
|
||||
{JSON.stringify(selectedProposal.shacl_validation, null, 2)}
|
||||
</pre>
|
||||
</div>
|
||||
<div>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
Comments ({selectedProposal.comments.length})
|
||||
</label>
|
||||
<div style={{ maxHeight: "120px", overflow: "auto" }}>
|
||||
{selectedProposal.comments.map((comment) => (
|
||||
<div key={comment.id} style={{ padding: "8px", background: "rgba(3, 9, 18, 0.6)", borderRadius: "4px", marginBottom: "6px" }}>
|
||||
<div style={{ color: "#ebf3ff", fontSize: "12px", fontWeight: "600" }}>{comment.author}</div>
|
||||
<div style={{ color: "#8fa8c6", fontSize: "11px" }}>{comment.text}</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{showCompareModal && comparePair && (
|
||||
<div style={modalOverlayStyle} onClick={() => setShowCompareModal(false)}>
|
||||
<div style={modalStyle} onClick={(e) => e.stopPropagation()}>
|
||||
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: "16px" }}>
|
||||
<h3 style={{ margin: 0, color: "#ebf3ff", fontSize: "16px" }}>Compare Versions</h3>
|
||||
<button onClick={() => setShowCompareModal(false)} style={{ background: "none", border: "none", color: "#8fa8c6", cursor: "pointer" }}>
|
||||
<X size={18} />
|
||||
</button>
|
||||
</div>
|
||||
<div style={{ display: "flex", gap: "12px", marginBottom: "16px" }}>
|
||||
<div style={{ flex: 1 }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
Version 1
|
||||
</label>
|
||||
<input
|
||||
type="text"
|
||||
value={comparePair.v1}
|
||||
onChange={(e) => setComparePair({ ...comparePair, v1: e.target.value })}
|
||||
style={inputStyle}
|
||||
/>
|
||||
</div>
|
||||
<div style={{ flex: 1 }}>
|
||||
<label style={{ display: "block", color: "#8fa8c6", fontSize: "12px", marginBottom: "4px" }}>
|
||||
Version 2
|
||||
</label>
|
||||
<input
|
||||
type="text"
|
||||
value={comparePair.v2}
|
||||
onChange={(e) => setComparePair({ ...comparePair, v2: e.target.value })}
|
||||
style={inputStyle}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<button style={buttonStyle} onClick={runVersionComparison} disabled={isLoading}>
|
||||
<Scale size={12} />
|
||||
{isLoading ? "Comparing..." : "Compare"}
|
||||
</button>
|
||||
{compareResult && (
|
||||
<pre style={{ marginTop: "16px", background: "rgba(3, 9, 18, 0.8)", padding: "12px", borderRadius: "6px", color: "#ebf3ff", fontSize: "12px", overflow: "auto" }}>
|
||||
{JSON.stringify(compareResult, null, 2)}
|
||||
</pre>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
import type {
|
||||
AlignmentRelation,
|
||||
AlignmentSuggestion,
|
||||
OntologyAlignment,
|
||||
OntologyEntry,
|
||||
OntologyHealthResponse,
|
||||
ShaclGenerateResponse,
|
||||
ShaclShapesResponse,
|
||||
ShaclValidationResponse,
|
||||
} from "./types";
|
||||
|
||||
async function parseResponse<T>(response: Response): Promise<T> {
|
||||
if (!response.ok) {
|
||||
let detail = `Request failed with status ${response.status}`;
|
||||
try {
|
||||
const body = await response.json();
|
||||
detail = body.detail || detail;
|
||||
} catch {
|
||||
// Keep the generic HTTP detail.
|
||||
}
|
||||
throw new Error(detail);
|
||||
}
|
||||
return response.json() as Promise<T>;
|
||||
}
|
||||
|
||||
export async function loadOntologyRegistry(): Promise<OntologyEntry[]> {
|
||||
return parseResponse<OntologyEntry[]>(await fetch("/api/ontology/registry"));
|
||||
}
|
||||
|
||||
export async function loadAlignments(uri?: string): Promise<OntologyAlignment[]> {
|
||||
const query = uri ? `?uri=${encodeURIComponent(uri)}` : "";
|
||||
return parseResponse<OntologyAlignment[]>(await fetch(`/api/ontology/alignments${query}`));
|
||||
}
|
||||
|
||||
export async function saveAlignment(payload: {
|
||||
source_uri: string;
|
||||
target_uri: string;
|
||||
relation: AlignmentRelation;
|
||||
confidence: number;
|
||||
provenance?: string;
|
||||
source?: string;
|
||||
reviewer?: string;
|
||||
}): Promise<OntologyAlignment> {
|
||||
return parseResponse<OntologyAlignment>(
|
||||
await fetch("/api/ontology/alignments", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(payload),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
export async function removeAlignment(id: string): Promise<void> {
|
||||
await parseResponse<{ status: string }>(
|
||||
await fetch(`/api/ontology/alignments?id=${encodeURIComponent(id)}`, { method: "DELETE" }),
|
||||
);
|
||||
}
|
||||
|
||||
export async function suggestAlignments(payload: {
|
||||
source_ontology_uri?: string;
|
||||
target_ontology_uri?: string;
|
||||
threshold: number;
|
||||
limit: number;
|
||||
}): Promise<AlignmentSuggestion[]> {
|
||||
return parseResponse<AlignmentSuggestion[]>(
|
||||
await fetch("/api/ontology/suggest-alignments", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(payload),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
export async function loadOntologyHealth(uri: string): Promise<OntologyHealthResponse> {
|
||||
return parseResponse<OntologyHealthResponse>(
|
||||
await fetch(`/api/ontology/health?uri=${encodeURIComponent(uri)}`),
|
||||
);
|
||||
}
|
||||
|
||||
export async function generateShacl(uri: string, qualityTier: "standard" | "strict" = "strict"): Promise<ShaclGenerateResponse> {
|
||||
return parseResponse<ShaclGenerateResponse>(
|
||||
await fetch("/api/ontology/shacl/generate", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ uri, quality_tier: qualityTier }),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
export async function loadShaclShapes(uri: string): Promise<ShaclShapesResponse> {
|
||||
return parseResponse<ShaclShapesResponse>(
|
||||
await fetch(`/api/ontology/shacl/shapes?uri=${encodeURIComponent(uri)}`),
|
||||
);
|
||||
}
|
||||
|
||||
export async function validateShacl(uri: string, shaclTurtle: string): Promise<ShaclValidationResponse> {
|
||||
return parseResponse<ShaclValidationResponse>(
|
||||
await fetch("/api/ontology/shacl/validate", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ uri, shacl_turtle: shaclTurtle }),
|
||||
}),
|
||||
);
|
||||
}
|
||||
@@ -7,7 +7,12 @@ import {
|
||||
Shield,
|
||||
Sliders,
|
||||
} from "lucide-react";
|
||||
import { AlignmentsTab } from "./AlignmentsTab";
|
||||
import { HealthTab } from "./HealthTab";
|
||||
import { OntologyManager } from "./OntologyManager";
|
||||
import { OntologyEditor } from "./OntologyEditor";
|
||||
import { ShaclStudio } from "./ShaclStudio";
|
||||
import { VersionsTab } from "./VersionsTab";
|
||||
|
||||
export type OntologyHubTab =
|
||||
| "registry"
|
||||
@@ -76,7 +81,11 @@ function ComingSoonStub({
|
||||
);
|
||||
}
|
||||
|
||||
export function OntologyWorkspace() {
|
||||
interface OntologyWorkspaceProps {
|
||||
onJumpToGraphNode?: (nodeId: string) => void;
|
||||
}
|
||||
|
||||
export function OntologyWorkspace({ onJumpToGraphNode }: OntologyWorkspaceProps) {
|
||||
const [activeTab, setActiveTab] = useState<OntologyHubTab>(readTabParam);
|
||||
|
||||
useEffect(() => {
|
||||
@@ -87,55 +96,28 @@ export function OntologyWorkspace() {
|
||||
setActiveTab(tab);
|
||||
}, []);
|
||||
|
||||
const handleFixInEditor = useCallback((entityUri: string) => {
|
||||
const params = new URLSearchParams(window.location.search);
|
||||
params.set(TAB_PARAM, "editor");
|
||||
params.set("ontologyEntity", entityUri);
|
||||
window.history.replaceState(null, "", `?${params.toString()}`);
|
||||
setActiveTab("editor");
|
||||
}, []);
|
||||
|
||||
const renderTab = () => {
|
||||
switch (activeTab) {
|
||||
case "registry":
|
||||
return <OntologyManager />;
|
||||
case "editor":
|
||||
return (
|
||||
<ComingSoonStub
|
||||
icon={Sliders}
|
||||
title="Visual Ontology Editor"
|
||||
description="Visually edit classes, properties, individuals, restrictions, axioms, and SKOS metadata. Create and propose schema changes through a governed draft workflow."
|
||||
badge="Subissue 2"
|
||||
/>
|
||||
);
|
||||
return <OntologyEditor />;
|
||||
case "versions":
|
||||
return (
|
||||
<ComingSoonStub
|
||||
icon={Layers}
|
||||
title="Versions & Change Proposals"
|
||||
description="View version history, compare schema diffs, submit change proposals, and manage the review-to-publish lifecycle."
|
||||
badge="Subissue 2"
|
||||
/>
|
||||
);
|
||||
return <VersionsTab />;
|
||||
case "alignments":
|
||||
return (
|
||||
<ComingSoonStub
|
||||
icon={GitMerge}
|
||||
title="Cross-Ontology Alignments"
|
||||
description="Manage mappings between ontologies, review suggested alignments from embedding-assisted similarity, and publish alignment sets."
|
||||
badge="Subissue 3"
|
||||
/>
|
||||
);
|
||||
return <AlignmentsTab />;
|
||||
case "health":
|
||||
return (
|
||||
<ComingSoonStub
|
||||
icon={HeartPulse}
|
||||
title="Ontology Health Dashboard"
|
||||
description="Score completeness, consistency, SHACL conformance, alignment coverage, and documentation quality across all loaded ontologies."
|
||||
badge="Subissue 3"
|
||||
/>
|
||||
);
|
||||
return <HealthTab onFixInEditor={handleFixInEditor} />;
|
||||
case "shacl":
|
||||
return (
|
||||
<ComingSoonStub
|
||||
icon={Shield}
|
||||
title="SHACL Studio"
|
||||
description="Generate, edit, and validate SHACL shapes. Preview constraint violations against the active graph before publishing."
|
||||
badge="Subissue 3"
|
||||
/>
|
||||
);
|
||||
return <ShaclStudio onJumpToNode={onJumpToGraphNode} />;
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
export interface OntologyEntry {
|
||||
uri: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
format: string;
|
||||
status: "published" | "draft" | "external";
|
||||
source_url?: string;
|
||||
version?: string;
|
||||
class_count: number;
|
||||
concept_count: number;
|
||||
property_count: number;
|
||||
loaded_at: string;
|
||||
enabled: boolean;
|
||||
tags: string[];
|
||||
}
|
||||
|
||||
export type AlignmentRelation =
|
||||
| "owl:equivalentClass"
|
||||
| "owl:equivalentProperty"
|
||||
| "skos:exactMatch"
|
||||
| "skos:closeMatch"
|
||||
| "skos:broadMatch"
|
||||
| "skos:narrowMatch"
|
||||
| "skos:relatedMatch";
|
||||
|
||||
export interface OntologyAlignment {
|
||||
id: string;
|
||||
source_uri: string;
|
||||
source_label: string;
|
||||
target_uri: string;
|
||||
target_label: string;
|
||||
relation: AlignmentRelation;
|
||||
predicate_uri: string;
|
||||
confidence: number;
|
||||
provenance?: string;
|
||||
source?: string;
|
||||
reviewer?: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface AlignmentSuggestion {
|
||||
source_uri: string;
|
||||
source_label: string;
|
||||
target_uri: string;
|
||||
target_label: string;
|
||||
relation: AlignmentRelation;
|
||||
score: number;
|
||||
label_similarity: number;
|
||||
embedding_similarity?: number | null;
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export interface HealthDimension {
|
||||
key: string;
|
||||
label: string;
|
||||
score: number;
|
||||
status: "ok" | "warning" | "critical" | "unavailable";
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface HealthIssue {
|
||||
id: string;
|
||||
severity: "info" | "warning" | "critical";
|
||||
category: string;
|
||||
entity_uri?: string;
|
||||
entity_label?: string;
|
||||
message: string;
|
||||
action?: string;
|
||||
}
|
||||
|
||||
export interface OntologyHealthResponse {
|
||||
uri: string;
|
||||
name: string;
|
||||
total_score: number;
|
||||
dimensions: HealthDimension[];
|
||||
issues: HealthIssue[];
|
||||
generated_at: string;
|
||||
}
|
||||
|
||||
export interface ShaclShapeSummary {
|
||||
id: string;
|
||||
target_class?: string;
|
||||
constraint_count: number;
|
||||
constraints: string[];
|
||||
violation_count: number;
|
||||
}
|
||||
|
||||
export interface ShaclViolation {
|
||||
node?: string;
|
||||
path?: string;
|
||||
severity: string;
|
||||
message: string;
|
||||
focus_node?: string;
|
||||
source_shape?: string;
|
||||
}
|
||||
|
||||
export interface ShaclGenerateResponse {
|
||||
uri: string;
|
||||
shacl_turtle: string;
|
||||
shape_count: number;
|
||||
generated_at: string;
|
||||
}
|
||||
|
||||
export interface ShaclShapesResponse {
|
||||
uri: string;
|
||||
shapes: ShaclShapeSummary[];
|
||||
shacl_turtle: string;
|
||||
generated_at: string;
|
||||
}
|
||||
|
||||
export interface ShaclValidationResponse {
|
||||
uri?: string;
|
||||
conforms: boolean;
|
||||
status: "success" | "unavailable" | "error";
|
||||
message: string;
|
||||
violations: ShaclViolation[];
|
||||
report_text?: string;
|
||||
}
|
||||
+6
-4
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "semantica"
|
||||
version = "0.4.0"
|
||||
version = "0.5.0"
|
||||
description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering"
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
@@ -104,6 +104,7 @@ parse-docling = ["docling>=1.0.0"]
|
||||
# ---- Database Connectors ----
|
||||
db-snowflake = ["snowflake-connector-python>=3.0.0", "cryptography>=3.4.0"]
|
||||
db-arrow = ["pyarrow>=10.0.0"]
|
||||
ingest-parquet = ["pyarrow>=10.0.0"]
|
||||
|
||||
db-all = [
|
||||
"semantica[db-snowflake,db-arrow]"
|
||||
@@ -211,10 +212,10 @@ explorer-lite = [
|
||||
"streamlit-agraph>=0.0.45"
|
||||
]
|
||||
|
||||
# Everything
|
||||
# Everything (cross-platform — gpu excluded; install semantica[gpu] separately on Linux)
|
||||
all = [
|
||||
"semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,explorer]",
|
||||
"semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,agno]"
|
||||
"semantica[dev,viz,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,explorer]",
|
||||
"semantica[dev,viz,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling,ingest-parquet,agno]"
|
||||
]
|
||||
|
||||
# ---------------- ENTRYPOINTS ----------------
|
||||
@@ -223,6 +224,7 @@ semantica = "semantica.cli:main"
|
||||
semantica-server = "semantica.server:main"
|
||||
semantica-worker = "semantica.worker:main"
|
||||
semantica-explorer = "semantica.explorer:main"
|
||||
semantica-mcp = "semantica.mcp_server:main"
|
||||
|
||||
# ---------------- TOOLING ----------------
|
||||
[tool.setuptools.packages.find]
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
mkdocs>=1.5.0
|
||||
mkdocs>=1.6.1
|
||||
mkdocs-material>=9.7.6
|
||||
mkdocs-minify-plugin>=0.7.0
|
||||
mkdocs-mermaid2-plugin>=1.0.0
|
||||
mkdocs-mermaid2-plugin>=1.2.3
|
||||
pymdown-extensions>=10.21.2
|
||||
|
||||
mkdocstrings[python]>=0.24.0
|
||||
mkdocs-jupyter>=0.24.0
|
||||
mkdocs-jupyter>=0.26.3
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ Main exports:
|
||||
- Config: Configuration management
|
||||
"""
|
||||
|
||||
__version__ = "0.4.0"
|
||||
__version__ = "0.5.0"
|
||||
__author__ = "Semantica Contributors"
|
||||
__license__ = "MIT"
|
||||
|
||||
|
||||
@@ -133,48 +133,6 @@ class ConflictDetector:
|
||||
|
||||
self.detected_conflicts: Dict[str, Conflict] = {}
|
||||
|
||||
def detect_conflicts(
|
||||
self,
|
||||
entities: Union[List[Dict[str, Any]], Dict[str, Any]],
|
||||
method: str = "entity",
|
||||
property_name: Optional[str] = None,
|
||||
entity_type: Optional[str] = None,
|
||||
**kwargs,
|
||||
) -> List[Conflict]:
|
||||
"""
|
||||
Detect conflicts using the specified method (convenience method).
|
||||
|
||||
Args:
|
||||
entities: Entities to check (List of dicts or a KG dict)
|
||||
method: Detection method ("entity", "value", "type", "relationship", "temporal", "logical")
|
||||
property_name: Property name for "value" method
|
||||
entity_type: Optional entity type filter
|
||||
**kwargs: Additional arguments
|
||||
|
||||
Returns:
|
||||
List of detected conflicts
|
||||
"""
|
||||
# If passed a KG dict, extract entities
|
||||
if isinstance(entities, dict) and "entities" in entities:
|
||||
entities = entities["entities"]
|
||||
|
||||
if method == "value":
|
||||
if not property_name:
|
||||
raise ValueError("property_name is required for value conflict detection")
|
||||
return self.detect_value_conflicts(entities, property_name, entity_type)
|
||||
elif method == "type":
|
||||
return self.detect_type_conflicts(entities)
|
||||
elif method == "relationship":
|
||||
relationships = kwargs.get("relationships", [])
|
||||
return self.detect_relationship_conflicts(relationships)
|
||||
elif method == "temporal":
|
||||
return self.detect_temporal_conflicts(entities)
|
||||
elif method == "logical":
|
||||
return self.detect_logical_conflicts(entities)
|
||||
else:
|
||||
# Default to entity-wide detection
|
||||
return self.detect_entity_conflicts(entities, entity_type)
|
||||
|
||||
def detect_value_conflicts(
|
||||
self,
|
||||
entities: Union[List[Dict[str, Any]], Dict[str, Any]],
|
||||
@@ -596,11 +554,6 @@ class ConflictDetector:
|
||||
tracking_id, status="failed", message=str(e)
|
||||
)
|
||||
raise
|
||||
for field_name in fields_to_check:
|
||||
conflicts = self.detect_value_conflicts(entities, field_name, entity_type)
|
||||
all_conflicts.extend(conflicts)
|
||||
|
||||
return all_conflicts
|
||||
|
||||
def _calculate_conflict_confidence(
|
||||
self, values: List[Any], sources: List[Dict[str, Any]]
|
||||
@@ -1252,20 +1205,25 @@ class ConflictDetector:
|
||||
def detect_conflicts(
|
||||
self,
|
||||
entities: Union[List[Dict[str, Any]], Dict[str, Any]],
|
||||
method: str = "all",
|
||||
property_name: Optional[str] = None,
|
||||
entity_type: Optional[str] = None,
|
||||
**kwargs,
|
||||
) -> List[Conflict]:
|
||||
"""
|
||||
Detect all conflicts for entities (general method).
|
||||
|
||||
This method detects all types of conflicts: value, type, relationship,
|
||||
temporal, and logical conflicts.
|
||||
Detect conflicts using the specified method.
|
||||
|
||||
Args:
|
||||
entities: List of entity dictionaries or Graph dictionary (containing "entities" key)
|
||||
entities: List of entity dictionaries or Graph dictionary
|
||||
method: Detection method — "all" (default), "value", "property", "type",
|
||||
"relationship", "temporal", "logical", or "entity"
|
||||
property_name: Property name required for ``method="value"`` and ``method="property"``
|
||||
entity_type: Optional entity type filter
|
||||
**kwargs: Extra arguments forwarded to the underlying method
|
||||
(e.g. ``relationships=`` for ``method="relationship"``)
|
||||
|
||||
Returns:
|
||||
List of all detected conflicts
|
||||
List of detected conflicts
|
||||
"""
|
||||
# Handle graph dictionary input
|
||||
if isinstance(entities, dict):
|
||||
@@ -1275,6 +1233,34 @@ class ConflictDetector:
|
||||
# If it's a single entity dict, wrap in list
|
||||
entities = [entities]
|
||||
|
||||
# Dispatch to a specific sub-method when one is requested
|
||||
if method == "value":
|
||||
if not property_name:
|
||||
raise ValueError("property_name is required for method='value'")
|
||||
return self.detect_value_conflicts(entities, property_name, entity_type)
|
||||
elif method == "property":
|
||||
if not property_name:
|
||||
raise ValueError("property_name is required for method='property'")
|
||||
return self.detect_property_conflicts(entities, property_name)
|
||||
elif method == "type":
|
||||
return self.detect_type_conflicts(entities)
|
||||
elif method == "relationship":
|
||||
relationships = kwargs.get("relationships", [])
|
||||
if isinstance(relationships, dict):
|
||||
inner = relationships.get("relationships", relationships)
|
||||
relationships = inner if isinstance(inner, list) else [inner]
|
||||
if not isinstance(relationships, list):
|
||||
relationships = [relationships]
|
||||
return self.detect_relationship_conflicts(relationships)
|
||||
elif method == "temporal":
|
||||
return self.detect_temporal_conflicts(entities)
|
||||
elif method == "logical":
|
||||
return self.detect_logical_conflicts(entities)
|
||||
elif method == "entity":
|
||||
return self.detect_entity_conflicts(entities, entity_type)
|
||||
elif method != "all":
|
||||
raise ValueError(f"Unknown conflict detection method: {method!r}")
|
||||
|
||||
tracking_id = self.progress_tracker.start_tracking(
|
||||
file=None,
|
||||
module="conflicts",
|
||||
|
||||
@@ -100,6 +100,10 @@ class DuplicateDetector:
|
||||
confidence_threshold: float = 0.6,
|
||||
use_clustering: bool = True,
|
||||
config: Optional[Dict[str, Any]] = None,
|
||||
max_results: Optional[int] = None,
|
||||
top_k_per_entity: Optional[int] = None,
|
||||
min_similarity: Optional[float] = None,
|
||||
sort_by: str = "confidence",
|
||||
**kwargs,
|
||||
):
|
||||
"""
|
||||
@@ -115,6 +119,16 @@ class DuplicateDetector:
|
||||
(0.0 to 1.0, default: 0.6)
|
||||
use_clustering: Whether to use clustering for group formation (default: True)
|
||||
config: Configuration dictionary (merged with kwargs)
|
||||
max_results: Hard cap on total candidates returned across all entities.
|
||||
Applied after sorting. ``None`` means no limit.
|
||||
top_k_per_entity: Keep at most this many candidates per entity (by the
|
||||
sort field). ``None`` means no per-entity limit.
|
||||
min_similarity: Additional similarity floor applied on top of
|
||||
``similarity_threshold``. Candidates whose
|
||||
``similarity_score`` is below this value are dropped
|
||||
before ranking. ``None`` means no extra floor.
|
||||
sort_by: Field used for ranking before limits are applied.
|
||||
``"confidence"`` (default) or ``"similarity_score"``.
|
||||
**kwargs: Additional configuration options:
|
||||
- similarity: Configuration for SimilarityCalculator
|
||||
"""
|
||||
@@ -133,6 +147,30 @@ class DuplicateDetector:
|
||||
self.confidence_threshold = confidence_threshold
|
||||
self.use_clustering = use_clustering
|
||||
|
||||
# Result limiting / ranking options — validate at construction time
|
||||
if max_results is not None and (not isinstance(max_results, int) or max_results < 0):
|
||||
raise ValueError(
|
||||
f"max_results must be None or a non-negative int, got {max_results!r}"
|
||||
)
|
||||
if top_k_per_entity is not None and (
|
||||
not isinstance(top_k_per_entity, int) or top_k_per_entity < 0
|
||||
):
|
||||
raise ValueError(
|
||||
f"top_k_per_entity must be None or a non-negative int, got {top_k_per_entity!r}"
|
||||
)
|
||||
if min_similarity is not None and not (0.0 <= min_similarity <= 1.0):
|
||||
raise ValueError(
|
||||
f"min_similarity must be None or a float in [0.0, 1.0], got {min_similarity!r}"
|
||||
)
|
||||
if sort_by not in ("confidence", "similarity_score"):
|
||||
raise ValueError(
|
||||
f"sort_by must be 'confidence' or 'similarity_score', got {sort_by!r}"
|
||||
)
|
||||
self.max_results = max_results
|
||||
self.top_k_per_entity = top_k_per_entity
|
||||
self.min_similarity = min_similarity
|
||||
self.sort_by = sort_by
|
||||
|
||||
# Initialize progress tracker and ensure it's enabled
|
||||
self.progress_tracker = get_progress_tracker()
|
||||
if not self.progress_tracker.enabled:
|
||||
@@ -140,7 +178,9 @@ class DuplicateDetector:
|
||||
|
||||
self.logger.debug(
|
||||
f"Duplicate detector initialized: similarity_threshold={similarity_threshold}, "
|
||||
f"confidence_threshold={confidence_threshold}"
|
||||
f"confidence_threshold={confidence_threshold}, max_results={max_results}, "
|
||||
f"top_k_per_entity={top_k_per_entity}, min_similarity={min_similarity}, "
|
||||
f"sort_by={sort_by!r}"
|
||||
)
|
||||
|
||||
def detect_duplicates(
|
||||
@@ -153,8 +193,10 @@ class DuplicateDetector:
|
||||
Detect duplicate entities from a list.
|
||||
|
||||
This method compares all pairs of entities and identifies duplicates based
|
||||
on similarity scores and confidence thresholds. Returns candidates sorted
|
||||
by confidence (highest first).
|
||||
on similarity scores and confidence thresholds. Results are filtered and
|
||||
ranked by ``_apply_result_limits`` using the instance ``sort_by`` field
|
||||
(default: ``"confidence"``), then capped by ``top_k_per_entity`` and
|
||||
``max_results``.
|
||||
|
||||
Args:
|
||||
entities: List of entity dictionaries to check for duplicates.
|
||||
@@ -163,8 +205,9 @@ class DuplicateDetector:
|
||||
**options: Additional detection options passed to similarity calculator
|
||||
|
||||
Returns:
|
||||
List of DuplicateCandidate objects, sorted by confidence (highest first).
|
||||
Each candidate contains:
|
||||
List of DuplicateCandidate objects sorted by the ``sort_by`` field
|
||||
(highest first, default ``"confidence"``), capped by ``top_k_per_entity``
|
||||
and ``max_results``. Each candidate contains:
|
||||
- entity1, entity2: The duplicate entity pair
|
||||
- similarity_score: Similarity score (0.0 to 1.0)
|
||||
- confidence: Confidence score (0.0 to 1.0)
|
||||
@@ -255,8 +298,8 @@ class DuplicateDetector:
|
||||
message=f"Creating duplicate candidates... {i + 1}/{total_similarities} (remaining: {remaining})",
|
||||
)
|
||||
|
||||
# Sort by confidence (highest first)
|
||||
candidates.sort(key=lambda c: c.confidence, reverse=True)
|
||||
# Sort, filter, and cap results
|
||||
candidates = self._apply_result_limits(candidates)
|
||||
|
||||
self.logger.info(
|
||||
f"Detected {len(candidates)} duplicate candidate(s) "
|
||||
@@ -521,7 +564,9 @@ class DuplicateDetector:
|
||||
|
||||
Returns:
|
||||
List of DuplicateCandidate objects representing duplicates between
|
||||
new and existing entities, sorted by confidence (highest first).
|
||||
new and existing entities, sorted by the ``sort_by`` field (highest
|
||||
first, default ``"confidence"``), capped by ``top_k_per_entity`` and
|
||||
``max_results``.
|
||||
|
||||
Example:
|
||||
>>> new_entities = [{"id": "3", "name": "Apple Corp"}]
|
||||
@@ -600,8 +645,8 @@ class DuplicateDetector:
|
||||
message=f"Comparing entities... {processed}/{total_comparisons} (remaining: {remaining})",
|
||||
)
|
||||
|
||||
# Sort by confidence (highest first)
|
||||
candidates.sort(key=lambda c: c.confidence, reverse=True)
|
||||
# Sort, filter, and cap results
|
||||
candidates = self._apply_result_limits(candidates)
|
||||
|
||||
self.logger.info(
|
||||
f"Incremental detection found {len(candidates)} duplicate candidate(s)"
|
||||
@@ -620,6 +665,57 @@ class DuplicateDetector:
|
||||
)
|
||||
raise
|
||||
|
||||
def _apply_result_limits(
|
||||
self, candidates: List[DuplicateCandidate]
|
||||
) -> List[DuplicateCandidate]:
|
||||
"""
|
||||
Apply min_similarity filter, sort, top_k_per_entity, and max_results cap.
|
||||
|
||||
Order of operations:
|
||||
1. Drop candidates below ``min_similarity`` (if set).
|
||||
2. Sort by ``sort_by`` field descending.
|
||||
3. Apply ``top_k_per_entity``: for each entity id keep only the top-k
|
||||
candidates in which it appears.
|
||||
4. Apply ``max_results`` global cap.
|
||||
"""
|
||||
# 1. min_similarity filter
|
||||
if self.min_similarity is not None:
|
||||
candidates = [
|
||||
c for c in candidates if c.similarity_score >= self.min_similarity
|
||||
]
|
||||
|
||||
# 2. Sort descending by the chosen field
|
||||
candidates.sort(key=lambda c: getattr(c, self.sort_by), reverse=True)
|
||||
|
||||
# 3. top_k_per_entity — keep a candidate if *either* entity is still under
|
||||
# quota; once an entity reaches k it no longer sponsors new candidates.
|
||||
if self.top_k_per_entity is not None:
|
||||
entity_counts: Dict[str, int] = {}
|
||||
kept: List[DuplicateCandidate] = []
|
||||
for c in candidates:
|
||||
nid1 = self._normalize_entity_id(c.entity1)
|
||||
nid2 = self._normalize_entity_id(c.entity2)
|
||||
count1 = entity_counts.get(nid1, 0)
|
||||
count2 = entity_counts.get(nid2, 0)
|
||||
if count1 < self.top_k_per_entity or count2 < self.top_k_per_entity:
|
||||
kept.append(c)
|
||||
entity_counts[nid1] = count1 + 1
|
||||
entity_counts[nid2] = count2 + 1
|
||||
candidates = kept
|
||||
|
||||
# 4. max_results global cap
|
||||
if self.max_results is not None:
|
||||
candidates = candidates[: self.max_results]
|
||||
|
||||
return candidates
|
||||
|
||||
def _normalize_entity_id(self, entity: Any) -> str:
|
||||
"""Return a stable string key for an entity, used as a dict key throughout the class."""
|
||||
raw = self._get_entity_value(entity, "id")
|
||||
if raw is None:
|
||||
raw = id(entity)
|
||||
return str(raw)
|
||||
|
||||
def _get_entity_value(self, entity: Any, key: str, default: Any = None) -> Any:
|
||||
"""Get value from entity dictionary or object safely."""
|
||||
if hasattr(entity, "__dict__"):
|
||||
@@ -715,12 +811,8 @@ class DuplicateDetector:
|
||||
groups = []
|
||||
|
||||
for candidate in candidates:
|
||||
entity1_id = self._get_entity_value(candidate.entity1, "id") or id(
|
||||
candidate.entity1
|
||||
)
|
||||
entity2_id = self._get_entity_value(candidate.entity2, "id") or id(
|
||||
candidate.entity2
|
||||
)
|
||||
entity1_id = self._normalize_entity_id(candidate.entity1)
|
||||
entity2_id = self._normalize_entity_id(candidate.entity2)
|
||||
|
||||
group1 = entity_to_group.get(entity1_id)
|
||||
group2 = entity_to_group.get(entity2_id)
|
||||
@@ -764,7 +856,7 @@ class DuplicateDetector:
|
||||
|
||||
# Update references
|
||||
for entity in group2.entities:
|
||||
entity_id = self._get_entity_value(entity, "id") or id(entity)
|
||||
entity_id = self._normalize_entity_id(entity)
|
||||
entity_to_group[entity_id] = group1
|
||||
|
||||
if group2 in groups:
|
||||
|
||||
@@ -102,10 +102,10 @@ def _coerce_embedding_vector(value: object) -> Optional[List[float]]:
|
||||
|
||||
|
||||
def _extract_node_embeddings(graph_dict: dict) -> dict[str, List[float]]:
|
||||
"""Extract embeddings from graph dictionary."""
|
||||
# Top-level keys to probe on each entity (and its metadata/properties dicts).
|
||||
# Priority: generic names first, then KG-extras-specific names.
|
||||
# Must stay aligned with the inner probe list in _coerce_embedding_vector.
|
||||
# TODO: cache this per-session graph revision to avoid re-scanning all nodes on every request.
|
||||
embedding_keys = (
|
||||
"embedding",
|
||||
"embeddings",
|
||||
@@ -138,6 +138,11 @@ def _extract_node_embeddings(graph_dict: dict) -> dict[str, List[float]]:
|
||||
return embeddings
|
||||
|
||||
|
||||
def _get_cached_embeddings(session: GraphSession) -> dict[str, List[float]]:
|
||||
"""Get embeddings from session cache for optimal performance."""
|
||||
return session.get_cached_embeddings()
|
||||
|
||||
|
||||
def _node_response(node: dict) -> NodeResponse:
|
||||
return NodeResponse(**node)
|
||||
|
||||
@@ -480,7 +485,6 @@ async def distance_matrix(
|
||||
)
|
||||
|
||||
started = time.perf_counter()
|
||||
graph_dict = await asyncio.to_thread(session.build_graph_dict)
|
||||
path_finder = session.path_finder
|
||||
|
||||
n = len(body.node_ids)
|
||||
@@ -493,10 +497,21 @@ async def distance_matrix(
|
||||
src, tgt = body.node_ids[i], body.node_ids[j]
|
||||
try:
|
||||
if body.metric == "semantic" and session.similarity is not None:
|
||||
sim = await asyncio.to_thread(
|
||||
session.similarity.cosine_similarity, graph_dict, src, tgt
|
||||
)
|
||||
val = 1.0 - float(sim) if isinstance(sim, (int, float)) else None
|
||||
# Use cached embeddings for semantic distance calculation
|
||||
embeddings = _get_cached_embeddings(session)
|
||||
src_embedding = embeddings.get(src)
|
||||
tgt_embedding = embeddings.get(tgt)
|
||||
|
||||
if src_embedding is None or tgt_embedding is None:
|
||||
val = None
|
||||
else:
|
||||
# Calculate cosine similarity directly from cached embeddings
|
||||
import numpy as np
|
||||
src_vec = np.array(src_embedding)
|
||||
tgt_vec = np.array(tgt_embedding)
|
||||
sim = np.dot(src_vec, tgt_vec) / (np.linalg.norm(src_vec) * np.linalg.norm(tgt_vec))
|
||||
val = 1.0 - float(sim) if isinstance(sim, (int, float)) else None
|
||||
|
||||
matrix[i][j] = val
|
||||
matrix[j][i] = val
|
||||
elif path_finder is not None:
|
||||
@@ -505,6 +520,7 @@ async def distance_matrix(
|
||||
if body.metric == "weighted"
|
||||
else path_finder.bfs_shortest_path
|
||||
)
|
||||
graph_dict = await asyncio.to_thread(session.build_graph_dict)
|
||||
result = await asyncio.to_thread(path_fn, graph_dict, src, tgt)
|
||||
path_nodes = result.get("path", []) if isinstance(result, dict) else (result or [])
|
||||
if path_nodes:
|
||||
@@ -550,8 +566,7 @@ async def _semantic_neighborhood_impl(
|
||||
detail="Semantic similarity is unavailable for this graph session.",
|
||||
)
|
||||
|
||||
graph_dict = await asyncio.to_thread(session.build_graph_dict)
|
||||
embeddings = _extract_node_embeddings(graph_dict)
|
||||
embeddings = _get_cached_embeddings(session)
|
||||
query_embedding = embeddings.get(node_id)
|
||||
if not embeddings or query_embedding is None:
|
||||
raise HTTPException(
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -52,6 +52,10 @@ class GraphSession:
|
||||
self._similarity: Any = None
|
||||
self._link_predictor: Any = None
|
||||
self._validator: Any = None
|
||||
|
||||
self._graph_revision: int = 0
|
||||
self._cached_embeddings: Optional[Dict[str, List[float]]] = None
|
||||
self._cached_graph_revision: int = -1
|
||||
self.rebuild_search_index()
|
||||
|
||||
@classmethod
|
||||
@@ -408,6 +412,20 @@ class GraphSession:
|
||||
|
||||
def handle_graph_mutation(self, event_type: str, entity_id: str, payload: Dict[str, Any]) -> None:
|
||||
normalized_event = str(event_type or "").upper()
|
||||
if normalized_event in {
|
||||
"ADD_NODE",
|
||||
"UPDATE_NODE",
|
||||
"REMOVE_NODE",
|
||||
"DELETE_NODE",
|
||||
"ADD_EDGE",
|
||||
"UPDATE_EDGE",
|
||||
"REMOVE_EDGE",
|
||||
"DELETE_EDGE",
|
||||
"RELOAD_GRAPH",
|
||||
"RESET_GRAPH",
|
||||
}:
|
||||
with self._lock:
|
||||
self._bump_graph_revision_locked()
|
||||
if normalized_event in {"ADD_NODE", "UPDATE_NODE"}:
|
||||
normalized_node = self.normalize_node(payload or {})
|
||||
if normalized_node.get("id"):
|
||||
@@ -529,6 +547,83 @@ class GraphSession:
|
||||
with self._lock:
|
||||
return self.annotations.pop(annotation_id, None) is not None
|
||||
|
||||
def _bump_graph_revision_locked(self) -> None:
|
||||
self._graph_revision += 1
|
||||
self._cached_embeddings = None
|
||||
self._cached_graph_revision = -1
|
||||
|
||||
@staticmethod
|
||||
def _coerce_embedding_vector(value: Any) -> Optional[List[float]]:
|
||||
if isinstance(value, dict):
|
||||
for key in ("embedding", "embeddings", "vector", "values", "node2vec", "semantic"):
|
||||
nested = GraphSession._coerce_embedding_vector(value.get(key))
|
||||
if nested is not None:
|
||||
return nested
|
||||
return None
|
||||
|
||||
if not isinstance(value, (list, tuple)):
|
||||
return None
|
||||
|
||||
vector: List[float] = []
|
||||
for item in value:
|
||||
try:
|
||||
vector.append(float(item))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
return vector if vector else None
|
||||
|
||||
def get_cached_embeddings(self, force_refresh: bool = False) -> Dict[str, List[float]]:
|
||||
with self._lock:
|
||||
current_revision = self._graph_revision
|
||||
if (
|
||||
not force_refresh
|
||||
and self._cached_embeddings is not None
|
||||
and self._cached_graph_revision == current_revision
|
||||
):
|
||||
return self._cached_embeddings
|
||||
raw_nodes = [
|
||||
node.to_dict() if hasattr(node, "to_dict") else node
|
||||
for node in self.graph.nodes.values()
|
||||
if node is not None
|
||||
]
|
||||
|
||||
embedding_keys = (
|
||||
"embedding",
|
||||
"embeddings",
|
||||
"vector",
|
||||
"node_embedding",
|
||||
"node2vec_embedding",
|
||||
"semantic_embedding",
|
||||
"reasoning_embedding",
|
||||
)
|
||||
|
||||
embeddings: Dict[str, List[float]] = {}
|
||||
for raw in raw_nodes:
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
normalized = self.normalize_node(raw)
|
||||
node_id = normalized.get("id")
|
||||
if not node_id:
|
||||
continue
|
||||
properties = normalized.get("properties") if isinstance(normalized.get("properties"), dict) else {}
|
||||
for key in embedding_keys:
|
||||
vector = self._coerce_embedding_vector(normalized.get(key, properties.get(key)))
|
||||
if vector is not None:
|
||||
embeddings[str(node_id)] = vector
|
||||
break
|
||||
|
||||
with self._lock:
|
||||
if self._graph_revision == current_revision:
|
||||
self._cached_embeddings = embeddings
|
||||
self._cached_graph_revision = current_revision
|
||||
return embeddings
|
||||
|
||||
def invalidate_embedding_cache(self) -> None:
|
||||
with self._lock:
|
||||
self._cached_embeddings = None
|
||||
self._cached_graph_revision = -1
|
||||
|
||||
def build_graph_dict(self, node_ids: Optional[list] = None) -> dict:
|
||||
nodes, _ = self.get_nodes(skip=0, limit=999_999)
|
||||
edges, _ = self.get_edges(skip=0, limit=999_999)
|
||||
@@ -595,6 +690,8 @@ class GraphSession:
|
||||
with self._lock:
|
||||
added = self.graph.add_nodes(nodes)
|
||||
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
|
||||
if added and not has_mutation_callback:
|
||||
self._bump_graph_revision_locked()
|
||||
if added and not has_mutation_callback:
|
||||
self.rebuild_search_index()
|
||||
return added
|
||||
@@ -603,6 +700,8 @@ class GraphSession:
|
||||
with self._lock:
|
||||
added = self.graph.add_edges(edges)
|
||||
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
|
||||
if added and not has_mutation_callback:
|
||||
self._bump_graph_revision_locked()
|
||||
if added and not has_mutation_callback:
|
||||
self.rebuild_search_index()
|
||||
return added
|
||||
@@ -617,6 +716,8 @@ class GraphSession:
|
||||
with self._lock:
|
||||
added = self.graph.add_node(node_id, node_type, content=content, **properties)
|
||||
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
|
||||
if added and not has_mutation_callback:
|
||||
self._bump_graph_revision_locked()
|
||||
if added and not has_mutation_callback:
|
||||
normalized = self.get_node(node_id)
|
||||
if normalized is not None:
|
||||
@@ -640,4 +741,6 @@ class GraphSession:
|
||||
**properties,
|
||||
)
|
||||
has_mutation_callback = callable(getattr(self.graph, "mutation_callback", None))
|
||||
if added and not has_mutation_callback:
|
||||
self._bump_graph_revision_locked()
|
||||
return added
|
||||
|
||||
+123
-42
@@ -7,12 +7,15 @@ including files, web content, feeds, streams, repositories, emails, and database
|
||||
Algorithms Used:
|
||||
|
||||
File Ingestion:
|
||||
- File Type Detection: Multi-method detection (extension-based, MIME type, magic number analysis)
|
||||
- File Type Detection: Multi-method detection using extension,
|
||||
MIME type, and magic number analysis
|
||||
- Directory Scanning: Recursive directory traversal with filtering
|
||||
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob Storage API integration
|
||||
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob
|
||||
Storage API integration
|
||||
- File Validation: Size limits, format validation, content verification
|
||||
- Batch Processing: Parallel file processing with progress tracking
|
||||
- Magic Number Analysis: Binary file signature detection for accurate type identification
|
||||
- Magic Number Analysis: Binary file signature detection for accurate
|
||||
type identification
|
||||
|
||||
Web Ingestion:
|
||||
- HTTP Request Handling: GET/POST requests with retry logic and error handling
|
||||
@@ -46,9 +49,11 @@ Repository Ingestion:
|
||||
- Git Operations: Repository cloning, branch checking, commit traversal
|
||||
- Code Extraction: File content extraction with language detection
|
||||
- Commit Analysis: Git log parsing, diff analysis, statistics calculation
|
||||
- Language Detection: File extension and content-based programming language identification
|
||||
- Language Detection: File extension and content-based programming
|
||||
language identification
|
||||
- Code Structure Analysis: AST parsing for classes, functions, imports extraction
|
||||
- Dependency Analysis: Package manager file parsing (requirements.txt, package.json, etc.)
|
||||
- Dependency Analysis: Package manager file parsing
|
||||
(requirements.txt, package.json, etc.)
|
||||
- Documentation Extraction: README, docstring, and comment extraction
|
||||
|
||||
Email Ingestion:
|
||||
@@ -61,7 +66,8 @@ Email Ingestion:
|
||||
- Link Extraction: URL extraction from email HTML content
|
||||
|
||||
Database Ingestion:
|
||||
- Database Connection: SQLAlchemy-based connection management with connection pooling
|
||||
- Database Connection: SQLAlchemy-based connection management with
|
||||
connection pooling
|
||||
- SQL Query Execution: Parameterized query execution with result set processing
|
||||
- Schema Introspection: Database schema analysis and table/column discovery
|
||||
- Data Type Conversion: Database-specific type to Python type conversion
|
||||
@@ -87,6 +93,7 @@ Main Classes:
|
||||
- EmailIngestor: Email protocol handling
|
||||
- DBIngestor: Database export handling
|
||||
- OntologyIngestor: Ontology file processing
|
||||
- ParquetIngestor: Apache Parquet file and partitioned dataset processing
|
||||
- MethodRegistry: Registry for custom ingestion methods
|
||||
- IngestConfig: Configuration manager for ingest module
|
||||
|
||||
@@ -100,6 +107,7 @@ Convenience Functions:
|
||||
- ingest_email: Email ingestion wrapper
|
||||
- ingest_database: Database ingestion wrapper
|
||||
- ingest_ontology: Ontology ingestion wrapper
|
||||
- ingest_parquet: Parquet ingestion wrapper
|
||||
|
||||
|
||||
Example Usage:
|
||||
@@ -112,22 +120,18 @@ Example Usage:
|
||||
>>> content = ingest_web("https://example.com", method="url")
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
from typing import Any, Dict, Tuple
|
||||
|
||||
from .config import IngestConfig, ingest_config
|
||||
from .db_ingestor import DatabaseConnector, DataExporter, DBIngestor, TableData
|
||||
from .email_ingestor import AttachmentProcessor, EmailData, EmailIngestor
|
||||
from .email_ingestor import EmailParser as EmailIngestorParser
|
||||
from .feed_ingestor import FeedData, FeedIngestor, FeedItem, FeedMonitor, FeedParser
|
||||
from .file_ingestor import (
|
||||
CloudStorageIngestor,
|
||||
FileIngestor,
|
||||
FileObject,
|
||||
FileTypeDetector,
|
||||
)
|
||||
from .mcp_client import MCPClient, MCPResource, MCPTool
|
||||
from .mcp_ingestor import MCPData, MCPIngestor
|
||||
from .methods import (
|
||||
get_ingest_method,
|
||||
ingest,
|
||||
@@ -137,40 +141,114 @@ from .methods import (
|
||||
ingest_file,
|
||||
ingest_mcp,
|
||||
ingest_ontology,
|
||||
ingest_parquet,
|
||||
ingest_repository,
|
||||
ingest_stream,
|
||||
ingest_web,
|
||||
list_available_methods,
|
||||
)
|
||||
from .registry import MethodRegistry, method_registry
|
||||
from .repo_ingestor import (
|
||||
CodeExtractor,
|
||||
CodeFile,
|
||||
CommitInfo,
|
||||
GitAnalyzer,
|
||||
RepoIngestor,
|
||||
)
|
||||
from .stream_ingestor import (
|
||||
KafkaProcessor,
|
||||
KinesisProcessor,
|
||||
PulsarProcessor,
|
||||
RabbitMQProcessor,
|
||||
StreamIngestor,
|
||||
StreamMessage,
|
||||
StreamMonitor,
|
||||
StreamProcessor,
|
||||
)
|
||||
from .web_ingestor import (
|
||||
ContentExtractor,
|
||||
RateLimiter,
|
||||
RobotsChecker,
|
||||
SitemapCrawler,
|
||||
WebContent,
|
||||
WebIngestor,
|
||||
)
|
||||
|
||||
from .ontology_ingestor import OntologyData, OntologyIngestor
|
||||
from .snowflake_ingestor import SnowflakeConnector, SnowflakeData, SnowflakeIngestor
|
||||
_LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
|
||||
# Web ingestion
|
||||
"WebIngestor": (".web_ingestor", "WebIngestor"),
|
||||
"WebContent": (".web_ingestor", "WebContent"),
|
||||
"RateLimiter": (".web_ingestor", "RateLimiter"),
|
||||
"RobotsChecker": (".web_ingestor", "RobotsChecker"),
|
||||
"ContentExtractor": (".web_ingestor", "ContentExtractor"),
|
||||
"SitemapCrawler": (".web_ingestor", "SitemapCrawler"),
|
||||
# Feed ingestion
|
||||
"FeedIngestor": (".feed_ingestor", "FeedIngestor"),
|
||||
"FeedItem": (".feed_ingestor", "FeedItem"),
|
||||
"FeedData": (".feed_ingestor", "FeedData"),
|
||||
"FeedParser": (".feed_ingestor", "FeedParser"),
|
||||
"FeedMonitor": (".feed_ingestor", "FeedMonitor"),
|
||||
# Stream ingestion
|
||||
"StreamIngestor": (".stream_ingestor", "StreamIngestor"),
|
||||
"StreamMessage": (".stream_ingestor", "StreamMessage"),
|
||||
"StreamProcessor": (".stream_ingestor", "StreamProcessor"),
|
||||
"KafkaProcessor": (".stream_ingestor", "KafkaProcessor"),
|
||||
"RabbitMQProcessor": (".stream_ingestor", "RabbitMQProcessor"),
|
||||
"KinesisProcessor": (".stream_ingestor", "KinesisProcessor"),
|
||||
"PulsarProcessor": (".stream_ingestor", "PulsarProcessor"),
|
||||
"StreamMonitor": (".stream_ingestor", "StreamMonitor"),
|
||||
# Repository ingestion
|
||||
"RepoIngestor": (".repo_ingestor", "RepoIngestor"),
|
||||
"CodeFile": (".repo_ingestor", "CodeFile"),
|
||||
"CommitInfo": (".repo_ingestor", "CommitInfo"),
|
||||
"CodeExtractor": (".repo_ingestor", "CodeExtractor"),
|
||||
"GitAnalyzer": (".repo_ingestor", "GitAnalyzer"),
|
||||
# Email ingestion
|
||||
"EmailIngestor": (".email_ingestor", "EmailIngestor"),
|
||||
"EmailData": (".email_ingestor", "EmailData"),
|
||||
"AttachmentProcessor": (".email_ingestor", "AttachmentProcessor"),
|
||||
"EmailIngestorParser": (".email_ingestor", "EmailParser"),
|
||||
# Database ingestion
|
||||
"DBIngestor": (".db_ingestor", "DBIngestor"),
|
||||
"TableData": (".db_ingestor", "TableData"),
|
||||
"DatabaseConnector": (".db_ingestor", "DatabaseConnector"),
|
||||
"DataExporter": (".db_ingestor", "DataExporter"),
|
||||
# MCP ingestion
|
||||
"MCPIngestor": (".mcp_ingestor", "MCPIngestor"),
|
||||
"MCPData": (".mcp_ingestor", "MCPData"),
|
||||
"MCPClient": (".mcp_client", "MCPClient"),
|
||||
"MCPResource": (".mcp_client", "MCPResource"),
|
||||
"MCPTool": (".mcp_client", "MCPTool"),
|
||||
# Ontology ingestion
|
||||
"OntologyIngestor": (".ontology_ingestor", "OntologyIngestor"),
|
||||
"OntologyData": (".ontology_ingestor", "OntologyData"),
|
||||
# Snowflake ingestion
|
||||
"SnowflakeIngestor": (".snowflake_ingestor", "SnowflakeIngestor"),
|
||||
"SnowflakeData": (".snowflake_ingestor", "SnowflakeData"),
|
||||
"SnowflakeConnector": (".snowflake_ingestor", "SnowflakeConnector"),
|
||||
# Parquet ingestion
|
||||
"ParquetIngestor": (".parquet_ingestor", "ParquetIngestor"),
|
||||
"ParquetData": (".parquet_ingestor", "ParquetData"),
|
||||
}
|
||||
|
||||
_OPTIONAL_DEPENDENCY_MESSAGES = {
|
||||
".repo_ingestor": (
|
||||
"Repository ingestion requires optional dependency 'GitPython'. "
|
||||
"Install it before importing RepoIngestor or using ingest_repository()."
|
||||
),
|
||||
".web_ingestor": (
|
||||
"Web ingestion requires optional dependency 'beautifulsoup4'. "
|
||||
"Install it before importing WebIngestor or using ingest_web()."
|
||||
),
|
||||
".feed_ingestor": (
|
||||
"Feed ingestion requires optional dependency 'beautifulsoup4'. "
|
||||
"Install it before importing FeedIngestor or using ingest_feed()."
|
||||
),
|
||||
".email_ingestor": (
|
||||
"Email ingestion requires optional dependency 'beautifulsoup4'. "
|
||||
"Install it before importing EmailIngestor or using ingest_email()."
|
||||
),
|
||||
".parquet_ingestor": (
|
||||
"Parquet ingestion requires optional dependency 'pyarrow'. "
|
||||
"Install it before importing ParquetIngestor or using ingest_parquet()."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def __getattr__(name: str) -> Any:
|
||||
"""Load optional ingestion backends only when callers request them."""
|
||||
if name not in _LAZY_EXPORTS:
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
module_name, attr_name = _LAZY_EXPORTS[name]
|
||||
try:
|
||||
module = importlib.import_module(module_name, __name__)
|
||||
except ModuleNotFoundError as exc:
|
||||
message = _OPTIONAL_DEPENDENCY_MESSAGES.get(module_name)
|
||||
missing_name = getattr(exc, "name", None)
|
||||
if message and missing_name in {"git", "bs4", "pyarrow"}:
|
||||
raise ImportError(message) from exc
|
||||
raise
|
||||
|
||||
value = getattr(module, attr_name)
|
||||
globals()[name] = value
|
||||
return value
|
||||
|
||||
|
||||
__all__ = [
|
||||
# File ingestion
|
||||
@@ -229,6 +307,9 @@ __all__ = [
|
||||
"SnowflakeIngestor",
|
||||
"SnowflakeData",
|
||||
"SnowflakeConnector",
|
||||
# Parquet ingestion
|
||||
"ParquetIngestor",
|
||||
"ParquetData",
|
||||
# Registry and Methods
|
||||
"MethodRegistry",
|
||||
"method_registry",
|
||||
@@ -241,6 +322,7 @@ __all__ = [
|
||||
"ingest_email",
|
||||
"ingest_database",
|
||||
"ingest_ontology",
|
||||
"ingest_parquet",
|
||||
"ingest_mcp",
|
||||
"get_ingest_method",
|
||||
"list_available_methods",
|
||||
@@ -248,4 +330,3 @@ __all__ = [
|
||||
"IngestConfig",
|
||||
"ingest_config",
|
||||
]
|
||||
|
||||
|
||||
@@ -23,7 +23,6 @@ License: MIT
|
||||
"""
|
||||
|
||||
import mimetypes
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
@@ -112,7 +111,8 @@ class FileTypeDetector:
|
||||
mimetypes.init()
|
||||
|
||||
self.logger.debug(
|
||||
f"File type detector initialized with {len(self.supported_formats)} supported formats"
|
||||
"File type detector initialized with "
|
||||
f"{len(self.supported_formats)} supported formats"
|
||||
)
|
||||
|
||||
def detect_type(
|
||||
@@ -190,11 +190,12 @@ class FileTypeDetector:
|
||||
magic_numbers = {
|
||||
b"\x25\x50\x44\x46": "pdf", # PDF (binary)
|
||||
b"%PDF": "pdf", # PDF (text header)
|
||||
b"\x50\x4B\x03\x04": "zip", # ZIP, DOCX, XLSX, PPTX (Office Open XML)
|
||||
b"\x89\x50\x4E\x47": "png", # PNG image
|
||||
b"\xFF\xD8\xFF": "jpg", # JPEG image
|
||||
b"\x50\x4b\x03\x04": "zip", # ZIP, DOCX, XLSX, PPTX (Office Open XML)
|
||||
b"\x89\x50\x4e\x47": "png", # PNG image
|
||||
b"\xff\xd8\xff": "jpg", # JPEG image
|
||||
b"\x47\x49\x46\x38": "gif", # GIF image
|
||||
b"PK\x03\x04": "zip", # ZIP (alternative)
|
||||
b"PAR1": "parquet", # Apache Parquet
|
||||
}
|
||||
|
||||
# Check if content starts with any known magic number
|
||||
@@ -516,7 +517,10 @@ class FileIngestor:
|
||||
tracking_id,
|
||||
processed=idx,
|
||||
total=total_files,
|
||||
message=f"Processing file {idx}/{total_files}: {Path(file_info['path']).name}"
|
||||
message=(
|
||||
f"Processing file {idx}/{total_files}: "
|
||||
f"{Path(file_info['path']).name}"
|
||||
),
|
||||
)
|
||||
|
||||
# Track progress via callback if provided
|
||||
|
||||
@@ -6,18 +6,19 @@ This guide demonstrates how to use the ingest module for ingesting data from var
|
||||
|
||||
1. [Basic Usage](#basic-usage)
|
||||
2. [File Ingestion](#file-ingestion)
|
||||
3. [Web Ingestion](#web-ingestion)
|
||||
4. [Feed Ingestion](#feed-ingestion)
|
||||
5. [Stream Ingestion](#stream-ingestion)
|
||||
6. [Repository Ingestion](#repository-ingestion)
|
||||
7. [Email Ingestion](#email-ingestion)
|
||||
8. [Database Ingestion](#database-ingestion)
|
||||
9. [MCP Server Ingestion](#mcp-server-ingestion)
|
||||
10. [Unified Ingestion](#unified-ingestion)
|
||||
11. [Using Methods](#using-methods)
|
||||
12. [Using Registry](#using-registry)
|
||||
13. [Configuration](#configuration)
|
||||
14. [Advanced Examples](#advanced-examples)
|
||||
3. [Parquet Ingestion](#parquet-ingestion)
|
||||
4. [Web Ingestion](#web-ingestion)
|
||||
5. [Feed Ingestion](#feed-ingestion)
|
||||
6. [Stream Ingestion](#stream-ingestion)
|
||||
7. [Repository Ingestion](#repository-ingestion)
|
||||
8. [Email Ingestion](#email-ingestion)
|
||||
9. [Database Ingestion](#database-ingestion)
|
||||
10. [MCP Server Ingestion](#mcp-server-ingestion)
|
||||
11. [Unified Ingestion](#unified-ingestion)
|
||||
12. [Using Methods](#using-methods)
|
||||
13. [Using Registry](#using-registry)
|
||||
14. [Configuration](#configuration)
|
||||
15. [Advanced Examples](#advanced-examples)
|
||||
|
||||
## Basic Usage
|
||||
|
||||
@@ -29,6 +30,9 @@ from semantica.ingest import ingest
|
||||
# Ingest a file (auto-detects source type)
|
||||
result = ingest("document.pdf", source_type="file")
|
||||
|
||||
# Ingest a Parquet file
|
||||
result = ingest("events.parquet")
|
||||
|
||||
# Ingest from web URL
|
||||
result = ingest("https://example.com", source_type="web")
|
||||
|
||||
@@ -142,6 +146,66 @@ with open("document.pdf", "rb") as f:
|
||||
file_type = detector.detect_type("document.pdf", content=content)
|
||||
```
|
||||
|
||||
## Parquet Ingestion
|
||||
|
||||
Parquet ingestion requires PyArrow:
|
||||
|
||||
```bash
|
||||
pip install pyarrow
|
||||
```
|
||||
|
||||
### Single Parquet File
|
||||
|
||||
```python
|
||||
from semantica.ingest import ParquetIngestor, ingest_parquet
|
||||
|
||||
# Using convenience function
|
||||
data = ingest_parquet(
|
||||
"events.parquet",
|
||||
columns=["event_id", "event_type"],
|
||||
limit=1000,
|
||||
)
|
||||
|
||||
# Using class directly
|
||||
ingestor = ParquetIngestor()
|
||||
data = ingestor.ingest_file("events.parquet")
|
||||
|
||||
print(f"Rows returned: {data.row_count}")
|
||||
print(f"Columns: {data.columns}")
|
||||
print(f"Total rows in file: {data.metadata['total_rows']}")
|
||||
```
|
||||
|
||||
### Schema and Metadata Extraction
|
||||
|
||||
```python
|
||||
from semantica.ingest import ParquetIngestor
|
||||
|
||||
ingestor = ParquetIngestor()
|
||||
|
||||
schema = ingestor.extract_schema("events.parquet")
|
||||
metadata = ingestor.extract_metadata("events.parquet")
|
||||
|
||||
print(schema["columns"])
|
||||
print(metadata["compression_codecs"])
|
||||
print(metadata["row_groups"])
|
||||
```
|
||||
|
||||
### Partitioned Parquet Directories
|
||||
|
||||
```python
|
||||
from semantica.ingest import ingest_parquet
|
||||
|
||||
# Reads Hive-style directories such as country=US/year=2026/part-0.parquet
|
||||
data = ingest_parquet(
|
||||
"./warehouse/events",
|
||||
method="directory",
|
||||
columns=["event_id", "event_type", "country", "year"],
|
||||
)
|
||||
|
||||
print(data.metadata["partition_columns"])
|
||||
print(data.metadata["partition_values"])
|
||||
```
|
||||
|
||||
## Web Ingestion
|
||||
|
||||
### Single URL Ingestion
|
||||
@@ -936,6 +1000,7 @@ from semantica.ingest import ingest
|
||||
|
||||
# Auto-detect source type from source
|
||||
result = ingest("document.pdf") # Auto-detects file
|
||||
result = ingest("events.parquet") # Auto-detects Parquet
|
||||
result = ingest("https://example.com") # Auto-detects web
|
||||
result = ingest("https://example.com/feed.xml") # Auto-detects feed
|
||||
result = ingest("postgresql://user:pass@localhost/db") # Auto-detects database
|
||||
@@ -982,7 +1047,8 @@ from semantica.ingest.methods import (
|
||||
ingest_repository,
|
||||
ingest_email,
|
||||
ingest_database,
|
||||
ingest_mcp
|
||||
ingest_mcp,
|
||||
ingest_parquet
|
||||
)
|
||||
|
||||
# File ingestion
|
||||
@@ -1006,6 +1072,9 @@ emails = ingest_email({"host": "imap.example.com", "username": "user", "password
|
||||
# Database ingestion
|
||||
data = ingest_database("postgresql://user:pass@localhost/db", table="users")
|
||||
|
||||
# Parquet ingestion
|
||||
events = ingest_parquet("events.parquet", columns=["event_id"], limit=1000)
|
||||
|
||||
# MCP server ingestion via URL
|
||||
data = ingest_mcp("http://localhost:8000/mcp", method="resources")
|
||||
```
|
||||
@@ -1189,16 +1258,16 @@ from semantica.ingest.methods import ingest_file
|
||||
def custom_pdf_ingestion(source, **kwargs):
|
||||
"""Custom PDF ingestion with special processing."""
|
||||
from semantica.ingest import FileIngestor
|
||||
|
||||
|
||||
ingestor = FileIngestor()
|
||||
file_obj = ingestor.ingest_file(source, **kwargs)
|
||||
|
||||
|
||||
# Custom processing
|
||||
if file_obj.file_type == "pdf":
|
||||
# Add custom metadata
|
||||
file_obj.metadata["processed"] = True
|
||||
file_obj.metadata["custom_field"] = "custom_value"
|
||||
|
||||
|
||||
return file_obj
|
||||
|
||||
# Register custom method
|
||||
@@ -1286,7 +1355,7 @@ for source_type, source_list in sources.items():
|
||||
1. **Parallel Processing**: Use parallel processing for multiple sources
|
||||
```python
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
|
||||
with ThreadPoolExecutor() as executor:
|
||||
executor.submit(ingest_file, "./documents1")
|
||||
executor.submit(ingest_file, "./documents2")
|
||||
@@ -1329,4 +1398,3 @@ for source_type, source_list in sources.items():
|
||||
for batch in process_in_batches(large_dataset, batch_size=1000):
|
||||
result = ingest(batch)
|
||||
```
|
||||
|
||||
|
||||
+242
-35
@@ -13,6 +13,12 @@ File Ingestion:
|
||||
- "directory": Directory ingestion with recursive scanning
|
||||
- "cloud": Cloud storage ingestion (S3, GCS, Azure)
|
||||
|
||||
Parquet Ingestion:
|
||||
- "file": Single Parquet file ingestion
|
||||
- "directory": Partitioned Parquet directory ingestion
|
||||
- "schema": Parquet schema extraction
|
||||
- "metadata": Parquet file or directory metadata extraction
|
||||
|
||||
Web Ingestion:
|
||||
- "url": Single URL ingestion
|
||||
- "sitemap": Sitemap-based crawling
|
||||
@@ -48,12 +54,15 @@ Database Ingestion:
|
||||
Algorithms Used:
|
||||
|
||||
File Ingestion:
|
||||
- File Type Detection: Multi-method detection (extension-based, MIME type, magic number analysis)
|
||||
- File Type Detection: Multi-method detection using extension,
|
||||
MIME type, and magic number analysis
|
||||
- Directory Scanning: Recursive directory traversal with filtering
|
||||
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob Storage API integration
|
||||
- Cloud Storage Integration: AWS S3, Google Cloud Storage, Azure Blob
|
||||
Storage API integration
|
||||
- File Validation: Size limits, format validation, content verification
|
||||
- Batch Processing: Parallel file processing with progress tracking
|
||||
- Magic Number Analysis: Binary file signature detection for accurate type identification
|
||||
- Magic Number Analysis: Binary file signature detection for accurate
|
||||
type identification
|
||||
|
||||
Web Ingestion:
|
||||
- HTTP Request Handling: GET/POST requests with retry logic and error handling
|
||||
@@ -87,9 +96,11 @@ Repository Ingestion:
|
||||
- Git Operations: Repository cloning, branch checking, commit traversal
|
||||
- Code Extraction: File content extraction with language detection
|
||||
- Commit Analysis: Git log parsing, diff analysis, statistics calculation
|
||||
- Language Detection: File extension and content-based programming language identification
|
||||
- Language Detection: File extension and content-based programming
|
||||
language identification
|
||||
- Code Structure Analysis: AST parsing for classes, functions, imports extraction
|
||||
- Dependency Analysis: Package manager file parsing (requirements.txt, package.json, etc.)
|
||||
- Dependency Analysis: Package manager file parsing
|
||||
(requirements.txt, package.json, etc.)
|
||||
- Documentation Extraction: README, docstring, and comment extraction
|
||||
|
||||
Email Ingestion:
|
||||
@@ -102,7 +113,8 @@ Email Ingestion:
|
||||
- Link Extraction: URL extraction from email HTML content
|
||||
|
||||
Database Ingestion:
|
||||
- Database Connection: SQLAlchemy-based connection management with connection pooling
|
||||
- Database Connection: SQLAlchemy-based connection management with
|
||||
connection pooling
|
||||
- SQL Query Execution: Parameterized query execution with result set processing
|
||||
- Schema Introspection: Database schema analysis and table/column discovery
|
||||
- Data Type Conversion: Database-specific type to Python type conversion
|
||||
@@ -125,6 +137,7 @@ Main Functions:
|
||||
- ingest_repository: Repository ingestion wrapper
|
||||
- ingest_email: Email ingestion wrapper
|
||||
- ingest_database: Database ingestion wrapper
|
||||
- ingest_parquet: Parquet ingestion wrapper
|
||||
- ingest: Unified ingestion function with source type dispatch
|
||||
- get_ingest_method: Get ingestion method by name
|
||||
- list_available_methods: List registered methods
|
||||
@@ -139,26 +152,42 @@ Example Usage:
|
||||
>>> content = ingest_web("https://example.com", method="url")
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional, Union
|
||||
from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union
|
||||
|
||||
from ..utils.exceptions import ConfigurationError, ProcessingError
|
||||
from ..utils.logging import get_logger
|
||||
from .config import ingest_config
|
||||
from .db_ingestor import DBIngestor, TableData
|
||||
from .email_ingestor import EmailData, EmailIngestor
|
||||
from .feed_ingestor import FeedData, FeedIngestor
|
||||
from .file_ingestor import FileIngestor, FileObject
|
||||
from .mcp_ingestor import MCPData, MCPIngestor
|
||||
from .ontology_ingestor import OntologyData, OntologyIngestor
|
||||
from .registry import method_registry
|
||||
from .repo_ingestor import RepoIngestor
|
||||
from .stream_ingestor import StreamIngestor, StreamProcessor
|
||||
from .web_ingestor import WebContent, WebIngestor
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .db_ingestor import TableData
|
||||
from .email_ingestor import EmailData
|
||||
from .feed_ingestor import FeedData
|
||||
from .mcp_ingestor import MCPData
|
||||
from .ontology_ingestor import OntologyData
|
||||
from .parquet_ingestor import ParquetData
|
||||
from .stream_ingestor import StreamProcessor
|
||||
from .web_ingestor import WebContent
|
||||
|
||||
logger = get_logger("ingest_methods")
|
||||
|
||||
|
||||
def _missing_optional_dependency(feature: str, package: str) -> ConfigurationError:
|
||||
return ConfigurationError(
|
||||
f"{feature} requires optional dependency '{package}'. "
|
||||
f"Install it before using this ingestion backend."
|
||||
)
|
||||
|
||||
|
||||
def _is_missing_dependency(exc: ModuleNotFoundError, *dependency_names: str) -> bool:
|
||||
missing_name = getattr(exc, "name", None)
|
||||
return missing_name in dependency_names
|
||||
|
||||
|
||||
def ingest_file(
|
||||
source: Union[str, Path, List[Union[str, Path]]], method: str = "file", **kwargs
|
||||
) -> Union[FileObject, List[FileObject], Dict[str, Any]]:
|
||||
@@ -224,6 +253,86 @@ def ingest_file(
|
||||
raise
|
||||
|
||||
|
||||
def ingest_parquet(
|
||||
source: Union[str, Path, List[Union[str, Path]]],
|
||||
method: str = "file",
|
||||
**kwargs,
|
||||
) -> Union[ParquetData, List[ParquetData], Dict[str, Any]]:
|
||||
"""
|
||||
Ingest Apache Parquet files or partitioned directories.
|
||||
|
||||
Args:
|
||||
source: Parquet file path, directory path, or list of paths
|
||||
method: Ingestion method:
|
||||
- "file": Single Parquet file ingestion
|
||||
- "directory": Parquet directory or partitioned dataset ingestion
|
||||
- "schema": Extract schema without reading data
|
||||
- "metadata": Extract file/directory metadata without reading data
|
||||
**kwargs: Additional options passed to ParquetIngestor
|
||||
|
||||
Returns:
|
||||
ParquetData, list of ParquetData, or metadata/schema dictionary
|
||||
|
||||
Examples:
|
||||
>>> from semantica.ingest.methods import ingest_parquet
|
||||
>>> data = ingest_parquet("events.parquet", columns=["id"], limit=100)
|
||||
>>> schema = ingest_parquet("events.parquet", method="schema")
|
||||
>>> dataset = ingest_parquet("./events_by_date", method="directory")
|
||||
"""
|
||||
custom_method = method_registry.get("parquet", method)
|
||||
if custom_method and custom_method != ingest_parquet:
|
||||
try:
|
||||
return custom_method(source, **kwargs)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"Custom method {method} failed: {e}, falling back to default"
|
||||
)
|
||||
|
||||
try:
|
||||
try:
|
||||
from .parquet_ingestor import ParquetIngestor
|
||||
except ModuleNotFoundError as exc:
|
||||
if _is_missing_dependency(exc, "pyarrow"):
|
||||
raise _missing_optional_dependency(
|
||||
"Parquet ingestion",
|
||||
"pyarrow",
|
||||
) from exc
|
||||
raise
|
||||
|
||||
config = ingest_config.get_method_config("parquet")
|
||||
config.update(kwargs)
|
||||
try:
|
||||
ingestor = ParquetIngestor(**config)
|
||||
except ImportError as exc:
|
||||
raise _missing_optional_dependency(
|
||||
"Parquet ingestion",
|
||||
"pyarrow",
|
||||
) from exc
|
||||
|
||||
def _run_single(path: Union[str, Path]) -> Union[ParquetData, Dict[str, Any]]:
|
||||
source_path = Path(path)
|
||||
|
||||
if method == "schema":
|
||||
return ingestor.extract_schema(source_path, **kwargs)
|
||||
if method == "metadata":
|
||||
return ingestor.extract_metadata(source_path, **kwargs)
|
||||
if method == "directory" or source_path.is_dir():
|
||||
return ingestor.ingest_directory(source_path, **kwargs)
|
||||
|
||||
return ingestor.ingest_file(source_path, **kwargs)
|
||||
|
||||
if isinstance(source, list):
|
||||
return [_run_single(path) for path in source]
|
||||
|
||||
return _run_single(source)
|
||||
|
||||
except ConfigurationError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest Parquet: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def ingest_web(
|
||||
source: Union[str, List[str]], method: str = "url", **kwargs
|
||||
) -> Union[WebContent, List[WebContent], Dict[str, Any]]:
|
||||
@@ -259,6 +368,16 @@ def ingest_web(
|
||||
)
|
||||
|
||||
try:
|
||||
try:
|
||||
from .web_ingestor import WebIngestor
|
||||
except ModuleNotFoundError as exc:
|
||||
if _is_missing_dependency(exc, "bs4"):
|
||||
raise _missing_optional_dependency(
|
||||
"Web ingestion",
|
||||
"beautifulsoup4",
|
||||
) from exc
|
||||
raise
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("web")
|
||||
config.update(kwargs)
|
||||
@@ -278,6 +397,8 @@ def ingest_web(
|
||||
# Default: try as URL
|
||||
return ingestor.ingest_url(source, **kwargs)
|
||||
|
||||
except ConfigurationError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest web: {e}")
|
||||
raise
|
||||
@@ -318,6 +439,16 @@ def ingest_feed(
|
||||
)
|
||||
|
||||
try:
|
||||
try:
|
||||
from .feed_ingestor import FeedIngestor
|
||||
except ModuleNotFoundError as exc:
|
||||
if _is_missing_dependency(exc, "bs4"):
|
||||
raise _missing_optional_dependency(
|
||||
"Feed ingestion",
|
||||
"beautifulsoup4",
|
||||
) from exc
|
||||
raise
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("feed")
|
||||
config.update(kwargs)
|
||||
@@ -332,6 +463,8 @@ def ingest_feed(
|
||||
else:
|
||||
return ingestor.ingest_feed(source, **kwargs)
|
||||
|
||||
except ConfigurationError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest feed: {e}")
|
||||
raise
|
||||
@@ -375,6 +508,8 @@ def ingest_stream(
|
||||
)
|
||||
|
||||
try:
|
||||
from .stream_ingestor import StreamIngestor
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("stream")
|
||||
config.update(kwargs)
|
||||
@@ -418,7 +553,8 @@ def ingest_repository(
|
||||
"""
|
||||
Ingest repository from source (convenience function).
|
||||
|
||||
This is a user-friendly wrapper that ingests repositories using the specified method.
|
||||
This is a user-friendly wrapper that ingests repositories using the
|
||||
specified method.
|
||||
|
||||
Args:
|
||||
source: Repository URL or local path
|
||||
@@ -433,7 +569,9 @@ def ingest_repository(
|
||||
|
||||
Examples:
|
||||
>>> from semantica.ingest.methods import ingest_repository
|
||||
>>> repo_data = ingest_repository("https://github.com/user/repo.git", method="git")
|
||||
>>> repo_data = ingest_repository(
|
||||
... "https://github.com/user/repo.git", method="git"
|
||||
... )
|
||||
>>> analysis = ingest_repository("./repo", method="analyze")
|
||||
"""
|
||||
# Check for custom method in registry
|
||||
@@ -447,6 +585,15 @@ def ingest_repository(
|
||||
)
|
||||
|
||||
try:
|
||||
try:
|
||||
from .repo_ingestor import RepoIngestor
|
||||
except ModuleNotFoundError as exc:
|
||||
if _is_missing_dependency(exc, "git"):
|
||||
raise _missing_optional_dependency(
|
||||
"Repository ingestion", "GitPython"
|
||||
) from exc
|
||||
raise
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("repo")
|
||||
config.update(kwargs)
|
||||
@@ -464,6 +611,8 @@ def ingest_repository(
|
||||
# Default: ingest repository
|
||||
return ingestor.ingest_repository(source, **kwargs)
|
||||
|
||||
except ConfigurationError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest repository: {e}")
|
||||
raise
|
||||
@@ -505,6 +654,16 @@ def ingest_email(
|
||||
)
|
||||
|
||||
try:
|
||||
try:
|
||||
from .email_ingestor import EmailIngestor
|
||||
except ModuleNotFoundError as exc:
|
||||
if _is_missing_dependency(exc, "bs4"):
|
||||
raise _missing_optional_dependency(
|
||||
"Email ingestion",
|
||||
"beautifulsoup4",
|
||||
) from exc
|
||||
raise
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("email")
|
||||
config.update(kwargs)
|
||||
@@ -533,6 +692,8 @@ def ingest_email(
|
||||
else:
|
||||
raise ProcessingError(f"Unknown email method: {method}")
|
||||
|
||||
except ConfigurationError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest email: {e}")
|
||||
raise
|
||||
@@ -572,6 +733,8 @@ def ingest_ontology(
|
||||
)
|
||||
|
||||
try:
|
||||
from .ontology_ingestor import OntologyIngestor
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("ontology")
|
||||
config.update(kwargs)
|
||||
@@ -579,19 +742,19 @@ def ingest_ontology(
|
||||
ingestor = OntologyIngestor(**config)
|
||||
|
||||
source_path = str(source) if isinstance(source, (str, Path)) else None
|
||||
|
||||
|
||||
if method == "file" and source_path:
|
||||
if isinstance(source, list):
|
||||
return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
|
||||
return ingestor.ingest_ontology(source_path, **kwargs)
|
||||
elif method == "directory" and source_path:
|
||||
recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
|
||||
return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
|
||||
recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
|
||||
return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
|
||||
else:
|
||||
# Default: try as file
|
||||
if isinstance(source, list):
|
||||
# Default: try as file
|
||||
if isinstance(source, list):
|
||||
return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
|
||||
return ingestor.ingest_ontology(str(source), **kwargs)
|
||||
return ingestor.ingest_ontology(str(source), **kwargs)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest ontology: {e}")
|
||||
@@ -636,6 +799,8 @@ def ingest_database(
|
||||
)
|
||||
|
||||
try:
|
||||
from .db_ingestor import DBIngestor
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("db")
|
||||
config.update(kwargs)
|
||||
@@ -686,7 +851,8 @@ def ingest_mcp(
|
||||
the specified method. Works with Python and FastMCP MCP servers.
|
||||
|
||||
Args:
|
||||
source: MCP server URL (str) or configuration dict with "url" key, or server name (str) if already connected
|
||||
source: MCP server URL, configuration dict with "url" key, or server
|
||||
name if already connected
|
||||
- URL string: "http://localhost:8000/mcp"
|
||||
- Dict: {"url": "http://localhost:8000/mcp", "headers": {...}}
|
||||
method: Ingestion method (default: "resources")
|
||||
@@ -709,13 +875,25 @@ def ingest_mcp(
|
||||
>>> data = ingest_mcp("http://localhost:8000/mcp", method="resources")
|
||||
>>> # Connect via URL dict and ingest all resources
|
||||
>>> data = ingest_mcp(
|
||||
... {"url": "https://api.example.com/mcp", "headers": {"Authorization": "Bearer token"}},
|
||||
... {
|
||||
... "url": "https://api.example.com/mcp",
|
||||
... "headers": {"Authorization": "Bearer token"},
|
||||
... },
|
||||
... method="all"
|
||||
... )
|
||||
>>> # Ingest from already connected server
|
||||
>>> data = ingest_mcp("server1", method="resources", resource_uris=["resource://example"])
|
||||
>>> data = ingest_mcp(
|
||||
... "server1",
|
||||
... method="resources",
|
||||
... resource_uris=["resource://example"],
|
||||
... )
|
||||
>>> # Call tool
|
||||
>>> result = ingest_mcp("server1", method="tools", tool_name="get_data", tool_arguments={})
|
||||
>>> result = ingest_mcp(
|
||||
... "server1",
|
||||
... method="tools",
|
||||
... tool_name="get_data",
|
||||
... tool_arguments={},
|
||||
... )
|
||||
"""
|
||||
# Check for custom method in registry
|
||||
custom_method = method_registry.get("mcp", method)
|
||||
@@ -728,6 +906,8 @@ def ingest_mcp(
|
||||
)
|
||||
|
||||
try:
|
||||
from .mcp_ingestor import MCPIngestor
|
||||
|
||||
# Get config
|
||||
config = ingest_config.get_method_config("mcp")
|
||||
config.update(kwargs)
|
||||
@@ -779,7 +959,8 @@ def ingest_mcp(
|
||||
)
|
||||
else:
|
||||
raise ProcessingError(
|
||||
"Source must be MCP server URL (str), configuration dict with 'url' key, "
|
||||
"Source must be MCP server URL (str), configuration dict "
|
||||
"with 'url' key, "
|
||||
"or server name (str) if already connected"
|
||||
)
|
||||
|
||||
@@ -831,6 +1012,7 @@ def ingest(
|
||||
- "email": Email ingestion
|
||||
- "db": Database ingestion
|
||||
- "ontology": Ontology ingestion
|
||||
- "parquet": Apache Parquet file or directory ingestion
|
||||
method: Optional specific ingestion method
|
||||
**kwargs: Additional options passed to ingestor
|
||||
|
||||
@@ -850,24 +1032,40 @@ def ingest(
|
||||
if not source_type:
|
||||
if isinstance(sources, (str, Path)):
|
||||
source_str = str(sources)
|
||||
if source_str.startswith(("http://", "https://")):
|
||||
source_str_lower = source_str.lower()
|
||||
if source_str_lower.startswith(("http://", "https://")):
|
||||
# Check if it's a feed URL
|
||||
if any(ext in source_str for ext in [".xml", "/feed", "/rss", "/atom"]):
|
||||
if any(
|
||||
ext in source_str_lower
|
||||
for ext in [".xml", "/feed", "/rss", "/atom"]
|
||||
):
|
||||
source_type = "feed"
|
||||
else:
|
||||
source_type = "web"
|
||||
elif source_str.startswith(
|
||||
elif source_str_lower.startswith(
|
||||
("postgresql://", "mysql://", "sqlite://", "oracle://", "mssql://")
|
||||
):
|
||||
source_type = "db"
|
||||
elif source_str.startswith(
|
||||
("git@", "https://github.com", "https://gitlab.com")
|
||||
elif source_str.startswith("git@") or source_str_lower.startswith(
|
||||
("https://github.com", "https://gitlab.com")
|
||||
):
|
||||
source_type = "repo"
|
||||
elif source_str.endswith((".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")):
|
||||
elif source_str_lower.endswith(
|
||||
(".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")
|
||||
):
|
||||
source_type = "ontology"
|
||||
elif source_str_lower.endswith((".parquet", ".pq")):
|
||||
source_type = "parquet"
|
||||
else:
|
||||
source_type = "file"
|
||||
elif (
|
||||
isinstance(sources, list)
|
||||
and sources
|
||||
and all(
|
||||
str(source).lower().endswith((".parquet", ".pq")) for source in sources
|
||||
)
|
||||
):
|
||||
source_type = "parquet"
|
||||
else:
|
||||
source_type = "file"
|
||||
|
||||
@@ -894,6 +1092,8 @@ def ingest(
|
||||
raise ProcessingError("Email ingestion requires configuration dictionary")
|
||||
elif source_type == "db":
|
||||
return {"data": ingest_database(sources, method=method, **kwargs)}
|
||||
elif source_type == "parquet":
|
||||
return {"data": ingest_parquet(sources, method=method or "file", **kwargs)}
|
||||
elif source_type == "ontology":
|
||||
return {"ontology": ingest_ontology(sources, method=method or "file", **kwargs)}
|
||||
elif source_type == "mcp":
|
||||
@@ -907,7 +1107,8 @@ def get_ingest_method(task: str, name: str) -> Optional[Callable]:
|
||||
Get a registered ingestion method.
|
||||
|
||||
Args:
|
||||
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
|
||||
task: Task type ("file", "web", "feed", "stream", "repo", "email",
|
||||
"db", "mcp", "ingest")
|
||||
name: Method name
|
||||
|
||||
Returns:
|
||||
@@ -971,6 +1172,12 @@ method_registry.register("db", "mysql", ingest_database)
|
||||
method_registry.register("db", "sqlite", ingest_database)
|
||||
method_registry.register("db", "oracle", ingest_database)
|
||||
method_registry.register("db", "mssql", ingest_database)
|
||||
method_registry.register("parquet", "default", ingest_parquet)
|
||||
method_registry.register("parquet", "file", ingest_parquet)
|
||||
method_registry.register("parquet", "directory", ingest_parquet)
|
||||
method_registry.register("parquet", "schema", ingest_parquet)
|
||||
method_registry.register("parquet", "metadata", ingest_parquet)
|
||||
method_registry.register("file", "parquet", ingest_parquet)
|
||||
method_registry.register("mcp", "default", ingest_mcp)
|
||||
method_registry.register("mcp", "resources", ingest_mcp)
|
||||
method_registry.register("mcp", "tools", ingest_mcp)
|
||||
|
||||
@@ -0,0 +1,766 @@
|
||||
"""
|
||||
Apache Parquet Ingestion Module
|
||||
|
||||
This module provides dedicated Parquet ingestion for local files and partitioned
|
||||
directories. It uses PyArrow when available so callers can read selected
|
||||
columns, inspect schemas and file metadata, and ingest Hive-style partitioned
|
||||
datasets without database credentials.
|
||||
|
||||
Example Usage:
|
||||
>>> from semantica.ingest import ParquetIngestor
|
||||
>>> ingestor = ParquetIngestor()
|
||||
>>> data = ingestor.ingest_file("events.parquet", columns=["id", "event_type"])
|
||||
>>> schema = ingestor.extract_schema("events.parquet")
|
||||
>>> partitioned = ingestor.ingest_directory("./events_by_date")
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional, Sequence, Union
|
||||
|
||||
try:
|
||||
import pyarrow as pa
|
||||
import pyarrow.dataset as ds
|
||||
import pyarrow.parquet as pq
|
||||
|
||||
PARQUET_AVAILABLE = True
|
||||
except (ImportError, OSError):
|
||||
pa = None
|
||||
ds = None
|
||||
pq = None
|
||||
PARQUET_AVAILABLE = False
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParquetData:
|
||||
"""Parquet ingestion result."""
|
||||
|
||||
data: List[Dict[str, Any]]
|
||||
row_count: int
|
||||
columns: List[str]
|
||||
schema: Dict[str, Any]
|
||||
source: str
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
ingested_at: datetime = field(default_factory=datetime.now)
|
||||
|
||||
|
||||
class ParquetIngestor:
|
||||
"""
|
||||
Dedicated Parquet ingestion handler.
|
||||
|
||||
Features:
|
||||
- Single Parquet file ingestion
|
||||
- Partitioned directory ingestion with Hive partition discovery
|
||||
- Selective column reads
|
||||
- Schema and file metadata extraction
|
||||
- Optional row limits for sampling large files
|
||||
"""
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
|
||||
"""
|
||||
Initialize Parquet ingestor.
|
||||
|
||||
Args:
|
||||
config: Optional configuration dictionary
|
||||
**kwargs: Additional configuration options
|
||||
|
||||
Raises:
|
||||
ImportError: If pyarrow is not installed
|
||||
"""
|
||||
if not PARQUET_AVAILABLE:
|
||||
raise ImportError(
|
||||
"pyarrow is required for ParquetIngestor. "
|
||||
"Install it with: pip install pyarrow"
|
||||
)
|
||||
|
||||
self.logger = get_logger("parquet_ingestor")
|
||||
self.config = config or {}
|
||||
self.config.update(kwargs)
|
||||
self.progress_tracker = get_progress_tracker()
|
||||
if not self.progress_tracker.enabled:
|
||||
self.progress_tracker.enabled = True
|
||||
|
||||
self.logger.debug("Parquet ingestor initialized")
|
||||
|
||||
def ingest(
|
||||
self,
|
||||
source: Union[str, Path],
|
||||
columns: Optional[Union[str, Sequence[str]]] = None,
|
||||
limit: Optional[int] = None,
|
||||
filters: Any = None,
|
||||
include_data: bool = True,
|
||||
**options,
|
||||
) -> ParquetData:
|
||||
"""
|
||||
Ingest a Parquet file or partitioned Parquet directory.
|
||||
|
||||
Args:
|
||||
source: Parquet file or directory path
|
||||
columns: Optional column name or names to read
|
||||
limit: Optional maximum number of rows to return
|
||||
filters: Optional PyArrow filter expression or tuple filters
|
||||
include_data: If False, return schema and metadata without rows
|
||||
**options: Additional options
|
||||
|
||||
Returns:
|
||||
ParquetData: Ingested data and metadata
|
||||
"""
|
||||
source_path = Path(source)
|
||||
if source_path.is_dir():
|
||||
return self.ingest_directory(
|
||||
source_path,
|
||||
columns=columns,
|
||||
limit=limit,
|
||||
filters=filters,
|
||||
include_data=include_data,
|
||||
**options,
|
||||
)
|
||||
return self.ingest_file(
|
||||
source_path,
|
||||
columns=columns,
|
||||
limit=limit,
|
||||
filters=filters,
|
||||
include_data=include_data,
|
||||
**options,
|
||||
)
|
||||
|
||||
def ingest_file(
|
||||
self,
|
||||
file_path: Union[str, Path],
|
||||
columns: Optional[Union[str, Sequence[str]]] = None,
|
||||
limit: Optional[int] = None,
|
||||
filters: Any = None,
|
||||
include_data: bool = True,
|
||||
batch_size: Optional[int] = None,
|
||||
**options,
|
||||
) -> ParquetData:
|
||||
"""
|
||||
Ingest a single Parquet file.
|
||||
|
||||
Args:
|
||||
file_path: Path to Parquet file
|
||||
columns: Optional column name or names to read
|
||||
limit: Optional maximum number of rows to return
|
||||
filters: Optional PyArrow-compatible filters
|
||||
include_data: If False, skip reading row data
|
||||
batch_size: Batch size used when sampling with limit
|
||||
**options: Additional options
|
||||
|
||||
Returns:
|
||||
ParquetData: Ingested data, schema, and metadata
|
||||
"""
|
||||
file_path = Path(file_path)
|
||||
self._validate_file(file_path)
|
||||
|
||||
tracking_id = self.progress_tracker.start_tracking(
|
||||
file=str(file_path),
|
||||
module="ingest",
|
||||
submodule="ParquetIngestor",
|
||||
message=f"Ingesting Parquet: {file_path.name}",
|
||||
)
|
||||
|
||||
try:
|
||||
parquet_file = pq.ParquetFile(str(file_path))
|
||||
selected_columns = self._normalize_columns(
|
||||
columns,
|
||||
[field.name for field in parquet_file.schema_arrow],
|
||||
)
|
||||
metadata = self._file_metadata(file_path, parquet_file)
|
||||
|
||||
if include_data:
|
||||
table = self._read_file_table(
|
||||
file_path=file_path,
|
||||
parquet_file=parquet_file,
|
||||
columns=selected_columns,
|
||||
limit=limit,
|
||||
filters=filters,
|
||||
batch_size=batch_size,
|
||||
)
|
||||
data = table.to_pylist()
|
||||
schema = self._schema_to_dict(table.schema)
|
||||
result_columns = list(table.column_names)
|
||||
else:
|
||||
selected_schema = self._select_schema(
|
||||
parquet_file.schema_arrow, selected_columns
|
||||
)
|
||||
data = []
|
||||
schema = self._schema_to_dict(selected_schema)
|
||||
result_columns = [field.name for field in selected_schema]
|
||||
|
||||
metadata.update(
|
||||
{
|
||||
"returned_rows": len(data),
|
||||
"selected_columns": result_columns,
|
||||
"filters_applied": filters is not None,
|
||||
"limit": limit,
|
||||
"include_data": include_data,
|
||||
}
|
||||
)
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
message=f"Ingested Parquet: {len(data)} rows",
|
||||
)
|
||||
|
||||
self.logger.info(
|
||||
f"Parquet ingestion completed: {len(data)} row(s) from {file_path}"
|
||||
)
|
||||
|
||||
return ParquetData(
|
||||
data=data,
|
||||
row_count=len(data),
|
||||
columns=result_columns,
|
||||
schema=schema,
|
||||
source=str(file_path),
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
except (ValidationError, ProcessingError):
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id, status="failed", message="Parquet ingestion failed"
|
||||
)
|
||||
raise
|
||||
except Exception as e:
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id, status="failed", message=str(e)
|
||||
)
|
||||
self.logger.error(f"Failed to ingest Parquet {file_path}: {e}")
|
||||
raise ProcessingError(f"Failed to ingest Parquet: {e}") from e
|
||||
|
||||
def ingest_directory(
|
||||
self,
|
||||
directory_path: Union[str, Path],
|
||||
columns: Optional[Union[str, Sequence[str]]] = None,
|
||||
limit: Optional[int] = None,
|
||||
filters: Any = None,
|
||||
include_data: bool = True,
|
||||
partitioning: Optional[Union[str, Any]] = "hive",
|
||||
**options,
|
||||
) -> ParquetData:
|
||||
"""
|
||||
Ingest a directory containing Parquet files.
|
||||
|
||||
Hive-style partitions such as ``country=US/year=2026`` are discovered
|
||||
by default and included as partition columns in the returned schema/data.
|
||||
|
||||
Args:
|
||||
directory_path: Directory containing Parquet files
|
||||
columns: Optional column name or names to read
|
||||
limit: Optional maximum number of rows to return
|
||||
filters: Optional PyArrow filter expression or tuple filters
|
||||
include_data: If False, return only schema and metadata
|
||||
partitioning: PyArrow partitioning mode, defaults to "hive"
|
||||
**options: Additional options
|
||||
|
||||
Returns:
|
||||
ParquetData: Ingested dataset data and metadata
|
||||
"""
|
||||
directory_path = Path(directory_path)
|
||||
parquet_files = self._validate_directory(directory_path)
|
||||
if limit is not None and limit < 0:
|
||||
raise ValidationError("limit must be greater than or equal to 0")
|
||||
|
||||
tracking_id = self.progress_tracker.start_tracking(
|
||||
file=str(directory_path),
|
||||
module="ingest",
|
||||
submodule="ParquetIngestor",
|
||||
message=f"Ingesting Parquet directory: {directory_path.name}",
|
||||
)
|
||||
|
||||
try:
|
||||
dataset = ds.dataset(
|
||||
str(directory_path),
|
||||
format="parquet",
|
||||
partitioning=partitioning,
|
||||
)
|
||||
selected_columns = self._normalize_columns(
|
||||
columns,
|
||||
[field.name for field in dataset.schema],
|
||||
)
|
||||
filter_expression = self._dataset_filter(filters)
|
||||
metadata = self._directory_metadata(
|
||||
directory_path,
|
||||
parquet_files,
|
||||
partitioning=partitioning,
|
||||
)
|
||||
|
||||
if include_data:
|
||||
if limit is not None:
|
||||
table = dataset.head(
|
||||
limit,
|
||||
columns=selected_columns,
|
||||
filter=filter_expression,
|
||||
)
|
||||
else:
|
||||
table = dataset.to_table(
|
||||
columns=selected_columns,
|
||||
filter=filter_expression,
|
||||
)
|
||||
data = table.to_pylist()
|
||||
schema = self._schema_to_dict(table.schema)
|
||||
result_columns = list(table.column_names)
|
||||
else:
|
||||
selected_schema = self._select_schema(dataset.schema, selected_columns)
|
||||
data = []
|
||||
schema = self._schema_to_dict(selected_schema)
|
||||
result_columns = [field.name for field in selected_schema]
|
||||
|
||||
metadata.update(
|
||||
{
|
||||
"returned_rows": len(data),
|
||||
"selected_columns": result_columns,
|
||||
"filters_applied": filters is not None,
|
||||
"limit": limit,
|
||||
"include_data": include_data,
|
||||
}
|
||||
)
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
message=f"Ingested Parquet directory: {len(data)} rows",
|
||||
)
|
||||
|
||||
self.logger.info(
|
||||
"Parquet directory ingestion completed: "
|
||||
f"{len(data)} row(s) from {directory_path}"
|
||||
)
|
||||
|
||||
return ParquetData(
|
||||
data=data,
|
||||
row_count=len(data),
|
||||
columns=result_columns,
|
||||
schema=schema,
|
||||
source=str(directory_path),
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
except (ValidationError, ProcessingError):
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="failed",
|
||||
message="Parquet directory ingestion failed",
|
||||
)
|
||||
raise
|
||||
except Exception as e:
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id, status="failed", message=str(e)
|
||||
)
|
||||
self.logger.error(
|
||||
f"Failed to ingest Parquet directory {directory_path}: {e}"
|
||||
)
|
||||
raise ProcessingError(f"Failed to ingest Parquet directory: {e}") from e
|
||||
|
||||
def read_columns(
|
||||
self,
|
||||
source: Union[str, Path],
|
||||
columns: Union[str, Sequence[str]],
|
||||
**options,
|
||||
) -> ParquetData:
|
||||
"""
|
||||
Read selected columns from a Parquet file or directory.
|
||||
|
||||
Args:
|
||||
source: Parquet file or directory path
|
||||
columns: Column name or names to read
|
||||
**options: Additional ingestion options
|
||||
|
||||
Returns:
|
||||
ParquetData: Ingested data containing only selected columns
|
||||
"""
|
||||
return self.ingest(source, columns=columns, **options)
|
||||
|
||||
def extract_schema(self, source: Union[str, Path], **options) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract schema from a Parquet file or directory.
|
||||
|
||||
Args:
|
||||
source: Parquet file or directory path
|
||||
**options: Additional options
|
||||
|
||||
Returns:
|
||||
dict: Schema with column names, types, nullability, and metadata
|
||||
"""
|
||||
source_path = Path(source)
|
||||
if source_path.is_dir():
|
||||
self._validate_directory(source_path)
|
||||
dataset = ds.dataset(
|
||||
str(source_path),
|
||||
format="parquet",
|
||||
partitioning=options.get("partitioning", "hive"),
|
||||
)
|
||||
return self._schema_to_dict(dataset.schema)
|
||||
|
||||
self._validate_file(source_path)
|
||||
parquet_file = pq.ParquetFile(str(source_path))
|
||||
return self._schema_to_dict(parquet_file.schema_arrow)
|
||||
|
||||
def extract_metadata(self, source: Union[str, Path], **options) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract Parquet file or directory metadata without reading row data.
|
||||
|
||||
Args:
|
||||
source: Parquet file or directory path
|
||||
**options: Additional options
|
||||
|
||||
Returns:
|
||||
dict: Row counts, row groups, compression, partitions, and file info
|
||||
"""
|
||||
source_path = Path(source)
|
||||
if source_path.is_dir():
|
||||
parquet_files = self._validate_directory(source_path)
|
||||
return self._directory_metadata(
|
||||
source_path,
|
||||
parquet_files,
|
||||
partitioning=options.get("partitioning", "hive"),
|
||||
)
|
||||
|
||||
self._validate_file(source_path)
|
||||
parquet_file = pq.ParquetFile(str(source_path))
|
||||
return self._file_metadata(source_path, parquet_file)
|
||||
|
||||
def _read_file_table(
|
||||
self,
|
||||
file_path: Path,
|
||||
parquet_file: Any,
|
||||
columns: Optional[List[str]],
|
||||
limit: Optional[int],
|
||||
filters: Any,
|
||||
batch_size: Optional[int],
|
||||
) -> Any:
|
||||
"""Read a Parquet file, using batches when a simple limit is requested."""
|
||||
if limit is not None and limit < 0:
|
||||
raise ValidationError("limit must be greater than or equal to 0")
|
||||
|
||||
if limit == 0:
|
||||
return pa.Table.from_batches(
|
||||
[],
|
||||
schema=self._select_schema(parquet_file.schema_arrow, columns),
|
||||
)
|
||||
|
||||
if limit is not None and filters is None:
|
||||
return self._read_file_limited(parquet_file, columns, limit, batch_size)
|
||||
|
||||
table = pq.read_table(str(file_path), columns=columns, filters=filters)
|
||||
if limit is not None:
|
||||
table = table.slice(0, limit)
|
||||
return table
|
||||
|
||||
def _read_file_limited(
|
||||
self,
|
||||
parquet_file: Any,
|
||||
columns: Optional[List[str]],
|
||||
limit: int,
|
||||
batch_size: Optional[int],
|
||||
) -> Any:
|
||||
"""Read at most ``limit`` rows from a file without loading the full file."""
|
||||
batches = []
|
||||
remaining = limit
|
||||
effective_batch_size = batch_size or min(max(limit, 1), 65_536)
|
||||
|
||||
for batch in parquet_file.iter_batches(
|
||||
batch_size=effective_batch_size,
|
||||
columns=columns,
|
||||
):
|
||||
if batch.num_rows > remaining:
|
||||
batch = batch.slice(0, remaining)
|
||||
batches.append(batch)
|
||||
remaining -= batch.num_rows
|
||||
if remaining <= 0:
|
||||
break
|
||||
|
||||
return pa.Table.from_batches(
|
||||
batches,
|
||||
schema=self._select_schema(parquet_file.schema_arrow, columns),
|
||||
)
|
||||
|
||||
def _validate_file(self, file_path: Path) -> None:
|
||||
"""Validate a local Parquet file path."""
|
||||
if not file_path.exists():
|
||||
raise ValidationError(f"Parquet file not found: {file_path}")
|
||||
if not file_path.is_file():
|
||||
raise ValidationError(f"Path is not a file: {file_path}")
|
||||
if file_path.suffix.lower() not in {".parquet", ".pq"}:
|
||||
raise ValidationError(f"File is not a Parquet file: {file_path}")
|
||||
|
||||
def _validate_directory(self, directory_path: Path) -> List[Path]:
|
||||
"""Validate a Parquet directory and return contained Parquet files."""
|
||||
if not directory_path.exists():
|
||||
raise ValidationError(f"Parquet directory not found: {directory_path}")
|
||||
if not directory_path.is_dir():
|
||||
raise ValidationError(f"Path is not a directory: {directory_path}")
|
||||
|
||||
parquet_files = self._parquet_files(directory_path)
|
||||
if not parquet_files:
|
||||
raise ValidationError(
|
||||
f"No Parquet files found in directory: {directory_path}"
|
||||
)
|
||||
return parquet_files
|
||||
|
||||
def _parquet_files(self, directory_path: Path) -> List[Path]:
|
||||
"""Return Parquet files under a directory."""
|
||||
return sorted(
|
||||
path
|
||||
for path in directory_path.rglob("*")
|
||||
if path.is_file() and path.suffix.lower() in {".parquet", ".pq"}
|
||||
)
|
||||
|
||||
def _normalize_columns(
|
||||
self,
|
||||
columns: Optional[Union[str, Sequence[str]]],
|
||||
available_columns: Sequence[str],
|
||||
) -> Optional[List[str]]:
|
||||
"""Normalize and validate optional selected columns."""
|
||||
if columns is None:
|
||||
configured_columns = self.config.get("columns")
|
||||
if configured_columns is None:
|
||||
return None
|
||||
columns = configured_columns
|
||||
|
||||
if isinstance(columns, str):
|
||||
normalized = [columns]
|
||||
else:
|
||||
normalized = list(columns)
|
||||
|
||||
missing = [column for column in normalized if column not in available_columns]
|
||||
if missing:
|
||||
raise ValidationError(
|
||||
"Column(s) not found in Parquet schema: "
|
||||
f"{', '.join(missing)}. Available columns: "
|
||||
f"{', '.join(available_columns)}"
|
||||
)
|
||||
return normalized
|
||||
|
||||
def _select_schema(self, schema: Any, columns: Optional[List[str]]) -> Any:
|
||||
"""Return schema limited to selected columns when provided."""
|
||||
if columns is None:
|
||||
return schema
|
||||
fields = [schema.field(column) for column in columns]
|
||||
return pa.schema(fields, metadata=schema.metadata)
|
||||
|
||||
def _schema_to_dict(self, schema: Any) -> Dict[str, Any]:
|
||||
"""Convert PyArrow schema to serializable metadata."""
|
||||
fields = []
|
||||
for schema_field in schema:
|
||||
fields.append(
|
||||
{
|
||||
"name": schema_field.name,
|
||||
"type": str(schema_field.type),
|
||||
"nullable": schema_field.nullable,
|
||||
"metadata": self._decode_metadata_map(schema_field.metadata),
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"columns": [field_info["name"] for field_info in fields],
|
||||
"fields": fields,
|
||||
"metadata": self._decode_metadata_map(schema.metadata),
|
||||
}
|
||||
|
||||
def _file_metadata(self, file_path: Path, parquet_file: Any) -> Dict[str, Any]:
|
||||
"""Extract metadata for a single Parquet file."""
|
||||
metadata = parquet_file.metadata
|
||||
compression_by_column = self._compression_by_column(metadata)
|
||||
|
||||
return {
|
||||
"format": "parquet",
|
||||
"source_type": "file",
|
||||
"file": str(file_path),
|
||||
"file_size": file_path.stat().st_size,
|
||||
"total_rows": metadata.num_rows,
|
||||
"row_groups": metadata.num_row_groups,
|
||||
"created_by": metadata.created_by,
|
||||
"format_version": getattr(metadata, "format_version", None),
|
||||
"serialized_size": getattr(metadata, "serialized_size", None),
|
||||
"schema_metadata": self._decode_metadata_map(metadata.metadata),
|
||||
"compression": {
|
||||
column: sorted(codecs)
|
||||
for column, codecs in compression_by_column.items()
|
||||
},
|
||||
"compression_codecs": sorted(
|
||||
{codec for codecs in compression_by_column.values() for codec in codecs}
|
||||
),
|
||||
}
|
||||
|
||||
def _directory_metadata(
|
||||
self,
|
||||
directory_path: Path,
|
||||
parquet_files: Sequence[Path],
|
||||
partitioning: Optional[Union[str, Any]],
|
||||
) -> Dict[str, Any]:
|
||||
"""Extract aggregate metadata for a Parquet directory."""
|
||||
file_entries = []
|
||||
total_rows = 0
|
||||
total_row_groups = 0
|
||||
compression_by_column: Dict[str, set] = {}
|
||||
partition_columns = set()
|
||||
partition_values: Dict[str, set] = {}
|
||||
|
||||
for parquet_path in parquet_files:
|
||||
parquet_file = pq.ParquetFile(str(parquet_path))
|
||||
file_metadata = self._file_metadata(parquet_path, parquet_file)
|
||||
partitions = self._partition_values(directory_path, parquet_path)
|
||||
|
||||
total_rows += file_metadata["total_rows"]
|
||||
total_row_groups += file_metadata["row_groups"]
|
||||
for column, codecs in file_metadata["compression"].items():
|
||||
compression_by_column.setdefault(column, set()).update(codecs)
|
||||
|
||||
for key, value in partitions.items():
|
||||
partition_columns.add(key)
|
||||
partition_values.setdefault(key, set()).add(value)
|
||||
|
||||
file_entries.append(
|
||||
{
|
||||
"path": str(parquet_path),
|
||||
"relative_path": str(parquet_path.relative_to(directory_path)),
|
||||
"rows": file_metadata["total_rows"],
|
||||
"row_groups": file_metadata["row_groups"],
|
||||
"file_size": file_metadata["file_size"],
|
||||
"partitions": partitions,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"format": "parquet",
|
||||
"source_type": "directory",
|
||||
"directory": str(directory_path),
|
||||
"file_count": len(parquet_files),
|
||||
"files": file_entries,
|
||||
"total_rows": total_rows,
|
||||
"row_groups": total_row_groups,
|
||||
"partitioning": partitioning,
|
||||
"partition_columns": sorted(partition_columns),
|
||||
"partition_values": {
|
||||
key: sorted(values) for key, values in partition_values.items()
|
||||
},
|
||||
"compression": {
|
||||
column: sorted(codecs)
|
||||
for column, codecs in compression_by_column.items()
|
||||
},
|
||||
"compression_codecs": sorted(
|
||||
{codec for codecs in compression_by_column.values() for codec in codecs}
|
||||
),
|
||||
}
|
||||
|
||||
def _compression_by_column(self, metadata: Any) -> Dict[str, set]:
|
||||
"""Return compression codecs used for each column across row groups."""
|
||||
compression_by_column: Dict[str, set] = {}
|
||||
for row_group_index in range(metadata.num_row_groups):
|
||||
row_group = metadata.row_group(row_group_index)
|
||||
for column_index in range(row_group.num_columns):
|
||||
column_chunk = row_group.column(column_index)
|
||||
column_name = column_chunk.path_in_schema
|
||||
compression = str(column_chunk.compression)
|
||||
compression_by_column.setdefault(column_name, set()).add(compression)
|
||||
return compression_by_column
|
||||
|
||||
def _partition_values(self, root: Path, parquet_path: Path) -> Dict[str, str]:
|
||||
"""Extract Hive-style partition key/value pairs from a file path."""
|
||||
partitions = {}
|
||||
relative_parent = parquet_path.parent.relative_to(root)
|
||||
for part in relative_parent.parts:
|
||||
if "=" not in part:
|
||||
continue
|
||||
key, value = part.split("=", 1)
|
||||
if key:
|
||||
partitions[key] = value
|
||||
return partitions
|
||||
|
||||
def _decode_metadata_map(
|
||||
self, metadata: Optional[Dict[Any, Any]]
|
||||
) -> Dict[str, str]:
|
||||
"""Decode PyArrow metadata bytes to strings."""
|
||||
if not metadata:
|
||||
return {}
|
||||
|
||||
decoded = {}
|
||||
for key, value in metadata.items():
|
||||
decoded[self._decode_metadata_value(key)] = self._decode_metadata_value(
|
||||
value
|
||||
)
|
||||
return decoded
|
||||
|
||||
def _decode_metadata_value(self, value: Any) -> str:
|
||||
"""Decode a metadata key or value."""
|
||||
if isinstance(value, bytes):
|
||||
return value.decode("utf-8", errors="replace")
|
||||
return str(value)
|
||||
|
||||
def _dataset_filter(self, filters: Any) -> Any:
|
||||
"""Convert simple tuple filters to a PyArrow dataset expression."""
|
||||
if filters is None:
|
||||
return None
|
||||
if self._is_filter_tuple(filters):
|
||||
return self._comparison_expression(*filters)
|
||||
if isinstance(filters, list):
|
||||
if all(self._is_filter_tuple(item) for item in filters):
|
||||
return self._and_expressions(
|
||||
self._comparison_expression(*item) for item in filters
|
||||
)
|
||||
if all(isinstance(group, list) for group in filters):
|
||||
return self._or_expressions(
|
||||
self._and_expressions(
|
||||
self._comparison_expression(*item) for item in group
|
||||
)
|
||||
for group in filters
|
||||
)
|
||||
return filters
|
||||
|
||||
def _is_filter_tuple(self, value: Any) -> bool:
|
||||
"""Return whether value is a simple (column, operator, value) filter."""
|
||||
return (
|
||||
isinstance(value, tuple)
|
||||
and len(value) == 3
|
||||
and isinstance(value[0], str)
|
||||
and isinstance(value[1], str)
|
||||
)
|
||||
|
||||
def _comparison_expression(self, column: str, operator: str, value: Any) -> Any:
|
||||
"""Create a PyArrow dataset comparison expression."""
|
||||
field = ds.field(column)
|
||||
if operator in {"=", "=="}:
|
||||
return field == value
|
||||
if operator == "!=":
|
||||
return field != value
|
||||
if operator == ">":
|
||||
return field > value
|
||||
if operator == ">=":
|
||||
return field >= value
|
||||
if operator == "<":
|
||||
return field < value
|
||||
if operator == "<=":
|
||||
return field <= value
|
||||
if operator.lower() == "in":
|
||||
return field.isin(value)
|
||||
if operator.lower() in {"not in", "not_in"}:
|
||||
return ~field.isin(value)
|
||||
raise ValidationError(f"Unsupported Parquet filter operator: {operator}")
|
||||
|
||||
def _and_expressions(self, expressions: Iterable[Any]) -> Any:
|
||||
"""Combine expressions with AND."""
|
||||
expression_list = list(expressions)
|
||||
if not expression_list:
|
||||
return None
|
||||
combined = expression_list[0]
|
||||
for expression in expression_list[1:]:
|
||||
combined = combined & expression
|
||||
return combined
|
||||
|
||||
def _or_expressions(self, expressions: Iterable[Any]) -> Any:
|
||||
"""Combine expressions with OR."""
|
||||
expression_list = [expr for expr in expressions if expr is not None]
|
||||
if not expression_list:
|
||||
return None
|
||||
combined = expression_list[0]
|
||||
for expression in expression_list[1:]:
|
||||
combined = combined | expression
|
||||
return combined
|
||||
@@ -13,6 +13,7 @@ Supported Registration Types:
|
||||
* "repo": Repository ingestion methods
|
||||
* "email": Email ingestion methods
|
||||
* "db": Database ingestion methods
|
||||
* "parquet": Parquet file and dataset ingestion methods
|
||||
* "ingest": General ingestion methods
|
||||
|
||||
Algorithms Used:
|
||||
@@ -24,7 +25,7 @@ Algorithms Used:
|
||||
|
||||
Key Features:
|
||||
- Method registry for custom ingestion methods
|
||||
- Task-based method organization (file, web, feed, stream, repo, email, db, ingest)
|
||||
- Task-based method organization by source category
|
||||
- Dynamic registration and unregistration
|
||||
- Easy discovery of available methods
|
||||
- Support for community-contributed extensions
|
||||
@@ -37,11 +38,13 @@ Global Instances:
|
||||
|
||||
Example Usage:
|
||||
>>> from semantica.ingest.registry import method_registry
|
||||
>>> method_registry.register("file", "custom_method", custom_file_ingestion_function)
|
||||
>>> method_registry.register(
|
||||
... "file", "custom_method", custom_file_ingestion_function
|
||||
... )
|
||||
>>> available = method_registry.list_all("file")
|
||||
"""
|
||||
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
from typing import Callable, Dict, List, Optional
|
||||
|
||||
|
||||
class MethodRegistry:
|
||||
@@ -56,6 +59,7 @@ class MethodRegistry:
|
||||
"email": {},
|
||||
"db": {},
|
||||
"mcp": {},
|
||||
"parquet": {},
|
||||
"ingest": {},
|
||||
}
|
||||
|
||||
@@ -65,7 +69,8 @@ class MethodRegistry:
|
||||
Register a custom ingestion method.
|
||||
|
||||
Args:
|
||||
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
|
||||
task: Task type such as "file", "web", "feed", "stream",
|
||||
"repo", "email", "db", "mcp", "parquet", or "ingest"
|
||||
name: Method name
|
||||
method_func: Method function
|
||||
"""
|
||||
@@ -79,7 +84,8 @@ class MethodRegistry:
|
||||
Get method by task and name.
|
||||
|
||||
Args:
|
||||
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
|
||||
task: Task type such as "file", "web", "feed", "stream",
|
||||
"repo", "email", "db", "mcp", "parquet", or "ingest"
|
||||
name: Method name
|
||||
|
||||
Returns:
|
||||
@@ -108,7 +114,8 @@ class MethodRegistry:
|
||||
Unregister a method.
|
||||
|
||||
Args:
|
||||
task: Task type ("file", "web", "feed", "stream", "repo", "email", "db", "mcp", "ingest")
|
||||
task: Task type such as "file", "web", "feed", "stream",
|
||||
"repo", "email", "db", "mcp", "parquet", or "ingest"
|
||||
name: Method name
|
||||
"""
|
||||
if task in cls._methods and name in cls._methods[task]:
|
||||
|
||||
@@ -11,6 +11,15 @@ Usage
|
||||
Configure in your tool's MCP settings:
|
||||
|
||||
Claude Desktop / Windsurf / Cline / Continue / VS Code:
|
||||
{
|
||||
"mcpServers": {
|
||||
"semantica": {
|
||||
"command": "semantica-mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Or using python -m:
|
||||
{
|
||||
"mcpServers": {
|
||||
"semantica": {
|
||||
@@ -20,7 +29,9 @@ Configure in your tool's MCP settings:
|
||||
}
|
||||
}
|
||||
|
||||
Run directly:
|
||||
Run directly for testing:
|
||||
semantica-mcp
|
||||
# or
|
||||
python -m semantica.mcp_server
|
||||
|
||||
Environment variables:
|
||||
@@ -71,9 +82,7 @@ def _tool_extract_entities(args: dict) -> dict:
|
||||
if not text:
|
||||
return {"error": "text is required"}
|
||||
from semantica.semantic_extract import NamedEntityRecognizer
|
||||
from semantica.semantic_extract.cache import _result_cache
|
||||
_result_cache.clear()
|
||||
entities = NamedEntityRecognizer().extract(text)
|
||||
entities = NamedEntityRecognizer().extract_entities(text)
|
||||
return {
|
||||
"entities": [
|
||||
{"label": getattr(e, "label", str(e)),
|
||||
@@ -91,10 +100,8 @@ def _tool_extract_relations(args: dict) -> dict:
|
||||
if not text:
|
||||
return {"error": "text is required"}
|
||||
from semantica.semantic_extract import RelationExtractor, TripletExtractor
|
||||
from semantica.semantic_extract.cache import _result_cache
|
||||
_result_cache.clear()
|
||||
relations = RelationExtractor().extract(text)
|
||||
triplets = TripletExtractor().extract(text)
|
||||
relations = RelationExtractor().extract_relations(text)
|
||||
triplets = TripletExtractor().extract_triplets(text)
|
||||
return {
|
||||
"relations": [
|
||||
{"source": getattr(r, "source", None),
|
||||
@@ -600,7 +607,3 @@ def _run_stdio():
|
||||
|
||||
def main():
|
||||
_run_stdio()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,8 @@
|
||||
"""
|
||||
Entry point for python -m semantica.mcp_server
|
||||
"""
|
||||
|
||||
from semantica.mcp_server import main
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -47,73 +47,97 @@ Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
from __future__ import annotations
|
||||
|
||||
from .config import Config, config
|
||||
from .coreference_resolver import (
|
||||
CoreferenceChain,
|
||||
CoreferenceChainBuilder,
|
||||
CoreferenceResolver,
|
||||
EntityCoreferenceDetector,
|
||||
Mention,
|
||||
PronounResolver,
|
||||
)
|
||||
from .event_detector import (
|
||||
Event,
|
||||
EventClassifier,
|
||||
EventDetector,
|
||||
EventRelationshipExtractor,
|
||||
TemporalEventProcessor,
|
||||
)
|
||||
from .extraction_validator import ExtractionValidator, ValidationResult
|
||||
from .llm_extraction import LLMEnhancer, LLMExtraction, LLMResponse
|
||||
from .methods import get_entity_method, get_relation_method, get_triplet_method
|
||||
from .named_entity_recognizer import (
|
||||
CustomEntityDetector,
|
||||
EntityClassifier,
|
||||
EntityConfidenceScorer,
|
||||
NamedEntityRecognizer,
|
||||
)
|
||||
from .ner_extractor import Entity, NERExtractor
|
||||
from .providers import (
|
||||
AnthropicProvider,
|
||||
BaseProvider,
|
||||
GeminiProvider,
|
||||
GroqProvider,
|
||||
HuggingFaceLLMProvider,
|
||||
HuggingFaceModelLoader,
|
||||
OllamaProvider,
|
||||
OpenAIProvider,
|
||||
create_provider,
|
||||
)
|
||||
from .registry import (
|
||||
MethodRegistry,
|
||||
ProviderRegistry,
|
||||
method_registry,
|
||||
provider_registry,
|
||||
)
|
||||
from .relation_extractor import Relation, RelationExtractor
|
||||
from .semantic_analyzer import (
|
||||
RoleLabeler,
|
||||
SemanticAnalyzer,
|
||||
SemanticCluster,
|
||||
SemanticClusterer,
|
||||
SemanticRole,
|
||||
SimilarityAnalyzer,
|
||||
)
|
||||
from .semantic_network_extractor import (
|
||||
SemanticEdge,
|
||||
SemanticNetwork,
|
||||
SemanticNetworkExtractor,
|
||||
SemanticNode,
|
||||
)
|
||||
from .triplet_extractor import (
|
||||
RDFSerializer,
|
||||
Triplet,
|
||||
TripletExtractor,
|
||||
TripletQualityChecker,
|
||||
TripletValidator,
|
||||
)
|
||||
import importlib
|
||||
from typing import Any, Dict, Tuple
|
||||
|
||||
|
||||
_LAZY_EXPORTS: Dict[str, Tuple[str, str]] = {
|
||||
# Named Entity Recognition
|
||||
"NamedEntityRecognizer": (".named_entity_recognizer", "NamedEntityRecognizer"),
|
||||
"EntityClassifier": (".named_entity_recognizer", "EntityClassifier"),
|
||||
"EntityConfidenceScorer": (".named_entity_recognizer", "EntityConfidenceScorer"),
|
||||
"CustomEntityDetector": (".named_entity_recognizer", "CustomEntityDetector"),
|
||||
"NERExtractor": (".ner_extractor", "NERExtractor"),
|
||||
"Entity": (".types", "Entity"),
|
||||
# Relation Extraction
|
||||
"RelationExtractor": (".relation_extractor", "RelationExtractor"),
|
||||
"Relation": (".types", "Relation"),
|
||||
# Event Detection
|
||||
"EventDetector": (".event_detector", "EventDetector"),
|
||||
"Event": (".event_detector", "Event"),
|
||||
"EventClassifier": (".event_detector", "EventClassifier"),
|
||||
"TemporalEventProcessor": (".event_detector", "TemporalEventProcessor"),
|
||||
"EventRelationshipExtractor": (".event_detector", "EventRelationshipExtractor"),
|
||||
# Coreference Resolution
|
||||
"CoreferenceResolver": (".coreference_resolver", "CoreferenceResolver"),
|
||||
"Mention": (".coreference_resolver", "Mention"),
|
||||
"CoreferenceChain": (".coreference_resolver", "CoreferenceChain"),
|
||||
"PronounResolver": (".coreference_resolver", "PronounResolver"),
|
||||
"EntityCoreferenceDetector": (".coreference_resolver", "EntityCoreferenceDetector"),
|
||||
"CoreferenceChainBuilder": (".coreference_resolver", "CoreferenceChainBuilder"),
|
||||
# Triplet Extraction
|
||||
"TripletExtractor": (".triplet_extractor", "TripletExtractor"),
|
||||
"TripleExtractor": (".triplet_extractor", "TripletExtractor"),
|
||||
"Triplet": (".types", "Triplet"),
|
||||
"TripletValidator": (".triplet_extractor", "TripletValidator"),
|
||||
"RDFSerializer": (".triplet_extractor", "RDFSerializer"),
|
||||
"TripletQualityChecker": (".triplet_extractor", "TripletQualityChecker"),
|
||||
# Semantic Analysis
|
||||
"SemanticAnalyzer": (".semantic_analyzer", "SemanticAnalyzer"),
|
||||
"SemanticRole": (".semantic_analyzer", "SemanticRole"),
|
||||
"SemanticCluster": (".semantic_analyzer", "SemanticCluster"),
|
||||
"SimilarityAnalyzer": (".semantic_analyzer", "SimilarityAnalyzer"),
|
||||
"RoleLabeler": (".semantic_analyzer", "RoleLabeler"),
|
||||
"SemanticClusterer": (".semantic_analyzer", "SemanticClusterer"),
|
||||
# Semantic Network
|
||||
"SemanticNetworkExtractor": (".semantic_network_extractor", "SemanticNetworkExtractor"),
|
||||
"SemanticNode": (".semantic_network_extractor", "SemanticNode"),
|
||||
"SemanticEdge": (".semantic_network_extractor", "SemanticEdge"),
|
||||
"SemanticNetwork": (".semantic_network_extractor", "SemanticNetwork"),
|
||||
# LLM Enhancement
|
||||
"LLMExtraction": (".llm_extraction", "LLMExtraction"),
|
||||
"LLMEnhancer": (".llm_extraction", "LLMEnhancer"),
|
||||
"LLMResponse": (".llm_extraction", "LLMResponse"),
|
||||
# Validation
|
||||
"ExtractionValidator": (".extraction_validator", "ExtractionValidator"),
|
||||
"ValidationResult": (".extraction_validator", "ValidationResult"),
|
||||
# Providers
|
||||
"BaseProvider": (".providers", "BaseProvider"),
|
||||
"OpenAIProvider": (".providers", "OpenAIProvider"),
|
||||
"GeminiProvider": (".providers", "GeminiProvider"),
|
||||
"GroqProvider": (".providers", "GroqProvider"),
|
||||
"AnthropicProvider": (".providers", "AnthropicProvider"),
|
||||
"OllamaProvider": (".providers", "OllamaProvider"),
|
||||
"HuggingFaceLLMProvider": (".providers", "HuggingFaceLLMProvider"),
|
||||
"HuggingFaceModelLoader": (".providers", "HuggingFaceModelLoader"),
|
||||
"create_provider": (".providers", "create_provider"),
|
||||
# Registry
|
||||
"ProviderRegistry": (".registry", "ProviderRegistry"),
|
||||
"MethodRegistry": (".registry", "MethodRegistry"),
|
||||
"provider_registry": (".registry", "provider_registry"),
|
||||
"method_registry": (".registry", "method_registry"),
|
||||
# Config
|
||||
"Config": (".config", "Config"),
|
||||
"config": (".config", "config"),
|
||||
# Methods
|
||||
"get_entity_method": (".methods", "get_entity_method"),
|
||||
"get_relation_method": (".methods", "get_relation_method"),
|
||||
"get_triplet_method": (".methods", "get_triplet_method"),
|
||||
}
|
||||
|
||||
|
||||
def __getattr__(name: str) -> Any:
|
||||
"""Lazily load semantic extraction exports to avoid import cycles and optional deps."""
|
||||
if name not in _LAZY_EXPORTS:
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
|
||||
module_name, attr_name = _LAZY_EXPORTS[name]
|
||||
module = importlib.import_module(module_name, __name__)
|
||||
value = getattr(module, attr_name)
|
||||
globals()[name] = value
|
||||
return value
|
||||
|
||||
__all__ = [
|
||||
# Named Entity Recognition
|
||||
|
||||
@@ -113,13 +113,11 @@ from typing import Any, Dict, List, Optional, Tuple, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError
|
||||
from ..utils.logging import get_logger
|
||||
from .ner_extractor import Entity
|
||||
from .providers import HuggingFaceModelLoader, create_provider
|
||||
from .registry import method_registry
|
||||
from .relation_extractor import Relation
|
||||
from .triplet_extractor import Triplet
|
||||
from .cache import ExtractionCache
|
||||
from .config import config
|
||||
from .types import Entity, Relation, Triplet
|
||||
|
||||
try:
|
||||
from .schemas import (
|
||||
|
||||
@@ -70,29 +70,17 @@ Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError
|
||||
from ..utils.helpers import safe_import
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .types import Entity
|
||||
|
||||
spacy, SPACY_AVAILABLE = safe_import("spacy")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entity:
|
||||
"""Entity representation."""
|
||||
|
||||
text: str
|
||||
label: str
|
||||
start_char: int
|
||||
end_char: int
|
||||
confidence: float = 1.0
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class NERExtractor:
|
||||
"""Named Entity Recognition extractor."""
|
||||
|
||||
|
||||
@@ -70,25 +70,12 @@ License: MIT
|
||||
"""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union
|
||||
|
||||
from ..utils.exceptions import ProcessingError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .ner_extractor import Entity
|
||||
|
||||
|
||||
@dataclass
|
||||
class Relation:
|
||||
"""Relation representation."""
|
||||
|
||||
subject: Entity
|
||||
predicate: str
|
||||
object: Entity
|
||||
confidence: float = 1.0
|
||||
context: str = ""
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
from .types import Entity, Relation
|
||||
|
||||
|
||||
class RelationExtractor:
|
||||
|
||||
@@ -70,34 +70,13 @@ Author: Semantica Contributors
|
||||
License: MIT
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
from urllib.parse import quote
|
||||
|
||||
from ..utils.exceptions import ProcessingError, ValidationError
|
||||
from ..utils.logging import get_logger
|
||||
from ..utils.progress_tracker import get_progress_tracker
|
||||
from .ner_extractor import Entity
|
||||
from .relation_extractor import Relation
|
||||
|
||||
|
||||
@dataclass
|
||||
class Triplet:
|
||||
"""RDF triplet representation."""
|
||||
|
||||
subject: str
|
||||
predicate: str
|
||||
object: str
|
||||
confidence: float = 1.0
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def get(self, key: str, default: Any = None) -> Any:
|
||||
"""Get attribute value like a dictionary."""
|
||||
return getattr(self, key, default)
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
"""Get item like a dictionary."""
|
||||
return getattr(self, key)
|
||||
from .types import Entity, Relation, Triplet
|
||||
|
||||
|
||||
class TripletExtractor:
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
"""
|
||||
Shared semantic extraction data types.
|
||||
|
||||
These lightweight dataclasses live outside the extractor implementations so
|
||||
method dispatchers and extractors can share result models without import cycles.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entity:
|
||||
"""Entity representation."""
|
||||
|
||||
text: str
|
||||
label: str
|
||||
start_char: int
|
||||
end_char: int
|
||||
confidence: float = 1.0
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Relation:
|
||||
"""Relation representation."""
|
||||
|
||||
subject: Entity
|
||||
predicate: str
|
||||
object: Entity
|
||||
confidence: float = 1.0
|
||||
context: str = ""
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Triplet:
|
||||
"""RDF triplet representation."""
|
||||
|
||||
subject: str
|
||||
predicate: str
|
||||
object: str
|
||||
confidence: float = 1.0
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def get(self, key: str, default: Any = None) -> Any:
|
||||
"""Get attribute value like a dictionary."""
|
||||
return getattr(self, key, default)
|
||||
|
||||
def __getitem__(self, key: str) -> Any:
|
||||
"""Get item like a dictionary."""
|
||||
return getattr(self, key)
|
||||
+7
-5
@@ -152,10 +152,11 @@ if EXPLORER_AVAILABLE:
|
||||
enrich,
|
||||
export_import,
|
||||
graph,
|
||||
ontology,
|
||||
temporal,
|
||||
vocabulary,
|
||||
provenance,
|
||||
sparql
|
||||
provenance,
|
||||
sparql
|
||||
)
|
||||
|
||||
app.include_router(analytics.router)
|
||||
@@ -164,12 +165,13 @@ if EXPLORER_AVAILABLE:
|
||||
app.include_router(enrich.router)
|
||||
app.include_router(export_import.router)
|
||||
app.include_router(graph.router)
|
||||
app.include_router(ontology.router)
|
||||
app.include_router(temporal.router)
|
||||
app.include_router(vocabulary.router)
|
||||
app.include_router(provenance.router)
|
||||
app.include_router(sparql.router)
|
||||
app.include_router(provenance.router)
|
||||
app.include_router(sparql.router)
|
||||
|
||||
logging.info("Explorer, Vocabulary, SPARQL, and Provenance API routes successfully mounted.")
|
||||
logging.info("Explorer, Vocabulary, SPARQL, Provenance, and Ontology API routes successfully mounted.")
|
||||
except Exception as exc:
|
||||
logging.error(f"Failed to mount explorer routes: {exc}")
|
||||
else:
|
||||
|
||||
@@ -33,10 +33,10 @@ Example Usage:
|
||||
>>> from semantica.utils import SUPPORTED_DOCUMENT_FORMATS, DEFAULT_CONFIG
|
||||
>>> if file_extension in SUPPORTED_DOCUMENT_FORMATS:
|
||||
... process_document(file_path)
|
||||
>>>
|
||||
>>>
|
||||
>>> config = DEFAULT_CONFIG.copy()
|
||||
>>> config["processing"]["batch_size"] = 200
|
||||
>>>
|
||||
>>>
|
||||
>>> from semantica.utils import ERROR_CODES, PERFORMANCE_THRESHOLDS
|
||||
>>> error_code = ERROR_CODES["VALIDATION_ERROR"]
|
||||
>>> max_time = PERFORMANCE_THRESHOLDS["max_processing_time"]
|
||||
@@ -57,6 +57,8 @@ SUPPORTED_DOCUMENT_FORMATS = [
|
||||
"csv",
|
||||
"xlsx",
|
||||
"pptx",
|
||||
"parquet",
|
||||
"pq",
|
||||
]
|
||||
|
||||
SUPPORTED_IMAGE_FORMATS = ["jpg", "jpeg", "png", "gif", "bmp", "tiff", "webp", "svg"]
|
||||
|
||||
@@ -253,21 +253,21 @@ class ConsoleProgressDisplay(ProgressDisplay):
|
||||
# If we have pipeline items, show all of them
|
||||
if pipeline_items:
|
||||
# Clear and show all pipeline items
|
||||
sys.stdout.write("\r" + " " * 150 + "\r")
|
||||
|
||||
self._safe_write("\r" + " " * 150 + "\r")
|
||||
|
||||
# Show header if first time
|
||||
if not hasattr(self, '_pipeline_header_shown'):
|
||||
if self.use_emoji:
|
||||
sys.stdout.write("🧠 Semantica - 📊 Current Progress\n")
|
||||
self._safe_write("🧠 Semantica - 📊 Current Progress\n")
|
||||
else:
|
||||
sys.stdout.write("Semantica - Current Progress\n")
|
||||
sys.stdout.write("=" * 150 + "\n")
|
||||
self._safe_write("Semantica - Current Progress\n")
|
||||
self._safe_write("=" * 150 + "\n")
|
||||
self._pipeline_header_shown = True
|
||||
|
||||
|
||||
# Display all pipeline items
|
||||
for pipeline_item in pipeline_items:
|
||||
self._display_item_line(pipeline_item)
|
||||
sys.stdout.write("\n")
|
||||
self._safe_write("\n")
|
||||
|
||||
sys.stdout.flush()
|
||||
else:
|
||||
|
||||
@@ -1,13 +1,18 @@
|
||||
import sys
|
||||
import unittest
|
||||
from typing import Dict, Any, List
|
||||
from semantica.deduplication.similarity_calculator import SimilarityCalculator
|
||||
from semantica.deduplication.duplicate_detector import DuplicateDetector
|
||||
from semantica.deduplication.duplicate_detector import (
|
||||
DuplicateCandidate,
|
||||
DuplicateDetector,
|
||||
)
|
||||
from semantica.deduplication.entity_merger import EntityMerger
|
||||
from semantica.deduplication.merge_strategy import MergeStrategy
|
||||
from semantica.deduplication.cluster_builder import ClusterBuilder
|
||||
from semantica.deduplication.registry import MethodRegistry
|
||||
from semantica.deduplication.config import DeduplicationConfig
|
||||
from semantica.deduplication.methods import get_deduplication_method
|
||||
from semantica.utils.progress_tracker import ConsoleProgressDisplay
|
||||
|
||||
class TestDeduplication(unittest.TestCase):
|
||||
|
||||
@@ -203,5 +208,402 @@ class TestDeduplication(unittest.TestCase):
|
||||
self.assertIsNone(invalid)
|
||||
|
||||
|
||||
class TestProgressTrackerEncoding(unittest.TestCase):
|
||||
"""Regression tests for issue #531 — Unicode crash on cp1252 Windows consoles."""
|
||||
|
||||
def _make_cp1252_stdout(self):
|
||||
"""Return a stdout-like object that raises UnicodeEncodeError for non-cp1252 chars."""
|
||||
class CP1252Writer:
|
||||
encoding = "cp1252"
|
||||
def write(self, text):
|
||||
text.encode("cp1252") # raises on emoji / block chars
|
||||
def flush(self):
|
||||
pass
|
||||
return CP1252Writer()
|
||||
|
||||
def test_safe_write_does_not_crash_on_cp1252(self):
|
||||
"""_safe_write must not raise UnicodeEncodeError on a cp1252 console."""
|
||||
display = ConsoleProgressDisplay()
|
||||
orig = sys.stdout
|
||||
sys.stdout = self._make_cp1252_stdout()
|
||||
try:
|
||||
display._safe_write("🧠 Semantica - 📊 Current Progress\n")
|
||||
except UnicodeEncodeError:
|
||||
self.fail("_safe_write raised UnicodeEncodeError on cp1252 stdout")
|
||||
finally:
|
||||
sys.stdout = orig
|
||||
|
||||
def test_update_pipeline_header_does_not_crash_on_cp1252(self):
|
||||
"""update() pipeline header write must not crash on a cp1252 console (issue #531)."""
|
||||
from semantica.utils.progress_tracker import ProgressItem
|
||||
display = ConsoleProgressDisplay()
|
||||
display.use_emoji = True # force emoji path to exercise the fixed branch
|
||||
orig = sys.stdout
|
||||
sys.stdout = self._make_cp1252_stdout()
|
||||
try:
|
||||
display._safe_write("🧠 Semantica - 📊 Current Progress\n")
|
||||
display._safe_write("=" * 150 + "\n")
|
||||
except UnicodeEncodeError:
|
||||
self.fail("Pipeline header write raised UnicodeEncodeError on cp1252 stdout")
|
||||
finally:
|
||||
sys.stdout = orig
|
||||
|
||||
def test_emoji_detection_disables_on_cp1252(self):
|
||||
"""ConsoleProgressDisplay should auto-disable emoji when stdout is cp1252."""
|
||||
orig = sys.stdout
|
||||
sys.stdout = self._make_cp1252_stdout()
|
||||
try:
|
||||
display = ConsoleProgressDisplay()
|
||||
self.assertFalse(display.use_emoji, "use_emoji should be False on cp1252 stdout")
|
||||
finally:
|
||||
sys.stdout = orig
|
||||
|
||||
|
||||
class TestResultLimiting(unittest.TestCase):
|
||||
"""Tests for issue #534 — max_results, top_k_per_entity, min_similarity, sort_by."""
|
||||
|
||||
def setUp(self):
|
||||
# Six entities: three near-duplicate Apple variants + two Microsoft variants + one Google.
|
||||
# Lower thresholds so all intra-brand pairs clear the bar.
|
||||
self.entities = [
|
||||
{"id": "a1", "name": "Apple Inc.", "type": "Company",
|
||||
"properties": {"industry": "Technology"}},
|
||||
{"id": "a2", "name": "Apple", "type": "Company",
|
||||
"properties": {"industry": "Tech"}},
|
||||
{"id": "a3", "name": "Apple Corp", "type": "Company",
|
||||
"properties": {"industry": "Technology"}},
|
||||
{"id": "b1", "name": "Microsoft Corporation", "type": "Company",
|
||||
"properties": {"industry": "Software"}},
|
||||
{"id": "b2", "name": "Microsoft Corp", "type": "Company",
|
||||
"properties": {"industry": "Software"}},
|
||||
{"id": "c1", "name": "Google LLC", "type": "Company",
|
||||
"properties": {"industry": "Internet"}},
|
||||
]
|
||||
self.threshold = 0.3
|
||||
|
||||
def _base_detector(self, **kwargs):
|
||||
return DuplicateDetector(
|
||||
similarity_threshold=self.threshold,
|
||||
confidence_threshold=self.threshold,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# max_results
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_max_results_caps_output(self):
|
||||
detector = self._base_detector(max_results=1)
|
||||
results = detector.detect_duplicates(self.entities)
|
||||
self.assertLessEqual(len(results), 1)
|
||||
|
||||
def test_max_results_two(self):
|
||||
detector = self._base_detector(max_results=2)
|
||||
results = detector.detect_duplicates(self.entities)
|
||||
self.assertLessEqual(len(results), 2)
|
||||
|
||||
def test_max_results_none_no_cap(self):
|
||||
uncapped = self._base_detector()
|
||||
large_cap = self._base_detector(max_results=999)
|
||||
self.assertEqual(
|
||||
len(uncapped.detect_duplicates(self.entities)),
|
||||
len(large_cap.detect_duplicates(self.entities)),
|
||||
)
|
||||
|
||||
def test_max_results_zero_returns_empty(self):
|
||||
detector = self._base_detector(max_results=0)
|
||||
self.assertEqual(detector.detect_duplicates(self.entities), [])
|
||||
|
||||
def test_max_results_returns_highest_confidence_first(self):
|
||||
"""When capped, the kept candidates must be the highest-confidence ones."""
|
||||
n = 2
|
||||
all_results = self._base_detector().detect_duplicates(self.entities)
|
||||
capped = self._base_detector(max_results=n).detect_duplicates(self.entities)
|
||||
if len(all_results) >= n:
|
||||
expected_ids = {
|
||||
(c.entity1["id"], c.entity2["id"]) for c in all_results[:n]
|
||||
}
|
||||
actual_ids = {
|
||||
(c.entity1["id"], c.entity2["id"]) for c in capped
|
||||
}
|
||||
self.assertEqual(expected_ids, actual_ids)
|
||||
|
||||
def test_max_results_empty_input(self):
|
||||
detector = self._base_detector(max_results=5)
|
||||
self.assertEqual(detector.detect_duplicates([]), [])
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# top_k_per_entity
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_top_k_per_entity_k1(self):
|
||||
# OR semantics: keep if EITHER entity is under quota.
|
||||
# A popular entity can appear > k times (each new partner brings it back).
|
||||
# Invariant: no (entity1, entity2) pair is returned more than once.
|
||||
k = 1
|
||||
results = self._base_detector(top_k_per_entity=k).detect_duplicates(self.entities)
|
||||
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
|
||||
self.assertEqual(len(pairs), len(set(pairs)), "No duplicate pairs should appear")
|
||||
# OR gives >= results than AND — at least 1 result when matches exist
|
||||
and_results = self._base_detector().detect_duplicates(self.entities)
|
||||
if and_results:
|
||||
self.assertGreater(len(results), 0)
|
||||
|
||||
def test_top_k_per_entity_k2(self):
|
||||
# Same OR semantics: no pair appears twice; result is bounded below by k=1 count
|
||||
k = 2
|
||||
results = self._base_detector(top_k_per_entity=k).detect_duplicates(self.entities)
|
||||
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
|
||||
self.assertEqual(len(pairs), len(set(pairs)), "No duplicate pairs should appear")
|
||||
k1_results = self._base_detector(top_k_per_entity=1).detect_duplicates(self.entities)
|
||||
self.assertGreaterEqual(len(results), len(k1_results))
|
||||
|
||||
def test_top_k_per_entity_large_k_same_as_none(self):
|
||||
uncapped = self._base_detector().detect_duplicates(self.entities)
|
||||
large_k = self._base_detector(top_k_per_entity=999).detect_duplicates(self.entities)
|
||||
self.assertEqual(len(uncapped), len(large_k))
|
||||
|
||||
def test_top_k_per_entity_empty_input(self):
|
||||
detector = self._base_detector(top_k_per_entity=2)
|
||||
self.assertEqual(detector.detect_duplicates([]), [])
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# min_similarity
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_min_similarity_all_results_above_floor(self):
|
||||
floor = 0.6
|
||||
results = self._base_detector(min_similarity=floor).detect_duplicates(self.entities)
|
||||
for c in results:
|
||||
self.assertGreaterEqual(
|
||||
c.similarity_score, floor,
|
||||
f"Candidate score {c.similarity_score} is below min_similarity={floor}",
|
||||
)
|
||||
|
||||
def test_min_similarity_very_high_returns_only_exact(self):
|
||||
results = self._base_detector(min_similarity=1.0).detect_duplicates(self.entities)
|
||||
for c in results:
|
||||
self.assertEqual(c.similarity_score, 1.0)
|
||||
|
||||
def test_min_similarity_zero_does_not_over_filter(self):
|
||||
no_floor = self._base_detector().detect_duplicates(self.entities)
|
||||
zero_floor = self._base_detector(min_similarity=0.0).detect_duplicates(self.entities)
|
||||
self.assertEqual(len(no_floor), len(zero_floor))
|
||||
|
||||
def test_min_similarity_stricter_than_threshold_reduces_results(self):
|
||||
"""A min_similarity above similarity_threshold must not increase the result count."""
|
||||
base = self._base_detector().detect_duplicates(self.entities)
|
||||
stricter = self._base_detector(min_similarity=0.8).detect_duplicates(self.entities)
|
||||
self.assertLessEqual(len(stricter), len(base))
|
||||
|
||||
def test_min_similarity_empty_input(self):
|
||||
detector = self._base_detector(min_similarity=0.5)
|
||||
self.assertEqual(detector.detect_duplicates([]), [])
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# sort_by
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_sort_by_confidence_descending(self):
|
||||
results = self._base_detector(sort_by="confidence").detect_duplicates(self.entities)
|
||||
scores = [c.confidence for c in results]
|
||||
self.assertEqual(scores, sorted(scores, reverse=True))
|
||||
|
||||
def test_sort_by_similarity_score_descending(self):
|
||||
results = self._base_detector(sort_by="similarity_score").detect_duplicates(self.entities)
|
||||
scores = [c.similarity_score for c in results]
|
||||
self.assertEqual(scores, sorted(scores, reverse=True))
|
||||
|
||||
def test_sort_by_default_is_confidence(self):
|
||||
default = self._base_detector().detect_duplicates(self.entities)
|
||||
explicit = self._base_detector(sort_by="confidence").detect_duplicates(self.entities)
|
||||
self.assertEqual(
|
||||
[(c.entity1["id"], c.entity2["id"]) for c in default],
|
||||
[(c.entity1["id"], c.entity2["id"]) for c in explicit],
|
||||
)
|
||||
|
||||
def test_sort_by_invalid_raises_at_construction(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._base_detector(sort_by="bogus_field")
|
||||
|
||||
def test_sort_by_invalid_message_contains_field_name(self):
|
||||
with self.assertRaises(ValueError, msg="bogus_field") as ctx:
|
||||
self._base_detector(sort_by="bogus_field")
|
||||
self.assertIn("bogus_field", str(ctx.exception))
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Input validation (bug_002 / bug_003)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_max_results_negative_raises(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._base_detector(max_results=-1)
|
||||
|
||||
def test_max_results_float_raises(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._base_detector(max_results=1.5)
|
||||
|
||||
def test_top_k_per_entity_negative_raises(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._base_detector(top_k_per_entity=-1)
|
||||
|
||||
def test_top_k_per_entity_float_raises(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._base_detector(top_k_per_entity=2.5)
|
||||
|
||||
def test_min_similarity_above_1_raises(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._base_detector(min_similarity=1.1)
|
||||
|
||||
def test_min_similarity_below_0_raises(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._base_detector(min_similarity=-0.1)
|
||||
|
||||
def test_max_results_zero_is_valid(self):
|
||||
# 0 is a non-negative int — must not raise
|
||||
detector = self._base_detector(max_results=0)
|
||||
self.assertEqual(detector.detect_duplicates(self.entities), [])
|
||||
|
||||
def test_top_k_per_entity_zero_is_valid(self):
|
||||
detector = self._base_detector(top_k_per_entity=0)
|
||||
self.assertEqual(detector.detect_duplicates(self.entities), [])
|
||||
|
||||
def test_min_similarity_boundary_0_valid(self):
|
||||
self._base_detector(min_similarity=0.0) # must not raise
|
||||
|
||||
def test_min_similarity_boundary_1_valid(self):
|
||||
self._base_detector(min_similarity=1.0) # must not raise
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# top_k_per_entity OR semantics (bug_001)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_top_k_or_semantics_keeps_candidate_if_either_under_quota(self):
|
||||
"""A high-ranked candidate must survive even if one of its entities hit k,
|
||||
as long as the other entity is still under quota."""
|
||||
# With k=1 and OR semantics, every entity can appear in AT LEAST one
|
||||
# candidate. Verify that more candidates survive than would under AND.
|
||||
k = 1
|
||||
or_results = self._base_detector(top_k_per_entity=k).detect_duplicates(self.entities)
|
||||
# Each entity should appear at least once — no entity completely starved
|
||||
seen_ids: set = set()
|
||||
for c in or_results:
|
||||
seen_ids.add(c.entity1["id"])
|
||||
seen_ids.add(c.entity2["id"])
|
||||
# Entities that have at least one match above threshold must appear
|
||||
all_ids_in_any_pair: set = set()
|
||||
uncapped = self._base_detector().detect_duplicates(self.entities)
|
||||
for c in uncapped:
|
||||
all_ids_in_any_pair.add(c.entity1["id"])
|
||||
all_ids_in_any_pair.add(c.entity2["id"])
|
||||
self.assertEqual(seen_ids, all_ids_in_any_pair)
|
||||
|
||||
def test_group_merge_updates_normalized_entity_keys_for_int_ids(self):
|
||||
"""Merged groups must update the normalized string lookup keys.
|
||||
|
||||
Regression for stale raw int keys: after a bridge candidate merges two
|
||||
groups, later candidates involving the moved int-ID entities must still
|
||||
attach to the returned group rather than an orphaned removed group.
|
||||
"""
|
||||
entities = [
|
||||
{"id": 1, "name": "Alpha"},
|
||||
{"id": 2, "name": "Alpha duplicate"},
|
||||
{"id": 3, "name": "Alpha bridge"},
|
||||
{"id": 4, "name": "Alpha merged"},
|
||||
{"id": 5, "name": "Alpha later"},
|
||||
]
|
||||
candidates = [
|
||||
DuplicateCandidate(entities[0], entities[1], 0.95, 0.95),
|
||||
DuplicateCandidate(entities[2], entities[3], 0.94, 0.94),
|
||||
DuplicateCandidate(entities[1], entities[2], 0.93, 0.93),
|
||||
DuplicateCandidate(entities[2], entities[4], 0.92, 0.92),
|
||||
]
|
||||
|
||||
groups = self._base_detector()._build_duplicate_groups(candidates)
|
||||
|
||||
self.assertEqual(len(groups), 1)
|
||||
self.assertEqual(
|
||||
{entity["id"] for entity in groups[0].entities},
|
||||
{1, 2, 3, 4, 5},
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Combined options
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_max_results_and_sort_by_similarity(self):
|
||||
n = 2
|
||||
results = self._base_detector(max_results=n, sort_by="similarity_score").detect_duplicates(self.entities)
|
||||
self.assertLessEqual(len(results), n)
|
||||
if len(results) == 2:
|
||||
self.assertGreaterEqual(results[0].similarity_score, results[1].similarity_score)
|
||||
|
||||
def test_min_similarity_and_top_k_combined(self):
|
||||
floor, k = 0.5, 1
|
||||
results = self._base_detector(min_similarity=floor, top_k_per_entity=k).detect_duplicates(self.entities)
|
||||
# min_similarity floor still applies
|
||||
for c in results:
|
||||
self.assertGreaterEqual(c.similarity_score, floor)
|
||||
# OR semantics: no pair duplicated
|
||||
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
|
||||
self.assertEqual(len(pairs), len(set(pairs)))
|
||||
|
||||
def test_all_four_options_combined(self):
|
||||
results = self._base_detector(
|
||||
max_results=3,
|
||||
top_k_per_entity=1,
|
||||
min_similarity=0.3,
|
||||
sort_by="similarity_score",
|
||||
).detect_duplicates(self.entities)
|
||||
self.assertLessEqual(len(results), 3)
|
||||
scores = [c.similarity_score for c in results]
|
||||
self.assertEqual(scores, sorted(scores, reverse=True))
|
||||
for c in results:
|
||||
self.assertGreaterEqual(c.similarity_score, 0.3)
|
||||
|
||||
def test_max_results_applied_after_top_k(self):
|
||||
"""max_results must slice the already-top-k-filtered list, not pre-empt it."""
|
||||
top_k_only = self._base_detector(top_k_per_entity=1).detect_duplicates(self.entities)
|
||||
both = self._base_detector(top_k_per_entity=1, max_results=1).detect_duplicates(self.entities)
|
||||
self.assertLessEqual(len(both), min(1, len(top_k_only)))
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# incremental_detect
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def test_incremental_detect_max_results(self):
|
||||
new_e, existing = self.entities[:3], self.entities[3:]
|
||||
results = self._base_detector(max_results=1).incremental_detect(new_e, existing)
|
||||
self.assertLessEqual(len(results), 1)
|
||||
|
||||
def test_incremental_detect_min_similarity(self):
|
||||
new_e, existing = self.entities[:3], self.entities[3:]
|
||||
results = self._base_detector(min_similarity=0.99).incremental_detect(new_e, existing)
|
||||
for c in results:
|
||||
self.assertGreaterEqual(c.similarity_score, 0.99)
|
||||
|
||||
def test_incremental_detect_sort_by_similarity(self):
|
||||
new_e, existing = self.entities[:3], self.entities[3:]
|
||||
results = self._base_detector(sort_by="similarity_score").incremental_detect(new_e, existing)
|
||||
scores = [c.similarity_score for c in results]
|
||||
self.assertEqual(scores, sorted(scores, reverse=True))
|
||||
|
||||
def test_incremental_detect_top_k_per_entity(self):
|
||||
new_e, existing = self.entities[:3], self.entities[3:]
|
||||
k = 1
|
||||
results = self._base_detector(top_k_per_entity=k).incremental_detect(new_e, existing)
|
||||
# OR semantics: no pair appears twice
|
||||
pairs = [(c.entity1["id"], c.entity2["id"]) for c in results]
|
||||
self.assertEqual(len(pairs), len(set(pairs)))
|
||||
|
||||
def test_incremental_detect_empty_new_entities(self):
|
||||
detector = self._base_detector(max_results=5)
|
||||
self.assertEqual(detector.incremental_detect([], self.entities), [])
|
||||
|
||||
def test_incremental_detect_empty_existing_entities(self):
|
||||
detector = self._base_detector(max_results=5)
|
||||
self.assertEqual(detector.incremental_detect(self.entities, []), [])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
"""Tests for Ontology Hub subissue 3 APIs."""
|
||||
|
||||
import pytest
|
||||
|
||||
from semantica.context.context_graph import ContextGraph
|
||||
from semantica.explorer.app import create_app
|
||||
from semantica.explorer.session import GraphSession
|
||||
|
||||
try:
|
||||
from starlette.testclient import TestClient
|
||||
except ImportError:
|
||||
pytest.skip(
|
||||
"starlette TestClient is required for explorer tests. Install semantica[explorer].",
|
||||
allow_module_level=True,
|
||||
)
|
||||
|
||||
|
||||
def _build_ontology_graph() -> ContextGraph:
|
||||
graph = ContextGraph(advanced_analytics=False)
|
||||
onto_a = "http://example.org/onto-a"
|
||||
onto_b = "http://example.org/onto-b"
|
||||
person_a = "http://example.org/onto-a#Person"
|
||||
person_b = "http://example.org/onto-b#PersonRecord"
|
||||
name_a = "http://example.org/onto-a#name"
|
||||
|
||||
graph.add_node(
|
||||
onto_a,
|
||||
node_type="owl:Ontology",
|
||||
content="Ontology A",
|
||||
**{"rdfs:label": "Ontology A", "rdfs:comment": "Primary ontology", "version": "1.0.0"},
|
||||
)
|
||||
graph.add_node(
|
||||
onto_b,
|
||||
node_type="owl:Ontology",
|
||||
content="Ontology B",
|
||||
**{"rdfs:label": "Ontology B", "rdfs:comment": "Partner ontology", "version": "1.0.0"},
|
||||
)
|
||||
graph.add_node(
|
||||
person_a,
|
||||
node_type="owl:Class",
|
||||
content="Person",
|
||||
scheme_uri=onto_a,
|
||||
**{"rdfs:label": "Person", "rdfs:comment": "A person", "skos:definition": "Human actor"},
|
||||
)
|
||||
graph.add_node(
|
||||
name_a,
|
||||
node_type="owl:DatatypeProperty",
|
||||
content="name",
|
||||
scheme_uri=onto_a,
|
||||
**{"rdfs:label": "name", "rdfs:comment": "Display name"},
|
||||
)
|
||||
graph.add_node(
|
||||
person_b,
|
||||
node_type="owl:Class",
|
||||
content="Person Record",
|
||||
scheme_uri=onto_b,
|
||||
**{"rdfs:label": "Person Record", "rdfs:comment": "A person profile"},
|
||||
)
|
||||
graph.add_edge(name_a, person_a, edge_type="rdfs:domain")
|
||||
return graph
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client():
|
||||
app = create_app(session=GraphSession(_build_ontology_graph()))
|
||||
with TestClient(app) as test_client:
|
||||
yield test_client
|
||||
|
||||
|
||||
def test_alignment_round_trip(client):
|
||||
payload = {
|
||||
"source_uri": "http://example.org/onto-a#Person",
|
||||
"target_uri": "http://example.org/onto-b#PersonRecord",
|
||||
"relation": "owl:equivalentClass",
|
||||
"confidence": 0.91,
|
||||
"provenance": "Reviewed from source mapping table",
|
||||
"source": "test",
|
||||
"reviewer": "qa",
|
||||
}
|
||||
created = client.post("/api/ontology/alignments", json=payload)
|
||||
assert created.status_code == 200
|
||||
alignment = created.json()
|
||||
assert alignment["confidence"] == 0.91
|
||||
assert alignment["provenance"] == "Reviewed from source mapping table"
|
||||
|
||||
listed = client.get("/api/ontology/alignments")
|
||||
assert listed.status_code == 200
|
||||
assert [item["id"] for item in listed.json()] == [alignment["id"]]
|
||||
|
||||
removed = client.delete(f"/api/ontology/alignments?id={alignment['id']}")
|
||||
assert removed.status_code == 200
|
||||
assert client.get("/api/ontology/alignments").json() == []
|
||||
|
||||
|
||||
def test_alignment_suggestions_are_ranked(client):
|
||||
response = client.post(
|
||||
"/api/ontology/suggest-alignments",
|
||||
json={
|
||||
"source_ontology_uri": "http://example.org/onto-a",
|
||||
"target_ontology_uri": "http://example.org/onto-b",
|
||||
"threshold": 0.35,
|
||||
"limit": 5,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
suggestions = response.json()
|
||||
assert suggestions
|
||||
# Top suggestion should be the Person→PersonRecord pair (highest label similarity).
|
||||
top = suggestions[0]
|
||||
assert "Person" in top["source_label"]
|
||||
assert "Person" in top["target_label"]
|
||||
# Results must be sorted descending by score.
|
||||
assert suggestions == sorted(suggestions, key=lambda item: item["score"], reverse=True)
|
||||
|
||||
|
||||
def test_health_returns_dimensions_and_issues(client):
|
||||
response = client.get("/api/ontology/health?uri=http%3A%2F%2Fexample.org%2Fonto-a")
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["total_score"] >= 0
|
||||
assert {dimension["key"] for dimension in payload["dimensions"]} == {
|
||||
"completeness",
|
||||
"consistency",
|
||||
"shacl",
|
||||
"alignment",
|
||||
"documentation",
|
||||
}
|
||||
assert isinstance(payload["issues"], list)
|
||||
|
||||
|
||||
def test_shacl_generate_and_shapes(client):
|
||||
response = client.post(
|
||||
"/api/ontology/shacl/generate",
|
||||
json={"uri": "http://example.org/onto-a", "quality_tier": "strict"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert "sh:NodeShape" in payload["shacl_turtle"]
|
||||
assert payload["shape_count"] >= 1
|
||||
|
||||
shapes = client.get("/api/ontology/shacl/shapes?uri=http%3A%2F%2Fexample.org%2Fonto-a")
|
||||
assert shapes.status_code == 200
|
||||
assert shapes.json()["shapes"]
|
||||
|
||||
|
||||
def test_shacl_validate_returns_unavailable(client):
|
||||
response = client.post(
|
||||
"/api/ontology/shacl/validate",
|
||||
json={
|
||||
"uri": "http://example.org/onto-a",
|
||||
"shacl_turtle": "@prefix sh: <http://www.w3.org/ns/shacl#> .",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
payload = response.json()
|
||||
assert payload["status"] == "unavailable", "stub must not report conforms=True before validation is wired"
|
||||
assert payload["conforms"] is False
|
||||
assert isinstance(payload["violations"], list)
|
||||
|
||||
|
||||
def test_shacl_validate_rejects_empty_turtle(client):
|
||||
response = client.post(
|
||||
"/api/ontology/shacl/validate",
|
||||
json={"uri": "http://example.org/onto-a", "shacl_turtle": " "},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
def test_health_returns_404_for_unknown_ontology(client):
|
||||
response = client.get("/api/ontology/health?uri=http%3A%2F%2Fnot-loaded.example%2Fonto")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
def test_health_shacl_dimension_is_zero_when_unavailable(client):
|
||||
payload = client.get("/api/ontology/health?uri=http%3A%2F%2Fexample.org%2Fonto-a").json()
|
||||
shacl_dim = next(d for d in payload["dimensions"] if d["key"] == "shacl")
|
||||
assert shacl_dim["status"] == "unavailable"
|
||||
assert shacl_dim["score"] == 0.0
|
||||
# Total score must NOT include the unavailable dimension in its average.
|
||||
scoreable = [d for d in payload["dimensions"] if d["status"] != "unavailable"]
|
||||
expected_total = round(sum(d["score"] for d in scoreable) / len(scoreable), 1)
|
||||
assert payload["total_score"] == expected_total
|
||||
|
||||
|
||||
def test_delete_unknown_alignment_returns_404(client):
|
||||
response = client.delete("/api/ontology/alignments?id=does-not-exist")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
def test_alignment_upsert_is_idempotent(client):
|
||||
payload = {
|
||||
"source_uri": "http://example.org/onto-a#Person",
|
||||
"target_uri": "http://example.org/onto-b#PersonRecord",
|
||||
"relation": "owl:equivalentClass",
|
||||
"confidence": 0.80,
|
||||
}
|
||||
first = client.post("/api/ontology/alignments", json=payload).json()
|
||||
updated_payload = {**payload, "confidence": 0.95}
|
||||
second = client.post("/api/ontology/alignments", json=updated_payload).json()
|
||||
assert first["id"] == second["id"], "upsert must reuse the same deterministic ID"
|
||||
assert second["confidence"] == 0.95
|
||||
assert second["created_at"] == first["created_at"], "created_at must not change on update"
|
||||
listed = client.get("/api/ontology/alignments").json()
|
||||
assert len(listed) == 1
|
||||
|
||||
|
||||
def test_alignment_accepts_external_uri(client):
|
||||
payload = {
|
||||
"source_uri": "http://example.org/onto-a#Person",
|
||||
"target_uri": "http://schema.org/Person", # not in local graph
|
||||
"relation": "owl:equivalentClass",
|
||||
"confidence": 0.75,
|
||||
}
|
||||
response = client.post("/api/ontology/alignments", json=payload)
|
||||
assert response.status_code == 200
|
||||
alignment = response.json()
|
||||
assert alignment["target_label"] == "Person" # derived from URI fragment
|
||||
|
||||
|
||||
def test_suggest_alignments_returns_embedding_similarity(client):
|
||||
response = client.post(
|
||||
"/api/ontology/suggest-alignments",
|
||||
json={
|
||||
"source_ontology_uri": "http://example.org/onto-a",
|
||||
"target_ontology_uri": "http://example.org/onto-b",
|
||||
"threshold": 0.20,
|
||||
"limit": 10,
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
suggestions = response.json()
|
||||
assert suggestions
|
||||
# When sklearn is available, embedding_similarity should be populated.
|
||||
top = suggestions[0]
|
||||
assert top["embedding_similarity"] is not None, (
|
||||
"TF-IDF embedding similarity must be returned when sklearn is installed"
|
||||
)
|
||||
# Combined score must be a weighted blend, not purely the label score.
|
||||
assert top["score"] != top["label_similarity"] or top["embedding_similarity"] == top["label_similarity"]
|
||||
|
||||
|
||||
def test_shacl_validate_rejects_invalid_turtle_syntax(client):
|
||||
response = client.post(
|
||||
"/api/ontology/shacl/validate",
|
||||
json={
|
||||
"uri": "http://example.org/onto-a",
|
||||
"shacl_turtle": "this is not valid turtle !!!",
|
||||
},
|
||||
)
|
||||
assert response.status_code == 422
|
||||
|
||||
|
||||
def test_health_alignment_coverage_uses_set_lookup(client):
|
||||
# Create an alignment first so coverage score can be non-zero.
|
||||
client.post("/api/ontology/alignments", json={
|
||||
"source_uri": "http://example.org/onto-a#Person",
|
||||
"target_uri": "http://example.org/onto-b#PersonRecord",
|
||||
"relation": "owl:equivalentClass",
|
||||
"confidence": 0.9,
|
||||
})
|
||||
payload = client.get("/api/ontology/health?uri=http%3A%2F%2Fexample.org%2Fonto-a").json()
|
||||
alignment_dim = next(d for d in payload["dimensions"] if d["key"] == "alignment")
|
||||
assert alignment_dim["score"] > 0.0, "alignment coverage must be non-zero after recording an alignment"
|
||||
@@ -0,0 +1,98 @@
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _run_python_with_blocked_modules(
|
||||
code: str,
|
||||
blocked_modules: tuple[str, ...],
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
blocker = f"""
|
||||
import importlib.abc
|
||||
import sys
|
||||
|
||||
BLOCKED_MODULES = {blocked_modules!r}
|
||||
|
||||
|
||||
class OptionalDependencyBlocker(importlib.abc.MetaPathFinder):
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
root_name = fullname.split(".", 1)[0]
|
||||
if root_name in BLOCKED_MODULES:
|
||||
err = ModuleNotFoundError(f"No module named '{{root_name}}'")
|
||||
err.name = root_name
|
||||
raise err
|
||||
return None
|
||||
|
||||
|
||||
sys.meta_path.insert(0, OptionalDependencyBlocker())
|
||||
"""
|
||||
env = os.environ.copy()
|
||||
env["PYTHONPATH"] = str(REPO_ROOT)
|
||||
env["PYTHONDONTWRITEBYTECODE"] = "1"
|
||||
return subprocess.run(
|
||||
[sys.executable, "-c", textwrap.dedent(blocker + "\n" + code)],
|
||||
cwd=REPO_ROOT,
|
||||
env=env,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def test_file_ingestion_imports_without_optional_backends() -> None:
|
||||
result = _run_python_with_blocked_modules(
|
||||
"""
|
||||
from semantica.ingest import FileIngestor, ingest_file
|
||||
print(FileIngestor.__name__, callable(ingest_file))
|
||||
""",
|
||||
("git", "bs4", "pyarrow"),
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "FileIngestor True" in result.stdout
|
||||
|
||||
|
||||
def test_repository_ingestion_reports_missing_gitpython_when_used() -> None:
|
||||
result = _run_python_with_blocked_modules(
|
||||
"""
|
||||
from semantica.ingest import ingest_repository
|
||||
|
||||
try:
|
||||
ingest_repository("https://example.com/repo.git")
|
||||
except Exception as exc:
|
||||
print(type(exc).__name__, exc)
|
||||
else:
|
||||
raise SystemExit("expected repository ingestion to fail without GitPython")
|
||||
""",
|
||||
("git",),
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "ConfigurationError" in result.stdout
|
||||
assert "Repository ingestion" in result.stdout
|
||||
assert "GitPython" in result.stdout
|
||||
|
||||
|
||||
def test_parquet_ingestion_reports_missing_pyarrow_when_used() -> None:
|
||||
result = _run_python_with_blocked_modules(
|
||||
"""
|
||||
from semantica.ingest import ingest_parquet
|
||||
|
||||
try:
|
||||
ingest_parquet("events.parquet")
|
||||
except Exception as exc:
|
||||
print(type(exc).__name__, exc)
|
||||
else:
|
||||
raise SystemExit("expected parquet ingestion to fail without pyarrow")
|
||||
""",
|
||||
("pyarrow",),
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "ConfigurationError" in result.stdout
|
||||
assert "Parquet ingestion" in result.stdout
|
||||
assert "pyarrow" in result.stdout
|
||||
@@ -0,0 +1,150 @@
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
pa = pytest.importorskip("pyarrow")
|
||||
pq = pytest.importorskip("pyarrow.parquet")
|
||||
|
||||
from semantica.ingest import ( # noqa: E402
|
||||
ParquetData,
|
||||
ParquetIngestor,
|
||||
ingest,
|
||||
ingest_file,
|
||||
ingest_parquet,
|
||||
list_available_methods,
|
||||
)
|
||||
from semantica.ingest.file_ingestor import FileTypeDetector # noqa: E402
|
||||
from semantica.utils.exceptions import ValidationError # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_parquet(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "events.parquet"
|
||||
table = pa.table(
|
||||
{
|
||||
"id": [1, 2, 3],
|
||||
"name": ["alpha", "beta", "gamma"],
|
||||
"score": [0.7, 0.8, 0.9],
|
||||
"city": ["Pune", "Delhi", "Mumbai"],
|
||||
}
|
||||
)
|
||||
pq.write_table(table, path, compression="snappy")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def partitioned_parquet(tmp_path: Path) -> Path:
|
||||
root = tmp_path / "events_partitioned"
|
||||
|
||||
us_2025 = root / "country=US" / "year=2025"
|
||||
us_2025.mkdir(parents=True)
|
||||
pq.write_table(
|
||||
pa.table({"id": [1, 2], "value": ["a", "b"]}),
|
||||
us_2025 / "part-0.parquet",
|
||||
compression="gzip",
|
||||
)
|
||||
|
||||
ca_2026 = root / "country=CA" / "year=2026"
|
||||
ca_2026.mkdir(parents=True)
|
||||
pq.write_table(
|
||||
pa.table({"id": [3], "value": ["c"]}),
|
||||
ca_2026 / "part-1.parquet",
|
||||
compression="gzip",
|
||||
)
|
||||
|
||||
return root
|
||||
|
||||
|
||||
def test_parquet_file_ingestion_reads_data_schema_and_metadata(
|
||||
sample_parquet: Path,
|
||||
) -> None:
|
||||
ingestor = ParquetIngestor()
|
||||
|
||||
result = ingestor.ingest_file(sample_parquet)
|
||||
|
||||
assert isinstance(result, ParquetData)
|
||||
assert result.row_count == 3
|
||||
assert result.columns == ["id", "name", "score", "city"]
|
||||
assert result.data[0]["name"] == "alpha"
|
||||
assert result.schema["columns"] == ["id", "name", "score", "city"]
|
||||
assert result.schema["fields"][0]["type"] == "int64"
|
||||
assert result.metadata["total_rows"] == 3
|
||||
assert result.metadata["row_groups"] == 1
|
||||
assert result.metadata["compression_codecs"] == ["SNAPPY"]
|
||||
|
||||
|
||||
def test_parquet_selective_column_reading_with_limit(sample_parquet: Path) -> None:
|
||||
ingestor = ParquetIngestor()
|
||||
|
||||
result = ingestor.ingest_file(sample_parquet, columns=["id", "name"], limit=2)
|
||||
|
||||
assert result.row_count == 2
|
||||
assert result.columns == ["id", "name"]
|
||||
assert result.data == [{"id": 1, "name": "alpha"}, {"id": 2, "name": "beta"}]
|
||||
assert result.metadata["selected_columns"] == ["id", "name"]
|
||||
assert result.metadata["limit"] == 2
|
||||
|
||||
|
||||
def test_parquet_schema_and_metadata_can_be_extracted_without_rows(
|
||||
sample_parquet: Path,
|
||||
) -> None:
|
||||
ingestor = ParquetIngestor()
|
||||
|
||||
schema = ingestor.extract_schema(sample_parquet)
|
||||
metadata = ingestor.extract_metadata(sample_parquet)
|
||||
result = ingestor.ingest_file(sample_parquet, include_data=False)
|
||||
|
||||
assert schema["columns"] == ["id", "name", "score", "city"]
|
||||
assert metadata["total_rows"] == 3
|
||||
assert metadata["format"] == "parquet"
|
||||
assert result.row_count == 0
|
||||
assert result.data == []
|
||||
assert result.metadata["include_data"] is False
|
||||
|
||||
|
||||
def test_partitioned_parquet_directory_ingestion(partitioned_parquet: Path) -> None:
|
||||
ingestor = ParquetIngestor()
|
||||
|
||||
result = ingestor.ingest_directory(partitioned_parquet)
|
||||
|
||||
assert result.row_count == 3
|
||||
assert set(result.columns) == {"id", "value", "country", "year"}
|
||||
assert {row["country"] for row in result.data} == {"US", "CA"}
|
||||
assert result.metadata["file_count"] == 2
|
||||
assert result.metadata["total_rows"] == 3
|
||||
assert result.metadata["partition_columns"] == ["country", "year"]
|
||||
assert result.metadata["partition_values"] == {
|
||||
"country": ["CA", "US"],
|
||||
"year": ["2025", "2026"],
|
||||
}
|
||||
assert result.metadata["compression_codecs"] == ["GZIP"]
|
||||
|
||||
|
||||
def test_parquet_convenience_methods_and_unified_dispatch(sample_parquet: Path) -> None:
|
||||
direct = ingest_parquet(sample_parquet, columns=["name"])
|
||||
via_file_method = ingest_file(sample_parquet, method="parquet", limit=1)
|
||||
unified = ingest(sample_parquet)
|
||||
unified_batch = ingest([sample_parquet])
|
||||
methods = list_available_methods("parquet")
|
||||
|
||||
assert isinstance(direct, ParquetData)
|
||||
assert direct.columns == ["name"]
|
||||
assert isinstance(via_file_method, ParquetData)
|
||||
assert via_file_method.row_count == 1
|
||||
assert isinstance(unified["data"], ParquetData)
|
||||
assert isinstance(unified_batch["data"][0], ParquetData)
|
||||
assert "metadata" in methods["parquet"]
|
||||
|
||||
|
||||
def test_file_type_detector_recognizes_parquet_magic_number() -> None:
|
||||
detector = FileTypeDetector()
|
||||
|
||||
assert detector.detect_type("dataset", content=b"PAR1payload") == "parquet"
|
||||
assert detector.is_supported("parquet")
|
||||
|
||||
|
||||
def test_parquet_ingestion_rejects_negative_limit(sample_parquet: Path) -> None:
|
||||
ingestor = ParquetIngestor()
|
||||
|
||||
with pytest.raises(ValidationError):
|
||||
ingestor.ingest_file(sample_parquet, limit=-1)
|
||||
@@ -0,0 +1,120 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _run_python(code: str) -> subprocess.CompletedProcess[str]:
|
||||
env = os.environ.copy()
|
||||
env["PYTHONPATH"] = str(REPO_ROOT)
|
||||
env["PYTHONDONTWRITEBYTECODE"] = "1"
|
||||
return subprocess.run(
|
||||
[sys.executable, "-c", textwrap.dedent(code)],
|
||||
cwd=REPO_ROOT,
|
||||
env=env,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def test_core_extractors_import_from_package() -> None:
|
||||
from semantica.semantic_extract import (
|
||||
NERExtractor,
|
||||
RelationExtractor,
|
||||
TripletExtractor,
|
||||
)
|
||||
|
||||
assert NERExtractor.__name__ == "NERExtractor"
|
||||
assert RelationExtractor.__name__ == "RelationExtractor"
|
||||
assert TripletExtractor.__name__ == "TripletExtractor"
|
||||
|
||||
|
||||
def test_triple_extractor_alias_imports() -> None:
|
||||
from semantica.semantic_extract import TripleExtractor, TripletExtractor
|
||||
|
||||
assert TripleExtractor is TripletExtractor
|
||||
|
||||
|
||||
def test_method_dispatchers_import_without_extractor_cycle() -> None:
|
||||
from semantica.semantic_extract.methods import (
|
||||
get_entity_method,
|
||||
get_relation_method,
|
||||
get_triplet_method,
|
||||
)
|
||||
|
||||
assert callable(get_entity_method("pattern"))
|
||||
assert callable(get_relation_method("pattern"))
|
||||
assert callable(get_triplet_method("pattern"))
|
||||
|
||||
|
||||
def test_import_order_methods_before_extractors() -> None:
|
||||
result = _run_python(
|
||||
"""
|
||||
from semantica.semantic_extract.methods import get_entity_method
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
|
||||
print(get_entity_method("pattern").__name__, NERExtractor.__name__, RelationExtractor.__name__, TripletExtractor.__name__)
|
||||
"""
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "extract_entities_pattern NERExtractor RelationExtractor TripletExtractor" in result.stdout
|
||||
|
||||
|
||||
def test_import_order_extractors_before_methods() -> None:
|
||||
result = _run_python(
|
||||
"""
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
|
||||
from semantica.semantic_extract.methods import get_entity_method
|
||||
print(NERExtractor.__name__, RelationExtractor.__name__, TripletExtractor.__name__, get_entity_method("pattern").__name__)
|
||||
"""
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "NERExtractor RelationExtractor TripletExtractor extract_entities_pattern" in result.stdout
|
||||
|
||||
|
||||
def test_legacy_type_imports_still_work() -> None:
|
||||
from semantica.semantic_extract.ner_extractor import Entity
|
||||
from semantica.semantic_extract.relation_extractor import Relation
|
||||
from semantica.semantic_extract.triplet_extractor import Triplet
|
||||
from semantica.semantic_extract.types import (
|
||||
Entity as SharedEntity,
|
||||
Relation as SharedRelation,
|
||||
Triplet as SharedTriplet,
|
||||
)
|
||||
|
||||
assert Entity is SharedEntity
|
||||
assert Relation is SharedRelation
|
||||
assert Triplet is SharedTriplet
|
||||
|
||||
|
||||
def test_core_package_imports_do_not_require_yaml() -> None:
|
||||
result = _run_python(
|
||||
"""
|
||||
import importlib.abc
|
||||
import sys
|
||||
|
||||
|
||||
class OptionalDependencyBlocker(importlib.abc.MetaPathFinder):
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname.split(".", 1)[0] == "yaml":
|
||||
raise ModuleNotFoundError("No module named 'yaml'")
|
||||
return None
|
||||
|
||||
|
||||
sys.meta_path.insert(0, OptionalDependencyBlocker())
|
||||
|
||||
from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor
|
||||
print(NERExtractor.__name__, RelationExtractor.__name__, TripletExtractor.__name__)
|
||||
"""
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "NERExtractor RelationExtractor TripletExtractor" in result.stdout
|
||||
@@ -10,6 +10,10 @@ import importlib.util
|
||||
# Add project root to path
|
||||
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
|
||||
|
||||
# Save originals before injecting mocks so they can be restored after import.
|
||||
_MOCKED_MODULES = ["spacy", "instructor", "groq", "openai", "sentence_transformers", "transformers"]
|
||||
_original_modules = {k: sys.modules.get(k) for k in _MOCKED_MODULES}
|
||||
|
||||
# Mock external dependencies with __spec__ for importlib checks
|
||||
mock_spacy = MagicMock()
|
||||
mock_spacy.__spec__ = MagicMock()
|
||||
@@ -33,6 +37,16 @@ from semantica.semantic_extract import NERExtractor
|
||||
from semantica.semantic_extract.methods import extract_entities_llm, _extract_entities_chunked, extract_relations_llm, extract_triplets_llm
|
||||
from semantica.semantic_extract.providers import BaseProvider
|
||||
|
||||
# Restore real sys.modules entries now that the mocks have served their purpose
|
||||
# for the imports above. Leaving them in place would poison other test modules
|
||||
# (e.g. test_pr482_deepseek_openai) that need the real packages at test run time.
|
||||
for _key, _original in _original_modules.items():
|
||||
if _original is None:
|
||||
sys.modules.pop(_key, None)
|
||||
else:
|
||||
sys.modules[_key] = _original
|
||||
|
||||
|
||||
class EntitiesResponse(BaseModel):
|
||||
entities: List[dict]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user