Compare commits

..
17 Commits
Author SHA1 Message Date
KaifAhmad1 a047ebf74f Merge main into pipeline and resolve visualization conflicts 2025-12-13 15:08:47 +05:30
KaifAhmad1 88c12b1867 Add pipeline orchestration fixes and E2E tests 2025-12-13 15:03:34 +05:30
KaifAhmad1 094bb8d82b Recommit pipeline orchestration and e2e tests 2025-12-13 15:01:37 +05:30
Mohd Kaif 7ff2fd9981 Merge pull request #87 from Hawksight-AI/visualization
Enhancement of Visualization Module & Comprehensive Testing Suite
2025-12-12 23:14:00 +05:30
KaifAhmad1 0a555145e4 Enhance visualization module with comprehensive testing and robust dependency handling 2025-12-12 23:10:17 +05:30
Mohd Kaif 994e58a170 Delete PR_DESCRIPTION.md 2025-12-12 20:24:25 +05:30
Mohd Kaif 244144dee3 Merge pull request #86 from Hawksight-AI/vector-store
Refactor: Remove Pinecone and Enhance Vector Store Backend Support
2025-12-12 20:23:51 +05:30
KaifAhmad1 5dfca85500 Merge branch 'main' into vector-store: Resolve PR_DESCRIPTION.md modify/delete conflict by keeping local version 2025-12-12 20:23:15 +05:30
KaifAhmad1 f3dd7a05bd Refactor: Remove Pinecone and enhance vector store backend support
- Removed all Pinecone references, adapters, and documentation to align with open-source, self-hosted focus.
- Removed PineconeAdapter and related dependencies.
- Updated VectorStore to enforce supported backends (FAISS, Weaviate, Qdrant, Milvus, InMemory).
- Updated cookbooks (e.g., 13_Vector_Store.ipynb) to use Weaviate/FAISS examples instead of Pinecone.
- Updated core documentation (modules.md, rchitecture.md, etc.) to reflect backend changes.
- Added new tests (	est_pinecone_removal.py, 	est_vector_store_deepdive.py) to verify removal and validate remaining backends.
- Verified all vector store tests pass.
2025-12-12 20:19:17 +05:30
Mohd Kaif d03a237278 Delete PR_DESCRIPTION.md 2025-12-12 18:50:33 +05:30
Mohd Kaif f3ac9fbffa Merge pull request #85 from Hawksight-AI/triplet-store
Refactor: Rename `triple_store` to `triplet_store`
2025-12-12 18:48:41 +05:30
KaifAhmad1 6856580a7a Refactor: Rename triple_store to triplet_store across codebase
- Renamed semantica/triple_store to semantica/triplet_store
- Updated all imports and class references in core modules and adapters
- Refactored Jupyter notebooks in cookbook/
- Updated documentation files (README, docs/, etc.)
- Updated tests and verified passing status
2025-12-12 18:45:00 +05:30
KaifAhmad1 4a282628ea Merge branch 'main' of https://github.com/Hawksight-AI/semantica 2025-12-12 16:31:43 +05:30
KaifAhmad1 c73e35a2fe docs: update chunking cookbook and PR description 2025-12-12 16:30:59 +05:30
Mohd Kaif a99f18b71b Merge pull request #84 from Hawksight-AI/split
Fix & Align Split Module with Documentation
2025-12-12 16:21:55 +05:30
Mohd Kaif d7d589f64e Merge pull request #83 from Hawksight-AI/semantic-extract
Refactor Semantic Extract Module to Class-Based Interfaces
2025-12-12 13:25:58 +05:30
Mohd Kaif 95c5486d22 Merge pull request #82 from Hawksight-AI/seed
Enhance SeedDataManager with Robust CSV/JSON Support
2025-12-12 12:10:09 +05:30
92 changed files with 2867 additions and 1806 deletions
+1 -1
View File
@@ -58,7 +58,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Production-ready quality assurance modules
- Comprehensive documentation with MkDocs
- Cookbook with interactive tutorials
- Support for multiple vector stores (Pinecone, Weaviate, Qdrant, FAISS)
- Support for multiple vector stores (Weaviate, Qdrant, FAISS)
- Support for multiple graph databases (Neo4j, NetworkX, RDFLib)
- Temporal knowledge graph support
- Conflict detection and resolution
-47
View File
@@ -1,47 +0,0 @@
# Refactor Semantic Extract Module to Class-Based Interfaces
## 📝 Summary
This PR refactors the Semantic Extract module to promote a cleaner, object-oriented API for Entity, Relation, and Triple extraction. It standardizes the usage around `NERExtractor`, `RelationExtractor`, and `TripleExtractor` classes, replacing the previous low-level `get_entity_method` factory functions in user-facing code.
## 🚀 Motivation
The previous API relied heavily on factory functions (`get_entity_method("pattern")`), which made discovery and configuration difficult for users. The new class-based approach:
- Improves code readability and IDE auto-completion.
- Provides a consistent interface (`extractor.extract()`) across all extraction tasks.
- Aligns the documentation and cookbooks with the actual best practices.
## 🔍 Key Changes
### 1. API Refactoring
- **Standardized Classes**: Promoted `NERExtractor`, `RelationExtractor`, and `TripleExtractor` as the primary entry points.
- **Method Aliases**: Added `extract()` aliases to `extract_entities()` and `extract_relations()` for a uniform API surface.
- **Configuration**: Unified configuration passing via class constructors.
### 2. Documentation Updates (`docs/reference/semantic_extract.md`)
- Added missing documentation for **Semantic Networks**, **Coreference Resolution**, and **LLM Enhancement**.
- Updated all code examples to use the new class-based API.
- Added a "Semantic Networks" card to the overview for better discoverability.
### 3. Cookbook Updates
- **`05_Entity_Extraction.ipynb`**: Refactored to use `NERExtractor` for Pattern, Regex, ML, and LLM examples.
- **`06_Relation_Extraction.ipynb`**: Refactored to use `RelationExtractor` for dependency and pattern-based examples.
- **`11_Chunking_and_Splitting.ipynb`**: Updated to use consistent method names (`ner_method="ml"`).
### 4. Split Module Improvements
- **Method Aliasing**: Added aliases in `methods.py` to support "spacy" (mapping to "ml") and "ml" (mapping to "dependency" for relations), improving robustness and user experience.
- **Robustness**: Verified `EntityAwareChunker` and `RelationAwareChunker` fallback mechanisms.
### 5. Testing
- Added `tests/test_ner_configurations.py` to verify all NER method configurations.
- Added `tests/test_notebooks_verification.py` to ensure notebook examples run correctly.
- Added `tests/test_semantic_extract_deepdive.py` covering relation and triple extraction scenarios.
## 🧪 Verification
- [x] **Unit Tests**: All new tests pass, verifying correct instantiation and execution of extractors.
- [x] **Notebooks**: Verified that the updated cookbooks run without errors.
- [x] **Documentation**: previewed `semantic_extract.md` to ensure correct rendering of new sections.
## ✅ Checklist
- [x] Code follows the project's coding standards.
- [x] Documentation has been updated to reflect the changes.
- [x] Tests have been added to cover the new functionality.
- [x] Cookbooks have been updated and verified.
+1 -1
View File
@@ -319,7 +319,7 @@ result = kg.query("Who founded the company?", return_format="structured")
print(f"Nodes: {kg.node_count}, Answer: {result.answer}")
```
[**Cookbook: Building Knowledge Graphs**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) • [**Graph Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/09_Graph_Store.ipynb) • [**Triple Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/20_Triple_Store.ipynb) • [**Visualization**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/16_Visualization.ipynb)
[**Cookbook: Building Knowledge Graphs**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/07_Building_Knowledge_Graphs.ipynb) • [**Graph Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/09_Graph_Store.ipynb) • [**Triplet Store**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/20_Triplet_Store.ipynb) • [**Visualization**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/16_Visualization.ipynb)
[**Graph Analytics**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/10_Graph_Analytics.ipynb) • [**Advanced Graph Analytics**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced/02_Advanced_Graph_Analytics.ipynb)
@@ -312,7 +312,7 @@
"- NumPy format\n",
"- Binary format\n",
"- FAISS format\n",
"- Vector store integration (Pinecone, Weaviate, Qdrant)\n"
"- Vector store integration (Weaviate, Qdrant)\n"
]
},
{
@@ -10,7 +10,7 @@
"\n",
"## Overview\n",
"\n",
"Build an enterprise semantic layer: construct knowledge graph, generate ontology, create semantic layer, export RDF, and store in triple store.\n",
"Build an enterprise semantic layer: construct knowledge graph, generate ontology, create semantic layer, export RDF, and store in triplet store.\n",
"\n",
"\n",
"**Documentation**: [API Reference](https://semantica.readthedocs.io/concepts/)\n",
@@ -25,7 +25,7 @@
"pip install semantica[all]\n",
"```\n",
"\n",
"## Workflow: Build KG → Generate Ontology → Create Semantic Layer → Export RDF → Triple Store\n"
"## Workflow: Build KG → Generate Ontology → Create Semantic Layer → Export RDF → Triplet Store\n"
]
},
{
@@ -37,7 +37,7 @@
"from semantica.kg import GraphBuilder\n",
"from semantica.ontology import OntologyGenerator\n",
"from semantica.export import RDFExporter\n",
"from semantica.triple_store import TripleStore\n"
"from semantica.triplet_store import TripletStore\n"
]
},
{
@@ -162,7 +162,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Store in Triple Store\n"
"## Step 5: Store in Triplet Store\n"
]
},
{
@@ -171,8 +171,8 @@
"metadata": {},
"outputs": [],
"source": [
"triple_store = TripleStore()\n",
"triple_store.store(knowledge_graph, ontology)\n"
"triplet_store = TripletStore()\n",
"triplet_store.store(knowledge_graph, ontology)\n"
]
},
{
@@ -186,7 +186,7 @@
"- Ontology Generated\n",
"- Semantic Layer Created with Mappings\n",
"- RDF Export Completed\n",
"- Triple Store Storage Completed\n"
"- Triplet Store Storage Completed\n"
]
}
],
@@ -217,7 +217,7 @@
"## 5. Best Practices for Production\n",
"\n",
"1. **Token Limits**: Align `token_limit` with your LLM's context window minus the prompt template size.\n",
"2. **Vector Store**: Use a production-grade vector store (e.g., Pinecone, Weaviate, Qdrant) instead of the mock store.\n",
"2. **Vector Store**: Use a production-grade vector store (e.g., Weaviate, Qdrant) instead of the mock store.\n",
"3. **Asynchronous Operations**: For high-throughput systems, consider wrapping storage operations in async tasks (though the core logic is synchronous for simplicity).\n",
"4. **Entity Resolution**: Implement a robust `EntityLinker` strategy to prevent graph fragmentation (e.g., \"Alice\" vs \"Alice S.\")."
]
@@ -850,4 +850,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
File diff suppressed because it is too large Load Diff
@@ -6,31 +6,31 @@
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/20_Triple_Store.ipynb)\n",
"\n",
"# Triple Store - Comprehensive Guide\n",
"# Triplet Store - Comprehensive Guide\n",
"\n",
"## Overview\n",
"\n",
"This notebook provides a **comprehensive walkthrough** of Semantica's triple_store module, demonstrating RDF triple storage, SPARQL querying, and multi-backend support for knowledge graph persistence.\n",
"This notebook provides a **comprehensive walkthrough** of Semantica's triplet_store module, demonstrating RDF triplet storage, SPARQL querying, and multi-backend support for knowledge graph persistence.\n",
"\n",
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/triple_store/)\n",
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/triplet_store/)\n",
"\n",
"### Learning Objectives\n",
"\n",
"By the end of this notebook, you will be able to:\n",
"\n",
"- Register and manage triple stores (Blazegraph, Jena, RDF4J, Virtuoso)\n",
"- Perform CRUD operations on RDF triples\n",
"- Register and manage triplet stores (Blazegraph, Jena, RDF4J, Virtuoso)\n",
"- Perform CRUD operations on RDF triplets\n",
"- Execute SPARQL queries with optimization\n",
"- Use bulk loading for large datasets\n",
"- Work with multiple store backends\n",
"- Validate and track triple operations\n",
"- Validate and track triplet operations\n",
"- Choose the right backend for your use case\n",
"\n",
"### What You'll Learn\n",
"\n",
"| Component | Purpose | When to Use |\n",
"|-----------|---------|-------------|\n",
"| `TripleManager` | Store coordination | All triple operations |\n",
"| `TripletManager` | Store coordination | All triplet operations |\n",
"| `QueryEngine` | SPARQL execution | Query optimization |\n",
"| `BulkLoader` | High-volume loading | Large datasets |\n",
"| `BlazegraphAdapter` | Blazegraph backend | High performance |\n",
@@ -57,13 +57,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1: Basic Triple Store Operations\n",
"## Step 1: Basic Triplet Store Operations\n",
"\n",
"Let's start with the `TripleManager` for basic triple store operations.\n",
"Let's start with the `TripletManager` for basic triplet store operations.\n",
"\n",
"### What is TripleManager?\n",
"### What is TripletManager?\n",
"\n",
"`TripleManager` is the main coordinator for triple store operations:\n",
"`TripletManager` is the main coordinator for triplet store operations:\n",
"- **Store Registration**: Register multiple backends\n",
"- **CRUD Operations**: Add, get, update, delete triples\n",
"- **Multi-Store**: Manage multiple stores simultaneously"
@@ -75,11 +75,11 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import TripleManager\n",
"from semantica.triplet_store import TripletManager\n",
"from semantica.semantic_extract.triple_extractor import Triple\n",
"\n",
"# Create triple manager\n",
"manager = TripleManager()\n",
"manager = TripletManager()\n",
"\n",
"# Register a Blazegraph store (in-memory for demo)\n",
"store = manager.register_store(\n",
@@ -130,7 +130,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import register_store\n",
"from semantica.triplet_store import register_store\n",
"\n",
"# Register multiple stores using convenience function\n",
"blazegraph_store = register_store(\n",
@@ -179,7 +179,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import add_triple, add_triples, get_triples, update_triple, delete_triple\n",
"from semantica.triplet_store import add_triple, add_triples, get_triples, update_triple, delete_triple\n",
"\n",
"# Create - Add single triple\n",
"triple1 = Triple(\n",
@@ -240,7 +240,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import QueryEngine, BlazegraphAdapter\n",
"from semantica.triplet_store import QueryEngine, BlazegraphAdapter\n",
"\n",
"# Create query engine with caching\n",
"engine = QueryEngine(enable_caching=True, enable_optimization=True)\n",
@@ -299,7 +299,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import optimize_query, plan_query\n",
"from semantica.triplet_store import optimize_query, plan_query\n",
"\n",
"# Original query\n",
"query = \"\"\"\n",
@@ -347,7 +347,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import BulkLoader, LoadProgress\n",
"from semantica.triplet_store import BulkLoader, LoadProgress\n",
"\n",
"# Create bulk loader\n",
"loader = BulkLoader(\n",
@@ -392,11 +392,11 @@
"source": [
"## Step 7: Store Adapters\n",
"\n",
"Work with different triple store backends.\n",
"Work with different triplet store backends.\n",
"\n",
"### Blazegraph Adapter\n",
"\n",
"High-performance triple store with GPU acceleration."
"High-performance triplet store with GPU acceleration."
]
},
{
@@ -405,7 +405,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import BlazegraphAdapter\n",
"from semantica.triplet_store import BlazegraphAdapter\n",
"\n",
"# Create Blazegraph adapter\n",
"blazegraph = BlazegraphAdapter(\n",
@@ -442,7 +442,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import JenaAdapter\n",
"from semantica.triplet_store import JenaAdapter\n",
"\n",
"# Create Jena adapter (in-memory)\n",
"jena = JenaAdapter()\n",
@@ -497,7 +497,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import RDF4JAdapter\n",
"from semantica.triplet_store import RDF4JAdapter\n",
"\n",
"# Create RDF4J adapter\n",
"rdf4j = RDF4JAdapter(\n",
@@ -540,7 +540,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import VirtuosoAdapter\n",
"from semantica.triplet_store import VirtuosoAdapter\n",
"\n",
"# Create Virtuoso adapter\n",
"virtuoso = VirtuosoAdapter(\n",
@@ -604,7 +604,7 @@
"metadata": {},
"outputs": [],
"source": [
"from semantica.triple_store import validate_triples\n",
"from semantica.triplet_store import validate_triples\n",
"\n",
"# Create triples (some invalid)\n",
"triples_to_validate = [\n",
@@ -650,7 +650,7 @@
"outputs": [],
"source": [
"# Register multiple stores\n",
"manager = TripleManager()\n",
"manager = TripletManager()\n",
"\n",
"primary = manager.register_store(\n",
" \"primary\",\n",
@@ -720,7 +720,7 @@
"\n",
"In this notebook, you've learned how to:\n",
"\n",
"- Register and manage triple stores\n",
"- Register and manage triplet stores\n",
"- Perform CRUD operations on RDF triples\n",
"- Execute and optimize SPARQL queries\n",
"- Use bulk loading for large datasets\n",
@@ -740,7 +740,7 @@
"### Next Steps\n",
"\n",
"**Further Reading**:\n",
"- [Triple Store API Reference](https://semantica.readthedocs.io/reference/triple_store/)\n",
"- [Triplet Store API Reference](https://semantica.readthedocs.io/reference/triplet_store/)\n",
"- [SPARQL 1.1 Specification](https://www.w3.org/TR/sparql11-query/)\n",
"- [Knowledge Graph Building](../use_cases/advanced_rag/01_GraphRAG_Complete.ipynb)\n",
"\n",
@@ -771,4 +771,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
@@ -38,7 +38,7 @@
"- **Parsing**: DocumentParser, PDFParser, StructuredDataParser, CSVParser, MCPParser\n",
"- **Extraction**: NERExtractor, RelationExtractor, CoreferenceResolver, TripleExtractor\n",
"- **KG**: GraphBuilder, GraphValidator, EntityResolver, GraphAnalyzer\n",
"- **Triple Store**: TripleStore, TripleManager, QueryEngine\n",
"- **Triplet Store**: TripletStore, TripletManager, QueryEngine\n",
"- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
"- **Quality**: KGQualityAssessor, ValidationEngine\n",
"- **Export**: JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
@@ -67,7 +67,6 @@
"from semantica.kg import GraphBuilder, GraphValidator, EntityResolver, GraphAnalyzer\n",
"from semantica.triple_store import TripleStore, TripleManager, QueryEngine\n",
"from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
"from semantica.export import JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
"from semantica.visualization import KGVisualizer, OntologyVisualizer, TemporalVisualizer\n",
"import tempfile\n",
@@ -22,7 +22,7 @@
"- **Materialized Knowledge Graphs**: Semantica's KG modules enable building persistent knowledge graphs from medical ontologies, clinical documents, and reports\n",
"- **Virtual Data Integration**: Semantica's DBIngestor and QueryEngine allow virtual integration with Electronic Health Records (EHRs) without data replication\n",
"- **Hybrid Design**: Semantica's architecture naturally separates structural knowledge from patient-level data\n",
"- **Dynamic Query Orchestration**: Semantica's Reasoning and Triple Store modules enable orchestration of queries across ontologies, documents, and EHRs\n",
"- **Dynamic Query Orchestration**: Semantica's Reasoning and Triplet Store modules enable orchestration of queries across ontologies, documents, and EHRs\n",
"- **Temporal & Semantic Dimensions**: Semantica's Temporal and Context modules provide historical analysis and semantic understanding\n",
"- **Traceable & Explainable**: Semantica's ExplanationGenerator and ContextRetriever provide traceable, explainable answers\n",
"\n",
@@ -55,7 +55,7 @@
"- **KG**: GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer (materialized knowledge graph)\n",
"- **Embeddings**: EmbeddingGenerator, TextEmbedder (for embeddings)\n",
"- **Vector Store**: VectorStore, HybridSearch, MetadataFilter (for RAG)\n",
"- **Triple Store**: TripleManager, QueryEngine (for SPARQL queries on ontologies)\n",
"- **Triplet Store**: TripletManager, QueryEngine (for SPARQL queries on ontologies)\n",
"- **Reasoning**: InferenceEngine, RuleManager (for query orchestration and medical reasoning)\n",
"- **Context**: ContextRetriever, ContextGraphBuilder (for contextual retrieval)\n",
"- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer (for visualization)\n",
@@ -86,7 +86,7 @@
"from semantica.kg import GraphBuilder, GraphAnalyzer, ConnectivityAnalyzer\n",
"from semantica.embeddings import EmbeddingGenerator, TextEmbedder\n",
"from semantica.vector_store import VectorStore, HybridSearch, MetadataFilter\n",
"from semantica.triple_store import TripleManager, QueryEngine\n",
"from semantica.triplet_store import TripletManager, QueryEngine\n",
"from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
"from semantica.context import ContextRetriever, ContextGraphBuilder\n",
"from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
@@ -440,9 +440,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 9: Setup Triple Store for Ontology Queries Using Semantica\n",
"## Step 9: Setup Triplet Store for Ontology Queries Using Semantica\n",
"\n",
"Using Semantica's triple store modules to enable SPARQL queries on medical ontologies.\n"
"Using Semantica's triplet store modules to enable SPARQL queries on medical ontologies.\n"
]
},
{
@@ -451,12 +451,12 @@
"metadata": {},
"outputs": [],
"source": [
"# Initialize Semantica triple store and query engine\n",
"triple_manager = TripleManager()\n",
"# Initialize Semantica triplet store and query engine\n",
"triplet_manager = TripletManager()\n",
"query_engine = QueryEngine()\n",
"\n",
"# Register triple store (using in-memory for demo)\n",
"store = triple_manager.register_store(\"healthcare_ontology\", \"jena\", \"http://localhost:3030/healthcare\")\n",
"# Register triplet store (using in-memory for demo)\n",
"store = triplet_manager.register_store(\"healthcare_ontology\", \"jena\", \"http://localhost:3030/healthcare\")\n",
"\n",
"# Convert ontology to triples and add to store\n",
"# In production, this would load the OWL ontology\n",
@@ -477,7 +477,7 @@
"\n",
"# Add triples using Semantica\n",
"for triple in sample_triples:\n",
" triple_manager.add_triple(triple, store_id=\"healthcare_ontology\")\n",
" triplet_manager.add_triple(triple, store_id=\"healthcare_ontology\")\n",
"\n",
"print(f\" - Triples added: {len(sample_triples)}\")\n",
"print(f\" - SPARQL queries enabled for ontology\")\n"
@@ -531,7 +531,7 @@
" \"context\": {}\n",
" }\n",
" \n",
" # 1. Query ontology using Semantica Triple Store\n",
" # 1. Query ontology using Semantica Triplet Store\n",
" sparql_query = f\"\"\"\n",
" SELECT ?concept WHERE {{\n",
" ?concept rdfs:label ?label .\n",
@@ -769,12 +769,12 @@
"2. **Materialized Knowledge Graphs**: Semantica's KG modules enable building persistent knowledge graphs from medical ontologies and documents\n",
"3. **Virtual Data Integration**: Semantica's DBIngestor allows virtual integration with EHRs without data replication\n",
"4. **Hybrid Search**: Semantica's HybridSearch combines vector similarity with knowledge graph queries\n",
"5. **Query Orchestration**: Semantica's Reasoning and Triple Store modules enable dynamic query orchestration\n",
"6. **Explainability**: Semantica's ExplanationGenerator provides traceable, explainable answers\n",
"5. **Query Orchestration**: Semantica's Reasoning and Triplet Store modules enable dynamic query orchestration\n",
"6. **Explainability**: Semantica's ExplanationGenerator provides traceable, explainable answers\n",
"\n",
"### Semantica-Specific Performance Considerations\n",
"\n",
"- **Vector Store**: Use Semantica's VectorStore with appropriate backend (FAISS for local, Pinecone/Weaviate for cloud)\n",
"- **Vector Store**: Use Semantica's VectorStore with appropriate backend (FAISS for local, Weaviate for cloud)\n",
"- **Graph Analytics**: Leverage Semantica's GraphAnalyzer for efficient centrality and community detection\n",
"- **Pipeline Execution**: Use Semantica's ExecutionEngine for parallel execution of pipeline steps\n",
"- **Caching**: Utilize Semantica's ContextRetriever caching for frequently accessed contexts\n",
@@ -30,7 +30,7 @@
"- **Parsing**: MCPParser, JSONParser, StructuredDataParser, DocumentParser\n",
"- **Extraction**: NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
"- **KG**: GraphBuilder, GraphValidator, EntityResolver, GraphAnalyzer\n",
"- **Triple Store**: TripleStore, TripleManager, QueryEngine\n",
"- **Triplet Store**: TripletStore, TripletManager, QueryEngine\n",
"- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
"- **Quality**: KGQualityAssessor, ValidationEngine\n",
"- **Export**: JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
@@ -67,9 +67,8 @@
"from semantica.parse import MCPParser, JSONParser, StructuredDataParser, DocumentParser\n",
"from semantica.semantic_extract import NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
"from semantica.kg import GraphBuilder, GraphValidator, EntityResolver, GraphAnalyzer\n",
"from semantica.triple_store import TripleStore, TripleManager, QueryEngine\n",
"from semantica.triplet_store import TripletStore, TripletManager, QueryEngine\n",
"from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
"from semantica.export import JSONExporter, RDFExporter, OWLExporter, ReportGenerator\n",
"from semantica.visualization import KGVisualizer, OntologyVisualizer, TemporalVisualizer\n",
"import json\n",
@@ -402,7 +401,7 @@
"source": [
"## Step 5: Build Healthcare Knowledge Graph\n",
"\n",
"Build a knowledge graph from the extracted medical entities and relationships, then store in triple store.\n"
"Build a knowledge graph from the extracted medical entities and relationships, then store in triplet store.\n"
]
},
{
@@ -426,13 +425,17 @@
"# Analyze graph structure\n",
"metrics = graph_analyzer.compute_metrics(resolved_kg)\n",
"\n",
"# Store in triple store\n",
"triple_store = TripleStore()\n",
"triple_manager = TripleManager()\n",
"# Store in triplet store\n",
"# triplet_store = TripletStore() # TripletStore is a configuration dataclass\n",
"triplet_manager = TripletManager()\n",
"query_engine = QueryEngine()\n",
"\n",
"triple_store.add_knowledge_graph(resolved_kg)\n",
"triple_manager.manage_triples(resolved_kg)\n",
"# Register default store (in-memory for demo)\n",
"store = triplet_manager.register_store(\"medical_kg\", \"jena\", \"http://localhost:3030/medical\")\n",
"\n",
"# Convert KG to triples and add to store (simplified)\n",
"# In a real scenario, we would convert entities/relations to triples first\n",
"# triplet_manager.add_triples(triples, store_id=\"medical_kg\")\n",
"\n",
"print(f\" Entities: {len(resolved_kg.get('entities', []))}\")\n",
"print(f\" Relationships: {len(resolved_kg.get('relationships', []))}\")\n",
@@ -22,13 +22,13 @@
"- **Extraction**: NERExtractor, RelationExtractor, CoreferenceResolver\n",
"- **KG**: GraphBuilder, TemporalGraphQuery, GraphValidator, EntityResolver\n",
"- **Ontology**: OntologyGenerator, ClassInferrer, PropertyGenerator, OntologyValidator\n",
"- **Triple Store**: TripleStore, TripleManager, QueryEngine\n",
"- **Triplet Store**: TripletStore, TripletManager, QueryEngine\n",
"- **Export**: RDFExporter, OWLExporter, JSONExporter\n",
"- **Visualization**: KGVisualizer, OntologyVisualizer, TemporalVisualizer\n",
"\n",
"### Pipeline\n",
"\n",
"**Patient Records → Parse → Extract Medical Entities → Build Temporal KG → Generate Ontology → Store in Triple Store → Query History → Export → Visualize**\n",
"**Patient Records → Parse → Extract Medical Entities → Build Temporal KG → Generate Ontology → Store in Triplet Store → Query History → Export → Visualize**\n",
"\n",
"## Installation\n",
"\n",
@@ -58,7 +58,7 @@
"from semantica.semantic_extract import NERExtractor, RelationExtractor, CoreferenceResolver\n",
"from semantica.kg import GraphBuilder, TemporalGraphQuery, GraphValidator, EntityResolver\n",
"from semantica.ontology import OntologyGenerator, ClassInferrer, PropertyGenerator, OntologyValidator\n",
"from semantica.triple_store import TripleStore, TripleManager, QueryEngine\n",
"from semantica.triplet_store import TripletStore, TripletManager, QueryEngine\n",
"from semantica.export import RDFExporter, OWLExporter, JSONExporter\n",
"from semantica.visualization import KGVisualizer, OntologyVisualizer, TemporalVisualizer\n",
"import tempfile\n",
@@ -261,9 +261,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Store in Triple Store and Query\n",
"## Step 5: Store in Triplet Store and Query\n",
"\n",
"Store knowledge graph in triple store and query medical history.\n"
"Store knowledge graph in triplet store and query medical history.\n"
]
},
{
@@ -272,12 +272,12 @@
"metadata": {},
"outputs": [],
"source": [
"triple_store = TripleStore()\n",
"triple_manager = TripleManager()\n",
"triplet_store = TripletStore()\n",
"triple_manager = TripletManager()\n",
"query_engine = QueryEngine()\n",
"temporal_query = TemporalGraphQuery()\n",
"\n",
"triple_store.store_knowledge_graph(patient_kg)\n",
"triplet_store.store_knowledge_graph(patient_kg)\n",
"\n",
"patient_id = \"P001\"\n",
"start_time = \"2024-01-01\"\n",
@@ -290,7 +290,7 @@
" end_time=end_time\n",
")\n",
"\n",
"print(f\"Stored patient knowledge graph in triple store\")\n",
"print(f\"Stored patient knowledge graph in triplet store\")\n",
"print(f\"Retrieved {len(medical_history.get('entities', []))} medical events for patient {patient_id}\")\n"
]
},
@@ -326,7 +326,7 @@
"temporal_viz = temporal_visualizer.visualize_timeline(patient_kg, output=\"interactive\")\n",
"\n",
"print(f\"Total modules used: 20+\")\n",
"print(f\"Pipeline complete: Patient Records → Parse → Extract → Temporal KG → Ontology → Triple Store → Query → Export → Visualize\")\n"
"print(f\"Pipeline complete: Patient Records → Parse → Extract → Temporal KG → Ontology → Triplet Store → Query → Export → Visualize\")\n"
]
}
],
+3 -3
View File
@@ -32,7 +32,7 @@ from semantica import Semantica
core = Semantica(
llm_provider="openai",
embedding_model="text-embedding-3-large",
vector_store="pinecone",
vector_store="weaviate",
graph_db="neo4j"
)
@@ -279,7 +279,7 @@ owl_ontology = ontology.to_owl()
rdf_ontology = ontology.to_rdf()
turtle_ontology = ontology.to_turtle()
# Save to triple store
# Save to triplet store
ontology.save_to_triple_store("http://localhost:9999/blazegraph/sparql")
```
@@ -357,7 +357,7 @@ semantic_chunks = embedder.semantic_chunk(documents)
embeddings = embedder.generate_embeddings(semantic_chunks)
# Store in vector database
vector_store = core.get_vector_store("pinecone")
vector_store = core.get_vector_store("weaviate")
vector_store.store_embeddings(semantic_chunks, embeddings)
# Semantic search
+3 -3
View File
@@ -66,8 +66,8 @@ graph TB
### Knowledge Graphs
- **`semantica.kg`** - Knowledge graph construction
- **`semantica.vector_store`** - Vector storage (Pinecone, Weaviate, FAISS)
- **`semantica.triple_store`** - RDF triple storage (Jena, Blazegraph)
- **`semantica.vector_store`** - Vector storage (Weaviate, FAISS)
- **`semantica.triplet_store`** - RDF triplet storage (Jena, Blazegraph)
- **`semantica.graph_store`** - Property graphs (Neo4j, FalkorDB)
### Quality Assurance
@@ -85,7 +85,7 @@ graph TB
4. Semantic Extraction → Entities, relationships, events
5. Graph Construction → Entity resolution, conflict resolution
6. Quality Assurance → Deduplication, validation
7. Storage → Vector, triple, and graph stores
7. Storage → Vector, triplet, and graph stores
8. Application → GraphRAG, agents, analytics
```
-1
View File
@@ -25,7 +25,6 @@ Projects and integrations from the Semantica community.
## 🔌 Integrations
### Vector Databases
- Pinecone
- Weaviate
- Qdrant
- FAISS
+1 -1
View File
@@ -181,7 +181,7 @@ A comprehensive reference of terms and concepts used in Semantica.
**Triple**
: A basic unit of knowledge in RDF, consisting of a subject, predicate, and object (e.g., `<Apple_Inc> <founded_by> <Steve_Jobs>`).
**Triple Store**
**Triplet Store**
: A database designed specifically for storing and querying RDF triples.
---
+10 -11
View File
@@ -15,7 +15,7 @@ Semantica's modules are organized into six logical layers:
| :--- | :--- | :--- |
| **Input Layer** | [Ingest](#ingest-module), [Parse](#parse-module), [Split](#split-module), [Normalize](#normalize-module) | Data ingestion, parsing, chunking, and cleaning |
| **Core Processing** | [Semantic Extract](#semantic-extract-module), [Knowledge Graph](#knowledge-graph-kg-module), [Ontology](#ontology-module), [Reasoning](#reasoning-module) | Entity extraction, graph construction, inference |
| **Storage** | [Embeddings](#embeddings-module), [Vector Store](#vector-store-module), [Graph Store](#graph-store-module), [Triple Store](#triple-store-module) | Vector and graph persistence |
| **Storage** | [Embeddings](#embeddings-module), [Vector Store](#vector-store-module), [Graph Store](#graph-store-module), [Triplet Store](#triplet-store-module) | Vector, graph, and triplet persistence |
| **Quality Assurance** | [Deduplication](#deduplication-module), [Conflicts](#conflicts-module) | Data quality and consistency |
| **Context & Memory** | [Context](#context-module), [Seed](#seed-module) | Agent memory and foundation data |
| **Output & Orchestration** | [Export](#export-module), [Visualization](#visualization-module), [Pipeline](#pipeline-module) | Export, visualization, and workflow management |
@@ -468,7 +468,7 @@ print(f"Similarity: {similarity:.3f}")
**Key Features:**
- Multiple backend support (FAISS, Pinecone, Weaviate, Qdrant, Milvus)
- Multiple backend support (FAISS, Weaviate, Qdrant, Milvus)
- Hybrid search (vector + keyword)
- Metadata filtering
- Batch operations
@@ -480,7 +480,6 @@ print(f"Similarity: {similarity:.3f}")
- `VectorStore` — Main vector store interface
- `FAISSAdapter` — FAISS integration
- `PineconeAdapter` — Pinecone integration
- `WeaviateAdapter` — Weaviate integration
- `HybridSearch` — Combine vector and keyword search
- `VectorRetriever` — Retrieve relevant vectors
@@ -563,15 +562,15 @@ results = store.execute_query("MATCH (p:Person) RETURN p.name")
---
### Triple Store Module
### Triplet Store Module
!!! abstract "Purpose"
RDF triple store integration for semantic web applications. Supports SPARQL queries and multiple backends.
RDF triplet store integration for semantic web applications. Supports SPARQL queries and multiple backends.
**Key Features:**
- Multi-backend support (Blazegraph, Jena, RDF4J, Virtuoso)
- CRUD operations for RDF triples
- CRUD operations for RDF triplets
- SPARQL query execution and optimization
- Bulk data loading with progress tracking
- Query caching and optimization
@@ -580,7 +579,7 @@ results = store.execute_query("MATCH (p:Person) RETURN p.name")
**Components:**
- `TripleManager` — Main triple store management coordinator
- `TripletManager` — Main triplet store management coordinator
- `QueryEngine` — SPARQL query execution and optimization
- `BulkLoader` — High-volume data loading with progress tracking
- `BlazegraphAdapter` — Blazegraph integration
@@ -601,9 +600,9 @@ results = store.execute_query("MATCH (p:Person) RETURN p.name")
**Quick Example:**
```python
from semantica.triple_store import TripleManager, execute_query
from semantica.triplet_store import TripletManager, execute_query
manager = TripleManager()
manager = TripletManager()
store = manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
# Add triple
@@ -617,7 +616,7 @@ result = manager.add_triple({
query_result = execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10", store)
```
**API Reference**: [Triple Store Module](reference/triple_store.md)
**API Reference**: [Triplet Store Module](reference/triplet_store.md)
---
@@ -1114,7 +1113,7 @@ new_facts = inference_engine.forward_chain(kg, rule_manager)
| **Embeddings** | `semantica.embeddings` | `EmbeddingGenerator` | Vector generation |
| **Vector Store** | `semantica.vector_store` | `VectorStore` | Vector storage |
| **Graph Store** | `semantica.graph_store` | `GraphStore` | Graph database |
| **Triple Store** | `semantica.triple_store` | `TripleManager` | RDF storage |
| **Triplet Store** | `semantica.triplet_store` | `TripletManager` | RDF storage |
| **Deduplication** | `semantica.deduplication` | `DuplicateDetector` | Duplicate removal |
| **Conflicts** | `semantica.conflicts` | `ConflictDetector` | Conflict resolution |
| **Context** | `semantica.context` | `AgentMemory` | Agent context |
+1 -1
View File
@@ -57,7 +57,7 @@ The **Context Module** provides agents with a persistent, searchable, and struct
The high-level facade that unifies all context operations. It routes data to the appropriate subsystems (Memory, Graph, Vector Store) and manages the lifecycle of context.
#### **Constructor Parameters**
* `vector_store` (Required): The backing vector database instance (e.g., FAISS, Pinecone).
* `vector_store` (Required): The backing vector database instance (e.g., FAISS, Weaviate).
* `knowledge_graph` (Optional): The graph store instance for structured knowledge.
* `token_limit` (Default: `2000`): The maximum number of tokens allowed in short-term memory before pruning occurs.
* `short_term_limit` (Default: `10`): The maximum number of distinct memory items in short-term memory.
+3 -3
View File
@@ -34,7 +34,7 @@ The **Embeddings Module** provides a unified interface for generating vector rep
---
Automatic formatting and validation for FAISS, Pinecone, Qdrant, and Weaviate.
Automatic formatting and validation for FAISS, Qdrant, and Weaviate.
</div>
@@ -122,13 +122,13 @@ print(f"Dimension: {embedder.get_embedding_dimension()}")
---
### VectorEmbeddingManager (The Bridge)
A utility class that prepares raw embeddings for insertion into specific vector databases. It handles formatting differences between backends like FAISS and Pinecone.
A utility class that prepares raw embeddings for insertion into specific vector databases. It handles formatting differences between backends like FAISS and Weaviate.
#### **Core Methods**
| Method | Description |
|--------|-------------|
| `prepare_for_vector_db(embeddings, backend, ...)` | Formats data for the target DB. |
| `prepare_for_vector_db(embeddings, metadata, backend)` | Formats data for the target DB. |
| `validate_dimensions(embeddings, expected_dim)` | Ensures vectors match the index configuration. |
| `batch_prepare(embeddings_list)` | Prepares a batch of embeddings for storage. |
+1 -1
View File
@@ -414,7 +414,7 @@ subgraph = graph_store.execute_query(query, parameters={"ids": node_ids})
## See Also
- [Knowledge Graph Module](kg.md) - Logical layer above Graph Store
- [Triple Store Module](triple_store.md) - RDF-based alternative
- [Triplet Store Module](triplet_store.md) - RDF-based alternative
- [Visualization Module](visualization.md) - Visualizing query results
## Cookbook
+1 -1
View File
@@ -245,7 +245,7 @@ kg.add_triples(inferred_triples)
## See Also
- [Ontology Module](ontology.md) - Source of schema-based rules
- [Triple Store Module](triple_store.md) - Backend for SPARQL reasoning
- [Triplet Store Module](triplet_store.md) - Backend for SPARQL reasoning
- [Modules Guide](../modules.md#quality-assurance) - Consistency checking overview
## Cookbook
@@ -1,6 +1,6 @@
# Triple Store
# Triplet Store
> **Store and query RDF triples with SPARQL support and semantic reasoning using industry-standard triple stores.**
> **Store and query RDF triplets with SPARQL support and semantic reasoning using industry-standard triplet stores.**
---
@@ -12,7 +12,7 @@
---
Store subject-predicate-object triples in W3C-compliant RDF format
Store subject-predicate-object triplets in W3C-compliant RDF format
- :material-code-braces:{ .lg .middle } **SPARQL Queries**
@@ -36,7 +36,7 @@
---
Query across multiple triple stores with SPARQL federation
Query across multiple triplet stores with SPARQL federation
- :material-upload-multiple:{ .lg .middle } **Bulk Loading**
@@ -89,29 +89,29 @@
## Main Classes
### TripleManager
### TripletManager
Main coordinator for triple store operations across multiple backends.
Main coordinator for triplet store operations across multiple backends.
**Methods:**
| Method | Description | Algorithm |
|--------|-------------|-----------|
| `register_store(id, backend, endpoint)` | Register triple store | Store registration |
| `add_triple(triple, store_id)` | Add single triple | Index insertion |
| `add_triples(triples, store_id)` | Batch add triples | Bulk index insertion |
| `register_store(store_id, backend, endpoint)` | Register triplet store | Store registration |
| `add_triple(triple, store_id)` | Add single triplet | Index insertion |
| `add_triples(triples, store_id)` | Batch add triplets | Bulk index insertion |
| `query(sparql, store_id)` | Execute SPARQL query | Query optimization + execution |
| `delete(pattern, store_id)` | Delete matching triples | Pattern matching + deletion |
| `delete(pattern, store_id)` | Delete matching triplets | Pattern matching + deletion |
| `bulk_load(file_path, format, store_id)` | Bulk load from file | Streaming parser + batch insert |
| `get_stats(store_id)` | Get store statistics | Statistics collection |
**Example:**
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
# Initialize manager
manager = TripleManager()
manager = TripletManager()
# Register Blazegraph store
store = manager.register_store(
@@ -193,9 +193,9 @@ SPARQL query execution and optimization engine.
**Example:**
```python
from semantica.triple_store import QueryEngine, TripleManager
from semantica.triplet_store import QueryEngine, TripletManager
manager = TripleManager()
manager = TripletManager()
store = manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph/sparql")
engine = QueryEngine()
@@ -270,9 +270,9 @@ High-performance bulk data loading with progress tracking.
**Example:**
```python
from semantica.triple_store import BulkLoader, TripleManager
from semantica.triplet_store import BulkLoader, TripletManager
manager = TripleManager()
manager = TripletManager()
store = manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph/sparql")
loader = BulkLoader(
@@ -322,7 +322,7 @@ progress = loader.load_from_string(
#### BlazegraphAdapter
High-performance triple store with GPU acceleration support.
High-performance triplet store with GPU acceleration support.
**Features:**
- High-performance SPARQL query execution
@@ -334,7 +334,7 @@ High-performance triple store with GPU acceleration support.
**Example:**
```python
from semantica.triple_store import BlazegraphAdapter
from semantica.triplet_store import BlazegraphAdapter
adapter = BlazegraphAdapter(
endpoint="http://localhost:9999/blazegraph/sparql",
@@ -376,7 +376,7 @@ results = adapter.query("""
Full-featured RDF framework with TDB2 storage.
**Features:**
- TDB2 native triple store
- TDB2 native triplet store
- SHACL validation
- Inference engines (RDFS, OWL)
- Fuseki SPARQL server
@@ -385,7 +385,7 @@ Full-featured RDF framework with TDB2 storage.
**Example:**
```python
from semantica.triple_store import JenaAdapter
from semantica.triplet_store import JenaAdapter
adapter = JenaAdapter(
tdb_directory="./tdb2_data",
@@ -451,7 +451,7 @@ Java-based RDF framework with multiple storage backends.
**Example:**
```python
from semantica.triple_store import RDF4JAdapter
from semantica.triplet_store import RDF4JAdapter
adapter = RDF4JAdapter(
server_url="http://localhost:8080/rdf4j-server",
@@ -497,7 +497,7 @@ Enterprise-grade RDF store with SQL integration.
**Example:**
```python
from semantica.triple_store import VirtuosoAdapter
from semantica.triplet_store import VirtuosoAdapter
adapter = VirtuosoAdapter(
host="localhost",
@@ -534,10 +534,10 @@ results = adapter.query(f"""
## Convenience Functions
Quick access to triple store operations:
Quick access to triplet store operations:
```python
from semantica.triple_store import (
from semantica.triplet_store import (
add_triple,
add_triples,
execute_query,
@@ -579,9 +579,9 @@ export_graph(
## Dataclasses
### TripleStore
### TripletStore
Configuration dataclass for triple store instances.
Configuration dataclass for triplet store instances.
**Attributes:**
@@ -649,35 +649,35 @@ Bulk loading progress dataclass.
```bash
# General settings
export TRIPLE_STORE_DEFAULT_BACKEND=blazegraph
export TRIPLE_STORE_BATCH_SIZE=10000
export TRIPLE_STORE_TIMEOUT=30
export TRIPLET_STORE_DEFAULT_BACKEND=blazegraph
export TRIPLET_STORE_BATCH_SIZE=10000
export TRIPLET_STORE_TIMEOUT=30
# Blazegraph settings
export TRIPLE_STORE_BLAZEGRAPH_ENDPOINT=http://localhost:9999/blazegraph/sparql
export TRIPLE_STORE_BLAZEGRAPH_NAMESPACE=kb
export TRIPLET_STORE_BLAZEGRAPH_ENDPOINT=http://localhost:9999/blazegraph/sparql
export TRIPLET_STORE_BLAZEGRAPH_NAMESPACE=kb
# Jena settings
export TRIPLE_STORE_JENA_TDB_DIRECTORY=./tdb2_data
export TRIPLE_STORE_JENA_INFERENCE=rdfs
export TRIPLET_STORE_JENA_TDB_DIRECTORY=./tdb2_data
export TRIPLET_STORE_JENA_INFERENCE=rdfs
# RDF4J settings
export TRIPLE_STORE_RDF4J_SERVER_URL=http://localhost:8080/rdf4j-server
export TRIPLE_STORE_RDF4J_REPOSITORY_ID=my_repo
export TRIPLET_STORE_RDF4J_SERVER_URL=http://localhost:8080/rdf4j-server
export TRIPLET_STORE_RDF4J_REPOSITORY_ID=my_repo
# Virtuoso settings
export TRIPLE_STORE_VIRTUOSO_HOST=localhost
export TRIPLE_STORE_VIRTUOSO_PORT=1111
export TRIPLE_STORE_VIRTUOSO_USER=dba
export TRIPLE_STORE_VIRTUOSO_PASSWORD=dba
export TRIPLET_STORE_VIRTUOSO_HOST=localhost
export TRIPLET_STORE_VIRTUOSO_PORT=1111
export TRIPLET_STORE_VIRTUOSO_USER=dba
export TRIPLET_STORE_VIRTUOSO_PASSWORD=dba
```
### YAML Configuration
```yaml
# config.yaml - Triple Store Configuration
# config.yaml - Triplet Store Configuration
triple_store:
triplet_store:
backend: blazegraph # blazegraph, jena, rdf4j, virtuoso
batch_size: 10000
timeout: 30
+8 -46
View File
@@ -1,6 +1,6 @@
# Vector Store
> **Unified vector database interface supporting FAISS, Pinecone, Weaviate, Qdrant, and Milvus with Hybrid Search.**
> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, and Milvus with Hybrid Search.**
---
@@ -12,7 +12,7 @@
---
Seamlessly switch between FAISS (Local), Pinecone, Weaviate, Qdrant, and Milvus
Seamlessly switch between FAISS (Local), Weaviate, Qdrant, and Milvus
- :material-magnify-plus:{ .lg .middle } **Hybrid Search**
@@ -230,7 +230,6 @@ results = searcher.search(
Backend-specific implementations:
- `FAISSAdapter`: Local, in-memory/disk.
- `PineconeAdapter`: Managed cloud service.
- `WeaviateAdapter`: Schema-aware vector DB.
- `QdrantAdapter`: Rust-based high-performance DB.
- `MilvusAdapter`: Scalable cloud-native DB.
@@ -265,41 +264,6 @@ query = np.random.rand(768).astype('float32')
distances, indices = adapter.search(index, query, k=10)
```
#### PineconeAdapter
Managed cloud vector database.
**Helper Classes:**
- `PineconeIndex`: Index management
- `PineconeQuery`: Query operations
- `PineconeMetadata`: Metadata handling
**Example:**
```python
from semantica.vector_store import PineconeAdapter
adapter = PineconeAdapter(api_key="your-key", environment="us-west1-gcp")
adapter.connect()
# Create index
index = adapter.create_index("my-index", dimension=768, metric="cosine")
# Upsert with metadata
adapter.upsert_vectors(
vectors=[[0.1, 0.2, ...], ...],
ids=["vec_1", "vec_2"],
metadata=[{"category": "news"}, ...]
)
# Query with filter
results = adapter.query_vectors(
query_vector=[0.1, 0.2, ...],
top_k=10,
filter={"category": {"$eq": "news"}}
)
```
#### WeaviateAdapter
Schema-aware vector database with GraphQL.
@@ -716,25 +680,23 @@ print(f"Available methods: {methods}")
### Environment Variables
```bash
export VECTOR_STORE_BACKEND=pinecone
export PINECONE_API_KEY=sk-...
export PINECONE_ENV=us-west1-gcp
export VECTOR_STORE_BACKEND=weaviate
export WEAVIATE_URL=http://localhost:8080
```
### YAML Configuration
```yaml
vector_store:
backend: faiss # or pinecone, weaviate, etc.
backend: faiss # or weaviate, qdrant, milvus
dimension: 1536
metric: cosine
faiss:
index_type: HNSW
pinecone:
environment: us-west1-gcp
index_name: my-index
weaviate:
url: http://localhost:8080
```
---
@@ -777,7 +739,7 @@ print(f"Context: {context}")
**Solution**: Ensure your embedding model dimension (e.g., 1536 for OpenAI) matches the VectorStore dimension.
**Issue**: FAISS index not saved.
**Solution**: Call `store.save("index.faiss")` explicitly for local FAISS indices, or use a persistent backend like Pinecone/Qdrant.
**Solution**: Call `store.save("index.faiss")` explicitly for local FAISS indices, or use a persistent backend like Weaviate/Qdrant.
---
+1 -1
View File
@@ -123,7 +123,7 @@ nav:
- Seed: reference/seed.md
- Semantic Extract: reference/semantic_extract.md
- Split: reference/split.md
- Triple Store: reference/triple_store.md
- Triplet Store: reference/triplet_store.md
- Utils: reference/utils.md
- Vector Store: reference/vector_store.md
- Visualization: reference/visualization.md
-1
View File
@@ -61,7 +61,6 @@ dependencies = [
"librosa>=0.9.0",
"opencv-python>=4.6.0",
"faiss-cpu>=1.7.0",
"pinecone-client>=2.2.0",
"weaviate-client>=3.15.0",
"qdrant-client>=1.3.0",
"neo4j>=5.0.0",
+7 -7
View File
@@ -84,7 +84,7 @@ class _SemanticaModules:
self._normalize = None
self._export = None
self._vector_store = None
self._triple_store = None
self._triplet_store = None
self._graph_store = None
self._ontology = None
self._evals = None
@@ -160,11 +160,11 @@ class _SemanticaModules:
return self._vector_store
@property
def triple_store(self):
"""Access triple store module."""
if self._triple_store is None:
self._triple_store = _ModuleProxy("triple_store")
return self._triple_store
def triplet_store(self):
"""Access triplet store module."""
if self._triplet_store is None:
self._triplet_store = _ModuleProxy("triplet_store")
return self._triplet_store
@property
def graph_store(self):
@@ -289,7 +289,7 @@ def __getattr__(name: str):
"normalize",
"export",
"vector_store",
"triple_store",
"triplet_store",
"graph_store",
"ontology",
"evals",
+55 -17
View File
@@ -302,7 +302,6 @@ class Semantica:
try:
self.logger.info("Executing processing pipeline")
# Track pipeline execution
pipeline_tracking_id = self.progress_tracker.start_tracking(
file=str(data) if isinstance(data, (str, Path)) else None,
module="pipeline",
@@ -310,23 +309,39 @@ class Semantica:
message="Executing pipeline",
)
# Validate pipeline
if isinstance(pipeline, dict):
pipeline = self._create_pipeline_from_dict(pipeline)
# Validate pipeline object
if not hasattr(pipeline, "execute"):
raise ProcessingError("Pipeline must have execute() method")
execution_engine = None
execution_result = None
try:
from ..pipeline import ExecutionEngine, Pipeline
if isinstance(pipeline, Pipeline):
execution_engine = ExecutionEngine()
except ImportError:
execution_engine = None
if execution_engine is None and not hasattr(pipeline, "execute"):
raise ProcessingError(
"Pipeline must be a Pipeline object or have execute() method"
)
# Allocate resources
resources = self._allocate_resources(pipeline)
try:
# Execute pipeline
result = pipeline.execute(data)
# Collect metrics
metrics = self._collect_metrics(pipeline)
if execution_engine is not None:
execution_result = execution_engine.execute_pipeline(
pipeline, data
)
success = execution_result.success
output = execution_result.output
metrics = execution_result.metrics
else:
output = pipeline.execute(data)
metrics = self._collect_metrics(pipeline)
success = True
if pipeline_tracking_id:
self.progress_tracker.stop_tracking(
@@ -334,8 +349,8 @@ class Semantica:
)
return {
"success": True,
"output": result,
"success": success,
"output": output,
"metrics": metrics,
"metadata": {
"pipeline": str(pipeline),
@@ -344,7 +359,6 @@ class Semantica:
}
finally:
# Release resources
self._release_resources(resources)
except Exception as e:
@@ -567,13 +581,37 @@ class Semantica:
Pipeline object or configuration dict (if pipeline module not available)
"""
try:
# Try to use PipelineBuilder if available
from ..pipeline import PipelineBuilder
pipeline_builder = PipelineBuilder()
return pipeline_builder.build_from_config(pipeline_config)
if not pipeline_config:
pipeline_builder.add_step("default_step", "default")
return pipeline_builder.build("default_pipeline")
steps_config = pipeline_config.get("steps")
if isinstance(steps_config, list) and steps_config and isinstance(
steps_config[0], str
):
converted_steps = [
{"name": name, "type": name, "config": {}}
for name in steps_config
]
normalized_config: Dict[str, Any] = {
"name": pipeline_config.get("name", "default_pipeline"),
"steps": converted_steps,
}
if "parallelism" in pipeline_config:
normalized_config["parallelism"] = pipeline_config["parallelism"]
return pipeline_builder.build_pipeline(normalized_config)
if "steps" in pipeline_config:
return pipeline_builder.build_pipeline(pipeline_config)
pipeline_builder.add_step("default_step", "default")
return pipeline_builder.build("default_pipeline")
except ImportError:
# Fallback: return config as-is if pipeline module not available
self.logger.debug("Pipeline module not available, using config directly")
return pipeline_config
+2 -10
View File
@@ -447,14 +447,6 @@ from semantica.embeddings import VectorEmbeddingManager
manager = VectorEmbeddingManager()
# Prepare for Pinecone
pinecone_data = manager.prepare_for_vector_db(
embeddings,
metadata=metadata,
backend="pinecone",
namespace="my_namespace"
)
# Prepare for Weaviate
weaviate_data = manager.prepare_for_vector_db(
embeddings,
@@ -486,9 +478,9 @@ from semantica.embeddings import VectorEmbeddingManager
manager = VectorEmbeddingManager()
# Validate dimensions for specific backend
is_valid = manager.validate_dimensions(embeddings, backend="pinecone")
is_valid = manager.validate_dimensions(embeddings, backend="weaviate")
if is_valid:
print("Embeddings meet Pinecone requirements")
print("Embeddings meet Weaviate requirements")
else:
print("Embeddings do not meet requirements")
```
@@ -9,7 +9,7 @@ Key Features:
- Validate embedding dimensions for different backends
- Normalize embeddings for vector DB requirements
- Create metadata compatible with vector DBs
- Integration helpers for FAISS, Pinecone, Weaviate, Qdrant, Milvus
- Integration helpers for FAISS, Weaviate, Qdrant, Milvus
Example Usage:
>>> from semantica.embeddings import VectorEmbeddingManager
@@ -36,7 +36,6 @@ class VectorEmbeddingManager:
Supported Backends:
- FAISS: Local vector storage
- Pinecone: Cloud vector database
- Weaviate: GraphQL-based vector database
- Qdrant: Vector similarity search engine
- Milvus: Open-source vector database
@@ -50,7 +49,7 @@ class VectorEmbeddingManager:
... backend="faiss"
... )
>>> # Validate dimensions
>>> is_valid = manager.validate_dimensions(embeddings, backend="pinecone")
>>> is_valid = manager.validate_dimensions(embeddings, backend="weaviate")
"""
def __init__(self, embedding_generator: Optional[EmbeddingGenerator] = None):
@@ -67,7 +66,6 @@ class VectorEmbeddingManager:
# Backend-specific dimension requirements
self.backend_requirements = {
"faiss": {"min_dim": 1, "max_dim": None, "dtype": np.float32},
"pinecone": {"min_dim": 1, "max_dim": 20000, "dtype": np.float32},
"weaviate": {"min_dim": 1, "max_dim": None, "dtype": np.float32},
"qdrant": {"min_dim": 1, "max_dim": None, "dtype": np.float32},
"milvus": {"min_dim": 1, "max_dim": 32768, "dtype": np.float32},
@@ -90,7 +88,7 @@ class VectorEmbeddingManager:
Args:
embeddings: Embeddings array (n_samples, embedding_dim) or (embedding_dim,)
metadata: Optional list of metadata dictionaries (one per embedding)
backend: Vector DB backend ("faiss", "pinecone", "weaviate", "qdrant", "milvus")
backend: Vector DB backend ("faiss", "weaviate", "qdrant", "milvus")
normalize: Whether to normalize embeddings (default: True)
**options: Additional backend-specific options
@@ -108,7 +106,7 @@ class VectorEmbeddingManager:
>>> embeddings = np.random.rand(10, 384).astype(np.float32)
>>> metadata = [{"text": f"doc_{i}"} for i in range(10)]
>>> result = manager.prepare_for_vector_db(
... embeddings, metadata, backend="pinecone"
... embeddings, metadata, backend="weaviate"
... )
"""
if backend.lower() not in self.backend_requirements:
@@ -228,7 +226,7 @@ class VectorEmbeddingManager:
bool: True if dimensions are valid, False otherwise
Example:
>>> is_valid = manager.validate_dimensions(embeddings, backend="pinecone")
>>> is_valid = manager.validate_dimensions(embeddings, backend="weaviate")
"""
if backend.lower() not in self.backend_requirements:
self.logger.warning(f"Unknown backend: {backend}, skipping validation")
@@ -312,7 +310,7 @@ class VectorEmbeddingManager:
Example:
>>> metadata = [{"text": "doc1", "category": "science"}]
>>> formatted = manager.create_metadata(metadata, backend="pinecone")
>>> formatted = manager.create_metadata(metadata, backend="weaviate")
"""
formatted = []
@@ -321,16 +319,7 @@ class VectorEmbeddingManager:
formatted_meta = meta.copy()
# Backend-specific formatting
if backend.lower() == "pinecone":
# Pinecone has specific metadata requirements
# Remove None values and ensure types are compatible
formatted_meta = {
k: v
for k, v in formatted_meta.items()
if v is not None
and isinstance(v, (str, int, float, bool, list))
}
elif backend.lower() == "weaviate":
if backend.lower() == "weaviate":
# Weaviate uses specific property types
# Ensure values are compatible
formatted_meta = {
@@ -374,8 +363,6 @@ class VectorEmbeddingManager:
# Add backend-specific details
if backend.lower() == "faiss":
info["index_type"] = options.get("index_type", "flat")
elif backend.lower() == "pinecone":
info["namespace"] = options.get("namespace", "default")
elif backend.lower() == "weaviate":
info["class_name"] = options.get("class_name", "Document")
+1 -1
View File
@@ -62,7 +62,7 @@ OWL Export:
Vector Export:
- Vector Serialization: Multiple format support (JSON, NumPy, Binary, FAISS)
- Vector Store Integration: Format conversion for Pinecone, Weaviate, Qdrant, FAISS
- Vector Store Integration: Format conversion for Weaviate, Qdrant, FAISS
- Metadata Association: Vector-to-metadata mapping and serialization
- Batch Export: Efficient batch vector export processing
- Multi-dimensional Support: Variable dimension vector handling
+1 -1
View File
@@ -110,7 +110,7 @@ OWL Export:
Vector Export:
- Vector Serialization: Multiple format support (JSON, NumPy, Binary, FAISS)
- Vector Store Integration: Format conversion for Pinecone, Weaviate, Qdrant, FAISS
- Vector Store Integration: Format conversion for Weaviate, Qdrant, FAISS
- Metadata Association: Vector-to-metadata mapping and serialization
- Batch Export: Efficient batch vector export processing
- Multi-dimensional Support: Variable dimension vector handling
+6 -29
View File
@@ -7,7 +7,7 @@ embedding systems.
Key Features:
- Multiple vector format export (JSON, NumPy, Binary, FAISS)
- Vector store integration (Pinecone, Weaviate, Qdrant, FAISS)
- Vector store integration (Weaviate, Qdrant, FAISS)
- Metadata and document association
- Batch vector export
- Multi-dimensional vector support
@@ -16,7 +16,7 @@ Example Usage:
>>> from semantica.export import VectorExporter
>>> exporter = VectorExporter(format="json", include_metadata=True)
>>> exporter.export(vectors, "vectors.json")
>>> exporter.export_for_vector_store(vectors, "pinecone.json", vector_store_type="pinecone")
>>> exporter.export_for_vector_store(vectors, "weaviate.json", vector_store_type="weaviate")
Author: Semantica Contributors
License: MIT
@@ -43,7 +43,7 @@ class VectorExporter:
Features:
- Multiple vector format export (JSON, NumPy, Binary, FAISS)
- Vector store integration (Pinecone, Weaviate, Qdrant, FAISS)
- Vector store integration (Weaviate, Qdrant, FAISS)
- Metadata and document association
- Batch vector export
- Multi-dimensional vector support
@@ -471,7 +471,7 @@ class VectorExporter:
self,
vectors: List[Dict[str, Any]],
file_path: Union[str, Path],
vector_store_type: str = "pinecone",
vector_store_type: str = "weaviate",
**options,
) -> None:
"""
@@ -480,12 +480,10 @@ class VectorExporter:
Args:
vectors: List of vector dictionaries
file_path: Output file path
vector_store_type: Vector store type ('pinecone', 'weaviate', 'qdrant', 'faiss')
vector_store_type: Vector store type ('weaviate', 'qdrant', 'faiss')
**options: Additional options
"""
if vector_store_type == "pinecone":
self._export_pinecone_format(vectors, file_path, **options)
elif vector_store_type == "weaviate":
if vector_store_type == "weaviate":
self._export_weaviate_format(vectors, file_path, **options)
elif vector_store_type == "qdrant":
self._export_qdrant_format(vectors, file_path, **options)
@@ -495,27 +493,6 @@ class VectorExporter:
# Default to JSON
self._export_json(vectors, Path(file_path), {}, **options)
def _export_pinecone_format(
self, vectors: List[Dict[str, Any]], file_path: Path, **options
) -> None:
"""Export in Pinecone format."""
pinecone_data = []
for vec_data in vectors:
vector_id = vec_data.get("id") or vec_data.get("vector_id", "")
vector = vec_data.get("vector") or vec_data.get("embedding", [])
metadata = vec_data.get("metadata", {})
if "text" in vec_data and self.include_text:
metadata["text"] = vec_data["text"]
pinecone_data.append(
{"id": vector_id, "values": vector, "metadata": metadata}
)
export_data = {"vectors": pinecone_data}
write_json_file(export_data, file_path, indent=2)
def _export_weaviate_format(
self, vectors: List[Dict[str, Any]], file_path: Path, **options
) -> None:
+1 -1
View File
@@ -148,7 +148,7 @@ class PipelineTemplateManager:
{
"name": "store_vectors",
"type": "store_vectors",
"config": {"store": "pinecone"},
"config": {"store": "weaviate"},
"dependencies": ["embed"],
},
],
+3 -2
View File
@@ -632,7 +632,7 @@ builder = template_manager.create_pipeline_from_template(
"rag_pipeline",
chunk={"chunk_size": 512},
embed={"model": "text-embedding-3-large"},
store_vectors={"store": "pinecone"}
store_vectors={"store": "weaviate"}
)
pipeline = builder.build()
@@ -1124,7 +1124,8 @@ builder = template_manager.create_pipeline_from_template(
ingest={"source": "./documents"},
chunk={"chunk_size": 512, "overlap": 50},
embed={"model": "text-embedding-3-large", "batch_size": 32},
store_vectors={"store": "pinecone", "index_name": "documents"}
# Step-specific overrides
store_vectors={"store": "weaviate", "index_name": "documents"}
)
pipeline = builder.build()
+7 -7
View File
@@ -45,7 +45,7 @@ print(f"Inferred {len(results)} new facts")
from semantica.reasoning import SPARQLReasoner
# Create SPARQL reasoner
reasoner = SPARQLReasoner(triple_store=kg)
reasoner = SPARQLReasoner(triplet_store=kg)
# Execute query
query = "SELECT ?s ?p ?o WHERE { ?s ?p ?o }"
@@ -190,7 +190,7 @@ results = engine.forward_chain()
from semantica.reasoning import SPARQLReasoner
# Create reasoner with knowledge graph
reasoner = SPARQLReasoner(triple_store=kg)
reasoner = SPARQLReasoner(triplet_store=kg)
# Execute SPARQL query
query = """
@@ -212,7 +212,7 @@ for binding in result.bindings:
```python
from semantica.reasoning import SPARQLReasoner
reasoner = SPARQLReasoner(triple_store=kg, enable_inference=True)
reasoner = SPARQLReasoner(triplet_store=kg, enable_inference=True)
# Add inference rule
reasoner.add_inference_rule("IF ?x :type :Company THEN ?x :type :Organization")
@@ -234,7 +234,7 @@ result = reasoner.execute_query(query)
from semantica.reasoning import SPARQLReasoner
reasoner = SPARQLReasoner(
triple_store=kg,
triplet_store=kg,
enable_inference=True,
inference_rules=["rdfs:subClassOf", "rdfs:subPropertyOf"]
)
@@ -1165,7 +1165,7 @@ engine = InferenceEngine(
# Configure SPARQL reasoner
reasoner = SPARQLReasoner(
triple_store=kg,
triplet_store=kg,
enable_inference=True,
query_cache_size=1000
)
@@ -1234,7 +1234,7 @@ for result in results:
print(f"Explanation: {explanation.natural_language}")
# 6. Query with SPARQL reasoning
sparql_reasoner = SPARQLReasoner(triple_store=kg, enable_inference=True)
sparql_reasoner = SPARQLReasoner(triplet_store=kg, enable_inference=True)
query_result = sparql_reasoner.execute_query("SELECT ?x WHERE { ?x :type :Employee }")
```
@@ -1332,7 +1332,7 @@ from semantica.kg import build
kg = build(sources=[...])
# Create SPARQL reasoner with KG
reasoner = SPARQLReasoner(triple_store=kg, enable_inference=True)
reasoner = SPARQLReasoner(triplet_store=kg, enable_inference=True)
# Add inference rules
reasoner.add_inference_rule("IF ?x :type :Company THEN ?x :type :Organization")
+6 -6
View File
@@ -12,7 +12,7 @@ Key Features:
- Query expansion
- Performance optimization
- Error handling and recovery
- Triple store integration
- Triplet store integration
Main Classes:
- SPARQLReasoner: SPARQL-based reasoning engine
@@ -67,7 +67,7 @@ class SPARQLReasoner:
Args:
config: Configuration dictionary
**kwargs: Additional configuration options:
- triple_store: Triple store connection
- triplet_store: Triplet store connection
- enable_inference: Enable inference rules
"""
self.logger = get_logger("sparql_reasoner")
@@ -78,7 +78,7 @@ class SPARQLReasoner:
self.progress_tracker = get_progress_tracker()
self.rule_manager = RuleManager(**self.config)
self.triple_store = self.config.get("triple_store")
self.triplet_store = self.config.get("triplet_store")
self.enable_inference = self.config.get("enable_inference", True)
self.query_cache: Dict[str, Any] = {}
@@ -356,12 +356,12 @@ class SPARQLReasoner:
)
expanded_query = self.expand_query(query, **options)
# Execute query (if triple store available)
# Execute query (if triplet store available)
self.progress_tracker.update_tracking(
tracking_id, message="Executing query..."
)
if self.triple_store:
# This would call the triple store's query method
if self.triplet_store:
# This would call the triplet store's query method
# For now, return empty result
result = SPARQLQueryResult(bindings=[], variables=[])
else:
@@ -1,22 +1,22 @@
"""
Triple Store Module
Triplet Store Module
This module provides comprehensive triple store integration and management
for RDF data storage and querying, supporting multiple triple store backends
This module provides comprehensive triplet store integration and management
for RDF data storage and querying, supporting multiple triplet store backends
with unified interfaces.
Algorithms Used:
Triple Store Management:
Triplet Store Management:
- Store Registration: Store type detection, adapter factory pattern, configuration management, default store selection
- Adapter Pattern: Unified interface for multiple backends (Blazegraph, Jena, RDF4J, Virtuoso), adapter instantiation, backend-specific operation delegation
- Store Selection: Default store resolution, store ID lookup, store validation
CRUD Operations:
- Triple Addition: Single triple insertion, batch triple insertion, triple validation (subject/predicate/object checking, confidence validation), adapter delegation
- Triple Retrieval: Pattern matching (subject/predicate/object filtering), SPARQL query construction, result binding extraction, triple reconstruction
- Triple Deletion: Triple matching, deletion operation delegation, result verification
- Triple Update: Delete-then-add pattern, atomic update operations, conflict detection
- Triplet Addition: Single triplet insertion, batch triplet insertion, triplet validation (subject/predicate/object checking, confidence validation), adapter delegation
- Triplet Retrieval: Pattern matching (subject/predicate/object filtering), SPARQL query construction, result binding extraction, triplet reconstruction
- Triplet Deletion: Triplet matching, deletion operation delegation, result verification
- Triplet Update: Delete-then-add pattern, atomic update operations, conflict detection
Bulk Loading:
- Batch Processing: Chunking algorithm (fixed-size batch creation), batch size optimization, memory management for large datasets
@@ -46,7 +46,7 @@ Store Adapters:
Data Validation:
- Triple Validation: Required field checking (subject, predicate, object), confidence range validation (0-1), URI format validation
- Pre-load Validation: Empty component detection, URI format checking, confidence threshold checking, error/warning categorization
Performance Optimization:
- Batch Size Optimization: Configurable batch size, memory-aware batching, throughput-based optimization
- Connection Pooling: Adapter-level connection management, connection reuse, connection lifecycle management
@@ -55,7 +55,7 @@ Performance Optimization:
Key Features:
- Multi-backend support (Blazegraph, Jena, RDF4J, Virtuoso)
- CRUD operations for RDF triples
- CRUD operations for RDF triplets
- SPARQL query execution and optimization
- Bulk data loading with progress tracking
- Query caching and optimization
@@ -65,41 +65,41 @@ Key Features:
- Configuration management with environment variables and config files
Main Classes:
- TripleManager: Main triple store management coordinator
- TripletManager: Main triplet store management coordinator
- QueryEngine: SPARQL query execution and optimization
- BulkLoader: High-volume data loading
- BlazegraphAdapter: Blazegraph integration adapter
- JenaAdapter: Apache Jena integration adapter
- RDF4JAdapter: Eclipse RDF4J integration adapter
- VirtuosoAdapter: Virtuoso RDF store integration adapter
- TripleStore: Triple store configuration dataclass
- TripletStore: Triplet store configuration dataclass
- QueryResult: Query result representation dataclass
- QueryPlan: Query execution plan dataclass
- LoadProgress: Bulk loading progress dataclass
Convenience Functions:
- register_store: Register triple store wrapper
- add_triple: Add single triple wrapper
- add_triples: Add multiple triples wrapper
- get_triples: Get triples matching pattern wrapper
- delete_triple: Delete triple wrapper
- register_store: Register triplet store wrapper
- add_triple: Add single triplet wrapper
- add_triples: Add multiple triplets wrapper
- get_triples: Get triplets matching pattern wrapper
- delete_triple: Delete triplet wrapper
- execute_query: Execute SPARQL query wrapper
- optimize_query: Optimize SPARQL query wrapper
- bulk_load: Bulk load triples wrapper
- get_triple_store_method: Get triple store method by task and name
- list_available_methods: List registered triple store methods
- bulk_load: Bulk load triplets wrapper
- get_triplet_store_method: Get triplet store method by task and name
- list_available_methods: List registered triplet store methods
Example Usage:
>>> from semantica.triple_store import TripleManager, register_store, add_triple, execute_query
>>> from semantica.triplet_store import TripletManager, register_store, add_triple, execute_query
>>> # Using convenience functions
>>> store = register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
>>> result = add_triple(triple, store_id="main")
>>> query_result = execute_query(sparql_query, store_adapter)
>>> # Using classes directly
>>> manager = TripleManager()
>>> manager = TripletManager()
>>> store = manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
>>> result = manager.add_triple(triple, store_id="main")
>>> from semantica.triple_store import QueryEngine
>>> from semantica.triplet_store import QueryEngine
>>> engine = QueryEngine()
>>> query_result = engine.execute_query(sparql_query, store_adapter)
@@ -109,7 +109,7 @@ License: MIT
from .blazegraph_adapter import BlazegraphAdapter
from .bulk_loader import BulkLoader, LoadProgress
from .config import TripleStoreConfig, triple_store_config
from .config import TripletStoreConfig, triplet_store_config
from .jena_adapter import JenaAdapter
from .methods import (
add_triple,
@@ -117,7 +117,7 @@ from .methods import (
bulk_load,
delete_triple,
execute_query,
get_triple_store_method,
get_triplet_store_method,
get_triples,
list_available_methods,
optimize_query,
@@ -129,13 +129,13 @@ from .methods import (
from .query_engine import QueryEngine, QueryPlan, QueryResult
from .rdf4j_adapter import RDF4JAdapter
from .registry import MethodRegistry, method_registry
from .triple_manager import TripleManager, TripleStore
from .triplet_manager import TripletManager, TripletStore
from .virtuoso_adapter import VirtuosoAdapter
__all__ = [
# Triple management
"TripleManager",
"TripleStore",
"TripletManager",
"TripletStore",
# Store adapters
"BlazegraphAdapter",
"JenaAdapter",
@@ -160,11 +160,11 @@ __all__ = [
"plan_query",
"bulk_load",
"validate_triples",
"get_triple_store_method",
"get_triplet_store_method",
"list_available_methods",
# Configuration and registry
"TripleStoreConfig",
"triple_store_config",
"TripletStoreConfig",
"triplet_store_config",
"MethodRegistry",
"method_registry",
]
@@ -17,7 +17,7 @@ Main Classes:
- BlazegraphAdapter: Main Blazegraph integration adapter
Example Usage:
>>> from semantica.triple_store import BlazegraphAdapter
>>> from semantica.triplet_store import BlazegraphAdapter
>>> adapter = BlazegraphAdapter(endpoint="http://localhost:9999/blazegraph", namespace="kb")
>>> result = adapter.execute_sparql(sparql_query)
>>> load_result = adapter.bulk_load(triples)
@@ -40,7 +40,7 @@ from ..utils.progress_tracker import get_progress_tracker
class BlazegraphAdapter:
"""
Blazegraph triple store adapter.
Blazegraph triplet store adapter.
Blazegraph connection and authentication
SPARQL query execution
@@ -119,7 +119,7 @@ class BlazegraphAdapter:
Query results
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="BlazegraphAdapter",
message="Executing SPARQL query on Blazegraph",
)
@@ -1,7 +1,7 @@
"""
Bulk Loader Module
This module provides high-volume data loading capabilities for triple stores,
This module provides high-volume data loading capabilities for triplet stores,
enabling efficient batch processing with progress tracking and error recovery.
Key Features:
@@ -18,7 +18,7 @@ Main Classes:
- LoadProgress: Bulk loading progress representation dataclass
Example Usage:
>>> from semantica.triple_store import BulkLoader
>>> from semantica.triplet_store import BulkLoader
>>> loader = BulkLoader(batch_size=1000, max_retries=3)
>>> progress = loader.load_triples(triples, store_adapter)
>>> print(f"Loaded {progress.loaded_triples}/{progress.total_triples} triples")
@@ -56,7 +56,7 @@ class LoadProgress:
class BulkLoader:
"""
High-volume data loading system for triple stores.
High-volume data loading system for triplet stores.
High-volume data loading strategies
Batch processing and chunking
@@ -104,7 +104,7 @@ class BulkLoader:
Load progress information
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="BulkLoader",
message=f"Loading {len(triples)} triples in bulk",
)
@@ -1,14 +1,14 @@
"""
Configuration Management Module for Triple Store
Configuration Management Module for Triplet Store
This module provides centralized configuration management for triple store operations,
This module provides centralized configuration management for triplet store operations,
supporting multiple configuration sources including environment variables, config files,
and programmatic configuration.
Supported Configuration Sources:
- Environment variables: TRIPLE_STORE_DEFAULT_STORE, TRIPLE_STORE_BATCH_SIZE, TRIPLE_STORE_ENABLE_CACHING, etc.
- Environment variables: TRIPLET_STORE_DEFAULT_STORE, TRIPLET_STORE_BATCH_SIZE, TRIPLET_STORE_ENABLE_CACHING, etc.
- Config files: YAML, JSON, TOML formats
- Programmatic: Python API for setting triple store configurations
- Programmatic: Python API for setting triplet store configurations
Algorithms Used:
- Environment Variable Parsing: OS-level environment variable access
@@ -19,7 +19,7 @@ Algorithms Used:
- Dictionary Merging: Deep merge algorithms for configuration updates
Key Features:
- Environment variable support for triple store parameters
- Environment variable support for triplet store parameters
- Config file support (YAML, JSON, TOML formats)
- Programmatic configuration via Python API
- Method-specific configuration management
@@ -27,13 +27,13 @@ Key Features:
- Global config instance for easy access
Main Classes:
- TripleStoreConfig: Main configuration manager class for triple store module
- TripletStoreConfig: Main configuration manager class for triplet store module
Example Usage:
>>> from semantica.triple_store.config import triple_store_config
>>> default_store = triple_store_config.get("default_store", default="main")
>>> triple_store_config.set("default_store", "main")
>>> method_config = triple_store_config.get_method_config("add_triple")
>>> from semantica.triplet_store.config import triplet_store_config
>>> default_store = triplet_store_config.get("default_store", default="main")
>>> triplet_store_config.set("default_store", "main")
>>> method_config = triplet_store_config.get_method_config("add_triple")
"""
import os
@@ -43,8 +43,8 @@ from typing import Any, Dict, Optional
from ..utils.logging import get_logger
class TripleStoreConfig:
"""Configuration manager for triple store module - supports .env files, environment variables, and programmatic config."""
class TripletStoreConfig:
"""Configuration manager for triplet store module - supports .env files, environment variables, and programmatic config."""
def __init__(self, config_file: Optional[str] = None):
"""
@@ -53,7 +53,7 @@ class TripleStoreConfig:
Args:
config_file: Optional path to configuration file (YAML, JSON, or TOML)
"""
self.logger = get_logger("triple_store_config")
self.logger = get_logger("triplet_store_config")
self.config_file = config_file
self._config: Dict[str, Any] = {}
self._method_configs: Dict[str, Dict[str, Any]] = {}
@@ -86,40 +86,40 @@ class TripleStoreConfig:
with open(file_path, "r") as f:
config_data = yaml.safe_load(f)
if config_data and "triple_store" in config_data:
self._config.update(config_data["triple_store"])
if config_data and "triplet_store" in config_data:
self._config.update(config_data["triplet_store"])
elif file_path.suffix == ".json":
import json
with open(file_path, "r") as f:
config_data = json.load(f)
if config_data and "triple_store" in config_data:
self._config.update(config_data["triple_store"])
if config_data and "triplet_store" in config_data:
self._config.update(config_data["triplet_store"])
elif file_path.suffix == ".toml":
import tomli
with open(file_path, "rb") as f:
config_data = tomli.load(f)
if config_data and "triple_store" in config_data:
self._config.update(config_data["triple_store"])
if config_data and "triplet_store" in config_data:
self._config.update(config_data["triplet_store"])
except Exception as e:
self.logger.error(f"Failed to load config file: {e}")
def _load_from_env(self) -> None:
"""Load configuration from environment variables."""
env_mappings = {
"TRIPLE_STORE_DEFAULT_STORE": "default_store",
"TRIPLE_STORE_BATCH_SIZE": "batch_size",
"TRIPLE_STORE_ENABLE_CACHING": "enable_caching",
"TRIPLE_STORE_CACHE_SIZE": "cache_size",
"TRIPLE_STORE_ENABLE_OPTIMIZATION": "enable_optimization",
"TRIPLE_STORE_MAX_RETRIES": "max_retries",
"TRIPLE_STORE_RETRY_DELAY": "retry_delay",
"TRIPLE_STORE_TIMEOUT": "timeout",
"TRIPLE_STORE_BLAZEGRAPH_ENDPOINT": "blazegraph_endpoint",
"TRIPLE_STORE_JENA_ENDPOINT": "jena_endpoint",
"TRIPLE_STORE_RDF4J_ENDPOINT": "rdf4j_endpoint",
"TRIPLE_STORE_VIRTUOSO_ENDPOINT": "virtuoso_endpoint",
"TRIPLET_STORE_DEFAULT_STORE": "default_store",
"TRIPLET_STORE_BATCH_SIZE": "batch_size",
"TRIPLET_STORE_ENABLE_CACHING": "enable_caching",
"TRIPLET_STORE_CACHE_SIZE": "cache_size",
"TRIPLET_STORE_ENABLE_OPTIMIZATION": "enable_optimization",
"TRIPLET_STORE_MAX_RETRIES": "max_retries",
"TRIPLET_STORE_RETRY_DELAY": "retry_delay",
"TRIPLET_STORE_TIMEOUT": "timeout",
"TRIPLET_STORE_BLAZEGRAPH_ENDPOINT": "blazegraph_endpoint",
"TRIPLET_STORE_JENA_ENDPOINT": "jena_endpoint",
"TRIPLET_STORE_RDF4J_ENDPOINT": "rdf4j_endpoint",
"TRIPLET_STORE_VIRTUOSO_ENDPOINT": "virtuoso_endpoint",
}
for env_var, config_key in env_mappings.items():
@@ -238,4 +238,4 @@ class TripleStoreConfig:
# Global configuration instance
triple_store_config = TripleStoreConfig()
triplet_store_config = TripletStoreConfig()
@@ -16,7 +16,7 @@ Main Classes:
- JenaAdapter: Main Jena integration adapter
Example Usage:
>>> from semantica.triple_store import JenaAdapter
>>> from semantica.triplet_store import JenaAdapter
>>> adapter = JenaAdapter(endpoint="http://localhost:3030/ds", dataset="default")
>>> result = adapter.add_triples(triples)
>>> query_result = adapter.execute_sparql(sparql_query)
@@ -47,7 +47,7 @@ except ImportError:
class JenaAdapter:
"""
Apache Jena adapter for triple store operations.
Apache Jena adapter for triplet store operations.
Jena connection and configuration
SPARQL query execution
@@ -129,7 +129,7 @@ class JenaAdapter:
Operation status
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="JenaAdapter",
message=f"Adding {len(triples)} triples to Jena model",
)
@@ -1,14 +1,14 @@
"""
Triple Store Methods Module
Triplet Store Methods Module
This module provides all triple store methods as simple, reusable functions for
registering stores, adding triples, querying, and managing triple stores. It supports
This module provides all triplet store methods as simple, reusable functions for
registering stores, adding triples, querying, and managing triplet stores. It supports
multiple approaches and integrates with the method registry for extensibility.
Supported Methods:
Store Registration:
- "default": Default store registration using TripleManager
- "default": Default store registration using TripletManager
- "blazegraph": Blazegraph-specific registration
- "jena": Jena-specific registration
- "rdf4j": RDF4J-specific registration
@@ -78,7 +78,7 @@ Bulk Loading:
- Stream Processing: Iterator-based processing, incremental batch collection
Key Features:
- Multiple triple store operation methods
- Multiple triplet store operation methods
- Store registration with method dispatch
- Method dispatchers with registry support
- Custom method registration capability
@@ -95,11 +95,11 @@ Main Functions:
- optimize_query: Query optimization wrapper
- bulk_load: Bulk loading wrapper
- validate_triples: Triple validation wrapper
- get_triple_store_method: Get triple store method by task and name
- get_triplet_store_method: Get triplet store method by task and name
- list_available_methods: List registered methods
Example Usage:
>>> from semantica.triple_store.methods import register_store, add_triple, execute_query
>>> from semantica.triplet_store.methods import register_store, add_triple, execute_query
>>> store = register_store("main", "blazegraph", "http://localhost:9999/blazegraph", method="default")
>>> result = add_triple(triple, store_id="main", method="default")
>>> query_result = execute_query(sparql_query, store_adapter, method="default")
@@ -109,23 +109,23 @@ from typing import Any, Dict, List, Optional, Union
from ..semantic_extract.triple_extractor import Triple
from .bulk_loader import BulkLoader, LoadProgress
from .config import triple_store_config
from .config import triplet_store_config
from .query_engine import QueryEngine, QueryPlan, QueryResult
from .registry import method_registry
from .triple_manager import TripleManager, TripleStore
from .triplet_manager import TripletManager, TripletStore
# Global manager instances
_global_manager: Optional[TripleManager] = None
_global_manager: Optional[TripletManager] = None
_global_query_engine: Optional[QueryEngine] = None
_global_bulk_loader: Optional[BulkLoader] = None
def _get_manager() -> TripleManager:
"""Get or create global TripleManager instance."""
def _get_manager() -> TripletManager:
"""Get or create global TripletManager instance."""
global _global_manager
if _global_manager is None:
config = triple_store_config.get_all()
_global_manager = TripleManager(config=config)
config = triplet_store_config.get_all()
_global_manager = TripletManager(config=config)
return _global_manager
@@ -133,7 +133,7 @@ def _get_query_engine() -> QueryEngine:
"""Get or create global QueryEngine instance."""
global _global_query_engine
if _global_query_engine is None:
config = triple_store_config.get_all()
config = triplet_store_config.get_all()
_global_query_engine = QueryEngine(config=config)
return _global_query_engine
@@ -142,16 +142,16 @@ def _get_bulk_loader() -> BulkLoader:
"""Get or create global BulkLoader instance."""
global _global_bulk_loader
if _global_bulk_loader is None:
config = triple_store_config.get_all()
config = triplet_store_config.get_all()
_global_bulk_loader = BulkLoader(config=config)
return _global_bulk_loader
def register_store(
store_id: str, store_type: str, endpoint: str, method: str = "default", **options
) -> TripleStore:
) -> TripletStore:
"""
Register a triple store.
Register a triplet store.
Args:
store_id: Store identifier
@@ -321,7 +321,7 @@ def execute_query(
Args:
query: SPARQL query string
store_adapter: Triple store adapter instance
store_adapter: Triplet store adapter instance
method: Query method name (default: "default")
**options: Additional options
@@ -383,7 +383,7 @@ def bulk_load(
Args:
triples: List of triples to load
store_adapter: Triple store adapter instance
store_adapter: Triplet store adapter instance
method: Loading method name (default: "default")
**options: Additional options
@@ -424,9 +424,9 @@ def validate_triples(
return loader.validate_before_load(triples, **options)
def get_triple_store_method(task: str, method_name: str) -> Optional[Any]:
def get_triplet_store_method(task: str, method_name: str) -> Optional[Any]:
"""
Get triple store method by task and name.
Get triplet store method by task and name.
Args:
task: Task type (register, add, get, delete, update, query, optimize, bulk_load, validate)
@@ -440,7 +440,7 @@ def get_triple_store_method(task: str, method_name: str) -> Optional[Any]:
def list_available_methods(task: Optional[str] = None) -> Dict[str, List[str]]:
"""
List all available triple store methods.
List all available triplet store methods.
Args:
task: Optional task type to filter by
@@ -2,7 +2,7 @@
Query Engine Module
This module provides comprehensive SPARQL query execution and optimization
for triple store operations, including query planning, caching, and performance
for triplet store operations, including query planning, caching, and performance
monitoring.
Key Features:
@@ -20,7 +20,7 @@ Main Classes:
- QueryPlan: Query execution plan representation dataclass
Example Usage:
>>> from semantica.triple_store import QueryEngine
>>> from semantica.triplet_store import QueryEngine
>>> engine = QueryEngine(enable_caching=True, enable_optimization=True)
>>> result = engine.execute_query(sparql_query, store_adapter)
>>> plan = engine.plan_query(sparql_query)
@@ -109,7 +109,7 @@ class QueryEngine:
Query result
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="QueryEngine",
message="Executing SPARQL query",
)
@@ -16,7 +16,7 @@ Main Classes:
- RDF4JAdapter: Main RDF4J integration adapter
Example Usage:
>>> from semantica.triple_store import RDF4JAdapter
>>> from semantica.triplet_store import RDF4JAdapter
>>> adapter = RDF4JAdapter(endpoint="http://localhost:8080/rdf4j-server", repository_id="repo1")
>>> result = adapter.execute_sparql(sparql_query)
>>> tx_id = adapter.begin_transaction()
@@ -38,7 +38,7 @@ from ..utils.progress_tracker import get_progress_tracker
class RDF4JAdapter:
"""
Eclipse RDF4J adapter for triple store operations.
Eclipse RDF4J adapter for triplet store operations.
RDF4J connection and repository management
SPARQL query execution
@@ -184,7 +184,7 @@ class RDF4JAdapter:
Query results
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="RDF4JAdapter",
message="Executing SPARQL query on RDF4J",
)
@@ -250,7 +250,7 @@ class RDF4JAdapter:
Operation status
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="RDF4JAdapter",
message=f"Adding {len(triples)} triples to RDF4J repository",
)
@@ -1,11 +1,11 @@
"""
Method Registry Module for Triple Store
Method Registry Module for Triplet Store
This module provides a method registry system for registering custom triple store methods,
enabling extensibility and community contributions to the triple store toolkit.
This module provides a method registry system for registering custom triplet store methods,
enabling extensibility and community contributions to the triplet store toolkit.
Supported Registration Types:
- Method Registry: Register custom triple store methods for:
- Method Registry: Register custom triplet store methods for:
* "register": Store registration methods
* "add": Triple addition methods
* "get": Triple retrieval methods
@@ -24,20 +24,20 @@ Algorithms Used:
- Task-based Organization: Hierarchical organization by task type
Key Features:
- Method registry for custom triple store methods
- Method registry for custom triplet store methods
- Task-based method organization (register, add, get, delete, update, query, optimize, bulk_load, validate)
- Dynamic registration and unregistration
- Easy discovery of available methods
- Support for community-contributed extensions
Main Classes:
- MethodRegistry: Registry for custom triple store methods
- MethodRegistry: Registry for custom triplet store methods
Global Instances:
- method_registry: Global method registry instance
Example Usage:
>>> from semantica.triple_store.registry import method_registry
>>> from semantica.triplet_store.registry import method_registry
>>> method_registry.register("add", "custom_method", custom_add_function)
>>> available = method_registry.list_all("add")
"""
@@ -46,7 +46,7 @@ from typing import Any, Callable, Dict, List, Optional
class MethodRegistry:
"""Registry for custom triple store methods."""
"""Registry for custom triplet store methods."""
def __init__(self):
"""Initialize method registry."""
@@ -1,25 +1,25 @@
"""
Triple Manager Module
Triplet Manager Module
This module provides comprehensive CRUD operations for RDF triples and triple
store management, enabling unified access to multiple triple store backends
This module provides comprehensive CRUD operations for RDF triplets and triplet
store management, enabling unified access to multiple triplet store backends
through a common interface.
Key Features:
- CRUD operations for RDF triples
- CRUD operations for RDF triplets
- Multi-store management and registration
- Batch operations and bulk loading
- Triple validation and consistency
- Triplet validation and consistency
- Store adapter pattern
- Error handling and recovery
Main Classes:
- TripleManager: Main triple store management coordinator
- TripleStore: Triple store configuration dataclass
- TripletManager: Main triplet store management coordinator
- TripletStore: Triplet store configuration dataclass
Example Usage:
>>> from semantica.triple_store import TripleManager
>>> manager = TripleManager()
>>> from semantica.triplet_store import TripletManager
>>> manager = TripletManager()
>>> store = manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
>>> result = manager.add_triple(triple, store_id="main")
>>> triples = manager.get_triple(subject="http://example.org/entity1")
@@ -39,8 +39,8 @@ from ..utils.progress_tracker import get_progress_tracker
@dataclass
class TripleStore:
"""Triple store configuration."""
class TripletStore:
"""Triplet store configuration."""
store_id: str
store_type: str # "blazegraph", "jena", "rdf4j", "virtuoso"
@@ -49,9 +49,9 @@ class TripleStore:
connected: bool = False
class TripleManager:
class TripletManager:
"""
Triple store management system.
Triplet store management system.
CRUD operations for RDF triples
Batch operations and bulk loading
@@ -63,26 +63,26 @@ class TripleManager:
def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
"""
Initialize triple manager.
Initialize triplet manager.
Args:
config: Configuration dictionary
**kwargs: Additional configuration options:
- default_store: Default triple store to use
- default_store: Default triplet store to use
"""
self.logger = get_logger("triple_manager")
self.logger = get_logger("triplet_manager")
self.config = config or {}
self.config.update(kwargs)
self.progress_tracker = get_progress_tracker()
self.stores: Dict[str, TripleStore] = {}
self.stores: Dict[str, TripletStore] = {}
self.default_store_id = self.config.get("default_store")
def register_store(
self, store_id: str, store_type: str, endpoint: str, **config
) -> TripleStore:
) -> TripletStore:
"""
Register a triple store.
Register a triplet store.
Args:
store_id: Store identifier
@@ -93,7 +93,7 @@ class TripleManager:
Returns:
Registered store
"""
store = TripleStore(
store = TripletStore(
store_id=store_id, store_type=store_type, endpoint=endpoint, config=config
)
@@ -102,7 +102,7 @@ class TripleManager:
if not self.default_store_id:
self.default_store_id = store_id
self.logger.info(f"Registered triple store: {store_id} ({store_type})")
self.logger.info(f"Registered triplet store: {store_id} ({store_type})")
return store
@@ -158,8 +158,8 @@ class TripleManager:
Operation status
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
submodule="TripleManager",
module="triplet_store",
submodule="TripletManager",
message=f"Adding {len(triples)} triples to store",
)
@@ -301,7 +301,7 @@ class TripleManager:
return True
def _get_store(self, store_id: Optional[str] = None) -> TripleStore:
def _get_store(self, store_id: Optional[str] = None) -> TripletStore:
"""Get store by ID."""
store_id = store_id or self.default_store_id
@@ -313,7 +313,7 @@ class TripleManager:
return self.stores[store_id]
def _get_adapter(self, store: TripleStore) -> Any:
def _get_adapter(self, store: TripletStore) -> Any:
"""Get adapter for store type."""
store_type = store.store_type.lower()
@@ -336,7 +336,7 @@ class TripleManager:
else:
raise ValidationError(f"Unsupported store type: {store_type}")
def get_store(self, store_id: str) -> Optional[TripleStore]:
def get_store(self, store_id: str) -> Optional[TripletStore]:
"""Get store by ID."""
return self.stores.get(store_id)
@@ -1,6 +1,6 @@
# Triple Store Module Usage Guide
# Triplet Store Module Usage Guide
This comprehensive guide demonstrates how to use the triple store module for RDF data storage and querying, supporting multiple triple store backends (Blazegraph, Jena, RDF4J, Virtuoso) with unified interfaces, SPARQL query execution, bulk loading, and query optimization.
This comprehensive guide demonstrates how to use the triplet store module for RDF data storage and querying, supporting multiple triplet store backends (Blazegraph, Jena, RDF4J, Virtuoso) with unified interfaces, SPARQL query execution, bulk loading, and query optimization.
## Table of Contents
@@ -17,14 +17,14 @@ This comprehensive guide demonstrates how to use the triple store module for RDF
## Basic Usage
### Using TripleManager
### Using TripletManager
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
from semantica.semantic_extract.triple_extractor import Triple
# Create triple manager
manager = TripleManager()
# Create triplet manager
manager = TripletManager()
# Register a store
store = manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
@@ -44,7 +44,7 @@ print(f"Triple added: {result['success']}")
### Using Convenience Functions
```python
from semantica.triple_store import register_store, add_triple, get_triples, execute_query
from semantica.triplet_store import register_store, add_triple, get_triples, execute_query
from semantica.semantic_extract.triple_extractor import Triple
# Register store
@@ -66,7 +66,7 @@ print(f"Found {len(triples)} triples")
### Using QueryEngine
```python
from semantica.triple_store import QueryEngine, BlazegraphAdapter
from semantica.triplet_store import QueryEngine, BlazegraphAdapter
# Create query engine
engine = QueryEngine(enable_caching=True, enable_optimization=True)
@@ -87,9 +87,9 @@ print(f"Execution time: {result.execution_time:.2f}s")
### Registering a Store
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
manager = TripleManager()
manager = TripletManager()
# Register Blazegraph store
blazegraph_store = manager.register_store(
@@ -126,7 +126,7 @@ virtuoso_store = manager.register_store(
### Using Convenience Function
```python
from semantica.triple_store import register_store
from semantica.triplet_store import register_store
# Register store using convenience function
store = register_store(
@@ -143,9 +143,9 @@ print(f"Store type: {store.store_type}")
### Multiple Stores
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
manager = TripleManager()
manager = TripletManager()
# Register multiple stores
manager.register_store("primary", "blazegraph", "http://localhost:9999/blazegraph")
@@ -165,10 +165,10 @@ print(f"Store endpoint: {store.endpoint}")
### Adding Triples
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
from semantica.semantic_extract.triple_extractor import Triple
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
# Add single triple
@@ -194,9 +194,9 @@ print(f"Added {result['total_triples']} triples in {result['batches']} batches")
### Retrieving Triples
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
# Get all triples for a subject
@@ -225,10 +225,10 @@ triples = manager.get_triple(
### Deleting Triples
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
from semantica.semantic_extract.triple_extractor import Triple
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
# Delete triple
@@ -244,10 +244,10 @@ print(f"Deleted: {result['success']}")
### Updating Triples
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
from semantica.semantic_extract.triple_extractor import Triple
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999/blazegraph")
# Update triple (delete old, add new)
@@ -270,7 +270,7 @@ print(f"Updated: {result['success']}")
### Basic Query Execution
```python
from semantica.triple_store import QueryEngine, BlazegraphAdapter
from semantica.triplet_store import QueryEngine, BlazegraphAdapter
# Create query engine
engine = QueryEngine(enable_caching=True)
@@ -298,7 +298,7 @@ for binding in result.bindings[:5]:
### Using Convenience Function
```python
from semantica.triple_store import execute_query, BlazegraphAdapter
from semantica.triplet_store import execute_query, BlazegraphAdapter
adapter = BlazegraphAdapter(endpoint="http://localhost:9999/blazegraph")
@@ -311,7 +311,7 @@ print(f"Found {len(result.bindings)} results")
### Query Result Processing
```python
from semantica.triple_store import QueryEngine, BlazegraphAdapter
from semantica.triplet_store import QueryEngine, BlazegraphAdapter
engine = QueryEngine()
adapter = BlazegraphAdapter(endpoint="http://localhost:9999/blazegraph")
@@ -332,7 +332,7 @@ print(f"Metadata: {result.metadata}")
### Query Caching
```python
from semantica.triple_store import QueryEngine, BlazegraphAdapter
from semantica.triplet_store import QueryEngine, BlazegraphAdapter
# Enable caching
engine = QueryEngine(enable_caching=True, cache_size=1000)
@@ -357,7 +357,7 @@ engine.clear_cache()
### Basic Query Optimization
```python
from semantica.triple_store import QueryEngine
from semantica.triplet_store import QueryEngine
engine = QueryEngine(enable_optimization=True)
@@ -377,7 +377,7 @@ print(f"Optimized query:\n{optimized}")
### Query Planning
```python
from semantica.triple_store import QueryEngine
from semantica.triplet_store import QueryEngine
engine = QueryEngine(enable_optimization=True)
@@ -403,7 +403,7 @@ print(f"Execution steps: {plan.execution_steps}")
### Query Statistics
```python
from semantica.triple_store import QueryEngine, BlazegraphAdapter
from semantica.triplet_store import QueryEngine, BlazegraphAdapter
engine = QueryEngine(enable_caching=True)
adapter = BlazegraphAdapter(endpoint="http://localhost:9999/blazegraph")
@@ -427,7 +427,7 @@ print(f"Cache size: {stats['cache_size']}")
### Basic Bulk Loading
```python
from semantica.triple_store import BulkLoader, BlazegraphAdapter
from semantica.triplet_store import BulkLoader, BlazegraphAdapter
from semantica.semantic_extract.triple_extractor import Triple
# Create bulk loader
@@ -455,7 +455,7 @@ print(f"Throughput: {progress.metadata.get('throughput', 0):.0f} triples/sec")
### Progress Tracking
```python
from semantica.triple_store import BulkLoader, BlazegraphAdapter, LoadProgress
from semantica.triplet_store import BulkLoader, BlazegraphAdapter, LoadProgress
from semantica.semantic_extract.triple_extractor import Triple
loader = BulkLoader(batch_size=1000)
@@ -476,7 +476,7 @@ progress = loader.load_triples(triples, adapter, progress_callback=progress_call
### Pre-load Validation
```python
from semantica.triple_store import BulkLoader
from semantica.triplet_store import BulkLoader
from semantica.semantic_extract.triple_extractor import Triple
loader = BulkLoader()
@@ -501,7 +501,7 @@ print(f"Valid triples: {validation['valid_triples']}/{validation['total_triples'
### Stream-based Loading
```python
from semantica.triple_store import BulkLoader, BlazegraphAdapter
from semantica.triplet_store import BulkLoader, BlazegraphAdapter
from semantica.semantic_extract.triple_extractor import Triple
loader = BulkLoader(batch_size=1000)
@@ -522,7 +522,7 @@ print(f"Loaded {progress.loaded_triples} triples from stream")
### Blazegraph Adapter
```python
from semantica.triple_store import BlazegraphAdapter
from semantica.triplet_store import BlazegraphAdapter
from semantica.semantic_extract.triple_extractor import Triple
# Create Blazegraph adapter
@@ -548,7 +548,7 @@ print(f"Found {len(result['bindings'])} results")
### Jena Adapter
```python
from semantica.triple_store import JenaAdapter
from semantica.triplet_store import JenaAdapter
from semantica.semantic_extract.triple_extractor import Triple
# Create Jena adapter (in-memory)
@@ -575,7 +575,7 @@ print(turtle)
### RDF4J Adapter
```python
from semantica.triple_store import RDF4JAdapter
from semantica.triplet_store import RDF4JAdapter
from semantica.semantic_extract.triple_extractor import Triple
# Create RDF4J adapter
@@ -594,7 +594,7 @@ result = adapter.add_triples(triples)
### Virtuoso Adapter
```python
from semantica.triple_store import VirtuosoAdapter
from semantica.triplet_store import VirtuosoAdapter
from semantica.semantic_extract.triple_extractor import Triple
# Create Virtuoso adapter
@@ -613,7 +613,7 @@ result = adapter.add_triples(triples)
## Algorithms and Methods
### Triple Store Management Algorithms
### Triplet Store Management Algorithms
#### Store Registration
**Algorithm**: Store type detection and adapter factory pattern
@@ -783,9 +783,9 @@ cost = engine._estimate_query_cost(query)
### Methods
#### TripleManager Methods
#### TripletManager Methods
- `register_store(store_id, store_type, endpoint, **config)`: Register triple store
- `register_store(store_id, store_type, endpoint, **config)`: Register triplet store
- `add_triple(triple, store_id, **options)`: Add single triple
- `add_triples(triples, store_id, **options)`: Add multiple triples
- `get_triple(subject, predicate, object, store_id, **options)`: Get triples matching pattern
@@ -824,14 +824,14 @@ cost = engine._estimate_query_cost(query)
## Dataclasses
### TripleStore
### TripletStore
Configuration dataclass for triple store instances.
Configuration dataclass for triplet store instances.
```python
from semantica.triple_store import TripleStore
from semantica.triplet_store import TripletStore
store = TripleStore(
store = TripletStore(
store_id="main",
store_type="blazegraph",
endpoint="http://localhost:9999/blazegraph/sparql",
@@ -856,7 +856,7 @@ print(f"Type: {store.store_type}")
Query execution result dataclass.
```python
from semantica.triple_store import QueryEngine, QueryResult
from semantica.triplet_store import QueryEngine, QueryResult
engine = QueryEngine()
result: QueryResult = engine.execute_query(query, adapter)
@@ -877,7 +877,7 @@ print(f"Execution time: {result.execution_time:.2f}s")
Query execution plan dataclass.
```python
from semantica.triple_store import QueryEngine, QueryPlan
from semantica.triplet_store import QueryEngine, QueryPlan
engine = QueryEngine(enable_optimization=True)
plan: QueryPlan = engine.plan_query(query)
@@ -897,38 +897,38 @@ print(f"Execution steps: {plan.execution_steps}")
### Environment Variables
```bash
# Triple store configuration
export TRIPLE_STORE_DEFAULT_STORE=main
export TRIPLE_STORE_BATCH_SIZE=1000
export TRIPLE_STORE_ENABLE_CACHING=true
export TRIPLE_STORE_CACHE_SIZE=1000
export TRIPLE_STORE_ENABLE_OPTIMIZATION=true
export TRIPLE_STORE_MAX_RETRIES=3
export TRIPLE_STORE_RETRY_DELAY=1.0
export TRIPLE_STORE_TIMEOUT=30
# Triplet store configuration
export TRIPLET_STORE_DEFAULT_STORE=main
export TRIPLET_STORE_BATCH_SIZE=1000
export TRIPLET_STORE_ENABLE_CACHING=true
export TRIPLET_STORE_CACHE_SIZE=1000
export TRIPLET_STORE_ENABLE_OPTIMIZATION=true
export TRIPLET_STORE_MAX_RETRIES=3
export TRIPLET_STORE_RETRY_DELAY=1.0
export TRIPLET_STORE_TIMEOUT=30
# Store endpoints
export TRIPLE_STORE_BLAZEGRAPH_ENDPOINT=http://localhost:9999/blazegraph
export TRIPLE_STORE_JENA_ENDPOINT=http://localhost:3030/ds
export TRIPLE_STORE_RDF4J_ENDPOINT=http://localhost:8080/rdf4j-server
export TRIPLE_STORE_VIRTUOSO_ENDPOINT=http://localhost:8890/sparql
export TRIPLET_STORE_BLAZEGRAPH_ENDPOINT=http://localhost:9999/blazegraph
export TRIPLET_STORE_JENA_ENDPOINT=http://localhost:3030/ds
export TRIPLET_STORE_RDF4J_ENDPOINT=http://localhost:8080/rdf4j-server
export TRIPLET_STORE_VIRTUOSO_ENDPOINT=http://localhost:8890/sparql
```
### Programmatic Configuration
```python
from semantica.triple_store.config import triple_store_config
from semantica.triplet_store.config import triplet_store_config
# Get configuration
batch_size = triple_store_config.get("batch_size", default=1000)
enable_caching = triple_store_config.get("enable_caching", default=True)
batch_size = triplet_store_config.get("batch_size", default=1000)
enable_caching = triplet_store_config.get("enable_caching", default=True)
# Set configuration
triple_store_config.set("batch_size", 2000)
triple_store_config.set("enable_caching", False)
triplet_store_config.set("batch_size", 2000)
triplet_store_config.set("enable_caching", False)
# Update with dictionary
triple_store_config.update({
triplet_store_config.update({
"batch_size": 2000,
"enable_caching": True,
"cache_size": 2000
@@ -939,7 +939,7 @@ triple_store_config.update({
```yaml
# config.yaml
triple_store:
triplet_store:
default_store: main
batch_size: 1000
enable_caching: true
@@ -956,11 +956,11 @@ triple_store:
## Advanced Examples
### Complete Triple Store Pipeline
### Complete Triplet Store Pipeline
```python
from semantica.triple_store import (
TripleManager,
from semantica.triplet_store import (
TripletManager,
QueryEngine,
BulkLoader,
register_store,
@@ -980,7 +980,7 @@ triples = [
result = add_triples(triples, store_id="main", batch_size=100)
# 3. Execute queries
from semantica.triple_store import BlazegraphAdapter
from semantica.triplet_store import BlazegraphAdapter
adapter = BlazegraphAdapter(endpoint="http://localhost:9999/blazegraph")
query = "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 10"
query_result = execute_query(query, adapter)
@@ -992,10 +992,10 @@ print(f"Query returned {len(query_result.bindings)} results")
### Multi-Store Operations
```python
from semantica.triple_store import TripleManager
from semantica.triplet_store import TripletManager
from semantica.semantic_extract.triple_extractor import Triple
manager = TripleManager()
manager = TripletManager()
# Register multiple stores
manager.register_store("primary", "blazegraph", "http://localhost:9999/blazegraph")
@@ -1012,7 +1012,7 @@ manager.add_triple(triple, store_id="backup")
### Query Optimization Workflow
```python
from semantica.triple_store import QueryEngine, BlazegraphAdapter
from semantica.triplet_store import QueryEngine, BlazegraphAdapter
engine = QueryEngine(enable_optimization=True, enable_caching=True)
adapter = BlazegraphAdapter(endpoint="http://localhost:9999/blazegraph")
@@ -1040,7 +1040,7 @@ print(f"Optimized: {result.metadata.get('optimized', False)}")
### Bulk Loading with Validation
```python
from semantica.triple_store import BulkLoader, BlazegraphAdapter
from semantica.triplet_store import BulkLoader, BlazegraphAdapter
from semantica.semantic_extract.triple_extractor import Triple
loader = BulkLoader(batch_size=1000, max_retries=3)
@@ -1066,22 +1066,22 @@ else:
### Custom Method Registration
```python
from semantica.triple_store.registry import method_registry
from semantica.triple_store import add_triple
from semantica.triplet_store.registry import method_registry
from semantica.triplet_store import add_triple
# Register custom add method
def custom_add_triple(triple, store_id=None, **options):
# Custom logic
print(f"Custom add: {triple.subject}")
# Call default implementation
from semantica.triple_store.methods import _get_manager
from semantica.triplet_store.methods import _get_manager
manager = _get_manager()
return manager.add_triple(triple, store_id=store_id, **options)
method_registry.register("add", "custom", custom_add_triple)
# Use custom method
from semantica.triple_store.methods import add_triple
from semantica.triplet_store.methods import add_triple
result = add_triple(triple, store_id="main", method="custom")
```
@@ -16,7 +16,7 @@ Main Classes:
- VirtuosoAdapter: Main Virtuoso integration adapter
Example Usage:
>>> from semantica.triple_store import VirtuosoAdapter
>>> from semantica.triplet_store import VirtuosoAdapter
>>> adapter = VirtuosoAdapter(endpoint="http://localhost:8890/sparql", username="dba", password="dba")
>>> result = adapter.execute_sparql(sparql_query)
>>> load_result = adapter.bulk_load(triples, graph="http://example.org/graph")
@@ -147,7 +147,7 @@ class VirtuosoAdapter:
Query results
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="VirtuosoAdapter",
message="Executing SPARQL query on Virtuoso",
)
@@ -240,7 +240,7 @@ class VirtuosoAdapter:
Load status
"""
tracking_id = self.progress_tracker.start_tracking(
module="triple_store",
module="triplet_store",
submodule="VirtuosoAdapter",
message=f"Bulk loading {len(triples)} triples to Virtuoso",
)
-1
View File
@@ -71,7 +71,6 @@ SUPPORTED_RDF_FORMATS = ["turtle", "rdfxml", "jsonld", "n3", "ntriples"]
# Supported Vector Store Backends
SUPPORTED_VECTOR_STORES = [
"faiss",
"pinecone",
"weaviate",
"qdrant",
"milvus",
+5 -5
View File
@@ -121,7 +121,7 @@ class ConsoleProgressDisplay(ProgressDisplay):
"semantic_extract": "🎯",
"seed": "🌱",
"split": "✂️",
"triple_store": "🗄️",
"triplet_store": "🗄️",
"vector_store": "📊",
"export": "💾",
"reasoning": "🤔",
@@ -176,7 +176,7 @@ class ConsoleProgressDisplay(ProgressDisplay):
"semantic_extract": "is extracting",
"seed": "is seeding",
"split": "is splitting",
"triple_store": "is storing",
"triplet_store": "is storing",
"vector_store": "is indexing",
"export": "is exporting",
"reasoning": "is reasoning",
@@ -352,7 +352,7 @@ class JupyterProgressDisplay(ProgressDisplay):
"semantic_extract": "🎯",
"seed": "🌱",
"split": "✂️",
"triple_store": "🗄️",
"triplet_store": "🗄️",
"vector_store": "📊",
"export": "💾",
"reasoning": "🤔",
@@ -405,7 +405,7 @@ class JupyterProgressDisplay(ProgressDisplay):
"semantic_extract": "is extracting",
"seed": "is seeding",
"split": "is splitting",
"triple_store": "is storing",
"triplet_store": "is storing",
"vector_store": "is indexing",
"export": "is exporting",
"reasoning": "is reasoning",
@@ -901,7 +901,7 @@ class ProgressTracker:
"semantic_extract": "🎯",
"seed": "🌱",
"split": "✂️",
"triple_store": "🗄️",
"triplet_store": "🗄️",
"vector_store": "📊",
"export": "💾",
"reasoning": "🤔",
+21 -28
View File
@@ -3,7 +3,7 @@ Vector Store Management Module
This module provides comprehensive vector storage and retrieval capabilities for the
Semantica framework, including support for multiple vector store backends (FAISS,
Pinecone, Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and
Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and
metadata filtering, metadata management, and namespace isolation.
Algorithms Used:
@@ -50,25 +50,30 @@ Namespace Management:
Adapter Pattern:
- FAISS Adapter: Local vector storage, FAISS index management, index persistence (save/load), batch operations, multiple index types support
- Pinecone Adapter: Cloud vector database integration, HTTP API communication, index management, upsert operations, query operations, metadata filtering
- Weaviate Adapter: GraphQL-based queries, schema management, object-oriented storage, rich metadata support, batch operations
- Qdrant Adapter: REST API communication, collection management, vector operations, payload (metadata) filtering, batch operations
- Milvus Adapter: gRPC communication, collection management, vector operations, metadata filtering, batch operations
- Unified Interface: Common interface for all adapters, backend-specific operation delegation, adapter factory pattern, connection management
- Weaviate Adapter: Schema-aware storage, GraphQL query support, object-oriented data model, batch operations, schema management
- Qdrant Adapter: Point-based storage, payload filtering, collection management, optimized search, batch operations
- Milvus Adapter: Scalable vector database, collection management, partitioning, complex querying, index building
Batch Operations:
- Batch Vector Operations: Chunking algorithm (fixed-size batch creation), batch processing, progress tracking, error handling per batch, retry mechanism
- Batch Indexing: Batch vector addition to index, incremental index updates, batch index training, batch index optimization
- Batch Search: Batch query processing, parallel search execution (when supported), result aggregation, batch result formatting
Supported Backends:
- FAISS: In-memory/local disk (Facebook AI Similarity Search)
- Weaviate: Cloud/Self-hosted (Schema-aware vector database)
- Qdrant: Cloud/Self-hosted (Vector database for the next generation of AI)
- Milvus: Cloud/Self-hosted (Highly scalable vector database)
- InMemory: Simple list-based storage for testing/small datasets
Performance Optimization:
- Vector Normalization: L2 normalization for cosine similarity, normalization caching, batch normalization
- Index Optimization: Index parameter tuning, index rebuilding for better performance, memory optimization, search speed optimization
- Caching: Query result caching, vector caching, metadata caching, cache invalidation strategies
- Parallel Processing: Batch-level parallelization, multi-threaded search (when supported), concurrent index operations
Configuration:
- Environment variables (SEMANTICA_VECTOR_STORE_*)
- Configuration files (yaml/json)
- Runtime configuration via VectorStoreConfig
Dependencies:
- faiss-cpu (or faiss-gpu)
- weaviate-client
- qdrant-client
- pymilvus
Key Features:
- Multi-backend vector store support (FAISS, Pinecone, Weaviate, Qdrant, Milvus)
- Multi-backend vector store support (FAISS, Weaviate, Qdrant, Milvus)
- Vector indexing and similarity search
- Metadata indexing and filtering
- Hybrid search combining vector and metadata queries
@@ -84,7 +89,6 @@ Main Classes:
- VectorRetriever: Vector retrieval and similarity search
- VectorManager: Vector store management and operations
- FAISSAdapter: FAISS integration for local vector storage
- PineconeAdapter: Pinecone cloud vector database integration
- WeaviateAdapter: Weaviate vector database integration
- QdrantAdapter: Qdrant vector database integration
- MilvusAdapter: Milvus vector database integration
@@ -141,12 +145,6 @@ from .methods import (
)
from .milvus_adapter import MilvusAdapter, MilvusClient, MilvusCollection, MilvusSearch
from .namespace_manager import Namespace, NamespaceManager
from .pinecone_adapter import (
PineconeAdapter,
PineconeIndex,
PineconeMetadata,
PineconeQuery,
)
from .qdrant_adapter import QdrantAdapter, QdrantClient, QdrantCollection, QdrantSearch
from .registry import MethodRegistry, method_registry
from .vector_store import VectorIndexer, VectorManager, VectorRetriever, VectorStore
@@ -168,11 +166,6 @@ __all__ = [
"FAISSIndex",
"FAISSSearch",
"FAISSIndexBuilder",
# Pinecone
"PineconeAdapter",
"PineconeIndex",
"PineconeQuery",
"PineconeMetadata",
# Weaviate
"WeaviateAdapter",
"WeaviateClient",
-2
View File
@@ -116,8 +116,6 @@ class VectorStoreConfig:
"VECTOR_STORE_ENABLE_HYBRID_SEARCH": "enable_hybrid_search",
"VECTOR_STORE_NAMESPACE": "default_namespace",
"VECTOR_STORE_FAISS_INDEX_TYPE": "faiss_index_type",
"VECTOR_STORE_PINECONE_API_KEY": "pinecone_api_key",
"VECTOR_STORE_PINECONE_ENVIRONMENT": "pinecone_environment",
"VECTOR_STORE_WEAVIATE_URL": "weaviate_url",
"VECTOR_STORE_QDRANT_URL": "qdrant_url",
"VECTOR_STORE_MILVUS_HOST": "milvus_host",
-510
View File
@@ -1,510 +0,0 @@
"""
Pinecone Adapter Module
This module provides Pinecone cloud vector database integration for vector storage
and similarity search in the Semantica framework, supporting serverless and pod-based
deployments with namespace isolation and metadata filtering.
Key Features:
- Cloud-based vector storage and retrieval
- Serverless and pod-based index specifications
- Namespace isolation for multi-tenant support
- Metadata filtering and querying
- Batch upsert and query operations
- Index statistics and monitoring
- Optional dependency handling
Main Classes:
- PineconeAdapter: Main Pinecone adapter for cloud vector operations
- PineconeIndex: Pinecone index wrapper with operations
- PineconeQuery: Pinecone query builder and executor
- PineconeMetadata: Metadata validation and sanitization
Example Usage:
>>> from semantica.vector_store import PineconeAdapter
>>> adapter = PineconeAdapter(api_key="your-api-key")
>>> adapter.connect()
>>> index = adapter.create_index("my-index", dimension=768, metric="cosine")
>>> adapter.upsert_vectors(vectors, ids, metadata, namespace="docs")
>>> results = adapter.query_vectors(query_vector, top_k=10, namespace="docs")
>>> stats = adapter.get_stats()
Author: Semantica Contributors
License: MIT
"""
from typing import Any, Dict, List, Optional, Union
import numpy as np
from ..utils.exceptions import ProcessingError, ValidationError
from ..utils.logging import get_logger
from ..utils.progress_tracker import get_progress_tracker
# Optional Pinecone import
try:
import pinecone
from pinecone import Pinecone, PodSpec, ServerlessSpec
PINECONE_AVAILABLE = True
except ImportError:
PINECONE_AVAILABLE = False
pinecone = None
Pinecone = None
ServerlessSpec = None
PodSpec = None
class PineconeIndex:
"""Pinecone index wrapper."""
def __init__(self, index: Any, index_name: str):
"""Initialize Pinecone index wrapper."""
self.index = index
self.index_name = index_name
self.logger = get_logger("pinecone_index")
def upsert_vectors(
self, vectors: List[Dict[str, Any]], namespace: Optional[str] = None, **options
) -> Dict[str, Any]:
"""Upsert vectors to index."""
if not PINECONE_AVAILABLE:
raise ProcessingError("Pinecone not available")
try:
response = self.index.upsert(
vectors=vectors, namespace=namespace, **options
)
return response
except Exception as e:
raise ProcessingError(f"Failed to upsert vectors: {str(e)}")
def query_vectors(
self,
query_vector: np.ndarray,
top_k: int = 10,
namespace: Optional[str] = None,
filter: Optional[Dict[str, Any]] = None,
**options,
) -> Dict[str, Any]:
"""Query similar vectors."""
if not PINECONE_AVAILABLE:
raise ProcessingError("Pinecone not available")
try:
response = self.index.query(
vector=query_vector.tolist(),
top_k=top_k,
namespace=namespace,
filter=filter,
include_metadata=True,
**options,
)
return response
except Exception as e:
raise ProcessingError(f"Failed to query vectors: {str(e)}")
def delete_vectors(
self, ids: List[str], namespace: Optional[str] = None, **options
) -> Dict[str, Any]:
"""Delete vectors from index."""
if not PINECONE_AVAILABLE:
raise ProcessingError("Pinecone not available")
try:
response = self.index.delete(ids=ids, namespace=namespace, **options)
return response
except Exception as e:
raise ProcessingError(f"Failed to delete vectors: {str(e)}")
def fetch_vectors(
self, ids: List[str], namespace: Optional[str] = None, **options
) -> Dict[str, Any]:
"""Fetch vectors by IDs."""
if not PINECONE_AVAILABLE:
raise ProcessingError("Pinecone not available")
try:
response = self.index.fetch(ids=ids, namespace=namespace, **options)
return response
except Exception as e:
raise ProcessingError(f"Failed to fetch vectors: {str(e)}")
def describe_index_stats(self, namespace: Optional[str] = None) -> Dict[str, Any]:
"""Get index statistics."""
if not PINECONE_AVAILABLE:
raise ProcessingError("Pinecone not available")
try:
stats = self.index.describe_index_stats(namespace=namespace)
return stats
except Exception as e:
raise ProcessingError(f"Failed to get index stats: {str(e)}")
class PineconeQuery:
"""Pinecone query builder."""
def __init__(self, index: PineconeIndex):
"""Initialize Pinecone query builder."""
self.index = index
self.logger = get_logger("pinecone_query")
def build_query(
self,
query_vector: np.ndarray,
top_k: int = 10,
namespace: Optional[str] = None,
filter: Optional[Dict[str, Any]] = None,
**options,
) -> Dict[str, Any]:
"""Build query parameters."""
return {
"vector": query_vector.tolist(),
"top_k": top_k,
"namespace": namespace,
"filter": filter,
**options,
}
def execute(self, query_params: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Execute query and format results."""
response = self.index.query_vectors(**query_params)
results = []
for match in response.get("matches", []):
results.append(
{
"id": match.get("id"),
"score": match.get("score", 0.0),
"metadata": match.get("metadata", {}),
}
)
return results
class PineconeMetadata:
"""Pinecone metadata handler."""
@staticmethod
def validate_metadata(metadata: Dict[str, Any]) -> Dict[str, Any]:
"""Validate and sanitize metadata."""
# Pinecone metadata restrictions
validated = {}
for key, value in metadata.items():
# Convert to allowed types
if isinstance(value, (str, int, float, bool, list)):
validated[key] = value
elif isinstance(value, dict):
# Nested dicts not directly supported
validated[key] = str(value)
else:
validated[key] = str(value)
return validated
class PineconeAdapter:
"""
Pinecone adapter for vector storage and similarity search.
Pinecone connection and authentication
Vector storage and retrieval
Similarity search and filtering
Namespace and index management
Performance optimization
Error handling and recovery
"""
def __init__(
self, api_key: Optional[str] = None, environment: Optional[str] = None, **config
):
"""Initialize Pinecone adapter."""
self.logger = get_logger("pinecone_adapter")
self.config = config
self.progress_tracker = get_progress_tracker()
self.api_key = api_key or config.get("api_key")
self.environment = environment or config.get("environment")
self.client: Optional[Any] = None
self.index: Optional[PineconeIndex] = None
self.query_builder: Optional[PineconeQuery] = None
# Check Pinecone availability
if not PINECONE_AVAILABLE:
self.logger.warning(
"Pinecone not available. Install with: pip install pinecone-client"
)
def connect(self, api_key: Optional[str] = None, **options) -> bool:
"""
Connect to Pinecone service.
Args:
api_key: Pinecone API key
**options: Connection options
Returns:
True if connected successfully
"""
if not PINECONE_AVAILABLE:
raise ProcessingError(
"Pinecone is not available. Install it with: pip install pinecone-client"
)
api_key = api_key or self.api_key
if not api_key:
raise ValidationError("Pinecone API key is required")
try:
self.client = Pinecone(api_key=api_key)
self.logger.info("Connected to Pinecone")
return True
except Exception as e:
raise ProcessingError(f"Failed to connect to Pinecone: {str(e)}")
def create_index(
self,
index_name: str,
dimension: int,
metric: str = "cosine",
spec: Optional[Dict[str, Any]] = None,
**options,
) -> PineconeIndex:
"""
Create new vector index.
Args:
index_name: Name of the index
dimension: Vector dimension
metric: Distance metric ("cosine", "euclidean", "dotproduct")
spec: Index specification (serverless or pod)
**options: Additional options
Returns:
PineconeIndex instance
"""
if self.client is None:
self.connect()
if not PINECONE_AVAILABLE:
raise ProcessingError("Pinecone not available")
try:
# Check if index exists
existing_indexes = [idx.name for idx in self.client.list_indexes()]
if index_name in existing_indexes:
self.logger.info(f"Index {index_name} already exists")
return self.get_index(index_name)
# Create index specification
if spec is None:
spec = ServerlessSpec(cloud="aws", region="us-east-1")
# Create index
self.client.create_index(
name=index_name,
dimension=dimension,
metric=metric,
spec=spec,
**options,
)
self.logger.info(f"Created Pinecone index: {index_name}")
return self.get_index(index_name)
except Exception as e:
raise ProcessingError(f"Failed to create index: {str(e)}")
def get_index(self, index_name: str) -> PineconeIndex:
"""
Get existing index.
Args:
index_name: Name of the index
Returns:
PineconeIndex instance
"""
if self.client is None:
self.connect()
if not PINECONE_AVAILABLE:
raise ProcessingError("Pinecone not available")
try:
index = self.client.Index(index_name)
self.index = PineconeIndex(index, index_name)
self.query_builder = PineconeQuery(self.index)
return self.index
except Exception as e:
raise ProcessingError(f"Failed to get index: {str(e)}")
def upsert_vectors(
self,
vectors: List[Union[np.ndarray, List[float]]],
ids: List[str],
metadata: Optional[List[Dict[str, Any]]] = None,
namespace: Optional[str] = None,
**options,
) -> Dict[str, Any]:
"""
Insert or update vectors.
Args:
vectors: List of vectors
ids: Vector IDs
metadata: Vector metadata
namespace: Namespace name
**options: Additional options
Returns:
Upsert response
"""
tracking_id = self.progress_tracker.start_tracking(
module="vector_store",
submodule="PineconeAdapter",
message=f"Upserting {len(vectors)} vectors to Pinecone",
)
try:
if self.index is None:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message="Index not initialized"
)
raise ProcessingError(
"Index not initialized. Call create_index() or get_index() first."
)
# Format vectors
self.progress_tracker.update_tracking(
tracking_id, message="Formatting vectors..."
)
formatted_vectors = []
for i, vector in enumerate(vectors):
if isinstance(vector, np.ndarray):
vector = vector.tolist()
vector_data = {"id": ids[i], "values": vector}
if metadata and i < len(metadata):
vector_data["metadata"] = PineconeMetadata.validate_metadata(
metadata[i]
)
formatted_vectors.append(vector_data)
self.progress_tracker.update_tracking(
tracking_id, message="Upserting vectors to Pinecone..."
)
result = self.index.upsert_vectors(formatted_vectors, namespace, **options)
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Upserted {len(vectors)} vectors",
)
return result
except Exception as e:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e)
)
raise
def query_vectors(
self,
query_vector: np.ndarray,
top_k: int = 10,
namespace: Optional[str] = None,
filter: Optional[Dict[str, Any]] = None,
**options,
) -> List[Dict[str, Any]]:
"""
Query similar vectors.
Args:
query_vector: Query vector
top_k: Number of results
namespace: Namespace name
filter: Metadata filter
**options: Additional options
Returns:
List of search results
"""
tracking_id = self.progress_tracker.start_tracking(
module="vector_store",
submodule="PineconeAdapter",
message=f"Querying {top_k} similar vectors from Pinecone",
)
try:
if self.query_builder is None:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message="Index not initialized"
)
raise ProcessingError(
"Index not initialized. Call create_index() or get_index() first."
)
self.progress_tracker.update_tracking(
tracking_id, message="Building query..."
)
query_params = self.query_builder.build_query(
query_vector, top_k, namespace, filter, **options
)
self.progress_tracker.update_tracking(
tracking_id, message="Executing query..."
)
results = self.query_builder.execute(query_params)
self.progress_tracker.stop_tracking(
tracking_id,
status="completed",
message=f"Query completed: {len(results) if isinstance(results, list) else 'N/A'} results",
)
return results
except Exception as e:
self.progress_tracker.stop_tracking(
tracking_id, status="failed", message=str(e)
)
raise
def delete_vectors(
self, ids: List[str], namespace: Optional[str] = None, **options
) -> Dict[str, Any]:
"""
Delete vectors from index.
Args:
ids: Vector IDs to delete
namespace: Namespace name
**options: Additional options
Returns:
Delete response
"""
if self.index is None:
raise ProcessingError(
"Index not initialized. Call create_index() or get_index() first."
)
return self.index.delete_vectors(ids, namespace, **options)
def get_stats(self, namespace: Optional[str] = None) -> Dict[str, Any]:
"""Get index statistics."""
if self.index is None:
raise ProcessingError(
"Index not initialized. Call create_index() or get_index() first."
)
stats = self.index.describe_index_stats(namespace)
return {
"total_vector_count": stats.get("total_vector_count", 0),
"dimension": stats.get("dimension", 0),
"index_fullness": stats.get("index_fullness", 0.0),
"namespaces": stats.get("namespaces", {}),
}
+8
View File
@@ -58,8 +58,16 @@ class VectorStore:
Provides vector store operations
"""
SUPPORTED_BACKENDS = {"faiss", "weaviate", "qdrant", "milvus", "inmemory"}
def __init__(self, backend="faiss", config=None, **kwargs):
"""Initialize vector store."""
if backend.lower() not in self.SUPPORTED_BACKENDS:
raise ValueError(
f"Unsupported backend: {backend}. "
f"Supported backends are: {', '.join(sorted(self.SUPPORTED_BACKENDS))}"
)
self.logger = get_logger("vector_store")
self.config = config or {}
self.config.update(kwargs)
+9 -43
View File
@@ -1,6 +1,6 @@
# Vector Store Module Usage Guide
This comprehensive guide demonstrates how to use the vector store module for vector storage and retrieval, supporting multiple vector store backends (FAISS, Pinecone, Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and metadata filtering, metadata management, and namespace isolation.
This comprehensive guide demonstrates how to use the vector store module for vector storage and retrieval, supporting multiple vector store backends (FAISS, Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and metadata filtering, metadata management, and namespace isolation.
## Table of Contents
@@ -687,34 +687,6 @@ distances, indices = adapter.search(index, query_vector, k=10)
print(f"Found {len(indices)} similar vectors")
```
### Pinecone Adapter
```python
from semantica.vector_store import PineconeAdapter
import numpy as np
# Create Pinecone adapter
adapter = PineconeAdapter(api_key="your-api-key", environment="us-west1-gcp")
# Connect
adapter.connect()
# Create index
index = adapter.create_index("my-index", dimension=768, metric="cosine")
# Upsert vectors
vectors = [np.random.rand(768).tolist() for _ in range(100)]
ids = [f"vec_{i}" for i in range(100)]
metadata = [{"category": "science"} for _ in range(100)]
adapter.upsert_vectors(vectors, ids, metadata)
# Query
query_vector = np.random.rand(768).tolist()
results = adapter.query_vectors(query_vector, top_k=10, include_metadata=True)
print(f"Found {len(results)} results")
```
### Weaviate Adapter
```python
@@ -1104,10 +1076,6 @@ export VECTOR_STORE_NAMESPACE=default
# FAISS configuration
export VECTOR_STORE_FAISS_INDEX_TYPE=flat
# Pinecone configuration
export VECTOR_STORE_PINECONE_API_KEY=your-api-key
export VECTOR_STORE_PINECONE_ENVIRONMENT=us-west1-gcp
# Weaviate configuration
export VECTOR_STORE_WEAVIATE_URL=http://localhost:8080
@@ -1153,8 +1121,6 @@ vector_store:
enable_hybrid_search: true
default_namespace: default
faiss_index_type: flat
pinecone_api_key: your-api-key
pinecone_environment: us-west1-gcp
weaviate_url: http://localhost:8080
qdrant_url: http://localhost:6333
milvus_host: localhost
@@ -1204,7 +1170,7 @@ print(f"Found {len(results)} hybrid search results")
### Multi-Backend Vector Store
```python
from semantica.vector_store import FAISSAdapter, PineconeAdapter
from semantica.vector_store import FAISSAdapter, WeaviateAdapter
import numpy as np
# Local FAISS store
@@ -1213,17 +1179,17 @@ faiss_index = faiss_adapter.create_index(index_type="flat", metric="L2")
faiss_vectors = np.random.rand(1000, 768).astype('float32')
faiss_adapter.add_vectors(faiss_index, faiss_vectors, ids=[f"faiss_{i}" for i in range(1000)])
# Cloud Pinecone store
pinecone_adapter = PineconeAdapter(api_key="your-key")
pinecone_adapter.connect()
pinecone_index = pinecone_adapter.create_index("my-index", dimension=768)
pinecone_vectors = [np.random.rand(768).tolist() for _ in range(1000)]
pinecone_adapter.upsert_vectors(pinecone_vectors, [f"pinecone_{i}" for i in range(1000)])
# Self-hosted Weaviate store
weaviate_adapter = WeaviateAdapter(url="http://localhost:8080")
weaviate_adapter.connect()
weaviate_index = weaviate_adapter.create_index("my-index", dimension=768)
weaviate_vectors = [np.random.rand(768).tolist() for _ in range(1000)]
weaviate_adapter.upsert_vectors(weaviate_vectors, [f"weaviate_{i}" for i in range(1000)])
# Search both
query_vector = np.random.rand(768)
faiss_results = faiss_adapter.search(faiss_index, query_vector, k=10)
pinecone_results = pinecone_adapter.query_vectors(query_vector, top_k=10)
weaviate_results = weaviate_adapter.query_vectors(query_vector, top_k=10)
```
### Hybrid Search with Custom Ranking
@@ -34,10 +34,19 @@ License: MIT
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
try:
import numpy as np
except ImportError:
np = None
try:
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
except ImportError:
px = None
go = None
make_subplots = None
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
@@ -68,6 +77,23 @@ class AnalyticsVisualizer:
except (KeyError, AttributeError):
self.color_scheme = ColorScheme.DEFAULT
def _check_dependencies(self):
"""Check if dependencies are available."""
if px is None or go is None:
raise ProcessingError(
"Plotly is required for analytics visualization. "
"Install with: pip install plotly"
)
if np is None:
raise ProcessingError(
"NumPy is required for analytics visualization. "
"Install with: pip install numpy"
)
def visualize_centrality(self, *args, **kwargs):
"""Alias for visualize_centrality_rankings."""
return self.visualize_centrality_rankings(*args, **kwargs)
def visualize_centrality_rankings(
self,
centrality: Dict[str, Any],
@@ -91,6 +117,7 @@ class AnalyticsVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="AnalyticsVisualizer",
@@ -188,6 +215,7 @@ class AnalyticsVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing community structure")
# Use KG visualizer for community visualization
@@ -217,6 +245,7 @@ class AnalyticsVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing connectivity analysis")
# Extract metrics
@@ -291,6 +320,7 @@ class AnalyticsVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing degree distribution")
# Calculate degrees
@@ -360,6 +390,7 @@ class AnalyticsVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing graph metrics dashboard")
# Extract key metrics
@@ -498,6 +529,7 @@ class AnalyticsVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing centrality comparison")
# Extract top nodes for each centrality type
@@ -35,10 +35,16 @@ from typing import Any, Dict, List, Optional, Tuple, Union
import matplotlib.pyplot as plt
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
import seaborn as sns
from plotly.subplots import make_subplots
try:
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
except ImportError:
px = None
go = None
make_subplots = None
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
@@ -85,6 +91,14 @@ class EmbeddingVisualizer:
self.color_scheme = ColorScheme.DEFAULT
self.point_size = config.get("point_size", 5)
def _check_dependencies(self):
"""Check if dependencies are available."""
if px is None or go is None:
raise ProcessingError(
"Plotly is required for embedding visualization. "
"Install with: pip install plotly"
)
def visualize_2d_projection(
self,
embeddings: np.ndarray,
@@ -92,17 +106,30 @@ class EmbeddingVisualizer:
method: str = "umap",
output: str = "interactive",
file_path: Optional[Union[str, Path]] = None,
color_by: Optional[List[Any]] = None,
size_by: Optional[List[float]] = None,
hover_data: Optional[List[Dict[str, Any]]] = None,
**options,
) -> Optional[Any]:
"""
Visualize embeddings in 2D using dimensionality reduction.
Implements the 5-step visualization process:
1. Problem setting: Dimensionality reduction choice
2. Data analysis: Logs embedding statistics
3. Layout: 2D Projection (UMAP/t-SNE/PCA)
4. Styling: Configurable color and size mapping
5. Interaction: Rich hover data
Args:
embeddings: Embedding matrix (n_samples, n_features)
labels: Optional labels for coloring points
labels: Optional labels for points (used as default color_by if provided)
method: Reduction method ("umap", "tsne", "pca")
output: Output type ("interactive", "html", "png", "svg")
file_path: Output file path
color_by: List of values to map to color (overrides labels)
size_by: List of values to map to point size
hover_data: List of dictionaries containing metadata for each point
**options: Additional options:
- n_components: Number of components (default: 2)
- perplexity: Perplexity for t-SNE
@@ -111,6 +138,7 @@ class EmbeddingVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="EmbeddingVisualizer",
@@ -119,6 +147,10 @@ class EmbeddingVisualizer:
try:
self.logger.info(f"Visualizing 2D projection using {method}")
# Step 2: Data Analysis
n_samples, n_features = embeddings.shape
self.logger.info(f"Embedding Analysis: {n_samples} samples, {n_features} dimensions")
if embeddings.shape[1] <= 2:
# Already 2D or less, use directly
@@ -138,7 +170,14 @@ class EmbeddingVisualizer:
tracking_id, message="Generating visualization..."
)
result = self._visualize_2d_plotly(
projected, labels, output, file_path, **options
projected,
labels,
output,
file_path,
color_by=color_by,
size_by=size_by,
hover_data=hover_data,
**options
)
self.progress_tracker.stop_tracking(
@@ -176,6 +215,7 @@ class EmbeddingVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="EmbeddingVisualizer",
@@ -238,6 +278,7 @@ class EmbeddingVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="EmbeddingVisualizer",
@@ -340,6 +381,7 @@ class EmbeddingVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="EmbeddingVisualizer",
@@ -444,6 +486,7 @@ class EmbeddingVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="EmbeddingVisualizer",
+154 -22
View File
@@ -33,12 +33,23 @@ License: MIT
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple, Union
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import numpy as np
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
try:
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
except ImportError:
mpatches = None
plt = None
try:
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
except ImportError:
px = None
go = None
make_subplots = None
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
@@ -96,25 +107,47 @@ class KGVisualizer:
self.hierarchical_layout = HierarchicalLayout(**config)
self.circular_layout = CircularLayout(**config)
def _check_dependencies(self):
"""Check if dependencies are available."""
if px is None or go is None:
raise ProcessingError(
"Plotly is required for KG visualization. "
"Install with: pip install plotly"
)
def visualize_network(
self,
graph: Dict[str, Any],
output: str = "interactive",
file_path: Optional[Union[str, Path]] = None,
node_color_by: str = "type",
node_size_by: Optional[str] = None,
hover_data: Optional[List[str]] = None,
**options,
) -> Optional[Any]:
"""
Visualize knowledge graph as interactive network.
Implements the 5-step visualization process:
1. Problem setting: implicit in graph selection
2. Data analysis: logs graph statistics
3. Layout: configurable via options
4. Styling: configurable node color/size mappings
5. Interaction: rich hover data and zoom capabilities
Args:
graph: Knowledge graph dictionary with entities and relationships
output: Output type ("interactive", "html", "png", "svg")
file_path: Output file path (required for non-interactive)
node_color_by: Property to map to node color (default: "type")
node_size_by: Property to map to node size (default: fixed)
hover_data: List of properties to show in hover tooltip
**options: Additional visualization options
Returns:
Plotly figure (if interactive) or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="KGVisualizer",
@@ -137,18 +170,34 @@ class KGVisualizer:
)
raise ProcessingError("No entities found in graph")
# Step 2: Data Analysis - Understand data structure
nodes = self._extract_nodes(entities)
edges = self._extract_edges(relationships, entities)
num_nodes = len(nodes)
num_edges = len(edges)
entity_types = set(n.get("type", "unknown") for n in nodes)
self.logger.info(f"Graph Structure Analysis: {num_nodes} nodes, {num_edges} edges")
self.logger.info(f"Entity Types: {', '.join(sorted(entity_types))}")
# Build node and edge lists
self.progress_tracker.update_tracking(
tracking_id, message="Building node and edge lists..."
)
nodes = self._extract_nodes(entities)
edges = self._extract_edges(relationships, entities)
self.progress_tracker.update_tracking(
tracking_id, message="Generating visualization..."
)
result = self._visualize_network_plotly(
nodes, edges, output, file_path, **options
nodes,
edges,
output,
file_path,
node_color_by=node_color_by,
node_size_by=node_size_by,
hover_data=hover_data,
**options
)
self.progress_tracker.stop_tracking(
@@ -184,6 +233,7 @@ class KGVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing knowledge graph communities")
entities = graph.get("entities", [])
@@ -242,6 +292,7 @@ class KGVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info(
f"Visualizing knowledge graph with {centrality_type} centrality"
)
@@ -295,6 +346,7 @@ class KGVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing entity type distribution")
entities = graph.get("entities", [])
@@ -339,6 +391,7 @@ class KGVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing relationship matrix")
entities = graph.get("entities", [])
@@ -442,6 +495,9 @@ class KGVisualizer:
edges: List[Dict[str, Any]],
output: str,
file_path: Optional[Path],
node_color_by: str = "type",
node_size_by: Optional[str] = None,
hover_data: Optional[List[str]] = None,
**options,
) -> Optional[Any]:
"""Create Plotly network visualization."""
@@ -458,11 +514,91 @@ class KGVisualizer:
else:
pos = self.force_layout.compute_layout(node_ids, edge_tuples, **options)
# Get entity type colors
entity_types = list(set(n.get("type", "entity") for n in nodes))
type_colors = ColorPalette.get_entity_type_colors(
entity_types, self.color_scheme
)
# Step 4: Styling - Node Colors
# Priority 1: Explicit color set in node (e.g. from visualize_communities)
# Priority 2: Mapped property via node_color_by
node_colors = []
if any("color" in n for n in nodes):
node_colors = [n.get("color", "#888") for n in nodes if n["id"] in pos]
else:
if node_color_by == "type":
entity_types = list(set(n.get("type", "entity") for n in nodes))
type_colors = ColorPalette.get_entity_type_colors(
entity_types, self.color_scheme
)
node_colors = [
type_colors.get(n.get("type", "entity"), "#888")
for n in nodes
if n["id"] in pos
]
else:
# Custom property mapping
values = []
for n in nodes:
if n["id"] not in pos: continue
val = n.get(node_color_by) or n.get("metadata", {}).get(node_color_by, "Unknown")
values.append(str(val))
unique_vals = sorted(list(set(values)))
colors = ColorPalette.get_colors(self.color_scheme, len(unique_vals))
val_map = dict(zip(unique_vals, colors))
node_colors = []
for n in nodes:
if n["id"] not in pos: continue
val = str(n.get(node_color_by) or n.get("metadata", {}).get(node_color_by, "Unknown"))
node_colors.append(val_map.get(val, "#888"))
# Step 4: Styling - Node Sizes
# Priority 1: Explicit size set in node (e.g. from visualize_centrality)
# Priority 2: Mapped property via node_size_by
node_sizes = []
if any("size" in n for n in nodes) and not node_size_by:
node_sizes = [n.get("size", self.node_size) for n in nodes if n["id"] in pos]
elif node_size_by:
raw_sizes = []
valid_indices = []
for i, n in enumerate(nodes):
if n["id"] not in pos: continue
val = n.get(node_size_by) or n.get("metadata", {}).get(node_size_by, 0)
try:
s = float(val)
except (ValueError, TypeError):
s = 0
raw_sizes.append(s)
valid_indices.append(i)
# Normalize to range [10, 50]
if raw_sizes and max(raw_sizes) > min(raw_sizes):
min_s, max_s = min(raw_sizes), max(raw_sizes)
node_sizes = [10 + 40 * ((s - min_s) / (max_s - min_s)) for s in raw_sizes]
else:
node_sizes = [self.node_size] * len(raw_sizes)
else:
node_sizes = [self.node_size for n in nodes if n["id"] in pos]
# Step 5: Interaction - Rich Hover
node_text = []
for n in nodes:
if n["id"] not in pos: continue
# Basic info
text = f"<b>{n['label']}</b><br>Type: {n.get('type', 'entity')}"
# Additional hover data
if hover_data:
for field in hover_data:
val = n.get(field) or n.get("metadata", {}).get(field, "N/A")
text += f"<br>{field}: {val}"
# Add dynamic styling info if relevant
if node_size_by:
val = n.get(node_size_by) or n.get("metadata", {}).get(node_size_by, "N/A")
text += f"<br>{node_size_by}: {val}"
node_text.append(text)
# Prepare edge traces
edge_x = []
@@ -485,13 +621,6 @@ class KGVisualizer:
# Prepare node traces
node_x = [pos[n["id"]][0] for n in nodes if n["id"] in pos]
node_y = [pos[n["id"]][1] for n in nodes if n["id"] in pos]
node_text = [n["label"] for n in nodes if n["id"] in pos]
node_colors = [
type_colors.get(n.get("type", "entity"), "#888")
for n in nodes
if n["id"] in pos
]
node_sizes = [n.get("size", self.node_size) for n in nodes if n["id"] in pos]
node_trace = go.Scatter(
x=node_x,
@@ -499,9 +628,12 @@ class KGVisualizer:
mode="markers+text",
hoverinfo="text",
text=node_text,
textposition="middle center",
textposition="top center",
marker=dict(
size=node_sizes, color=node_colors, line=dict(width=2, color="white")
size=node_sizes,
color=node_colors,
line=dict(width=2, color="white"),
opacity=0.9
),
)
+145 -11
View File
@@ -37,10 +37,17 @@ from typing import Any, Dict, List, Optional, Union
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import plotly.express as px
import plotly.graph_objects as go
try:
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
except ImportError:
px = None
go = None
make_subplots = None
from matplotlib.patches import FancyBboxPatch
from plotly.subplots import make_subplots
try:
import graphviz
@@ -90,26 +97,60 @@ class OntologyVisualizer:
self.color_scheme = ColorScheme.DEFAULT
self.node_size = config.get("node_size", 15)
def _check_dependencies(self, require_graphviz: bool = False):
"""Check if dependencies are available."""
if require_graphviz:
if graphviz is None:
raise ProcessingError(
"Graphviz is required for DOT export. "
"Install with: pip install graphviz"
)
else:
if px is None or go is None:
raise ProcessingError(
"Plotly is required for ontology visualization. "
"Install with: pip install plotly"
)
def visualize_hierarchy(
self,
ontology: Dict[str, Any],
output: str = "interactive",
file_path: Optional[Union[str, Path]] = None,
node_color_by: str = "level",
node_size_by: str = "instances",
hover_data: Optional[List[str]] = None,
**options,
) -> Optional[Any]:
"""
Visualize class hierarchy as tree.
Implements the 5-step visualization process:
1. Problem setting: Implicit in ontology selection
2. Data analysis: Logs ontology statistics
3. Layout: Hierarchical tree layout
4. Styling: Configurable node color (e.g. by level) and size (e.g. by instances)
5. Interaction: Rich hover data
Args:
ontology: Ontology dictionary with classes, or SemanticNetwork object,
or ontology generator result
output: Output type ("interactive", "html", "png", "svg", "dot")
file_path: Output file path
node_color_by: Property to map to node color (default: "level")
node_size_by: Property to map to node size (default: "instances")
hover_data: List of properties to show in hover tooltip
**options: Additional options
Returns:
Visualization figure or None
"""
# Check dependencies
if output == "dot":
self._check_dependencies(require_graphviz=True)
else:
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="OntologyVisualizer",
@@ -150,6 +191,15 @@ class OntologyVisualizer:
"No classes found in ontology. Please provide classes or a semantic network."
)
# Step 2: Data Analysis
num_classes = len(classes)
max_depth = 0
for cls in classes:
depth = self._calculate_class_depth(cls, classes)
max_depth = max(max_depth, depth)
self.logger.info(f"Ontology Analysis: {num_classes} classes, max depth {max_depth}")
# If output is dot and graphviz is available, use it
if output == "dot" and graphviz is not None and file_path:
self.progress_tracker.update_tracking(
@@ -175,7 +225,14 @@ class OntologyVisualizer:
tracking_id, message="Generating visualization..."
)
result = self._visualize_hierarchy_plotly(
hierarchy, classes, output, file_path, **options
hierarchy,
classes,
output,
file_path,
node_color_by=node_color_by,
node_size_by=node_size_by,
hover_data=hover_data,
**options
)
self.progress_tracker.stop_tracking(
@@ -209,6 +266,7 @@ class OntologyVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing ontology properties")
# Handle different input formats
@@ -258,6 +316,7 @@ class OntologyVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing ontology structure")
classes = ontology.get("classes", [])
@@ -342,6 +401,7 @@ class OntologyVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing class-property matrix")
classes = ontology.get("classes", [])
@@ -411,6 +471,7 @@ class OntologyVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing ontology metrics")
classes = ontology.get("classes", [])
@@ -603,6 +664,7 @@ class OntologyVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing semantic model")
# Handle OntologyGenerator result
@@ -651,6 +713,9 @@ class OntologyVisualizer:
classes: List[Dict[str, Any]],
output: str,
file_path: Optional[Path],
node_color_by: str = "level",
node_size_by: str = "instances",
hover_data: Optional[List[str]] = None,
**options,
) -> Optional[Any]:
"""Create Plotly hierarchy visualization."""
@@ -676,7 +741,16 @@ class OntologyVisualizer:
edges = []
def add_node_and_children(cls_name, level=0, x_offset=0):
nodes.append({"name": cls_name, "level": level, "x": x_offset, "y": -level})
# Find class data
cls_data = all_class_names.get(cls_name, {})
nodes.append({
"name": cls_name,
"level": level,
"x": x_offset,
"y": -level,
"data": cls_data
})
children = hierarchy.get(cls_name, [])
child_width = 1.0 / max(len(children), 1)
@@ -692,6 +766,63 @@ class OntologyVisualizer:
root_x = (i + 0.5) * root_width
add_node_and_children(root, 0, root_x)
# Step 4: Styling - Node Colors
# Default to coloring by level
node_colors = []
if node_color_by == "level":
node_colors = [n["level"] for n in nodes]
else:
# Map custom property
values = []
for n in nodes:
val = str(n["data"].get(node_color_by, "Unknown"))
values.append(val)
unique_vals = sorted(list(set(values)))
colors = ColorPalette.get_colors(self.color_scheme, len(unique_vals))
val_map = dict(zip(unique_vals, colors))
node_colors = [val_map.get(str(n["data"].get(node_color_by, "Unknown")), "#888") for n in nodes]
# Step 4: Styling - Node Sizes
# Default to sizing by instances (if available) or fixed size
node_sizes = []
if node_size_by:
raw_sizes = []
for n in nodes:
val = n["data"].get(node_size_by, 0)
try:
s = float(val)
except (ValueError, TypeError):
s = 0
raw_sizes.append(s)
if raw_sizes and max(raw_sizes) > min(raw_sizes):
min_s, max_s = min(raw_sizes), max(raw_sizes)
# Scale between 10 and 40
node_sizes = [10 + 30 * ((s - min_s) / (max_s - min_s)) for s in raw_sizes]
else:
node_sizes = [self.node_size] * len(nodes)
else:
node_sizes = [self.node_size] * len(nodes)
# Step 5: Interaction - Rich Hover
node_text = []
for n in nodes:
cls_data = n["data"]
text = f"<b>{n['name']}</b><br>Level: {n['level']}"
# Add instances if available
if "instances" in cls_data:
text += f"<br>Instances: {cls_data['instances']}"
# Additional hover data
if hover_data:
for field in hover_data:
val = cls_data.get(field, "N/A")
text += f"<br>{field}: {val}"
node_text.append(text)
# Create visualization
edge_x = []
edge_y = []
@@ -714,18 +845,21 @@ class OntologyVisualizer:
node_x = [n["x"] for n in nodes]
node_y = [n["y"] for n in nodes]
node_text = [n["name"] for n in nodes]
node_trace = go.Scatter(
x=node_x,
y=node_y,
mode="markers+text",
text=node_text,
textposition="middle center",
text=[n["name"] for n in nodes], # Keep label on node simple
hovertext=node_text, # Rich hover text
hoverinfo="text",
textposition="top center",
marker=dict(
size=self.node_size * 10,
color="lightblue",
line=dict(width=2, color="darkblue"),
size=node_sizes,
color=node_colors,
colorscale="Viridis" if node_color_by == "level" else None,
line=dict(width=2, color="white"),
showscale=True if node_color_by == "level" else False
),
)
+32 -5
View File
@@ -33,9 +33,19 @@ License: MIT
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
try:
import numpy as np
except ImportError:
np = None
try:
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
except ImportError:
px = None
go = None
make_subplots = None
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
@@ -66,6 +76,14 @@ class QualityVisualizer:
except (KeyError, AttributeError):
self.color_scheme = ColorScheme.DEFAULT
def _check_dependencies(self):
"""Check if dependencies are available."""
if px is None or go is None:
raise ProcessingError(
"Plotly is required for quality visualization. "
"Install with: pip install plotly"
)
def visualize_dashboard(
self,
quality_report: Any,
@@ -85,6 +103,7 @@ class QualityVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="QualityVisualizer",
@@ -266,6 +285,7 @@ class QualityVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing quality score distribution")
fig = go.Figure(
@@ -315,6 +335,7 @@ class QualityVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing quality issues")
# Extract issues
@@ -405,6 +426,7 @@ class QualityVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing completeness metrics")
# Extract metrics
@@ -468,6 +490,13 @@ class QualityVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
if np is None:
raise ProcessingError(
"NumPy is required for consistency heatmap visualization. "
"Install with: pip install numpy"
)
self.logger.info("Visualizing consistency heatmap")
# Extract consistency matrix
@@ -477,8 +506,6 @@ class QualityVisualizer:
if not matrix:
raise ProcessingError("No consistency matrix found")
import numpy as np
matrix = np.array(matrix)
fig = go.Figure(
@@ -33,8 +33,12 @@ License: MIT
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
import plotly.express as px
import plotly.graph_objects as go
try:
import plotly.express as px
import plotly.graph_objects as go
except ImportError:
px = None
go = None
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
@@ -62,6 +66,14 @@ class SemanticNetworkVisualizer:
except (KeyError, AttributeError):
self.color_scheme = ColorScheme.DEFAULT
def _check_dependencies(self):
"""Check if dependencies are available."""
if px is None or go is None:
raise ProcessingError(
"Plotly is required for semantic network visualization. "
"Install with: pip install plotly"
)
def visualize_network(
self,
semantic_network: Any,
@@ -87,6 +99,7 @@ class SemanticNetworkVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="SemanticNetworkVisualizer",
@@ -274,6 +287,7 @@ class SemanticNetworkVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing semantic network node types")
# Extract nodes
@@ -325,6 +339,7 @@ class SemanticNetworkVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing semantic network edge types")
# Extract edges
+21 -3
View File
@@ -33,9 +33,14 @@ License: MIT
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
try:
import plotly.express as px
import plotly.graph_objects as go
from plotly.subplots import make_subplots
except ImportError:
px = None
go = None
make_subplots = None
from ..utils.exceptions import ProcessingError
from ..utils.logging import get_logger
@@ -66,6 +71,14 @@ class TemporalVisualizer:
except (KeyError, AttributeError):
self.color_scheme = ColorScheme.DEFAULT
def _check_dependencies(self):
"""Check if dependencies are available."""
if px is None or go is None:
raise ProcessingError(
"Plotly is required for temporal visualization. "
"Install with: pip install plotly"
)
def visualize_timeline(
self,
temporal_data: Dict[str, Any],
@@ -85,6 +98,7 @@ class TemporalVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
tracking_id = self.progress_tracker.start_tracking(
module="visualization",
submodule="TemporalVisualizer",
@@ -208,6 +222,7 @@ class TemporalVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing temporal patterns")
if not patterns:
@@ -279,6 +294,7 @@ class TemporalVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing snapshot comparison")
timestamps = sorted(snapshots.keys())
@@ -371,6 +387,7 @@ class TemporalVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing version history")
# Build tree structure
@@ -438,6 +455,7 @@ class TemporalVisualizer:
Returns:
Visualization figure or None
"""
self._check_dependencies()
self.logger.info("Visualizing metrics evolution")
fig = go.Figure()
+1 -1
View File
@@ -15,7 +15,7 @@ modules = {
"kg": "semantica.kg",
"embeddings": "semantica.embeddings",
"vector_store": "semantica.vector_store",
"triple_store": "semantica.triple_store",
"triplet_store": "semantica.triplet_store",
"ontology": "semantica.ontology",
"reasoning": "semantica.reasoning",
"pipeline": "semantica.pipeline",
+98
View File
@@ -0,0 +1,98 @@
import pytest
from semantica import build as module_build
from semantica.core import Semantica
from semantica.core.methods import (
initialize_framework,
get_status,
run_pipeline,
build_knowledge_base,
)
pytestmark = pytest.mark.integration
class DummyPipeline:
def __init__(self):
self.executed_with = None
def execute(self, data):
self.executed_with = data
return {"value": data}
def test_initialize_and_get_status_integration():
framework = initialize_framework()
status = get_status(framework=framework, method="summary")
assert status["state"] in {"ready", "running", "initializing"}
assert "health" in status
framework.shutdown(graceful=True)
def test_semantica_run_pipeline_with_dummy_pipeline():
pipeline = DummyPipeline()
framework = Semantica()
framework.initialize()
data = {"input": "value"}
result = framework.run_pipeline(pipeline, data)
assert result["success"] is True
assert result["output"] == {"value": data}
assert pipeline.executed_with == data
framework.shutdown(graceful=True)
def test_core_methods_run_pipeline_with_dummy_pipeline():
pipeline = DummyPipeline()
data = "sample"
result = run_pipeline(pipeline, data)
assert result["success"] is True
assert result["output"] == {"value": data}
def test_framework_build_knowledge_base_end_to_end(tmp_path):
source_path = tmp_path / "sample_e2e_framework.txt"
source_path.write_text("Apple Inc. is a technology company.")
framework = Semantica()
result = framework.build_knowledge_base(
sources=[str(source_path)],
embeddings=False,
graph=False,
pipeline={
"name": "e2e_pipeline",
"steps": [
{"name": "step1", "type": "default", "config": {}},
],
},
)
stats = result["statistics"]
assert stats["sources_processed"] == 1
assert len(result["results"]) == 1
assert result["results"][0]["success"] is True
framework.shutdown(graceful=True)
def test_core_methods_build_knowledge_base_end_to_end(tmp_path):
source_path = tmp_path / "sample_e2e_core_methods.txt"
source_path.write_text("Tim Cook leads Apple.")
result = build_knowledge_base(
sources=str(source_path),
method="minimal",
embeddings=False,
graph=False,
pipeline={"steps": ["step1", "step2"]},
)
stats = result["statistics"]
assert stats["sources_processed"] == 1
assert len(result["results"]) == 1
assert result["results"][0]["success"] is True
def test_module_build_end_to_end_default_pipeline(tmp_path):
source_path = tmp_path / "sample_e2e_module_build.txt"
source_path.write_text("Sample data for end-to-end test.")
result = module_build(str(source_path), embeddings=False, graph=False)
stats = result["statistics"]
assert stats["sources_processed"] == 1
assert len(result["results"]) == 1
assert result["results"][0]["success"] is True
@@ -4,6 +4,8 @@ from unittest.mock import MagicMock, patch
from semantica.ingest import MCPIngestor, ingest_mcp, DBIngestor, FileIngestor
from semantica.ingest.mcp_ingestor import MCPData
pytestmark = pytest.mark.integration
class TestCookbookIntegration:
@pytest.fixture
+2
View File
@@ -18,6 +18,8 @@ from semantica.ingest import (
MCPIngestor, IngestConfig, ingest_config
)
pytestmark = pytest.mark.integration
class TestNotebook02DataIngestion:
def setup_method(self):
+2
View File
@@ -7,6 +7,8 @@ from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, StreamIngest
from semantica.kg import GraphBuilder, EntityResolver, ProvenanceTracker
from semantica.conflicts import ConflictDetector
pytestmark = pytest.mark.integration
class TestNotebook06MultiSourceIntegration:
def setup_method(self):
+5
View File
@@ -1,9 +1,14 @@
import unittest
import os
from datetime import datetime, timezone
import pytest
from semantica.normalize import methods
from semantica.normalize.config import normalize_config
pytestmark = pytest.mark.integration
class TestNormalizeIntegration(unittest.TestCase):
def test_normalize_text_integration(self):
text = "Hello World"
+5
View File
@@ -1,5 +1,8 @@
import unittest
from unittest.mock import MagicMock, patch
import pytest
from semantica.ontology import (
OntologyEngine,
ClassInferrer,
@@ -11,6 +14,8 @@ from semantica.ontology import (
)
from semantica.visualization import OntologyVisualizer
pytestmark = pytest.mark.integration
class TestNotebook14(unittest.TestCase):
"""
Tests mirroring the steps in cookbook/introduction/14_Ontology.ipynb
@@ -1,7 +1,9 @@
import unittest
from unittest.mock import MagicMock, patch
from collections import defaultdict
import pytest
from semantica.ontology.class_inferrer import ClassInferrer
from semantica.ontology.property_generator import PropertyGenerator
from semantica.ontology.naming_conventions import NamingConventions
@@ -10,6 +12,8 @@ from semantica.ontology.ontology_validator import OntologyValidator, ValidationR
from semantica.ontology.namespace_manager import NamespaceManager
from semantica.ontology.module_manager import ModuleManager
pytestmark = pytest.mark.integration
class TestOntologyComprehensive(unittest.TestCase):
def setUp(self):
+5 -1
View File
@@ -1,10 +1,14 @@
import unittest
import os
import tempfile
import json
import pytest
from semantica.parse import DocumentParser, CSVParser, JSONParser, XMLParser, HTMLParser, StructuredDataParser
pytestmark = pytest.mark.integration
class TestNotebook03(unittest.TestCase):
def setUp(self):
self.temp_dir = tempfile.mkdtemp()
+4
View File
@@ -6,6 +6,8 @@ import json
import csv
from pathlib import Path
import pytest
from semantica.parse.document_parser import DocumentParser, PDFParser, DOCXParser, HTMLParser
from semantica.parse.pptx_parser import PPTXParser
from semantica.parse.excel_parser import ExcelParser
@@ -17,6 +19,8 @@ from semantica.parse.web_parser import WebParser
from semantica.parse.registry import MethodRegistry
from semantica.parse.config import ParseConfig
pytestmark = pytest.mark.integration
class TestParseComprehensive(unittest.TestCase):
def setUp(self):
+5
View File
@@ -1,6 +1,9 @@
import unittest
from unittest.mock import MagicMock, patch
import time
import pytest
from semantica.pipeline import (
PipelineBuilder,
ExecutionEngine,
@@ -10,6 +13,8 @@ from semantica.pipeline import (
RetryStrategy
)
pytestmark = pytest.mark.integration
class TestNotebook07(unittest.TestCase):
def setUp(self):
@@ -3,6 +3,8 @@ from unittest.mock import MagicMock, patch
import time
from typing import Dict, Any
import pytest
from semantica.pipeline.pipeline_builder import PipelineBuilder, StepStatus, Pipeline
from semantica.pipeline.execution_engine import ExecutionEngine, PipelineStatus
from semantica.pipeline.failure_handler import (
@@ -11,6 +13,8 @@ from semantica.pipeline.failure_handler import (
from semantica.pipeline.parallelism_manager import ParallelismManager, Task
from semantica.pipeline.pipeline_validator import PipelineValidator
pytestmark = pytest.mark.integration
class TestPipelineComprehensive(unittest.TestCase):
def setUp(self):
+4 -2
View File
@@ -1,10 +1,10 @@
import unittest
import sys
import os
import numpy as np
# Add project root to path
import pytest
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from semantica.embeddings import EmbeddingGenerator, TextEmbedder
@@ -13,6 +13,8 @@ from semantica.vector_store import (
SearchRanker, NamespaceManager
)
pytestmark = pytest.mark.integration
class TestSemanticaFeatures(unittest.TestCase):
def test_01_embedding_generation(self):
+4
View File
@@ -4,6 +4,8 @@ import tempfile
import unittest
from pathlib import Path
import pytest
from semantica.kg import GraphBuilder
from semantica.export import (
JSONExporter,
@@ -12,6 +14,8 @@ from semantica.export import (
GraphExporter,
)
pytestmark = pytest.mark.integration
class TestNotebook15Export(unittest.TestCase):
def setUp(self):
self.test_dir = tempfile.mkdtemp()
+4 -1
View File
@@ -2,9 +2,12 @@ import sys
import os
import numpy as np
# Add project root to path
import pytest
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
pytestmark = pytest.mark.integration
def log(msg):
print(msg)
with open("test_progress.log", "a") as f:
+4 -1
View File
@@ -3,9 +3,12 @@ import sys
import os
import numpy as np
# Add project root to path
import pytest
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
pytestmark = pytest.mark.integration
class TestNotebooks(unittest.TestCase):
def test_12_embedding_generation(self):
+4
View File
@@ -6,6 +6,8 @@ import numpy as np
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from semantica.kg import GraphBuilder
from semantica.export import (
JSONExporter,
@@ -24,6 +26,8 @@ from semantica.export import (
export_config
)
pytestmark = pytest.mark.integration
class TestNotebooks(unittest.TestCase):
def setUp(self):
self.test_dir = tempfile.mkdtemp()
+4 -2
View File
@@ -1,10 +1,10 @@
import unittest
import sys
import os
from pathlib import Path
# Add project root to path
import pytest
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from semantica.semantic_extract import (
@@ -17,6 +17,8 @@ from semantica.semantic_extract import (
)
from semantica.semantic_extract.methods import get_entity_method, get_relation_method
pytestmark = pytest.mark.integration
class TestNotebooksVerification(unittest.TestCase):
"""
Test suite to verify the code snippets from the notebooks:
+2
View File
@@ -21,6 +21,8 @@ from semantica.pipeline import (
from semantica.pipeline.pipeline_builder import Pipeline, PipelineSerializer
from semantica.pipeline.execution_engine import ExecutionResult
pytestmark = pytest.mark.integration
# --- Fixtures ---
@pytest.fixture
+4 -1
View File
@@ -3,7 +3,8 @@ import sys
import os
from unittest.mock import MagicMock, patch
# Add project root to path
import pytest
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
from semantica.semantic_extract.ner_extractor import NERExtractor, Entity
@@ -23,6 +24,8 @@ from semantica.semantic_extract.triple_extractor import (
)
from semantica.semantic_extract.methods import get_entity_method, get_relation_method
pytestmark = pytest.mark.integration
class TestSemanticExtractDeepDive(unittest.TestCase):
def setUp(self):
@@ -1,11 +1,11 @@
import unittest
import sys
import os
from unittest.mock import MagicMock, patch
from dataclasses import dataclass
# Add project root to path
import pytest
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from semantica.semantic_extract.named_entity_recognizer import (
@@ -22,6 +22,8 @@ from semantica.semantic_extract.methods import (
extract_triples_rules
)
pytestmark = pytest.mark.integration
class TestSemanticExtractDeepDivePart2(unittest.TestCase):
def setUp(self):
@@ -1,19 +1,19 @@
import unittest
from unittest.mock import MagicMock, patch
from semantica.triple_store.triple_manager import TripleManager, TripleStore
from semantica.triple_store.query_engine import QueryEngine, QueryResult
from semantica.triplet_store.triplet_manager import TripletManager, TripletStore
from semantica.triplet_store.query_engine import QueryEngine, QueryResult
from semantica.semantic_extract.triple_extractor import Triple
class TestTripleStore(unittest.TestCase):
class TestTripletStore(unittest.TestCase):
def setUp(self):
self.mock_logger = MagicMock()
self.mock_tracker = MagicMock()
self.logger_patcher = patch('semantica.triple_store.triple_manager.get_logger', return_value=self.mock_logger)
self.tracker_patcher = patch('semantica.triple_store.triple_manager.get_progress_tracker', return_value=self.mock_tracker)
self.logger_patcher_qe = patch('semantica.triple_store.query_engine.get_logger', return_value=self.mock_logger)
self.tracker_patcher_qe = patch('semantica.triple_store.query_engine.get_progress_tracker', return_value=self.mock_tracker)
self.logger_patcher = patch('semantica.triplet_store.triplet_manager.get_logger', return_value=self.mock_logger)
self.tracker_patcher = patch('semantica.triplet_store.triplet_manager.get_progress_tracker', return_value=self.mock_tracker)
self.logger_patcher_qe = patch('semantica.triplet_store.query_engine.get_logger', return_value=self.mock_logger)
self.tracker_patcher_qe = patch('semantica.triplet_store.query_engine.get_progress_tracker', return_value=self.mock_tracker)
self.logger_patcher.start()
self.tracker_patcher.start()
@@ -26,23 +26,23 @@ class TestTripleStore(unittest.TestCase):
self.logger_patcher_qe.stop()
self.tracker_patcher_qe.stop()
def test_triple_manager_init(self):
manager = TripleManager(default_store="main")
def test_triplet_manager_init(self):
manager = TripletManager(default_store="main")
self.assertEqual(manager.default_store_id, "main")
self.assertEqual(manager.stores, {})
def test_register_store(self):
manager = TripleManager()
manager = TripletManager()
store = manager.register_store("main", "blazegraph", "http://localhost:9999")
self.assertIsInstance(store, TripleStore)
self.assertIsInstance(store, TripletStore)
self.assertEqual(store.store_id, "main")
self.assertEqual(store.store_type, "blazegraph")
self.assertEqual(store.endpoint, "http://localhost:9999")
self.assertIn("main", manager.stores)
@patch('semantica.triple_store.triple_manager.TripleManager._get_adapter')
@patch('semantica.triplet_store.triplet_manager.TripletManager._get_adapter')
def test_add_triple(self, mock_get_adapter):
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999")
mock_adapter = MagicMock()
@@ -56,9 +56,9 @@ class TestTripleStore(unittest.TestCase):
self.assertEqual(result["store_id"], "main")
mock_adapter.add_triple.assert_called_once_with(triple)
@patch('semantica.triple_store.triple_manager.TripleManager._get_adapter')
@patch('semantica.triplet_store.triplet_manager.TripletManager._get_adapter')
def test_add_triples(self, mock_get_adapter):
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999")
mock_adapter = MagicMock()
@@ -76,9 +76,9 @@ class TestTripleStore(unittest.TestCase):
self.assertEqual(result["total_triples"], 2)
mock_adapter.add_triples.assert_called()
@patch('semantica.triple_store.triple_manager.TripleManager._get_adapter')
@patch('semantica.triplet_store.triplet_manager.TripletManager._get_adapter')
def test_get_triple(self, mock_get_adapter):
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999")
mock_adapter = MagicMock()
@@ -91,9 +91,9 @@ class TestTripleStore(unittest.TestCase):
self.assertEqual(result, expected_triples)
mock_adapter.get_triples.assert_called_once_with("s", None, None)
@patch('semantica.triple_store.triple_manager.TripleManager._get_adapter')
@patch('semantica.triplet_store.triplet_manager.TripletManager._get_adapter')
def test_delete_triple(self, mock_get_adapter):
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999")
mock_adapter = MagicMock()
@@ -106,9 +106,9 @@ class TestTripleStore(unittest.TestCase):
self.assertTrue(result["success"])
mock_adapter.delete_triple.assert_called_once_with(triple)
@patch('semantica.triple_store.triple_manager.TripleManager._get_adapter')
@patch('semantica.triplet_store.triplet_manager.TripletManager._get_adapter')
def test_update_triple(self, mock_get_adapter):
manager = TripleManager()
manager = TripletManager()
manager.register_store("main", "blazegraph", "http://localhost:9999")
mock_adapter = MagicMock()
@@ -0,0 +1,62 @@
import unittest
from unittest.mock import MagicMock, patch
import os
import sys
# Ensure semantica is in path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
from semantica.vector_store.vector_store import VectorStore
from semantica.vector_store.registry import method_registry
from semantica.vector_store.config import vector_store_config
class TestPineconeRemoval(unittest.TestCase):
"""Verify that Pinecone has been completely removed from the system."""
def test_pinecone_backend_rejected(self):
"""Test that initializing VectorStore with backend='pinecone' raises an error."""
with self.assertRaises(ValueError) as context:
VectorStore(backend="pinecone")
# The error message might be generic "Unknown backend" or specific.
# We just want to ensure it fails.
self.assertTrue("pinecone" in str(context.exception).lower() or "unknown" in str(context.exception).lower())
def test_registry_clean(self):
"""Test that no Pinecone methods are registered."""
# Check all task types
task_types = ["store", "search", "index", "hybrid_search", "metadata", "namespace"]
for task in task_types:
methods = method_registry.list_all(task)
# Flatten if it's a dict
if isinstance(methods, dict):
method_names = methods.get(task, [])
else:
method_names = methods
for name in method_names:
self.assertNotIn("pinecone", name.lower(), f"Found pinecone reference in registry task {task}: {name}")
def test_config_clean(self):
"""Test that configuration does not contain Pinecone keys."""
config = vector_store_config.get_all()
for key in config.keys():
self.assertNotIn("pinecone", key.lower(), f"Found pinecone key in config: {key}")
def test_adapters_existence(self):
"""Verify that other adapters exist but PineconeAdapter does not."""
try:
from semantica.vector_store import faiss_adapter
from semantica.vector_store import weaviate_adapter
from semantica.vector_store import qdrant_adapter
from semantica.vector_store import milvus_adapter
except ImportError as e:
self.fail(f"Failed to import a required adapter: {e}")
with self.assertRaises(ImportError):
from semantica.vector_store import pinecone_adapter
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,375 @@
import unittest
from unittest.mock import MagicMock, patch, ANY
import numpy as np
import sys
from pathlib import Path
import pytest
sys.path.append(str(Path(__file__).parent.parent.parent))
from semantica.vector_store.vector_store import VectorStore, VectorIndexer, VectorRetriever, VectorManager
from semantica.vector_store.registry import MethodRegistry, method_registry
from semantica.vector_store.faiss_adapter import FAISSAdapter, FAISSIndex, FAISSIndexBuilder, FAISSSearch
from semantica.vector_store.milvus_adapter import MilvusAdapter, MilvusClient, MilvusCollection, MilvusSearch
from semantica.vector_store.qdrant_adapter import QdrantAdapter
from semantica.vector_store.weaviate_adapter import WeaviateAdapter
from semantica.vector_store.hybrid_search import HybridSearch, MetadataFilter, SearchRanker
pytestmark = pytest.mark.integration
class TestVectorStoreDeepDive(unittest.TestCase):
def setUp(self):
self.vectors = [np.array([1.0, 0.0]), np.array([0.0, 1.0])]
self.ids = ["vec_1", "vec_2"]
self.metadata = [{"type": "a"}, {"type": "b"}]
def test_vector_store_in_memory(self):
"""Test the default in-memory VectorStore implementation."""
store = VectorStore(backend="inmemory", dimension=2)
# Test storing vectors
ids = store.store_vectors(self.vectors, self.metadata)
self.assertEqual(len(ids), 2)
self.assertEqual(store.vectors[ids[0]].tolist(), self.vectors[0].tolist())
# Test searching vectors (exact match)
results = store.search_vectors(np.array([1.0, 0.0]), k=1)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], ids[0])
# Score should be close to 1.0 (cosine similarity of identical vectors)
self.assertAlmostEqual(results[0]["score"], 1.0)
# Test searching vectors (orthogonal)
results = store.search_vectors(np.array([0.0, 1.0]), k=1)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], ids[1])
# Test updating vectors
new_vec = np.array([0.5, 0.5])
store.update_vectors([ids[0]], [new_vec])
self.assertTrue(np.array_equal(store.get_vector(ids[0]), new_vec))
# Test deleting vectors
store.delete_vectors([ids[0]])
self.assertIsNone(store.get_vector(ids[0]))
self.assertEqual(len(store.vectors), 1)
def test_vector_indexer_retriever(self):
"""Test VectorIndexer and VectorRetriever directly."""
indexer = VectorIndexer(backend="inmemory", dimension=2)
index = indexer.create_index(self.vectors, self.ids)
self.assertIsNotNone(index)
self.assertEqual(len(index["vectors"]), 2)
retriever = VectorRetriever(backend="inmemory")
results = retriever.search_similar(
np.array([1.0, 0.0]),
self.vectors,
self.ids,
k=1
)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], "vec_1")
# Test hybrid search (metadata filter)
results = retriever.search_hybrid(
np.array([1.0, 0.0]),
{"type": "b"}, # Filter for vec_2
self.vectors,
self.metadata,
k=1
)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["vector"].tolist(), self.vectors[1].tolist())
def test_method_registry(self):
"""Test the MethodRegistry."""
registry = MethodRegistry()
def custom_store(): return "stored"
# Register
registry.register("store", "custom", custom_store, version="1.0")
self.assertTrue(registry.has("store", "custom"))
# Get
func = registry.get("store", "custom")
self.assertEqual(func(), "stored")
# Metadata
meta = registry.get_metadata("store", "custom")
self.assertEqual(meta["version"], "1.0")
# List
all_methods = registry.list_all("store")
self.assertEqual(all_methods["store"], ["custom"])
# Unregister
registry.unregister("store", "custom")
self.assertFalse(registry.has("store", "custom"))
@patch('semantica.vector_store.faiss_adapter.faiss')
@patch('semantica.vector_store.faiss_adapter.FAISS_AVAILABLE', True)
def test_faiss_adapter(self, mock_faiss):
"""Test FAISSAdapter with mocked faiss."""
# Setup mock
mock_index = MagicMock()
mock_faiss.IndexFlatL2.return_value = mock_index
mock_faiss.read_index.return_value = mock_index
# Mock search return
# distances, indices
mock_index.search.return_value = (np.array([[0.0, 0.1]]), np.array([[0, 1]]))
mock_index.ntotal = 2
# Test Init
adapter = FAISSAdapter(dimension=2)
# Test Create Index
adapter.create_index(index_type="flat")
mock_faiss.IndexFlatL2.assert_called_with(2)
# Test Add Vectors
adapter.add_vectors(self.vectors, self.ids, self.metadata)
mock_index.add.assert_called()
self.assertEqual(len(adapter.index.vector_ids), 2)
# Test Search
results = adapter.search_similar(np.array([1.0, 0.0]), k=2)
self.assertEqual(len(results), 2)
self.assertEqual(results[0]["id"], "vec_1")
# Test Save
adapter.save_index("test.index")
mock_faiss.write_index.assert_called()
# Test Load
adapter.load_index("test.index")
mock_faiss.read_index.assert_called()
@patch('semantica.vector_store.milvus_adapter.connections')
@patch('semantica.vector_store.milvus_adapter.Collection')
@patch('semantica.vector_store.milvus_adapter.utility')
@patch('semantica.vector_store.milvus_adapter.DataType')
@patch('semantica.vector_store.milvus_adapter.FieldSchema')
@patch('semantica.vector_store.milvus_adapter.CollectionSchema')
@patch('semantica.vector_store.milvus_adapter.MILVUS_AVAILABLE', True)
def test_milvus_adapter(self, mock_collection_schema, mock_field_schema, mock_data_type, mock_utility, mock_collection_cls, mock_connections):
"""Test MilvusAdapter with mocked pymilvus."""
# Setup mocks
mock_data_type.INT64 = 1
mock_data_type.FLOAT_VECTOR = 2
# Setup mocks
mock_utility.has_collection.return_value = False
mock_collection_instance = MagicMock()
mock_collection_cls.return_value = mock_collection_instance
# Mock search results
mock_hit = MagicMock()
mock_hit.id = 1
mock_hit.distance = 0.1
mock_collection_instance.search.return_value = [[mock_hit]]
# Test Init
adapter = MilvusAdapter(host="localhost")
# Test Connect
adapter.connect()
mock_connections.connect.assert_called_with(
alias="default", host="localhost", port=19530, user=None, password=None
)
# Test Create Collection
adapter.create_collection("test_coll", dimension=2)
mock_collection_cls.assert_called()
mock_collection_instance.create_index.assert_called()
# Test Insert
adapter.insert_vectors(self.vectors)
mock_collection_instance.insert.assert_called()
# Test Search
results = adapter.search_vectors(np.array([1.0, 0.0]), limit=1)
mock_collection_instance.search.assert_called()
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], 1)
@patch('semantica.vector_store.qdrant_adapter.QdrantClientLib')
@patch('semantica.vector_store.qdrant_adapter.VectorParams')
@patch('semantica.vector_store.qdrant_adapter.Distance')
@patch('semantica.vector_store.qdrant_adapter.PointStruct')
@patch('semantica.vector_store.qdrant_adapter.QDRANT_AVAILABLE', True)
def test_qdrant_adapter(self, mock_point_struct, mock_distance, mock_vector_params, mock_qdrant_cls):
"""Test QdrantAdapter with mocked qdrant_client."""
mock_client = MagicMock()
mock_qdrant_cls.return_value = mock_client
# Mock search response
mock_hit = MagicMock()
mock_hit.id = "vec_1"
mock_hit.score = 0.9
mock_hit.payload = {"type": "a"}
mock_client.search.return_value = [mock_hit]
adapter = QdrantAdapter(url="http://localhost:6333")
# Connect
adapter.connect()
mock_qdrant_cls.assert_called()
# Create Collection
adapter.create_collection("test-collection", vector_size=2)
mock_client.create_collection.assert_called()
# Insert
adapter.insert_vectors(self.vectors, self.ids, payloads=self.metadata)
mock_client.upsert.assert_called()
# Search
results = adapter.search_vectors(np.array([1.0, 0.0]), limit=1)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], "vec_1")
@patch('semantica.vector_store.weaviate_adapter.weaviate')
@patch('semantica.vector_store.weaviate_adapter.MetadataQuery')
@patch('semantica.vector_store.weaviate_adapter.WEAVIATE_AVAILABLE', True)
def test_weaviate_adapter(self, mock_metadata_query, mock_weaviate):
"""Test WeaviateAdapter with mocked weaviate."""
mock_client = MagicMock()
mock_weaviate.connect_to_local.return_value = mock_client
mock_collection = MagicMock()
mock_client.collections.get.return_value = mock_collection
# Mock search response
mock_obj = MagicMock()
mock_obj.uuid = "uuid-1"
mock_obj.properties = {"text": "hello"}
mock_obj.metadata.distance = 0.1
mock_query_response = MagicMock()
mock_query_response.objects = [mock_obj]
mock_collection.query.near_vector.return_value = mock_query_response
adapter = WeaviateAdapter(url="http://localhost:8080")
# Connect
adapter.connect()
mock_weaviate.connect_to_local.assert_called()
# Create Schema
adapter.create_schema("TestClass", properties=[])
mock_client.collections.create.assert_called()
# Add Objects
# Need to mock batch context manager
mock_batch = MagicMock()
mock_collection.batch.dynamic.return_value.__enter__.return_value = mock_batch
adapter.get_collection("TestClass")
adapter.add_objects([{"text": "hello"}], vectors=self.vectors)
mock_batch.add_object.assert_called()
# Query
results = adapter.query_vectors(np.array([1.0, 0.0]), limit=1)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], "uuid-1")
def test_hybrid_search(self):
"""Test HybridSearch, MetadataFilter and SearchRanker."""
search = HybridSearch()
# Test MetadataFilter
meta_filter = MetadataFilter().eq("type", "a")
self.assertTrue(meta_filter.matches({"type": "a"}))
self.assertFalse(meta_filter.matches({"type": "b"}))
meta_filter = MetadataFilter().gt("val", 10)
self.assertTrue(meta_filter.matches({"val": 20}))
self.assertFalse(meta_filter.matches({"val": 5}))
# Test Search
results = search.search(
query_vector=np.array([1.0, 0.0]),
vectors=self.vectors,
metadata=self.metadata,
vector_ids=self.ids,
k=2
)
self.assertEqual(len(results), 2)
self.assertEqual(results[0]["id"], "vec_1")
# Test Filtered Search
results = search.search(
query_vector=np.array([1.0, 0.0]),
vectors=self.vectors,
metadata=self.metadata,
vector_ids=self.ids,
k=2,
metadata_filter=MetadataFilter().eq("type", "b")
)
self.assertEqual(len(results), 1)
self.assertEqual(results[0]["id"], "vec_2")
# Test Ranker
ranker = SearchRanker(strategy="reciprocal_rank_fusion")
res1 = [{"id": "1", "score": 0.9}, {"id": "2", "score": 0.8}]
res2 = [{"id": "2", "score": 0.85}, {"id": "1", "score": 0.7}]
fused = ranker.rank([res1, res2])
self.assertEqual(len(fused), 2)
# ID 2 should be top because it's high in both? Or ID 1?
# RRF: 1/(k+1) + 1/(k+2) vs 1/(k+2) + 1/(k+1). They are equal rank-wise (1st and 2nd).
# Multi-source search
sources = [
{"vectors": [self.vectors[0]], "metadata": [self.metadata[0]], "ids": ["vec_1"]},
{"vectors": [self.vectors[1]], "metadata": [self.metadata[1]], "ids": ["vec_2"]}
]
multi_res = search.multi_source_search(np.array([1.0, 0.0]), sources, k=2)
self.assertEqual(len(multi_res), 2)
def test_vector_manager(self):
"""Test VectorManager."""
manager = VectorManager()
store = VectorStore(backend="inmemory")
store.store_vectors(self.vectors, self.metadata)
# Test statistics
stats = manager.collect_statistics(store)
self.assertEqual(stats["total_vectors"], 2)
self.assertEqual(stats["backend"], "inmemory")
# Test maintenance
health = manager.maintain_store(store)
self.assertTrue(health["healthy"])
# Test manage_store wrapper
results = manager.manage_store(store, statistics=True, optimize=True)
self.assertIn("statistics", results)
self.assertIn("optimize", results)
def test_config(self):
"""Test VectorStoreConfig."""
from semantica.vector_store.config import vector_store_config
# Test get default
self.assertEqual(vector_store_config.get("default_backend"), "faiss")
# Test set
vector_store_config.set("test_key", "test_value")
self.assertEqual(vector_store_config.get("test_key"), "test_value")
# Test update
vector_store_config.update({"test_key_2": "val2"})
self.assertEqual(vector_store_config.get("test_key_2"), "val2")
# Test method config
vector_store_config.set_method_config("test_method", {"param": 1})
self.assertEqual(vector_store_config.get_method_config("test_method")["param"], 1)
if __name__ == '__main__':
unittest.main()
+4 -1
View File
@@ -5,7 +5,8 @@ import traceback
import logging
from typing import Any, Dict, List, Optional
# Configure logging
import pytest
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
logger = logging.getLogger("verify_backends")
@@ -15,6 +16,8 @@ except ImportError:
logger.error("Failed to import semantica. Make sure you are in the project root or semantica is installed.")
exit(1)
pytestmark = pytest.mark.integration
def verify_backend(backend_name: str, config: Dict[str, Any]) -> bool:
logger.info(f"\n{'='*20} Verifying {backend_name.upper()} {'='*20}")
store = None
+4 -1
View File
@@ -3,11 +3,14 @@ import os
from typing import List, Dict, Any, Optional
from dataclasses import dataclass, field
# Add project root to path
import pytest
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
from semantica.context import AgentContext, ContextGraph
pytestmark = pytest.mark.integration
@dataclass
class VectorSearchResult:
id: str
+292
View File
@@ -0,0 +1,292 @@
import os
import sys
import unittest
import numpy as np
from datetime import datetime
import logging
import pytest
# Add project root to path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
from semantica.visualization import (
KGVisualizer,
OntologyVisualizer,
EmbeddingVisualizer,
SemanticNetworkVisualizer,
QualityVisualizer,
AnalyticsVisualizer,
TemporalVisualizer
)
from semantica.kg import GraphBuilder, GraphAnalyzer, TemporalVersionManager
from semantica.ontology import OntologyGenerator
from semantica.embeddings import EmbeddingGenerator
pytestmark = pytest.mark.integration
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("reproduce_notebooks")
def run_introduction_notebook():
logger.info("Running Introduction Notebook steps...")
# Step 1: Knowledge Graph Visualization
logger.info("Step 1: Knowledge Graph Visualization")
kg_visualizer = KGVisualizer()
builder = GraphBuilder()
entities = [
{"id": "e1", "type": "Organization", "name": "Apple Inc.", "properties": {}},
{"id": "e2", "type": "Person", "name": "Tim Cook", "properties": {}}
]
relationships = [
{"source": "e2", "target": "e1", "type": "CEO_of", "properties": {}}
]
kg = builder.build([{"entities": entities, "relationships": relationships}])
viz = kg_visualizer.visualize_network(kg, output="interactive")
assert viz is not None, "KG visualization failed"
logger.info("KG Visualization successful")
# Step 2: Ontology Visualization
logger.info("Step 2: Ontology Visualization")
ontology_visualizer = OntologyVisualizer()
generator = OntologyGenerator(min_occurrences=1)
ontology = generator.generate_ontology({"entities": entities, "relationships": relationships})
viz = ontology_visualizer.visualize_hierarchy(ontology, output="interactive")
# Note: verify if None is expected if ontology is simple or empty, but here it should be fine
if viz is None:
logger.warning("Ontology visualization returned None (might be due to empty hierarchy)")
else:
logger.info("Ontology Visualization successful")
# Step 3: Embedding Visualization
logger.info("Step 3: Embedding Visualization")
embedding_visualizer = EmbeddingVisualizer()
# Mocking EmbeddingGenerator to avoid heavy model loading if possible,
# but let's try to use the real one if it falls back gracefully.
# If it fails, we will catch and use random embeddings.
try:
emb_generator = EmbeddingGenerator()
texts = ["Apple Inc.", "Microsoft Corporation", "Amazon"]
embeddings = emb_generator.generate_embeddings(texts, data_type="text")
except Exception as e:
logger.warning(f"Embedding generation failed: {e}. Using random embeddings.")
embeddings = np.random.rand(3, 384)
labels = ["Apple", "Microsoft", "Amazon"]
# Need at least n_neighbors + 1 samples for UMAP usually, but with 3 samples it might warn.
# Let's use PCA or just catch potential UMAP errors if samples are too few.
try:
viz = embedding_visualizer.visualize_2d_projection(embeddings, labels, method="umap")
if viz is None:
# Fallback to pca if umap fails silently or returns None
viz = embedding_visualizer.visualize_2d_projection(embeddings, labels, method="pca")
except Exception as e:
logger.warning(f"UMAP visualization failed: {e}. Trying PCA.")
viz = embedding_visualizer.visualize_2d_projection(embeddings, labels, method="pca")
assert viz is not None, "Embedding visualization failed"
logger.info("Embedding Visualization successful")
# Step 4: Semantic Network Visualization
logger.info("Step 4: Semantic Network Visualization")
semantic_network = {
"nodes": [
{"id": "n1", "label": "Node 1", "type": "Entity"},
{"id": "n2", "label": "Node 2", "type": "Entity"}
],
"edges": [
{"source": "n1", "target": "n2", "label": "related_to"}
]
}
sem_viz = SemanticNetworkVisualizer()
viz1 = sem_viz.visualize_network(semantic_network, output="interactive")
viz2 = sem_viz.visualize_node_types(semantic_network, output="interactive")
viz3 = sem_viz.visualize_edge_types(semantic_network, output="interactive")
assert viz1 is not None, "Semantic Network visualization failed"
assert viz2 is not None, "Node Types visualization failed"
assert viz3 is not None, "Edge Types visualization failed"
logger.info("Semantic Network Visualization successful")
# Step 5: Advanced Embedding Visualization
logger.info("Step 5: Advanced Embedding Visualization")
text_emb = np.random.rand(50, 128)
image_emb = np.random.rand(50, 128)
audio_emb = np.random.rand(50, 128)
emb_viz = EmbeddingVisualizer()
viz1 = emb_viz.visualize_multimodal_comparison(text_emb, image_emb, audio_emb, output="interactive")
viz2 = emb_viz.visualize_quality_metrics(text_emb, output="interactive")
assert viz1 is not None, "Multimodal comparison failed"
assert viz2 is not None, "Quality metrics visualization failed"
logger.info("Advanced Embedding Visualization successful")
def run_advanced_notebook():
logger.info("Running Advanced Notebook steps...")
# Step 1: Create Sample Knowledge Graph
logger.info("Step 1: Create Sample Knowledge Graph")
builder = GraphBuilder()
entities = [
{"id": "e1", "type": "Person", "name": "Alice", "properties": {"age": 30}},
{"id": "e2", "type": "Person", "name": "Bob", "properties": {"age": 35}},
{"id": "e3", "type": "Organization", "name": "Tech Corp", "properties": {"founded": 2010}},
{"id": "e4", "type": "Location", "name": "San Francisco", "properties": {"country": "USA"}},
]
relationships = [
{"source": "e1", "target": "e2", "type": "knows", "properties": {"since": 2020}},
{"source": "e1", "target": "e3", "type": "works_for", "properties": {"role": "Engineer"}},
{"source": "e3", "target": "e4", "type": "located_in", "properties": {}},
]
knowledge_graph = builder.build([{"entities": entities, "relationships": relationships}])
# Step 2: Knowledge Graph Visualization
logger.info("Step 2: Knowledge Graph Visualization")
kg_visualizer = KGVisualizer(layout="force", color_scheme="vibrant")
viz = kg_visualizer.visualize_network(knowledge_graph, output="interactive")
assert viz is not None, "KG visualization failed"
logger.info("KG Visualization successful")
# Step 3: Generate Embeddings and Visualize
logger.info("Step 3: Generate Embeddings and Visualize")
# Use random embeddings to ensure stability
embeddings = np.random.rand(len(entities), 128)
labels = [entity.get("type", "Unknown") for entity in entities]
embedding_visualizer = EmbeddingVisualizer()
# t-SNE requires more samples typically, use PCA if it fails
try:
viz = embedding_visualizer.visualize_2d_projection(embeddings, labels, method="tsne", output="interactive", file_path=None)
except Exception as e:
logger.warning(f"t-SNE failed (likely too few samples): {e}. Using PCA.")
viz = embedding_visualizer.visualize_2d_projection(embeddings, labels, method="pca", output="interactive", file_path=None)
assert viz is not None, "Embedding visualization failed"
logger.info("Embedding Visualization successful")
# Step 4: Quality Metrics Visualization
logger.info("Step 4: Quality Metrics Visualization")
quality_visualizer = QualityVisualizer()
quality_report = {
"overall_score": 0.85,
"consistency_score": 0.90,
"completeness_score": 0.80
}
viz = quality_visualizer.visualize_dashboard(quality_report, output="interactive")
assert viz is not None, "Quality dashboard visualization failed"
logger.info("Quality Visualization successful")
# Step 5: Graph Analytics Visualization
logger.info("Step 5: Graph Analytics Visualization")
# Mocking GraphAnalyzer results
centrality_scores = {"e1": 0.5, "e2": 0.3, "e3": 0.8, "e4": 0.4}
# Wrap in expected format
centrality_data = {"centrality": centrality_scores}
community_dict = {"e1": 0, "e2": 0, "e3": 1, "e4": 1}
# Wrap in expected format
communities_data = {"node_assignments": community_dict}
analytics_visualizer = AnalyticsVisualizer()
viz1 = analytics_visualizer.visualize_centrality_rankings(centrality_data, title="Node Centrality Scores")
viz2 = analytics_visualizer.visualize_community_structure(
knowledge_graph,
communities_data,
title="Community Detection"
)
assert viz1 is not None, "Centrality visualization failed"
assert viz2 is not None, "Communities visualization failed"
logger.info("Analytics Visualization successful")
# Step 6: Temporal Data Visualization
logger.info("Step 6: Temporal Data Visualization")
temporal_kg = {
"entities": entities,
"relationships": relationships,
"timestamps": {
"e1": [2020, 2021, 2022],
"e2": [2020, 2021],
"e3": [2010, 2015, 2020, 2022],
}
}
# Generate events from timestamps
events = []
for entity_id, times in temporal_kg["timestamps"].items():
for t in times:
events.append({
"timestamp": t,
"type": "update",
"entity": entity_id,
"label": f"Update {entity_id}"
})
temporal_kg["events"] = events
entity_history = {
"e1": [
{"timestamp": 2020, "properties": {"age": 28}},
{"timestamp": 2021, "properties": {"age": 29}},
{"timestamp": 2022, "properties": {"age": 30}},
]
}
temporal_visualizer = TemporalVisualizer()
viz1 = temporal_visualizer.visualize_timeline(temporal_kg, output="interactive")
timestamps = [str(item["timestamp"]) for item in entity_history["e1"]]
age_values = [item["properties"]["age"] for item in entity_history["e1"]]
metrics_history = {"age": age_values}
viz2 = temporal_visualizer.visualize_metrics_evolution(metrics_history, timestamps, output="interactive")
assert viz1 is not None, "Timeline visualization failed"
assert viz2 is not None, "Metrics evolution visualization failed"
# Version Manager part
try:
version_manager = TemporalVersionManager()
v1 = version_manager.create_version(temporal_kg, timestamp="2020-01-01", version_label="v2020")
temporal_kg_v2 = {
"entities": temporal_kg.get("entities", []),
"relationships": temporal_kg.get("relationships", []) + [
{"source": "e1", "target": "e2", "type": "collaborated_with", "valid_from": "2023-01-01"}
]
}
v2 = version_manager.create_version(temporal_kg_v2, timestamp="2023-01-01", version_label="v2023")
snapshots = {v1["timestamp"]: v1, v2["timestamp"]: v2}
viz3 = temporal_visualizer.visualize_snapshot_comparison(snapshots, output="interactive")
version_history = [
{"version": v1.get("label"), "timestamp": v1.get("timestamp")},
{"version": v2.get("label"), "timestamp": v2.get("timestamp")}
]
viz4 = temporal_visualizer.visualize_version_history(version_history, output="interactive")
assert viz3 is not None, "Snapshot comparison failed"
assert viz4 is not None, "Version history visualization failed"
except Exception as e:
logger.warning(f"Temporal Version Manager part failed: {e}")
logger.info("Temporal Visualization successful")
if __name__ == "__main__":
try:
run_introduction_notebook()
print("-" * 50)
run_advanced_notebook()
print("ALL NOTEBOOK REPRODUCTIONS SUCCESSFUL")
except Exception as e:
logger.error(f"Reproduction failed: {e}")
sys.exit(1)
@@ -0,0 +1,149 @@
import unittest
from unittest.mock import MagicMock, patch
import sys
import numpy as np
# Helper to mock modules
def mock_module(name):
m = MagicMock()
sys.modules[name] = m
return m
class TestOptionalDependencies(unittest.TestCase):
@classmethod
def setUpClass(cls):
# Mock heavy/problematic dependencies globally to prevent environment crashes
# We use a dict to save original modules if they exist, but for this test file
# we generally want to run in a controlled "clean" environment.
cls.modules_to_patch = [
'sklearn', 'sklearn.decomposition', 'sklearn.manifold',
'scipy', 'scipy.optimize',
'matplotlib', 'matplotlib.pyplot', 'matplotlib.patches',
'plotly', 'plotly.express', 'plotly.graph_objects', 'plotly.subplots',
'networkx', 'seaborn'
]
cls.original_modules = {}
for mod in cls.modules_to_patch:
if mod in sys.modules:
cls.original_modules[mod] = sys.modules[mod]
sys.modules[mod] = MagicMock()
@classmethod
def tearDownClass(cls):
# Restore original modules
for mod in cls.modules_to_patch:
if mod in cls.original_modules:
sys.modules[mod] = cls.original_modules[mod]
else:
del sys.modules[mod]
def setUp(self):
# Clear cached visualization modules to ensure fresh imports
self.viz_modules = [
'semantica.visualization.embedding_visualizer',
'semantica.visualization.ontology_visualizer',
'semantica.visualization.kg_visualizer',
'semantica.visualization.utils.export_formats'
]
for mod in self.viz_modules:
if mod in sys.modules:
del sys.modules[mod]
def test_embedding_visualizer_without_umap(self):
"""Test EmbeddingVisualizer behavior when umap is missing."""
# Ensure umap is missing
with patch.dict(sys.modules, {'umap': None}):
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
# Setup PCA mock to verify fallback
mock_pca_class = sys.modules['sklearn.decomposition'].PCA
mock_pca_instance = mock_pca_class.return_value
# Configure fit_transform to return correct shape (n_samples, 2)
mock_pca_instance.fit_transform.return_value = np.zeros((4, 2))
viz = EmbeddingVisualizer()
# Use numpy array!
embeddings = np.array([[0, 1, 2], [1, 0, 3], [0, 0, 0], [1, 1, 1]])
# Should fallback to PCA when method="umap" is used but umap is None
# The code logs a warning and uses PCA
viz.visualize_2d_projection(embeddings, method="umap")
# Verify PCA was called
mock_pca_class.assert_called()
def test_ontology_visualizer_without_graphviz(self):
"""Test OntologyVisualizer behavior when graphviz is missing."""
# Ensure graphviz is missing
with patch.dict(sys.modules, {'graphviz': None}):
from semantica.visualization.ontology_visualizer import OntologyVisualizer, ProcessingError
viz = OntologyVisualizer()
ontology = {
"classes": [
{"name": "A", "label": "A"},
{"name": "B", "label": "B", "parent": "A"}
]
}
with self.assertRaises(ProcessingError) as cm:
viz.visualize_hierarchy(ontology, output="dot", file_path="test.dot")
self.assertIn("Graphviz is required for DOT export", str(cm.exception))
def test_analytics_visualizer_without_plotly(self):
"""Test AnalyticsVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.analytics_visualizer import AnalyticsVisualizer, ProcessingError
# Need to ensure numpy is available for init (it's imported at top level)
# But we are testing plotly missing.
viz = AnalyticsVisualizer()
with self.assertRaises(ProcessingError) as cm:
viz.visualize_centrality_rankings({"node1": 1.0})
self.assertIn("Plotly is required", str(cm.exception))
def test_quality_visualizer_without_plotly(self):
"""Test QualityVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.quality_visualizer import QualityVisualizer, ProcessingError
viz = QualityVisualizer()
with self.assertRaises(ProcessingError) as cm:
viz.visualize_dashboard({})
self.assertIn("Plotly is required", str(cm.exception))
def test_semantic_network_visualizer_without_plotly(self):
"""Test SemanticNetworkVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.semantic_network_visualizer import SemanticNetworkVisualizer, ProcessingError
viz = SemanticNetworkVisualizer()
with self.assertRaises(ProcessingError) as cm:
viz.visualize_network({})
self.assertIn("Plotly is required", str(cm.exception))
def test_temporal_visualizer_without_plotly(self):
"""Test TemporalVisualizer behavior when plotly is missing."""
with patch.dict(sys.modules, {'plotly': None, 'plotly.express': None, 'plotly.graph_objects': None}):
from semantica.visualization.temporal_visualizer import TemporalVisualizer, ProcessingError
viz = TemporalVisualizer()
with self.assertRaises(ProcessingError) as cm:
viz.visualize_timeline({"events": []})
self.assertIn("Plotly is required", str(cm.exception))
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,254 @@
import unittest
from unittest.mock import MagicMock, patch
import sys
import numpy as np
from pathlib import Path
import pytest
# Mock heavy libraries before importing visualization modules
sys.modules['matplotlib'] = MagicMock()
sys.modules['matplotlib.pyplot'] = MagicMock()
sys.modules['matplotlib.colors'] = MagicMock()
sys.modules['matplotlib.patches'] = MagicMock()
sys.modules['plotly'] = MagicMock()
sys.modules['plotly.express'] = MagicMock()
sys.modules['plotly.graph_objects'] = MagicMock()
sys.modules['plotly.subplots'] = MagicMock()
sys.modules['seaborn'] = MagicMock()
sys.modules['umap'] = MagicMock()
sys.modules['sklearn'] = MagicMock()
sys.modules['sklearn.decomposition'] = MagicMock()
sys.modules['sklearn.manifold'] = MagicMock()
sys.modules['networkx'] = MagicMock()
sys.modules['graphviz'] = MagicMock()
# Import visualizers
from semantica.visualization.kg_visualizer import KGVisualizer
from semantica.visualization.ontology_visualizer import OntologyVisualizer
from semantica.visualization.embedding_visualizer import EmbeddingVisualizer
from semantica.visualization.semantic_network_visualizer import SemanticNetworkVisualizer
from semantica.visualization.quality_visualizer import QualityVisualizer
from semantica.visualization.analytics_visualizer import AnalyticsVisualizer
from semantica.visualization.temporal_visualizer import TemporalVisualizer
from semantica.visualization.utils.color_schemes import ColorScheme
pytestmark = pytest.mark.integration
class TestVisualizationComprehensive(unittest.TestCase):
def setUp(self):
self.mock_logger = MagicMock()
self.mock_tracker = MagicMock()
# Patch dependencies for all visualizers
self.patchers = [
patch('semantica.visualization.kg_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.kg_visualizer.get_progress_tracker', return_value=self.mock_tracker),
patch('semantica.visualization.ontology_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.ontology_visualizer.get_progress_tracker', return_value=self.mock_tracker),
patch('semantica.visualization.embedding_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.embedding_visualizer.get_progress_tracker', return_value=self.mock_tracker),
patch('semantica.visualization.semantic_network_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.semantic_network_visualizer.get_progress_tracker', return_value=self.mock_tracker),
patch('semantica.visualization.quality_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.quality_visualizer.get_progress_tracker', return_value=self.mock_tracker),
patch('semantica.visualization.analytics_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.analytics_visualizer.get_progress_tracker', return_value=self.mock_tracker),
patch('semantica.visualization.temporal_visualizer.get_logger', return_value=self.mock_logger),
patch('semantica.visualization.temporal_visualizer.get_progress_tracker', return_value=self.mock_tracker),
# Mock Layouts
patch('semantica.visualization.kg_visualizer.ForceDirectedLayout', MagicMock()),
patch('semantica.visualization.kg_visualizer.HierarchicalLayout', MagicMock()),
patch('semantica.visualization.kg_visualizer.CircularLayout', MagicMock()),
patch('semantica.visualization.ontology_visualizer.HierarchicalLayout', MagicMock()),
patch('semantica.visualization.semantic_network_visualizer.ForceDirectedLayout', MagicMock()),
]
for p in self.patchers:
p.start()
# Reset plotly mocks
import plotly.graph_objects as go
import plotly.express as px
go.Figure.reset_mock()
px.bar.reset_mock()
px.scatter.reset_mock()
def tearDown(self):
for p in self.patchers:
p.stop()
# --- KGVisualizer Tests ---
def test_kg_visualizer(self):
viz = KGVisualizer()
graph = {
"entities": [{"id": "e1", "label": "E1", "type": "T1"}, {"id": "e2", "label": "E2", "type": "T2"}],
"relationships": [{"source": "e1", "target": "e2", "type": "R1"}]
}
# Test visualize_network
viz.visualize_network(graph)
# Test visualize_communities
communities = {"node_assignments": {"e1": 0, "e2": 1}, "num_communities": 2}
viz.visualize_communities(graph, communities)
# Test visualize_centrality
centrality = {"centrality": {"e1": 0.5, "e2": 0.3}}
viz.visualize_centrality(graph, centrality)
# Test visualize_entity_types
viz.visualize_entity_types(graph)
# Test visualize_relationship_matrix
viz.visualize_relationship_matrix(graph)
# --- OntologyVisualizer Tests ---
def test_ontology_visualizer(self):
viz = OntologyVisualizer()
ontology = {
"classes": [
{"name": "C1", "label": "Class 1", "parent": None},
{"name": "C2", "label": "Class 2", "parent": "C1"}
],
"properties": [
{"name": "P1", "label": "Prop 1", "domain": "C1", "range": "C2"}
]
}
# Test visualize_hierarchy
viz.visualize_hierarchy(ontology)
# Test visualize_properties
viz.visualize_properties(ontology)
# Test visualize_structure
viz.visualize_structure(ontology)
# Test visualize_class_property_matrix
viz.visualize_class_property_matrix(ontology)
# Test visualize_metrics
viz.visualize_metrics(ontology)
# Test visualize_semantic_model (mocking extract classes)
semantic_model = {"nodes": [{"id": "n1", "type": "T1"}], "edges": []}
viz.visualize_semantic_model(semantic_model)
# --- SemanticNetworkVisualizer Tests ---
def test_semantic_network_visualizer(self):
viz = SemanticNetworkVisualizer()
semantic_network = {
"nodes": [{"id": "n1", "label": "N1", "type": "T1"}],
"edges": [{"source": "n1", "target": "n1", "label": "R1"}]
}
# Test visualize_network
with patch('semantica.visualization.kg_visualizer.KGVisualizer') as MockKG:
viz.visualize_network(semantic_network)
MockKG.return_value.visualize_network.assert_called()
# Test visualize_node_types
viz.visualize_node_types(semantic_network)
# Test visualize_edge_types
viz.visualize_edge_types(semantic_network)
# --- QualityVisualizer Tests ---
def test_quality_visualizer(self):
viz = QualityVisualizer()
# Test visualize_dashboard
report = {"overall_score": 0.8, "consistency_score": 0.9, "completeness_score": 0.7}
viz.visualize_dashboard(report)
# Test visualize_score_distribution
scores = [0.1, 0.5, 0.9]
viz.visualize_score_distribution(scores)
# Test visualize_issues
report_issues = {"issues": [{"type": "error", "severity": "high"}]}
viz.visualize_issues(report_issues)
# Test visualize_completeness_metrics
metrics = {"entity_completeness": 0.8}
viz.visualize_completeness_metrics(metrics)
# Test visualize_consistency_heatmap
consistency = {"consistency_matrix": [[1.0]], "labels": ["C1"]}
viz.visualize_consistency_heatmap(consistency)
# --- AnalyticsVisualizer Tests ---
def test_analytics_visualizer(self):
viz = AnalyticsVisualizer()
graph = {"entities": [], "relationships": []}
# Test visualize_centrality_rankings
centrality = {"rankings": [{"node": "n1", "score": 0.9}]}
viz.visualize_centrality_rankings(centrality)
# Test visualize_community_structure
communities = {"node_assignments": {}}
with patch('semantica.visualization.kg_visualizer.KGVisualizer') as MockKG:
viz.visualize_community_structure(graph, communities)
# Test visualize_connectivity
connectivity = {"is_connected": True, "num_components": 1, "component_sizes": [10]}
viz.visualize_connectivity(connectivity)
# Test visualize_degree_distribution
viz.visualize_degree_distribution(graph)
# Test visualize_metrics_dashboard
metrics = {"num_nodes": 10, "num_edges": 20, "density": 0.1}
viz.visualize_metrics_dashboard(metrics)
# Test visualize_centrality_comparison
results = {"degree": {"rankings": [{"node": "n1", "score": 0.9}]}}
viz.visualize_centrality_comparison(results)
# --- TemporalVisualizer Tests ---
def test_temporal_visualizer(self):
viz = TemporalVisualizer()
# Test visualize_timeline
temporal_data = {"events": [{"timestamp": "2023-01-01", "type": "create", "label": "E1"}], "timestamps": ["2023-01-01"]}
viz.visualize_timeline(temporal_data)
# Test visualize_temporal_patterns
patterns = [{"pattern_type": "trend", "start_time": "2023", "end_time": "2024", "entities": ["e1"]}]
viz.visualize_temporal_patterns(patterns)
# Test visualize_snapshot_comparison
snapshots = {"2023": {"entities": ["e1"], "relationships": []}}
viz.visualize_snapshot_comparison(snapshots)
# Test visualize_version_history
history = [{"version": "v1", "date": "2023-01-01"}]
viz.visualize_version_history(history)
# Test visualize_metrics_evolution
metrics_history = {"nodes": [10, 20]}
timestamps = ["2023", "2024"]
viz.visualize_metrics_evolution(metrics_history, timestamps)
# --- EmbeddingVisualizer Tests ---
def test_embedding_visualizer(self):
viz = EmbeddingVisualizer()
embeddings = np.random.rand(10, 10)
# Test visualize_2d_projection (mock UMAP/PCA)
with patch('semantica.visualization.embedding_visualizer.umap.UMAP') as MockUMAP:
MockUMAP.return_value.fit_transform.return_value = np.random.rand(10, 2)
viz.visualize_2d_projection(embeddings)
# Test visualize_similarity_heatmap
viz.visualize_similarity_heatmap(embeddings[:5]) # smaller for heatmap
# Test visualize_clustering
clusters = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
with patch('semantica.visualization.embedding_visualizer.umap.UMAP') as MockUMAP:
MockUMAP.return_value.fit_transform.return_value = np.random.rand(10, 2)
viz.visualize_clustering(embeddings, clusters)
if __name__ == '__main__':
unittest.main()