chore: release v0.2.5

docs: update CONTRIBUTING.md and CONTRIBUTORS.md with improved format…

.github/FUNDING.yml

@@ -1,8 +1,3 @@
 # Funding options for Semantica
-# Uncomment and add your usernames/links below
-
-# github: [username]
-# patreon: username
-# ko_fi: username
-# custom: ["https://your-funding-page.com"]
+github: Hawksight-AI
 

.github/SUPPORT.md

@@ -32,6 +32,8 @@ For enterprise support, custom development, or consulting services:
 
 ## Sponsorship
 
+### Sponsor this project
+
 Support Semantica development:
 - [GitHub Sponsors](https://github.com/sponsors/Hawksight-AI)
 

.pre-commit-config.yaml

@@ -5,6 +5,7 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-yaml
+        exclude: 'neptune-setup\.yaml$'
       - id: check-json
       - id: check-toml
       - id: check-added-large-files
@@ -49,9 +50,15 @@ repos:
     hooks:
       - id: yamllint
         args: ['-d', '{extends: default, rules: {line-length: {max: 120}}}']
+        exclude: 'neptune-setup\.yaml$'
+
+  - repo: https://github.com/aws-cloudformation/cfn-lint
+    rev: v1.43.3
+    hooks:
+      - id: cfn-lint
+        files: 'neptune-setup\.yaml$'
 
   # Removed slow hooks for faster development:
   # - mypy: Type checking (can be run manually or in CI)
   # - bandit: Security scanning (can be run separately)
   # - pytest: Testing (should be run manually, not on every commit)
-

CHANGELOG.md

@@ -7,6 +7,106 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.5] - 2026-01-27
+
+### Added
+- **Pinecone Vector Store Support**:
+    - Implemented native Pinecone support (`PineconeStore`) with full CRUD capabilities.
+    - Added support for serverless and pod-based indexes, namespaces, and metadata filtering.
+    - Integrated with `VectorStore` unified interface and registry.
+    - (Closes #219, Resolves #220)
+- **Configurable LLM Retry Logic**:
+    - Exposed `max_retries` parameter in `NERExtractor`, `RelationExtractor`, `TripletExtractor` and low-level extraction methods (`extract_entities_llm`, `extract_relations_llm`, `extract_triplets_llm`).
+    - Defaults to 3 retries to prevent infinite loops during JSON validation failures or API timeouts.
+    - Propagated retry configuration through chunked processing helpers to ensure consistent behavior for long documents.
+    - Updated `03_Earnings_Call_Analysis.ipynb` to use `max_retries=3` by default.
+
+### Added
+- **Bring Your Own Model (BYOM) Support**:
+    - Enabled full support for custom Hugging Face models in `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Added support for custom tokenizers in `HuggingFaceModelLoader` to handle models with non-standard tokenization requirements.
+    - Implemented robust fallback logic for model selection: runtime options (`extract(model=...)`) now correctly override configuration defaults.
+- **Enhanced NER Implementation**:
+    - Added configurable aggregation strategies (`simple`, `first`, `average`, `max`) to `extract_entities_huggingface` for better sub-word token handling.
+    - Implemented robust IOB/BILOU parsing to reconstruct entities from raw model outputs when structured output is unavailable.
+    - Added confidence scoring for aggregated entities.
+- **Relation Extraction Improvements**:
+    - Implemented standard entity marker technique (wrapping subject/object with `<subj>`, `<obj>` tags) in `extract_relations_huggingface` for compatibility with sequence classification models.
+    - Added structured output parsing to convert raw model predictions into validated `Relation` objects.
+- **Triplet Extraction Completion**:
+    - Added specialized parsing for Seq2Seq models (e.g., REBEL) in `extract_triplets_huggingface` to generate structured triplets directly from text.
+    - Implemented post-processing logic to clean and validate generated triplets.
+
+### Fixed
+- **LLM Extraction Stability**:
+    - Fixed infinite retry loops in `BaseProvider` by strictly enforcing `max_retries` limit during structured output generation.
+    - Resolved stuck execution in earnings call analysis notebooks when using smaller models (e.g., Llama 3 8B) that frequently produce invalid JSON.
+- **Model Parameter Precedence**:
+    - Fixed issue where configuration defaults took precedence over runtime arguments in Hugging Face extractors. Runtime options now correctly override config values.
+- **Import Handling**:
+    - Fixed circular import issues in test suites by implementing robust mocking strategies.
+
+## [0.2.4] - 2026-01-22
+
+### Added
+- **Ontology Ingestion Module**:
+    - Implemented `OntologyIngestor` in `semantica.ingest` for parsing RDF/OWL files (Turtle, RDF/XML, JSON-LD, N3) into standardized `OntologyData` objects.
+    - Added `ingest_ontology` convenience function and integrated it into the unified `ingest(source_type="ontology")` interface.
+    - Added recursive directory scanning support for batch ontology ingestion.
+    - Exposed ingestion tools in `semantica.ontology` for better discoverability.
+    - Added `OntologyData` dataclass for consistent metadata handling (source path, format, timestamps).
+- **Documentation**:
+    - **Ontology Usage Guide**: Updated `ontology_usage.md` with comprehensive examples for single-file and directory ingestion.
+    - **API Reference**: Updated `ontology.md` with `OntologyIngestor` class documentation and method details.
+- **Tests**:
+    - **Comprehensive Test Suite**: Added `tests/ingest/test_ontology_ingestor.py` covering all supported formats, error handling, and unified interface integration.
+    - **Demo Script**: Added `examples/demo_ontology_ingest.py` for end-to-end usage demonstration.
+
+## [0.2.3] - 2026-01-20
+
+### Fixed
+- **LLM Relation Extraction Parsing**:
+    - Fixed relation extraction returning zero relations despite successful API calls to Groq and other providers
+    - Normalized typed responses from instructor/OpenAI/Groq to consistent dict format before parsing
+    - Added structured JSON fallback when typed generation yields zero relations to avoid silent empty outputs
+    - Removed acceptance of extra kwargs (`max_tokens`, `max_entities_prompt`) from relation extraction internals
+    - Filtered kwargs passed to provider LLM calls to only `temperature` and `verbose`
+- **API Parameter Handling**:
+    - Limited kwargs forwarded in chunked extraction helper to prevent parameter leakage
+    - Ensured minimal, safe parameters are passed to provider calls
+- **Pipeline Circular Import (Issues #192, #193)**:
+    - Fixed circular import between `pipeline_builder` and `pipeline_validator` triggered during `semantica.pipeline` import
+    - Lazy-loaded `PipelineValidator` inside `PipelineBuilder.__init__` and guarded type hints with `TYPE_CHECKING`
+    - Ensured `from semantica.deduplication import DuplicateDetector` no longer fails even when pipeline module is imported
+- **JupyterLab Progress Output (Issue #181)**:
+    - Added `SEMANTICA_DISABLE_JUPYTER_PROGRESS` environment variable to disable rich Jupyter/Colab progress tables
+    - When enabled, progress falls back to console-style output, preventing infinite scrolling and JupyterLab out-of-memory errors
+
+### Added
+- **Comprehensive Test Suite**:
+-    - Added unit tests (`tests/test_relations_llm.py`) with mocked LLM provider covering both typed and structured response paths
+-    - Added integration tests (`tests/integration/test_relations_groq.py`) for real Groq API calls with environment variable API key
+-    - Tests validate relation extraction completion and result parsing across different response formats
+- **Amazon Neptune Dev Environment**:
+-    - Added CloudFormation template (`cookbook/introduction/neptune-setup.yaml`) to provision a dev Neptune cluster with public endpoint and IAM auth enabled
+-    - Documented deployment, cost estimates, and IAM User vs IAM Role best practices in `cookbook/introduction/21_Amazon_Neptune_Store.ipynb`
+-    - Added `cfn-lint` to `.pre-commit-config.yaml` for validating CloudFormation templates while excluding `neptune-setup.yaml` from generic YAML linters
+- **Vector Store High-Performance Ingestion**:
+-    - Added `VectorStore.add_documents` for high-throughput ingestion with automatic embedding generation, batching, and parallel processing
+-    - Added `VectorStore.embed_batch` helper for generating embeddings for lists of texts without immediately storing them
+-    - Enabled default parallel ingestion in `VectorStore` with `max_workers=6` for common workloads
+-    - Added dedicated documentation page `docs/vector_store_usage.md` describing high-performance vector store usage and configuration
+-    - Added `tests/vector_store/test_vector_store_parallel.py` covering parallel vs sequential performance, error handling, and edge cases for `add_documents` and `embed_batch`
+
+### Changed
+- **Relation Extraction API**:
+-    - Simplified parameter interface by removing unused kwargs that were previously ignored
+-    - Improved error handling and verbose logging for debugging relation extraction issues
+-    - Enhanced robustness of post-response parsing across different LLM providers
+- **Vector Store Defaults and Examples**:
+-    - Standardized `VectorStore` default concurrency to `max_workers=6` for parallel ingestion
+-    - Updated vector store reference documentation and usage guides to rely on implicit defaults instead of requiring manual `max_workers` configuration in examples
+
 
 ## [0.2.2] - 2026-01-15
 

CONTRIBUTING.md

@@ -1,306 +1,266 @@
 # Contributing to Semantica
 
-Thank you for your interest in contributing to Semantica! This document provides guidelines and instructions for contributing to the project.
+Thank you for your interest in contributing! Every contribution, no matter how small, is valuable. 🎉
 
-## Table of Contents
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-- [Code of Conduct](#code-of-conduct)
-- [Getting Started](#getting-started)
-- [Development Setup](#development-setup)
-- [Code Style Guidelines](#code-style-guidelines)
-- [Testing Requirements](#testing-requirements)
-- [Commit Message Conventions](#commit-message-conventions)
-- [Pull Request Process](#pull-request-process)
-- [Documentation Standards](#documentation-standards)
-- [Types of Contributions](#types-of-contributions)
-- [Getting Help](#getting-help)
+> **New to contributing?** Start with a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/vqRt2qbx) community.
 
-## Code of Conduct
+---
 
-This project adheres to a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the maintainers.
+## 🚀 Quick Start
 
-## Getting Started
+1. Find a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue)
+2. [Fork Semantica](https://github.com/Hawksight-AI/semantica/fork) & clone the repository
+3. Make your changes
+4. Submit a pull request!
 
-1. **Fork the repository** on GitHub
-2. **Clone your fork** locally:
-   ```bash
-   git clone https://github.com/your-username/semantica.git
-   cd semantica
-   ```
-3. **Add the upstream remote**:
-   ```bash
-   git remote add upstream https://github.com/Hawksight-AI/semantica.git
-   ```
+**Need help?** Join [Discord](https://discord.gg/vqRt2qbx) or [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
 
-## Development Setup
+---
 
-### Prerequisites
+## 🎯 Ways to Contribute
 
-- Python 3.8 or higher (3.9+ recommended)
-- pip package manager
-- Git
+### 💻 Code
 
-### Installation
+**What you can do:**
+- Fix bugs
+- Add new features
+- Improve code quality (add type hints, docstrings, improve error messages)
+- Optimize performance
 
-1. **Create a virtual environment** (recommended):
-   ```bash
-   python -m venv venv
-   source venv/bin/activate  # On Windows: venv\Scripts\activate
-   ```
+**Where:** `semantica/` directory
 
-2. **Install the project in editable mode with dev dependencies**:
-   ```bash
-   pip install -e ".[dev]"
-   ```
+**Good first issues:** Add docstrings, type hints, or improve error messages
 
-3. **Install pre-commit hooks**:
-   ```bash
-   pre-commit install
-   ```
+---
 
-### Verify Installation
+### 📝 Documentation
+
+**What you can do:**
+- Fix typos and grammar errors
+- Improve clarity and readability
+- Add code examples and tutorials
+- Create new cookbook notebooks
+- Improve API documentation (docstrings)
+- Create troubleshooting guides
+- Update installation instructions
+- Add missing documentation
+
+**Where:** `README.md`, `docs/`, `cookbook/`, docstrings in code
+
+**Good first issues:** Fix typos, add examples, create cookbook tutorials, improve docstrings
+
+**Documentation formatting:**
+- Use clear, concise language
+- Include code examples where helpful
+- Follow markdown best practices
+- Use proper headings hierarchy
+- Add links to related sections
+- Include screenshots for UI-related docs
+
+---
+
+### 🧪 Testing
+
+**What you can do:**
+- Add unit tests
+- Improve test coverage
+- Add integration tests
+
+**Where:** `tests/` directory
+
+**Good first issues:** Add tests for specific functions or classes
+
+---
+
+### 🐛 Bug Reports
+
+**What:** Report bugs you find
+
+**How:** Use the [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
+
+**Include:** Description, steps to reproduce, expected vs actual behavior, environment details
+
+---
+
+### 💡 Feature Requests
+
+**What:** Suggest new features or improvements
+
+**How:** Use the [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
+
+**Include:** Problem statement, proposed solution, use cases
+
+---
+
+### 🎨 Cookbook & Examples
+
+**What:** Create tutorials and examples
+
+**Where:** `cookbook/` directory
+
+**Examples:** Create new notebooks, add examples, improve existing tutorials
+
+---
+
+### 💬 Community Support
+
+**What:** Help others in the community
+
+**Where:** [Discord](https://discord.gg/vqRt2qbx), [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
+
+**Examples:** Answer questions, review PRs, share your projects
+
+---
+
+### 🎓 Educational Content
+
+**What:** Create educational materials
+
+**Examples:** Blog posts, video tutorials, talks, workshops, case studies
+
+---
+
+### 🔧 Other Contributions
+
+- **Design & Graphics:** Logos, diagrams, visualizations
+- **Tools & Integrations:** CLI tools, integrations with other frameworks
+- **Infrastructure:** CI/CD improvements, Docker optimization
+- **Security:** Report security vulnerabilities (privately)
+
+---
+
+## 📋 Getting Started
+
+### 1. Fork & Clone
+
+First, [fork Semantica](https://github.com/Hawksight-AI/semantica/fork) on GitHub, then:
 
 ```bash
-python -c "import semantica; print(semantica.__version__)"
-pytest --version
-black --version
+git clone https://github.com/your-username/semantica.git
+cd semantica
+git remote add upstream https://github.com/Hawksight-AI/semantica.git
 ```
 
-## Code Style Guidelines
-
-We use several tools to maintain code quality and consistency:
-
-### Formatting
-
-- **Black**: Code formatting (line length: 88)
-  ```bash
-  black semantica/
-  ```
-
-- **isort**: Import sorting
-  ```bash
-  isort semantica/
-  ```
-
-### Linting
-
-- **flake8**: Style guide enforcement
-  ```bash
-  flake8 semantica/
-  ```
-
-- **mypy**: Static type checking
-  ```bash
-  mypy semantica/
-  ```
-
-### Running All Checks
+### 2. Set Up Environment
 
 ```bash
-# Format code
-black semantica/ tests/
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # Windows: venv\Scripts\activate
 
-# Sort imports
-isort semantica/ tests/
+# Install dev dependencies
+pip install -e ".[dev]"
 
-# Lint
-flake8 semantica/ tests/
-
-# Type check
-mypy semantica/
+# Install pre-commit hooks (optional)
+pre-commit install
 ```
 
-Or use pre-commit hooks (automatically runs on commit):
-```bash
-pre-commit run --all-files
-```
-
-## Testing Requirements
-
-### Running Tests
+### 3. Create Branch
 
 ```bash
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=semantica --cov-report=html
-
-# Run specific test file
-pytest tests/test_specific.py
-
-# Run with verbose output
-pytest -v
+git checkout -b feature/your-feature-name
+# or
+git checkout -b fix/bug-description
 ```
 
-### Test Coverage
+### 4. Make Changes
 
-- Minimum coverage: **80%**
-- Critical modules: **90%+**
-- Coverage reports are generated in `htmlcov/`
+- Follow code style (see below)
+- Add tests for new features
+- Update documentation
 
-### Writing Tests
+### 5. Run Checks
 
-- Follow pytest conventions
-- Use descriptive test names
-- Include docstrings for complex tests
-- Test both success and failure cases
-- Use fixtures for common setup
-
-Example:
-```python
-def test_entity_extraction():
-    """Test basic entity extraction functionality."""
-    from semantica.semantic_extract import NamedEntityRecognizer
-    
-    ner = NamedEntityRecognizer()
-    entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
-    
-    assert len(entities) > 0
-    assert any(e.text == "Apple Inc." for e in entities)
+```bash
+pytest                          # Run tests
+black semantica/ tests/        # Format code
+isort semantica/ tests/         # Sort imports
+flake8 semantica/ tests/        # Lint
 ```
 
-## Commit Message Conventions
+Or use pre-commit hooks: `pre-commit run --all-files`
 
-We follow [Conventional Commits](https://www.conventionalcommits.org/) specification:
+### 6. Commit & Push
 
-### Format
-
-```
-<type>(<scope>): <subject>
-
-<body>
-
-<footer>
+```bash
+git commit -m "feat(module): add new feature"
+git push origin feature/your-feature-name
 ```
 
-### Types
+Then create a pull request on GitHub!
 
-- `feat`: New feature
-- `fix`: Bug fix
-- `docs`: Documentation changes
-- `style`: Code style changes (formatting, etc.)
-- `refactor`: Code refactoring
-- `test`: Adding or updating tests
-- `chore`: Maintenance tasks
-- `perf`: Performance improvements
-- `ci`: CI/CD changes
+---
 
-### Examples
+## 📐 Code Style
+
+We use automated tools:
+
+| Tool     | Purpose                    | Command                    |
+|----------|----------------------------|----------------------------|
+| **Black** | Code formatting            | `black semantica/ tests/` |
+| **isort** | Import sorting             | `isort semantica/ tests/` |
+| **flake8** | Style enforcement          | `flake8 semantica/ tests/` |
+| **mypy** | Type checking              | `mypy semantica/`          |
+
+**Run all:** `black semantica/ tests/ && isort semantica/ tests/ && flake8 semantica/ tests/ && mypy semantica/`
+
+---
+
+## 🧪 Testing
+
+```bash
+pytest                          # Run all tests
+pytest --cov=semantica         # With coverage
+pytest tests/test_file.py      # Specific file
+```
+
+**Coverage goal:** 80% minimum, 90%+ for critical modules
+
+---
+
+## 📝 Commit Messages
+
+Use [Conventional Commits](https://www.conventionalcommits.org/):
 
 ```
 feat(kg): add temporal graph support
-
-Add support for temporal knowledge graphs with version tracking
-and time-based queries.
-
-Closes #123
+fix(parse): handle empty PDF files
+docs(readme): add installation guide
+test(extract): add unit tests
 ```
 
-```
-fix(parse): handle empty PDF files gracefully
+**Types:** `feat`, `fix`, `docs`, `test`, `refactor`, `perf`, `style`, `chore`
 
-Previously, empty PDF files would cause a crash. Now they return
-an empty document with appropriate warnings.
+---
 
-Fixes #456
-```
+## ✅ PR Checklist
 
-## Pull Request Process
-
-### Before Submitting
-
-1. **Update your fork**:
-   ```bash
-   git fetch upstream
-   git checkout main
-   git merge upstream/main
-   ```
-
-2. **Create a feature branch**:
-   ```bash
-   git checkout -b feature/your-feature-name
-   # or
-   git checkout -b fix/bug-description
-   ```
-
-3. **Make your changes** and commit following our conventions
-
-4. **Run all checks**:
-   ```bash
-   pytest
-   black semantica/ tests/
-   isort semantica/ tests/
-   flake8 semantica/ tests/
-   mypy semantica/
-   ```
-
-5. **Push to your fork**:
-   ```bash
-   git push origin feature/your-feature-name
-   ```
-
-### PR Checklist
+Before submitting:
 
 - [ ] Code follows style guidelines
 - [ ] Tests pass locally
-- [ ] New tests added for new features
+- [ ] New tests added (if applicable)
 - [ ] Documentation updated
 - [ ] Commit messages follow conventions
 - [ ] No merge conflicts
-- [ ] PR description is clear and complete
 
-### PR Description Template
+---
 
-```markdown
-## Description
-Brief description of changes
+## 📖 Documentation Standards
 
-## Type of Change
-- [ ] Bug fix
-- [ ] New feature
-- [ ] Breaking change
-- [ ] Documentation update
+### Code Documentation (Docstrings)
 
-## Related Issues
-Closes #123
-Related to #456
+**Format:** Use Google-style docstrings
 
-## Testing
-- [ ] Tests pass locally
-- [ ] Added new tests
-- [ ] Updated existing tests
-
-## Checklist
-- [ ] Code follows style guidelines
-- [ ] Self-review completed
-- [ ] Comments added for complex code
-- [ ] Documentation updated
-- [ ] No new warnings generated
-```
-
-## Documentation Standards
-
-### Code Documentation
-
-- Use Google-style docstrings
-- Include type hints
-- Document all public functions and classes
-- Include examples for complex functions
-
-Example:
 ```python
-def extract_entities(
-    text: str,
-    model: str = "transformer",
-    confidence_threshold: float = 0.7
-) -> List[Entity]:
+def extract_entities(text: str, model: str = "transformer") -> List[Entity]:
     """Extract named entities from text.
     
     Args:
         text: Input text to process
         model: NER model to use (default: "transformer")
-        confidence_threshold: Minimum confidence score (default: 0.7)
     
     Returns:
         List of extracted Entity objects
@@ -309,92 +269,98 @@ def extract_entities(
         ValueError: If text is empty or model is invalid
     
     Example:
-        >>> ner = NamedEntityRecognizer()
+        >>> from semantica.semantic_extract import NERExtractor
+        >>> ner = NERExtractor(method="ml", model="en_core_web_sm")
         >>> entities = ner.extract("Apple Inc. was founded in 1976.")
         >>> len(entities)
         2
     """
-    ...
 ```
 
-### Documentation Files
+### Markdown Documentation Formatting
 
-- Update relevant documentation in `docs/`
-- Add examples to cookbook if applicable
-- Update API reference if adding new public APIs
-- Keep README.md up to date
+**General Guidelines:**
+- Use clear headings (H1 for title, H2 for main sections, H3 for subsections)
+- Keep paragraphs short and focused
+- Use bullet points for lists
+- Add code blocks with syntax highlighting
+- Include links to related documentation
 
-## Types of Contributions
+**Code Blocks:**
+- Use triple backticks with language identifier: ` ```python `, ` ```bash `
+- Include comments in code examples
+- Show expected output when helpful
 
-### 💻 Code Contributions
+**Examples:**
 
-- **Bug Fixes**: Resolving issues reported in the issue tracker.
-- **New Features**: Implementing new capabilities (please discuss via an issue first!).
-- **Refactoring**: Improving code structure and maintainability without changing behavior.
-- **Algorithm Optimization**: Improving the efficiency of graph algorithms and vector search.
+```markdown
+## Section Title
 
-#### ⚡ Performance and Latency
-We deeply value efficiency. Contributions that make Semantica faster and lighter are highly appreciated!
+Brief introduction paragraph.
 
-- **Latency Reduction**: Optimize critical paths and RAG pipeline response times.
-- **Memory Optimization**: Reduce graph/vector processing memory footprint.
-- **Throughput**: Improve operations per second (bulk ingestion, parallel queries).
-- **Benchmarks**: Add performance benchmarks to track regressions.
-- **Async/Concurrency**: Enhance asynchronous execution and concurrency.
+### Subsection
 
-### 📚 Documentation Contributions
+- Bullet point 1
+- Bullet point 2
 
-- Fix typos and grammar
-- Improve clarity
-- Add examples
-- Create tutorials
-- Translate documentation
+**Code example:**
 
-### Testing Contributions
+```python
+from semantica import SomeClass
 
-- Add test coverage
-- Improve test quality
-- Add integration tests
-- Performance benchmarks
+instance = SomeClass()
+result = instance.method()
+```
 
-### Other Contributions
+**Note:** Additional context or warnings.
+```
 
-- Answer questions in discussions
-- Help with issues
-- Review pull requests
-- Share use cases
-- Report bugs
-- Suggest features
+**Best Practices:**
+- Start with an overview/introduction
+- Use consistent terminology
+- Include "See also" links
+- Add examples for complex concepts
+- Keep formatting consistent across docs
 
-## Getting Help
+---
 
-### Communication Channels
+## 🆘 Getting Help
 
-- **GitHub Discussions**: General questions and discussions
-- **GitHub Issues**: Bug reports and feature requests
-- **Discord**: Real-time chat and community support
+- 💬 [Discord](https://discord.gg/vqRt2qbx) - Real-time chat
+- 💭 [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) - Q&A
+- 🐛 [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) - Bug reports
 
-### Before Asking for Help
+**Before asking:** Check existing documentation, search issues/discussions, review cookbook examples
 
-1. Check existing documentation
-2. Search GitHub issues and discussions
-3. Review code examples in cookbook
-4. Check FAQ in documentation
+---
 
-### Asking Good Questions
+## 🏆 Recognition
 
-- Provide context and environment details
-- Include code examples
-- Show what you've tried
-- Include error messages and logs
-- Be specific about what you need
-
-## Recognition
-
-Contributors are recognized in:
+All contributors are recognized in:
 - [CONTRIBUTORS.md](CONTRIBUTORS.md)
 - GitHub contributors page
-- Release notes for significant contributions
+- Release notes
 
-Thank you for contributing to Semantica! 🎉
+We follow the [all-contributors](https://allcontributors.org) specification!
 
+---
+
+## 📜 Code of Conduct
+
+This project follows a [Code of Conduct](CODE_OF_CONDUCT.md). Be respectful and inclusive.
+
+---
+
+## 📚 Resources
+
+- [README.md](README.md) - Project overview
+- [Cookbook](cookbook/) - Tutorials and examples
+- [Documentation](docs/) - Comprehensive guides
+
+---
+
+**Thank you for contributing!** 🚀
+
+Every contribution matters - whether it's a single line of code, a typo fix, a helpful answer, or a bug report. We appreciate you! 🙏
+
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**

CONTRIBUTORS.md

@@ -4,44 +4,31 @@ Thank you to all the people who have contributed to Semantica! 🎉
 
 This project follows the [all-contributors](https://allcontributors.org) specification. Contributions of any kind are welcome!
 
-## How to Contribute
+⭐ **Give us a Star** • 🍴 **Fork us** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-We welcome contributions of all kinds! Whether you're:
-- Writing code
-- Improving documentation
-- Reporting bugs
-- Suggesting features
-- Answering questions
-- Reviewing pull requests
-- Sharing use cases
-- Creating examples
-
-All contributions are valuable and appreciated!
+---
 
 ## Contribution Types
 
+We recognize all types of contributions:
+
 - 💻 **Code**: Writing code, fixing bugs, implementing features
 - 📝 **Documentation**: Writing docs, tutorials, examples
+- 🧪 **Testing**: Writing tests, improving test coverage
 - 🐛 **Bug Reports**: Finding and reporting bugs
 - 💡 **Ideas**: Suggesting new features or improvements
 - 🎨 **Design**: UI/UX improvements, graphics, branding
-- 📖 **Examples**: Creating code examples and tutorials
-- 🔍 **Testing**: Writing tests, improving test coverage
-- 💬 **Answering Questions**: Helping others in discussions
-- 📢 **Talks**: Giving talks, presentations, workshops
-- 🌍 **Translation**: Translating documentation
+- 🎨 **Cookbook**: Creating tutorials and examples
+- 💬 **Community**: Answering questions, reviewing PRs
+- 🎓 **Education**: Blog posts, video tutorials, talks, workshops
 - 🔧 **Tools**: Creating tools, scripts, integrations
 - 📦 **Packaging**: Improving build, release, distribution
 - ⚠️ **Security**: Reporting security vulnerabilities
-- 🎓 **Education**: Teaching, mentoring, tutorials
-- 📹 **Video**: Creating video content, tutorials
-- 🎵 **Audio**: Podcasts, audio content
-- 📸 **Photography**: Screenshots, images
-- 🔬 **Research**: Research, analysis, studies
-- 💰 **Financial**: Sponsoring, funding
 - 🏗️ **Infrastructure**: CI/CD, hosting, infrastructure
 - 🚇 **Maintenance**: Maintenance, triage, project management
 
+---
+
 ## Contributors
 
 <!-- ALL-CONTRIBUTORS-LIST:START -->
@@ -50,48 +37,78 @@ All contributions are valuable and appreciated!
 
 <!-- ALL-CONTRIBUTORS-LIST:END -->
 
+---
+
 ## Recognition
 
-### Top Contributors
+All contributors are recognized in:
 
-Contributors are recognized based on their contributions to the project. Recognition includes:
+- This contributors list
+- [GitHub contributors page](https://github.com/Hawksight-AI/semantica/graphs/contributors)
+- Release notes for significant contributions
+- Community appreciation
 
-- Listing in this file
-- GitHub contributor statistics
-- Special mentions in release notes
-- Featured showcases for significant contributions
-
-### Hall of Fame
-
-Special recognition for exceptional contributions:
-
-- **Coming soon** - We'll feature outstanding contributors here!
+---
 
 ## How to Add Yourself
 
-If you've contributed to Semantica and want to be added to this list:
+### Automatic Recognition
 
-1. **Automatic**: If you've made a commit, you'll appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors)
-2. **Manual**: Open a PR adding yourself to this file, or use the [@all-contributors bot](https://allcontributors.org/docs/en/bot/usage)
+If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors).
 
-Example:
-```markdown
-- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
-```
+### Using All-Contributors Bot
 
-## All Contributors Bot
-
-We use the [all-contributors](https://allcontributors.org) bot to automatically recognize contributors. To add a contributor, comment on an issue or PR:
+Comment on any issue or PR with:
 
 ```
 @all-contributors please add @username for code, docs, bug
 ```
 
-## Thank You!
+**Examples:**
 
-Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community!
+```
+@all-contributors please add @johndoe for code
+@all-contributors please add @janedoe for docs, bug
+@all-contributors please add @devuser for code, test, maintenance
+```
+
+### Manual Addition
+
+Open a PR adding yourself to this file:
+
+```markdown
+- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
+```
 
 ---
 
-**Want to contribute?** Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
+## Contribution Type Codes
 
+When using the all-contributors bot, use these codes:
+
+- `code` - Code contributions
+- `doc` - Documentation
+- `test` - Testing
+- `bug` - Bug reports
+- `ideas` - Feature requests/ideas
+- `design` - Design work
+- `example` - Cookbook/examples
+- `question` - Answering questions
+- `talk` - Talks/presentations
+- `tool` - Tools/integrations
+- `packaging` - Packaging/distribution
+- `security` - Security reports
+- `infra` - Infrastructure
+- `maintenance` - Maintenance
+
+See [all-contributors specification](https://allcontributors.org/docs/en/emoji-key) for complete list.
+
+---
+
+## Thank You!
+
+Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community! 🙏
+
+**Want to contribute?** 
+
+⭐ Give us a Star • 🍴 [Fork us](https://github.com/Hawksight-AI/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!

README.md

@@ -1,204 +1,236 @@
 <div align="center">
 
-<img src="semantica_logo.png" alt="Semantica Logo" width="450" height="auto">
+<img src="semantica_logo.png" alt="Semantica Logo" width="460"/>
 
 # 🧠 Semantica
+### Open-Source Semantic Layer & Knowledge Engineering Framework
 
-[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyPI version](https://img.shields.io/pypi/v/semantica.svg)](https://pypi.org/project/semantica/)
+[![PyPI](https://img.shields.io/pypi/v/semantica.svg)](https://pypi.org/project/semantica/)
 [![Monthly Downloads](https://img.shields.io/pypi/dm/semantica)](https://pypi.org/project/semantica/)
 [![Total Downloads](https://static.pepy.tech/badge/semantica)](https://pepy.tech/project/semantica)
-[![Discord](https://img.shields.io/badge/Discord-Join%20Us-7289da?style=flat&logo=discord&logoColor=white)](https://discord.gg/pMHguUzG)
 [![CI](https://github.com/Hawksight-AI/semantica/workflows/CI/badge.svg)](https://github.com/Hawksight-AI/semantica/actions)
+[![Discord](https://img.shields.io/badge/Discord-Join-7289da?logo=discord&logoColor=white)](https://discord.gg/RgaGTj9J)
 
-<p align="center">
-    <a href="https://github.com/Hawksight-AI/semantica/stargazers">
-        <img src="https://img.shields.io/badge/Give%20a%20Star-%E2%AD%90-yellow?style=for-the-badge&labelColor=555555" alt="Give a Star">
-    </a>
-    &nbsp;&nbsp;
-    <a href="https://github.com/Hawksight-AI/semantica/fork">
-        <img src="https://img.shields.io/badge/Support%20Project-Fork%20Us-blue?style=for-the-badge&labelColor=555555" alt="Support Project">
-    </a>
-</p>
+### ⭐ Give us a Star • 🍴 Fork us • 💬 Join our Discord
 
-**Open Source Framework for Semantic Layer & Knowledge Engineering**
-
-> **Transform chaotic data into intelligent knowledge.**
-
-*The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.*
-
-**100% Open Source** • **MIT Licensed** • **Latest Version: 0.2.2** • **Production Ready** • **Community Driven**
-
-[**Discord**](https://discord.gg/pMHguUzG)
+> **Transform Choas into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.**
 
 </div>
 
-## What is Semantica?
-
-Semantica bridges the gap between raw data chaos and AI-ready knowledge. It's a **semantic intelligence platform** that transforms unstructured data into structured, queryable knowledge graphs powering GraphRAG, AI agents, and multi-agent systems.
-
-### What Makes Semantica Different?
-
-Unlike traditional approaches that process isolated documents and extract text into vectors, Semantica understands **semantic relationships across all content**, provides **automated ontology generation**, and builds a **unified semantic layer** with **production-grade QA**.
-
-| **Traditional Approaches** | **Semantica's Approach** |
-|:---------------------------|:-------------------------|
-| Process data as isolated documents | Understands semantic relationships across all content |
-| Extract text and store vectors | Builds knowledge graphs with meaningful connections |
-| Generic entity recognition | General-purpose ontology generation and validation |
-| Manual schema definition | Automatic semantic modeling from content patterns |
-| Disconnected data silos | Unified semantic layer across all data sources |
-| Basic quality checks | Production-grade QA with conflict detection & resolution |
 
 ---
 
-## 🎯 The Problem We Solve
+## 🚀 Why Semantica?
 
-### The Semantic Gap
+**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and compliant.
 
-Organizations today face a **fundamental mismatch** between how data exists and how AI systems need it.
-
-#### The Semantic Gap: Problem vs. Solution
-
-Organizations have **unstructured data** (PDFs, emails, logs), **messy data** (inconsistent formats, duplicates, conflicts), and **disconnected silos** (no shared context, missing relationships). AI systems need **clear rules** (formal ontologies), **structured entities** (validated, consistent), and **relationships** (semantic connections, context-aware reasoning).
-
-| **What Organizations Have** | **What AI Systems Require** |
-|:------------------------------|:------------------------------|
-| **Unstructured Data** | **Clear Rules** |
-| PDFs, emails, logs | Formal ontologies |
-| Mixed schemas | Graphs & Networks |
-| Conflicting facts | |
-| **Messy, Noisy Data** | **Structured Entities** |
-| Inconsistent formats | Validated entities |
-| Duplicate records | Domain Knowledge |
-| Missing relationships | |
-| **Disconnected, Siloed Data** | **Relationships** |
-| Data in separate systems | Semantic connections |
-| No shared context | Context-Aware Reasoning |
-| Isolated knowledge | |
-
-### **SEMANTICA FRAMEWORK**
-
-Semantica operates through three integrated layers that transform raw data into AI-ready knowledge:
-
-**Input Layer** — Universal ingestion from multiple data formats (PDFs, DOCX, HTML, JSON, CSV, databases, live feeds, APIs, streams, archives, multi-modal content) into a unified pipeline.
-
-**Semantic Layer** — Core intelligence engine performing entity extraction, relationship mapping, ontology generation, context engineering, and quality assurance. Includes **advanced entity deduplication** (Jaro-Winkler, disjoint property handling) to ensure a clean single source of truth.
-
-**Output Layer** — Production-ready knowledge graphs, vector embeddings, and validated ontologies that power GraphRAG systems, AI agents, and multi-agent systems.
-
-**Powers: GraphRAG, AI Agents, Multi-Agent Systems**
-
-
-
-
-
-### What Happens Without Semantics?
-
-**They Break** — Systems crash due to inconsistent formats and missing structure.
-
-**They Hallucinate** — AI models generate false information without semantic context to validate outputs.
-
-**They Fail Silently** — Systems return wrong answers without warnings, leading to bad decisions.
-
-**Why?** Systems have data — not semantics. They can't connect concepts, understand relationships, validate against domain rules, or detect conflicts.
+Perfect for **high-stakes domains** where mistakes have real consequences.
 
 ---
 
-## 💡 The Semantica Solution
+### ⚡ Get Started in 30 Seconds
 
-**Semantica** is an **open-source framework** that closes the semantic gap between real-world messy data and the structured semantic layers required by advanced AI systems — GraphRAG, agents, multi-agent systems, reasoning models, and more.
+```bash
+pip install semantica
+```
 
-### How Semantica Solves These Problems
+```python
+from semantica.semantic_extract import NERExtractor
+from semantica.kg import GraphBuilder
 
-**Efficient Embeddings** — Uses **FastEmbed** by default for high-performance, lightweight local embedding generation (faster than sentence-transformers).
+# Extract entities and build knowledge graph
+ner = NERExtractor(method="ml", model="en_core_web_sm")
+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
+kg = GraphBuilder().build({"entities": entities, "relationships": []})
 
-**Universal Data Ingestion** — Handles multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams) with unified pipeline, no custom parsers needed.
+print(f"Built KG with {len(kg.get('entities', []))} entities")
+```
 
-**Automated Semantic Extraction** — NER, relationship extraction, and triplet generation with LLM enhancement. Includes **auto-chunking** for long documents and **robust error handling** with automatic retry logic.
-
-**Knowledge Graph Construction** — Production-ready graphs with entity resolution, temporal support, and graph analytics. Queryable knowledge ready for AI applications.
-
-**GraphRAG Engine** — Hybrid vector + graph retrieval achieves 91% accuracy (30% improvement) via semantic search + graph traversal for multi-hop reasoning. Features LLM-generated responses grounded in knowledge graph context with reasoning traces. [See Comparison Benchmark](cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
-
-**AI Agent Context Engineering** — Persistent memory with RAG + knowledge graphs enables context maintenance, action validation, and structured knowledge access.
-
-**Automated Ontology Generation** — 6-stage LLM pipeline generates validated OWL ontologies with HermiT/Pellet validation, eliminating manual engineering.
-
-**Production-Grade QA** — Conflict detection, deduplication, quality scoring, and provenance tracking ensure trusted, production-ready knowledge graphs.
-
-**Pipeline Orchestration** — Flexible pipeline builder with parallel execution enables scalable processing via orchestrator-worker pattern.
-
-### Core Features at a Glance
-
-| **Feature Category** | **Capabilities** | **Key Benefits** |
-|:---------------------|:-----------------|:------------------|
-| **Data Ingestion** | Multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams, archives) | Universal ingestion, no custom parsers needed |
-| **Semantic Extraction** | NER, relations, triplets, LLM enhancement, **auto-chunking** | Automated discovery with robust error handling |
-| **Knowledge Graphs** | Entity resolution, temporal support, graph analytics, query interface | Production-ready, queryable knowledge structures |
-| **Ontology Generation** | 6-stage LLM pipeline, OWL generation, HermiT/Pellet validation | Automated ontology creation from documents |
-| **GraphRAG** | Hybrid vector + graph retrieval, multi-hop reasoning, LLM-generated responses | 91% accuracy, 30% improvement over vector-only, reasoning traces |
-| **LLM Providers** | Unified interface to 100+ LLMs (Groq, OpenAI, HuggingFace, LiteLLM) | Clean imports, multiple providers, structured output |
-| **Agent Memory** | Persistent memory (Save/Load), Hybrid Retrieval (Vector+Graph), FastEmbed support | Context-aware agents with semantic understanding |
-| **Pipeline Orchestration** | Parallel execution, custom steps, orchestrator-worker pattern | Scalable, flexible data processing |
-| **Quality Assurance** | Conflict detection, deduplication, quality scoring, provenance | Trusted knowledge graphs ready for production |
+**[📖 Full Quick Start](#-quick-start)** • **[🍳 Cookbook Examples](#-semantica-cookbook)** • **[💬 Join Discord](https://discord.gg/RgaGTj9J)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)**
 
 ---
 
-## 👥 Who Is This For?
+## Core Value Proposition
 
-Semantica is designed for **developers, data engineers, and organizations** building the next generation of AI applications that require semantic understanding and knowledge graphs.
+| **Trustworthy** | **Explainable** | **Auditable** |
+|:------------------:|:------------------:|:-----------------:|
+| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking |
+| Rule-based governance | Entity relationships & ontologies | Source-level provenance |
+| Production-grade QA | Multi-hop graph reasoning | Audit-ready compliance |
 
-### Who Uses Semantica
+---
 
-**AI/ML Engineers & Data Scientists** — Build GraphRAG systems, AI agents, and multi-agent systems.
+## Key Features & Benefits
 
-**Data Engineers** — Build scalable pipelines with semantic enrichment.
+### Not Just Another Agentic Framework
 
-**Knowledge Engineers & Ontologists** — Create knowledge graphs and ontologies with automated pipelines.
+**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with:
 
-**Enterprise Data Teams** — Unify semantic layers, improve data quality, resolve conflicts.
+| Feature | Benefit |
+|:--------|:--------|
+| **Auditable** | Complete provenance tracking with full audit trails |
+| **Explainable** | Transparent reasoning paths with entity relationships |
+| **Provenance-Aware** | Source-level provenance from documents to responses |
+| **Validated** | Built-in conflict detection, deduplication, QA |
+| **Governed** | Rule-based validation and semantic consistency |
 
-**Software & DevOps Engineers** — Build semantic APIs and infrastructure with production-ready SDK.
+### Perfect For High-Stakes Use Cases
 
-**Analysts & Researchers** — Transform data into queryable knowledge graphs for insights.
+| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** |
+|:-----------------:|:--------------:|:------------:|
+| Clinical decisions | Fraud detection | Evidence-backed research |
+| Drug interactions | Regulatory compliance | Contract analysis |
+| Patient safety | Risk assessment | Case law reasoning |
 
-**Security & Compliance Teams** — Threat intelligence, regulatory reporting, audit trails.
+| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** |
+|:-------------------:|:----------------:|:-------------------:|:-----------------:|
+| Threat attribution | Policy decisions | Power grids | Decision logs |
+| Incident response | Classified info | Transportation | Safety validation |
 
-**Product Teams & Startups** — Rapid prototyping of AI products and semantic features.
+### Powers Your AI Stack
+
+- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search
+- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory
+- **Reasoning Models** — Explainable AI decisions with reasoning paths
+- **Enterprise AI** — Governed, auditable platforms for compliance
+
+### Integrations
+
+- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)
+- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication
+- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)
+
+> **Built for environments where every answer must be explainable and governed.**
+
+
+---
+
+## 🚨 The Problem: The Semantic Gap
+
+### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.
+
+### Understanding the Semantic Gap
+
+The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).
+
+**Traditional AI approaches:**
+- Rely on statistical patterns and text similarity
+- Cannot understand relationships between entities
+- Cannot reason about domain-specific rules
+- Cannot explain why decisions were made
+- Cannot trace back to original sources with confidence
+
+**High-stakes AI requires:**
+- Semantic understanding of entities and their relationships
+- Domain knowledge encoded as formal rules (ontologies)
+- Explainable reasoning paths
+- Source-level provenance
+- Conflict detection and resolution
+
+**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.
+
+### What Organizations Have vs What They Need
+
+| **Current State** | **Required for High-Stakes AI** |
+|:---------------------|:-----------------------------------|
+| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |
+| APIs, databases, streams | Structured and validated entities |
+| Conflicting facts and duplicates | Explicit semantic relationships |
+| Siloed systems with no lineage | **Explainable reasoning paths** |
+| | **Source-level provenance** |
+| | **Audit-ready compliance** |
+
+### The Cost of Missing Semantics
+
+- **Decisions cannot be explained** — No transparency in AI reasoning
+- **Errors cannot be traced** — No way to debug or improve
+- **Conflicts go undetected** — Contradictory information causes failures
+- **Compliance becomes impossible** — No audit trails for regulations
+
+**Trustworthy AI requires semantic accountability.**
+
+---
+
+## 🆚 Semantica vs Traditional RAG
+
+| Feature | Traditional RAG | Semantica |
+|:--------|:----------------|:----------|
+| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths |
+| **Provenance** | ❌ No provenance | ✅ Source-level provenance |
+| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning |
+| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection |
+| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments |
+| **Compliance** | ❌ No audit trails | ✅ Audit-ready provenance |
+
+---
+
+## 🧩 Semantica Architecture
+
+### 1️⃣ Input Layer — Governed Ingestion
+- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX
+- 🔧 **Docling Support** — Docling parser for table extraction
+- 💾 **Data Sources** — Databases, APIs, streams, archives, web content
+- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction
+- 📊 **Single Pipeline** — Unified ingestion with metadata and source tracking
+
+### 2️⃣ Semantic Layer — Trust & Reasoning Engine
+- 🔍 **Entity Extraction** — NER, normalization, classification
+- 🔗 **Relationship Discovery** — Triplet generation, semantic links
+- 📐 **Ontology Induction** — Automated domain rule generation
+- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution
+- ✅ **Quality Assurance** — Conflict detection, validation
+- 📊 **Provenance Tracking** — Source, time, confidence metadata
+- 🧠 **Reasoning Traces** — Explainable inference paths
+
+### 3️⃣ Output Layer — Auditable Knowledge Assets
+- 📊 **Knowledge Graphs** — Queryable, temporal, explainable
+- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support
+- 🔢 **Vector Embeddings** — FastEmbed by default
+- ☁️ **AWS Neptune** — Amazon Neptune graph database support
+- 🔍 **Provenance** — Every AI response links back to:
+  - 📄 Source documents
+  - 🏷️ Extracted entities & relations
+  - 📐 Ontology rules applied
+  - 🧠 Reasoning steps used
+
+---
+
+## 🏥 Built for High-Stakes Domains
+
+Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:
+
+- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety compliance
+- **💰 Finance & Risk** — Fraud detection, regulatory compliance (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation
+- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change management, case law reasoning
+- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis
+- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence
+- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response
+- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation  
+
+---
+
+## 👥 Who Uses Semantica?
+
+- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents
+- **⚙️ Data Engineers** — Creating governed semantic pipelines
+- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale
+- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure
+- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems  
 
 ---
 
 ## 📦 Installation
 
-> **✅ Available on PyPI!** Semantica is now published on PyPI. Install it with a single command: `pip install semantica`
-
-**Prerequisites:** Python 3.8+ (3.9+ recommended) • pip (latest version)
-
 ### Install from PyPI (Recommended)
 
 ```bash
-# Install latest version from PyPI
 pip install semantica
-
-# Or install with optional dependencies
+# or
 pip install semantica[all]
-
-# GitHub Workaround (if PyPI version has issues)
-pip install git+https://github.com/Hawksight-AI/semantica.git@main
-
-# Verify installation
-python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')"
 ```
 
-**Current Version:** [![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/) • [View on PyPI](https://pypi.org/project/semantica/)
-
-!!! info "Windows PyTorch Note"
-    If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica.
-
-
-
 ### Install from Source (Development)
 
 ```bash
@@ -258,7 +290,7 @@ print(f" Ingested {len(sources)} sources")
 
 ### Document Parsing & Processing
 
-> **Multi-format parsing** • **Text normalization** • **Intelligent chunking**
+> **Multi-format parsing** • **Docling Support** • **Text normalization** • **Intelligent chunking**
 
 ```python
 from semantica.parse import DocumentParser, DoclingParser
@@ -269,7 +301,7 @@ from semantica.split import TextSplitter
 parser = DocumentParser()
 parsed = parser.parse("document.pdf", format="auto")
 
-# Enhanced parsing with Docling (recommended for complex layouts/tables)
+# Parsing with Docling (for complex layouts/tables)
 # Requires: pip install docling
 docling_parser = DoclingParser(enable_ocr=True)
 result = docling_parser.parse("complex_table.pdf")
@@ -360,7 +392,7 @@ results = vector_store.search(query="supply chain", top_k=5)
 
 ### Graph Store & Triplet Store
 
-> **Neo4j, FalkorDB, Amazon Neptune support** • **SPARQL queries** • **RDF triplets**
+> **Neo4j, FalkorDB, Amazon Neptune** • **SPARQL queries** • **RDF triplets**
 
 ```python
 from semantica.graph_store import GraphStore
@@ -398,15 +430,19 @@ results = triplet_store.execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT
 
 ### Ontology Generation & Management
 
-> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation
+> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation • **Custom Ontology Import** (OWL, RDF, Turtle, JSON-LD)
 
 ```python
 from semantica.ontology import OntologyGenerator
+from semantica.ingest import ingest_ontology
 
+# Generate ontology automatically
 generator = OntologyGenerator(llm_provider="openai", model="gpt-4")
 ontology = generator.generate_from_documents(sources=["domain_docs/"])
 
-print(f"Classes: {len(ontology.classes)}")
+# Or import your existing ontology
+custom_ontology = ingest_ontology("my_ontology.ttl")  # Supports OWL, RDF, Turtle, JSON-LD
+print(f"Classes: {len(custom_ontology.classes)}")
 ```
 
 [**Cookbook: Ontology**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/14_Ontology.ipynb)
@@ -476,7 +512,7 @@ reasoned_result = context.query_with_reasoning(
 
 ### Knowledge Graph-Powered RAG (GraphRAG)
 
-> **30% Accuracy Improvement** • Vector + Graph Hybrid Search • 91% Accuracy • **Multi-Hop Reasoning** • **LLM-Generated Responses**
+> **Vector + Graph Hybrid Search** • **Multi-Hop Reasoning** • **LLM-Generated Responses** • **Semantic Re-ranking**
 
 ```python
 from semantica.context import AgentContext
@@ -525,7 +561,7 @@ print(f"Confidence: {result['confidence']:.3f}")
 from semantica.llms import Groq, OpenAI, HuggingFaceLLM, LiteLLM
 import os
 
-# Groq - Fast inference
+# Groq
 groq = Groq(
     model="llama-3.1-8b-instant",
     api_key=os.getenv("GROQ_API_KEY")
@@ -555,7 +591,7 @@ structured = groq.generate_structured("Extract entities from: Apple Inc. was fou
 ```
 
 **Supported Providers:**
-- **Groq**: Fast inference with Llama models
+- **Groq**: Inference with Llama models
 - **OpenAI**: GPT-3.5, GPT-4, and other OpenAI models
 - **HuggingFace**: Local LLM inference with Transformers
 - **LiteLLM**: Unified interface to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, and more)
@@ -755,7 +791,7 @@ print(f"Found {len(results)} results")
 
 #### Cybersecurity
 - [**Real-Time Anomaly Detection**](cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) - CVE RSS, Kafka streams, temporal KGs, sentence chunking
-- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, enhanced GraphRAG, deduplication
+- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, GraphRAG, deduplication
 
 #### Intelligence & Law Enforcement
 - [**Criminal Network Analysis**](cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) - OSINT RSS, deduplication, network centrality, graph analytics
@@ -772,12 +808,16 @@ print(f"Found {len(results)} results")
 
 ## 🔬 Advanced Features
 
+**Docling Integration** — Document parsing with table extraction for PDFs, DOCX, PPTX, and XLSX files. Supports OCR and multiple export formats.
+
+**AWS Neptune Support** — Amazon Neptune graph database integration with IAM authentication and OpenCypher queries.
+
+**Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD, N3) and extend Schema.org, FOAF, Dublin Core, or custom ontologies.
+
 **Incremental Updates** — Real-time stream processing with Kafka, RabbitMQ, Kinesis for live updates.
 
 **Multi-Language Support** — Process multiple languages with automatic detection.
 
-**Custom Ontology Import** — Import and extend Schema.org and custom ontologies.
-
 **Advanced Reasoning** — Forward/backward chaining, Rete-based pattern matching, and automated explanation generation.
 
 **Graph Analytics** — Centrality, community detection, path finding, temporal analysis.
@@ -867,20 +907,11 @@ git push origin feature/your-feature
 4. **Feature Requests** - [Request feature](https://github.com/Hawksight-AI/semantica/issues/new)
 
 
-###  Contributors
-
-<a href="https://github.com/Hawksight-AI/semantica/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=Hawksight-AI/semantica" alt="Contributors" />
-</a>
-
 ## 📜 License
 
 Semantica is licensed under the **MIT License** - see the [LICENSE](https://github.com/Hawksight-AI/semantica/blob/main/LICENSE) file for details.
 
-<div align="center">
-
 **Built by the Semantica Community**
 
-[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/pMHguUzG)
 
-</div>
+[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/RgaGTj9J)

RELEASE.md

@@ -26,10 +26,10 @@ Before releasing, ensure:
 
 The project uses GitHub Actions for automated releases to PyPI.
 
-1. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.2`).
+1.29. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.3`).
    ```bash
-   git tag -a v0.2.2 -m "Release v0.2.2"
-   git push origin v0.2.2
+   git tag -a v0.2.3 -m "Release v0.2.3"
+   git push origin v0.2.3
    ```
 2. **GitHub Action**: The `Release` workflow will automatically trigger, build the package, create a GitHub Release, and publish to PyPI using Trusted Publishing.
 

SECURITY.md

@@ -6,6 +6,7 @@ We actively support the following versions of Semantica with security updates:
 
 | Version | Supported          |
 | ------- | ------------------ |
+| 0.2.3   | :white_check_mark: |
 | 0.2.2   | :white_check_mark: |
 | 0.2.1   | :white_check_mark: |
 | 0.2.0   | :white_check_mark: |

cookbook/introduction/21_Amazon_Neptune_Store.ipynb

@@ -25,6 +25,57 @@
     "- AWS credentials configured (boto3, environment variables, or IAM role)\n",
     "- Network access to your Neptune cluster (VPC, security groups)\n",
     "\n",
+    "#### Quick Setup with CloudFormation\n",
+    "\n",
+    "If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n",
+    "\n",
+    "```bash\n",
+    "# Deploy the Neptune stack (takes ~15-20 minutes)\n",
+    "aws cloudformation create-stack \\\n",
+    "  --stack-name semantica-neptune \\\n",
+    "  --template-body file://neptune-setup.yaml \\\n",
+    "  --capabilities CAPABILITY_NAMED_IAM\n",
+    "\n",
+    "# Wait for stack creation to complete\n",
+    "aws cloudformation wait stack-create-complete --stack-name semantica-neptune\n",
+    "\n",
+    "# Get the outputs (endpoint, port, credentials)\n",
+    "aws cloudformation describe-stacks --stack-name semantica-neptune \\\n",
+    "  --query 'Stacks[0].Outputs' --output table\n",
+    "```\n",
+    "\n",
+    "The template creates:\n",
+    "- VPC with public subnets and Internet Gateway\n",
+    "- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
+    "- IAM user with least-privilege access for OpenCypher queries\n",
+    "- Security group allowing Bolt protocol (port 8182) access\n",
+    "\n",
+    "> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
+    "\n",
+    "**Outputs:**\n",
+    "- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
+    "- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
+    "- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
+    "- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
+    "- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
+    "\n",
+    "**Cleanup:**\n",
+    "```bash\n",
+    "aws cloudformation delete-stack --stack-name semantica-neptune\n",
+    "```\n",
+    "\n",
+    "**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n",
+    "\n",
+    "| Resource | Cost (USD) |\n",
+    "| --- | --- |\n",
+    "| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n",
+    "| Storage (10 GB) | ~1/month |\n",
+    "| I/O requests | ~1-5/month |\n",
+    "| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
+    "| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
+    "\n",
+    "> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
+    "\n",
     "---"
    ]
   },
@@ -70,14 +121,21 @@
     "import os\n",
     "\n",
     "# Neptune cluster configuration - REPLACE WITH YOUR VALUES\n",
+    "# (Get these from CloudFormation stack outputs)\n",
     "os.environ[\"NEPTUNE_ENDPOINT\"] = \"your-cluster.us-east-1.neptune.amazonaws.com\"\n",
     "os.environ[\"NEPTUNE_PORT\"] = \"8182\"\n",
     "os.environ[\"AWS_REGION\"] = \"us-east-1\"\n",
     "\n",
-    "# AWS credentials (if using IAM Auth and not relying on IAM role or ~/.aws/credentials)\n",
-    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n",
-    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n",
-    "# os.environ[\"AWS_SESSION_TOKEN\"] = \"your-session-token\"\n",
+    "# AWS credentials for IAM Authentication\n",
+    "# Option 1: IAM User (static credentials from CloudFormation template)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"AKIA...\"  # From AwsAccessKeyId output\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # From AwsSecretAccessKey output\n",
+    "# Note: No AWS_SESSION_TOKEN needed for IAM users\n",
+    "\n",
+    "# Option 2: IAM Role / Temporary credentials (e.g., STS AssumeRole, EC2 instance role)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"ASIA...\"  # Temporary access key\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # Temporary secret key\n",
+    "# os.environ[\"AWS_SESSION_TOKEN\"] = \"...\"      # REQUIRED for temporary credentials\n",
     "\n",
     "print(f\"Neptune Endpoint: {os.environ.get('NEPTUNE_ENDPOINT')}\")\n",
     "print(f\"AWS Region: {os.environ.get('AWS_REGION')}\")"

cookbook/introduction/neptune-setup.yaml

@@ -0,0 +1,228 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: >
+  Amazon Neptune cluster with public endpoint, IAM authentication, and least-privilege
+  IAM user for Semantica cookbook. Uses db.t3.medium (most cost-effective Neptune instance type).
+
+Parameters:
+  EnvironmentName:
+    Type: String
+    Default: semantica-neptune
+    Description: Environment name prefix for resource naming
+
+Resources:
+  # =============================================================================
+  # VPC & NETWORKING
+  # =============================================================================
+
+  VPC:
+    Type: AWS::EC2::VPC
+    Properties:
+      CidrBlock: 10.0.0.0/16
+      EnableDnsHostnames: true
+      EnableDnsSupport: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-vpc
+
+  InternetGateway:
+    Type: AWS::EC2::InternetGateway
+    Properties:
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-igw
+
+  InternetGatewayAttachment:
+    Type: AWS::EC2::VPCGatewayAttachment
+    Properties:
+      InternetGatewayId: !Ref InternetGateway
+      VpcId: !Ref VPC
+
+  PublicSubnet1:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [0, !GetAZs '']
+      CidrBlock: 10.0.1.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-1
+
+  PublicSubnet2:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [1, !GetAZs '']
+      CidrBlock: 10.0.2.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-2
+
+  PublicRouteTable:
+    Type: AWS::EC2::RouteTable
+    Properties:
+      VpcId: !Ref VPC
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-rt
+
+  DefaultPublicRoute:
+    Type: AWS::EC2::Route
+    DependsOn: InternetGatewayAttachment
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      DestinationCidrBlock: 0.0.0.0/0
+      GatewayId: !Ref InternetGateway
+
+  PublicSubnet1RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet1
+
+  PublicSubnet2RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet2
+
+  # =============================================================================
+  # SECURITY GROUP
+  # =============================================================================
+
+  NeptuneSecurityGroup:
+    Type: AWS::EC2::SecurityGroup
+    Properties:
+      GroupName: !Sub ${EnvironmentName}-neptune-sg
+      GroupDescription: Security group for Neptune cluster - allows Bolt protocol access
+      VpcId: !Ref VPC
+      SecurityGroupIngress:
+        - IpProtocol: tcp
+          FromPort: 8182
+          ToPort: 8182
+          CidrIp: 0.0.0.0/0
+          Description: Allow Bolt protocol access from anywhere
+      SecurityGroupEgress:
+        - IpProtocol: -1
+          CidrIp: 0.0.0.0/0
+          Description: Allow all outbound traffic
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-neptune-sg
+
+  # =============================================================================
+  # NEPTUNE CLUSTER
+  # =============================================================================
+
+  NeptuneSubnetGroup:
+    Type: AWS::Neptune::DBSubnetGroup
+    Properties:
+      DBSubnetGroupDescription: Subnet group for Neptune cluster
+      DBSubnetGroupName: !Sub ${EnvironmentName}-subnet-group
+      SubnetIds:
+        - !Ref PublicSubnet1
+        - !Ref PublicSubnet2
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-subnet-group
+
+  NeptuneCluster:
+    Type: AWS::Neptune::DBCluster
+    Properties:
+      DBClusterIdentifier: !Sub ${EnvironmentName}-cluster
+      DBSubnetGroupName: !Ref NeptuneSubnetGroup
+      VpcSecurityGroupIds:
+        - !Ref NeptuneSecurityGroup
+      EngineVersion: '1.4.6.3'
+      IamAuthEnabled: true
+      StorageEncrypted: true
+      DeletionProtection: false
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-cluster
+
+  NeptuneInstance:
+    Type: AWS::Neptune::DBInstance
+    Properties:
+      DBInstanceIdentifier: !Sub ${EnvironmentName}-instance
+      DBInstanceClass: db.t3.medium
+      DBClusterIdentifier: !Ref NeptuneCluster
+      PubliclyAccessible: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-instance
+
+  # =============================================================================
+  # IAM USER WITH LEAST PRIVILEGES
+  # =============================================================================
+
+  NeptuneUser:
+    Type: AWS::IAM::User
+    Properties:
+      UserName: !Sub ${EnvironmentName}-user
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-user
+
+  NeptuneUserPolicy:
+    Type: AWS::IAM::Policy
+    Properties:
+      PolicyName: !Sub ${EnvironmentName}-neptune-access
+      Users:
+        - !Ref NeptuneUser
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Sid: NeptuneDataAccess
+            Effect: Allow
+            Action:
+              - neptune-db:connect
+              - neptune-db:ReadDataViaQuery
+              - neptune-db:WriteDataViaQuery
+              - neptune-db:DeleteDataViaQuery
+            Resource: !Sub
+              - arn:aws:neptune-db:${AWS::Region}:${AWS::AccountId}:${ClusterResourceId}/*
+              - ClusterResourceId: !GetAtt NeptuneCluster.ClusterResourceId
+
+  NeptuneUserAccessKey:
+    Type: AWS::IAM::AccessKey
+    Properties:
+      UserName: !Ref NeptuneUser
+
+# =============================================================================
+# OUTPUTS
+# =============================================================================
+
+Outputs:
+  NeptuneEndpoint:
+    Description: Neptune cluster endpoint (hostname only) - use as NEPTUNE_ENDPOINT
+    Value: !GetAtt NeptuneCluster.Endpoint
+
+  NeptunePort:
+    Description: Neptune cluster port - use as NEPTUNE_PORT
+    Value: !GetAtt NeptuneCluster.Port
+
+  AwsAccessKeyId:
+    Description: Access key ID for the Neptune IAM user - use as AWS_ACCESS_KEY_ID
+    Value: !Ref NeptuneUserAccessKey
+
+  AwsSecretAccessKey:
+    Description: Secret access key for the Neptune IAM user - use as AWS_SECRET_ACCESS_KEY
+    Value: !GetAtt NeptuneUserAccessKey.SecretAccessKey
+
+  AwsRegion:
+    Description: AWS region where Neptune is deployed - use as AWS_REGION
+    Value: !Ref AWS::Region
+
+  NeptuneClusterResourceId:
+    Description: Neptune cluster resource ID (for IAM policy reference)
+    Value: !GetAtt NeptuneCluster.ClusterResourceId
+
+  VpcId:
+    Description: VPC ID
+    Value: !Ref VPC
+
+  SecurityGroupId:
+    Description: Neptune security group ID
+    Value: !Ref NeptuneSecurityGroup

cookbook/use_cases/finance/03_Earnings_Call_Analysis.ipynb

@@ -288,6 +288,7 @@
     "    llm_model=\"llama-3.1-8b-instant\",\n",
     "    temperature=0.0,\n",
     "    api_key=GROQ_API_KEY,\n",
+    "    max_retries=3,\n",
     ")\n",
     "\n",
     "ENTITY_TYPES = [\"ORGANIZATION\", \"PERSON\", \"MONEY\", \"PERCENT\", \"DATE\", \"EVENT\"]\n",
@@ -371,8 +372,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "from concurrent.futures import ThreadPoolExecutor, TimeoutError\n",
     "from semantica.semantic_extract import RelationExtractor\n",
     "\n",
+    "MAX_ENTITIES = 30\n",
+    "CHUNK_TIMEOUT = 60\n",
+    "\n",
     "relation_extractor = RelationExtractor(\n",
     "    method=\"llm\",\n",
     "    confidence_threshold=0.6,\n",
@@ -385,22 +390,79 @@
     "        \"FOR_PERIOD\",\n",
     "        \"RELATED_TO\",\n",
     "    ],\n",
+    "    provider=\"groq\",\n",
+    "    llm_model=\"llama-3.1-8b-instant\",\n",
     "    api_key=GROQ_API_KEY,\n",
+    "    temperature=0.0,\n",
+    "    verbose=False,\n",
     ")\n",
     "\n",
-    "relationships = [\n",
-    "    r\n",
-    "    for c in chunks\n",
-    "    for r in relation_extractor.extract_relations(\n",
-    "        text=get_chunk_text(c),\n",
-    "        entities=all_entities,\n",
-    "        provider=\"groq\",\n",
-    "        llm_model=\"llama-3.1-8b-instant\",\n",
-    "        temperature=0.0,\n",
+    "\n",
+    "def filter_entities(text, entities):\n",
+    "    t = text.lower()\n",
+    "    return [e for e in entities if e.text.lower() in t]\n",
+    "\n",
+    "\n",
+    "def process_chunk(idx, chunk, total):\n",
+    "    text = get_chunk_text(chunk).strip()\n",
+    "\n",
+    "    remaining = total - (idx + 1)\n",
+    "\n",
+    "    if not text:\n",
+    "        print(f\"Chunk {idx+1}/{total} | remaining {remaining} | skipped (empty)\")\n",
+    "        return []\n",
+    "\n",
+    "    chunk_entities = filter_entities(text, all_entities)[:MAX_ENTITIES]\n",
+    "\n",
+    "    if len(chunk_entities) < 2:\n",
+    "        print(\n",
+    "            f\"Chunk {idx+1}/{total} | remaining {remaining} | \"\n",
+    "            f\"skipped (entities={len(chunk_entities)})\"\n",
+    "        )\n",
+    "        return []\n",
+    "\n",
+    "    print(\n",
+    "        f\"Chunk {idx+1}/{total} | remaining {remaining} | \"\n",
+    "        f\"entities={len(chunk_entities)}\"\n",
     "    )\n",
-    "    if get_chunk_text(c).strip()\n",
-    "]\n",
-    "print(\"Relationships:\", len(relationships))\n"
+    "\n",
+    "    return relation_extractor.extract_relations(\n",
+    "        text=text,\n",
+    "        entities=chunk_entities,\n",
+    "        verbose=False,\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "relationships = []\n",
+    "total_chunks = len(chunks)\n",
+    "\n",
+    "with ThreadPoolExecutor(max_workers=1) as executor:\n",
+    "    for i, c in enumerate(chunks):\n",
+    "        future = executor.submit(process_chunk, i, c, total_chunks)\n",
+    "\n",
+    "        try:\n",
+    "            rels = future.result(timeout=CHUNK_TIMEOUT)\n",
+    "            relationships.extend(rels)\n",
+    "            print(f\"  relations={len(rels)}\")\n",
+    "\n",
+    "        except TimeoutError:\n",
+    "            remaining = total_chunks - (i + 1)\n",
+    "            print(\n",
+    "                f\"Chunk {i+1}/{total_chunks} | remaining {remaining} | timed out\"\n",
+    "            )\n",
+    "\n",
+    "        except Exception as e:\n",
+    "            remaining = total_chunks - (i + 1)\n",
+    "            print(\n",
+    "                f\"Chunk {i+1}/{total_chunks} | remaining {remaining} | failed: {e}\"\n",
+    "            )\n",
+    "\n",
+    "print(f\"Done {total_chunks}/{total_chunks}\")\n",
+    "print(f\"Total relationships: {len(relationships)}\")\n",
+    "\n",
+    "if relationships:\n",
+    "    for r in relationships[:10]:\n",
+    "        print(f\"{r.subject.text} → {r.predicate} → {r.object.text}\")\n"
    ]
   },
   {
@@ -421,39 +483,69 @@
     "from semantica.conflicts import SourceTracker, SourceReference, ConflictDetector\n",
     "\n",
     "source_tracker = SourceTracker()\n",
+    "\n",
     "conflict_detector = ConflictDetector(\n",
     "    source_tracker=source_tracker,\n",
     "    similarity_threshold=0.8,\n",
     "    confidence_threshold=0.7,\n",
     ")\n",
     "\n",
-    "for entity in all_entities:\n",
-    "    entity_id = getattr(entity, \"id\", None) or getattr(entity, \"text\", \"\")\n",
-    "    entity_text = getattr(entity, \"text\", \"\")\n",
-    "    entity_label = getattr(entity, \"label\", \"UNKNOWN\")\n",
+    "entities = all_entities\n",
+    "extracted_relationships = relationships\n",
     "\n",
+    "for e in entities:\n",
+    "    entity_id = getattr(e, \"id\", None) or e.text\n",
     "    source_tracker.track_property_source(\n",
-    "        entity_id,\n",
-    "        \"name\",\n",
-    "        entity_text,\n",
-    "        # FIXED: Changed 'source' to 'document' to match SourceReference signature\n",
+    "        entity_id=entity_id,\n",
+    "        property_name=\"name\",\n",
+    "        value=e.text,\n",
     "        source=SourceReference(\n",
-    "            document=\"earnings_call\",  # Was incorrect: source=\"earnings_call\"\n",
+    "            document=\"earnings_call\",\n",
     "            timestamp=\"2024-Q1\",\n",
-    "            metadata={\"entity_type\": entity_label},\n",
+    "            metadata={\"entity_type\": getattr(e, \"label\", \"UNKNOWN\")},\n",
     "        ),\n",
     "    )\n",
     "\n",
-    "value_conflicts = conflict_detector.detect_value_conflicts(\n",
-    "    [{\"id\": getattr(e, \"id\", \"\"), \"name\": getattr(e, \"text\", \"\")} for e in all_entities],\n",
+    "entity_records = [\n",
+    "    {\n",
+    "        \"id\": getattr(e, \"id\", None) or e.text,\n",
+    "        \"name\": e.text,\n",
+    "    }\n",
+    "    for e in entities\n",
+    "]\n",
+    "\n",
+    "entity_value_conflicts = conflict_detector.detect_value_conflicts(\n",
+    "    entity_records,\n",
     "    property_name=\"name\",\n",
     ")\n",
     "\n",
-    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(relationships)\n",
+    "normalized_relationships = [\n",
+    "    {\n",
+    "        \"id\": getattr(r, \"id\", None),\n",
+    "        \"source_id\": getattr(r.subject, \"id\", None) or r.subject.text,\n",
+    "        \"target_id\": getattr(r.object, \"id\", None) or r.object.text,\n",
+    "        \"type\": r.predicate,\n",
+    "        \"confidence\": getattr(r, \"confidence\", 1.0),\n",
+    "        \"metadata\": {},\n",
+    "    }\n",
+    "    for r in extracted_relationships\n",
+    "]\n",
+    "\n",
+    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(\n",
+    "    normalized_relationships\n",
+    ")\n",
     "\n",
     "print(\"Conflict detection completed\")\n",
-    "print(\"Value conflicts:\", len(value_conflicts))\n",
-    "print(\"Relationship conflicts:\", len(relationship_conflicts))"
+    "print(\"Entity value conflicts:\", len(entity_value_conflicts))\n",
+    "print(\"Relationship conflicts:\", len(relationship_conflicts))\n",
+    "\n",
+    "if entity_value_conflicts:\n",
+    "    print(\"\\nSample entity conflict:\")\n",
+    "    print(entity_value_conflicts[0])\n",
+    "\n",
+    "if relationship_conflicts:\n",
+    "    print(\"\\nSample relationship conflict:\")\n",
+    "    print(relationship_conflicts[0])"
    ]
   },
   {
@@ -478,20 +570,36 @@
     "    source_tracker=source_tracker,\n",
     ")\n",
     "\n",
-    "resolved_conflicts = []\n",
+    "resolved_entity_value_conflicts = []\n",
+    "resolved_relationship_conflicts = []\n",
     "\n",
-    "for conflict in value_conflicts:\n",
-    "    resolved_conflicts.append(\n",
-    "        conflict_resolver.resolve_conflict(conflict, strategy=\"voting\")\n",
+    "for conflict in entity_value_conflicts:\n",
+    "    resolved_entity_value_conflicts.append(\n",
+    "        conflict_resolver.resolve_conflict(\n",
+    "            conflict,\n",
+    "            strategy=\"voting\",\n",
+    "        )\n",
     "    )\n",
     "\n",
     "for conflict in relationship_conflicts:\n",
-    "    resolved_conflicts.append(\n",
-    "        conflict_resolver.resolve_conflict(conflict, strategy=\"voting\")\n",
+    "    resolved_relationship_conflicts.append(\n",
+    "        conflict_resolver.resolve_conflict(\n",
+    "            conflict,\n",
+    "            strategy=\"voting\",\n",
+    "        )\n",
     "    )\n",
     "\n",
     "print(\"Conflict resolution completed\")\n",
-    "print(\"Total conflicts resolved:\", len(resolved_conflicts))\n"
+    "print(\"Entity value conflicts resolved:\", len(resolved_entity_value_conflicts))\n",
+    "print(\"Relationship conflicts resolved:\", len(resolved_relationship_conflicts))\n",
+    "\n",
+    "if resolved_entity_value_conflicts:\n",
+    "    print(\"\\nSample resolved entity conflict:\")\n",
+    "    print(resolved_entity_value_conflicts[0])\n",
+    "\n",
+    "if resolved_relationship_conflicts:\n",
+    "    print(\"\\nSample resolved relationship conflict:\")\n",
+    "    print(resolved_relationship_conflicts[0])"
    ]
   },
   {
@@ -510,38 +618,80 @@
    "outputs": [],
    "source": [
     "from semantica.deduplication import DuplicateDetector, EntityMerger\n",
+    "import time\n",
     "\n",
-    "duplicate_detector = DuplicateDetector(\n",
-    "    similarity_threshold=0.8,\n",
-    "    confidence_threshold=0.7,\n",
+    "start_time = time.time()\n",
+    "\n",
+    "raw = []\n",
+    "for i, e in enumerate(entities):\n",
+    "    raw.append({\n",
+    "        \"id\": getattr(e, \"id\", None) or f\"entity_{i}_{getattr(e, 'text', str(e))}\",\n",
+    "        \"name\": (getattr(e, \"text\", getattr(e, \"name\", \"\")) or \"\").strip(),\n",
+    "        \"type\": getattr(e, \"label\", \"UNKNOWN\"),\n",
+    "        \"confidence\": float(getattr(e, \"confidence\", 1.0) or 1.0),\n",
+    "        \"metadata\": getattr(e, \"metadata\", {}),\n",
+    "    })\n",
+    "\n",
+    "filtered = [r for r in raw if r[\"name\"] and len(r[\"name\"]) >= 3]\n",
+    "\n",
+    "collapsed = {}\n",
+    "for ent in filtered:\n",
+    "    key = (ent[\"type\"], ent[\"name\"].lower())\n",
+    "    best = collapsed.get(key)\n",
+    "    if best is None or ent[\"confidence\"] > best[\"confidence\"]:\n",
+    "        collapsed[key] = ent\n",
+    "\n",
+    "entity_dicts = list(collapsed.values())\n",
+    "\n",
+    "detector = DuplicateDetector(\n",
+    "    similarity_threshold=0.96,\n",
+    "    confidence_threshold=0.92,\n",
+    "    use_clustering=True,\n",
     ")\n",
     "\n",
-    "entity_dicts = [\n",
-    "    {\n",
-    "        \"id\": getattr(e, \"id\", \"\"),\n",
-    "        \"name\": getattr(e, \"text\", \"\"),\n",
-    "        \"type\": getattr(e, \"label\", \"UNKNOWN\"),\n",
-    "        \"confidence\": getattr(e, \"confidence\", 1.0),\n",
-    "        \"metadata\": getattr(e, \"metadata\", {}),\n",
-    "    }\n",
-    "    for e in resolved_entities\n",
-    "]\n",
+    "detector.detect_duplicate_groups(entity_dicts)\n",
     "\n",
-    "duplicates = duplicate_detector.detect_duplicates(entity_dicts)\n",
+    "merger = EntityMerger(\n",
+    "    preserve_provenance=True,\n",
+    "    detector={\n",
+    "        \"similarity_threshold\": 0.96,\n",
+    "        \"confidence_threshold\": 0.92,\n",
+    "        \"use_clustering\": True,\n",
+    "    },\n",
+    "    strategy={\"default_strategy\": \"keep_most_complete\"},\n",
+    ")\n",
     "\n",
-    "entity_merger = EntityMerger(preserve_provenance=True)\n",
-    "\n",
-    "merge_operations = entity_merger.merge_duplicates(\n",
-    "    entity_dicts,\n",
+    "merge_operations = merger.merge_duplicates(\n",
+    "    entities=entity_dicts,\n",
     "    strategy=\"keep_most_complete\",\n",
     ")\n",
     "\n",
-    "merged_entities = [op.merged_entity for op in merge_operations]\n",
+    "deduplicated_entities = [op.merged_entity for op in merge_operations] or entity_dicts\n",
     "\n",
-    "print(\"Entity deduplication completed\")\n",
-    "print(\"Original entities:\", len(entity_dicts))\n",
-    "print(\"Merged entities:\", len(merged_entities))\n",
-    "print(\"Duplicates removed:\", len(entity_dicts) - len(merged_entities))\n"
+    "entity_id_mapping = {}\n",
+    "for op in merge_operations:\n",
+    "    mid = op.merged_entity[\"id\"]\n",
+    "    for sid in op.source_ids:\n",
+    "        entity_id_mapping[sid] = mid\n",
+    "\n",
+    "deduplicated_relationships = []\n",
+    "for rel in normalized_relationships:\n",
+    "    s = entity_id_mapping.get(rel[\"source_id\"], rel[\"source_id\"])\n",
+    "    t = entity_id_mapping.get(rel[\"target_id\"], rel[\"target_id\"])\n",
+    "    if s != t:\n",
+    "        r = rel.copy()\n",
+    "        r[\"source_id\"], r[\"target_id\"] = s, t\n",
+    "        deduplicated_relationships.append(r)\n",
+    "\n",
+    "print({\n",
+    "    \"time_seconds\": round(time.time() - start_time, 2),\n",
+    "    \"entities_in\": len(raw),\n",
+    "    \"entities_after_filter\": len(filtered),\n",
+    "    \"entities_after_exact\": len(entity_dicts),\n",
+    "    \"entities_out\": len(deduplicated_entities),\n",
+    "    \"duplicates_removed\": len(entity_dicts) - len(deduplicated_entities),\n",
+    "    \"relationships_updated\": len(deduplicated_relationships),\n",
+    "})"
    ]
   },
   {
@@ -561,28 +711,17 @@
    "source": [
     "from semantica.kg import GraphBuilder\n",
     "\n",
+    "# Deduplication is already done; avoid additional entity resolution/merging\n",
     "graph_builder = GraphBuilder(\n",
-    "    merge_entities=True,\n",
-    "    entity_resolution_strategy=\"fuzzy\",\n",
+    "    merge_entities=False,\n",
+    "    entity_resolution_strategy=\"none\",\n",
     ")\n",
     "\n",
-    "triplet_relationships = [\n",
-    "    {\n",
-    "        \"source\": t.subject,\n",
-    "        \"predicate\": t.predicate,\n",
-    "        \"target\": t.object,\n",
-    "        \"confidence\": t.confidence,\n",
-    "        \"metadata\": t.metadata,\n",
-    "    }\n",
-    "    for t in validated_triplets\n",
-    "]\n",
-    "\n",
-    "final_relationships = resolved_relationships + triplet_relationships\n",
+    "final_relationships = deduplicated_relationships\n",
     "\n",
     "kg_data = {\n",
-    "    \"entities\": merged_entities,\n",
+    "    \"entities\": deduplicated_entities,\n",
     "    \"relationships\": final_relationships,\n",
-    "    \"triplets\": validated_triplets,\n",
     "    \"metadata\": {\n",
     "        \"source\": \"earnings_call_transcript\",\n",
     "        \"extraction_method\": \"Groq LLM\",\n",
@@ -591,12 +730,12 @@
     "\n",
     "knowledge_graph = graph_builder.build(\n",
     "    sources=[kg_data],\n",
-    "    merge_entities=True,\n",
+    "    merge_entities=False,\n",
     ")\n",
     "\n",
-    "print(\"Knowledge graph build completed\")\n",
+    "print(\"Knowledge graph build completed (no additional merging)\")\n",
     "print(\"Final entities:\", len(knowledge_graph.get(\"entities\", [])))\n",
-    "print(\"Final relationships:\", len(knowledge_graph.get(\"relationships\", [])))\n"
+    "print(\"Final relationships:\", len(knowledge_graph.get(\"relationships\", [])))"
    ]
   },
   {
@@ -641,12 +780,10 @@
     "connectivity = graph_analyzer.analyze_connectivity(knowledge_graph)\n",
     "metrics = graph_analyzer.compute_metrics(knowledge_graph)\n",
     "\n",
-    "top_entities = centrality.get(\"rankings\", [])[:5]\n",
     "num_communities = len(communities.get(\"communities\", []))\n",
     "\n",
     "print(\"Graph analysis completed\")\n",
-    "print(\"Communities:\", num_communities)\n",
-    "print(\"Top entities:\", len(top_entities))\n"
+    "print(\"Communities:\", num_communities)"
    ]
   },
   {
@@ -779,31 +916,62 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "import time\n",
     "from semantica.vector_store import VectorStore\n",
     "from semantica.context import ContextRetriever\n",
     "\n",
-    "vector_store = VectorStore(backend=\"faiss\")\n",
+    "if 'chunks' not in locals() or not chunks:\n",
+    "    raise ValueError(\"Chunks not found. Please run Step 3 first.\")\n",
     "\n",
-    "vector_store.add(\n",
-    "    texts=[parsed_doc[\"full_text\"]],\n",
-    "    metadata=[{\"source\": \"earnings_call\", \"type\": \"transcript\"}],\n",
+    "# Extract text content safely\n",
+    "chunk_texts = [getattr(c, \"content\", getattr(c, \"text\", \"\")) for c in chunks]\n",
+    "chunk_metadatas = [\n",
+    "    {\n",
+    "        \"source\": \"earnings_call\", \n",
+    "        \"type\": \"transcript\", \n",
+    "        \"chunk_index\": i,\n",
+    "        **(getattr(c, \"metadata\", {}) or {})\n",
+    "    }\n",
+    "    for i, c in enumerate(chunks)\n",
+    "]\n",
+    "\n",
+    "# Initialize Vector Store (Optimized for Speed)\n",
+    "# dimension=384 matches the default fast model (BAAI/bge-small-en-v1.5)\n",
+    "vector_store = VectorStore(\n",
+    "    backend=\"faiss\", \n",
+    "    dimension=384,  \n",
+    "    max_workers=16\n",
     ")\n",
     "\n",
+    "print(f\"Storing {len(chunks)} chunks with high-performance settings...\")\n",
+    "start_time = time.time()\n",
+    "\n",
+    "# Store in large batches with parallel processing\n",
+    "vector_ids = vector_store.add_documents(\n",
+    "    documents=chunk_texts,\n",
+    "    metadata=chunk_metadatas,\n",
+    "    batch_size=128,\n",
+    "    parallel=True\n",
+    ")\n",
+    "\n",
+    "print(f\"✅ Stored in {time.time() - start_time:.2f}s\")\n",
+    "\n",
+    "# Initialize Hybrid Retriever\n",
     "context_retriever = ContextRetriever(\n",
-    "    knowledge_graph=knowledge_graph,\n",
+    "    knowledge_graph=knowledge_graph, # Assumes knowledge_graph exists\n",
     "    vector_store=vector_store,\n",
     "    hybrid_alpha=0.6,\n",
     "    use_graph_expansion=True,\n",
     "    max_expansion_hops=2,\n",
     ")\n",
     "\n",
+    "# Test Retrieval\n",
     "queries = [\n",
     "    \"What was the company's revenue guidance?\",\n",
     "    \"What were the key financial metrics discussed?\",\n",
     "]\n",
     "\n",
     "retrieved_contexts = []\n",
-    "\n",
     "for query in queries:\n",
     "    results = context_retriever.retrieve(\n",
     "        query=query,\n",
@@ -813,8 +981,7 @@
     "    retrieved_contexts.append(results)\n",
     "\n",
     "print(\"Hybrid GraphRAG configured\")\n",
-    "print(\"Queries processed:\", len(queries))\n",
-    "print(\"Sample results:\", len(retrieved_contexts[0]) if retrieved_contexts else 0)\n"
+    "print(\"Sample results:\", len(retrieved_contexts[0]) if retrieved_contexts else 0)"
    ]
   },
   {
@@ -857,10 +1024,13 @@
     "    retention_days=30,\n",
     ")\n",
     "\n",
+    "entity_count = len(knowledge_graph.get(\"entities\", []))\n",
+    "relationship_count = len(knowledge_graph.get(\"relationships\", []))\n",
+    "\n",
     "memory_contents = [\n",
     "    f\"Earnings call transcript: {parsed_doc['metadata'].get('title', 'Earnings Call')}\",\n",
-    "    f\"Financial metrics extracted: {sum(len(v) for v in financial_metrics.values())}\",\n",
-    "    f\"Key entities identified: {len(merged_entities)}\",\n",
+    "    f\"Graph entities: {entity_count}\",\n",
+    "    f\"Graph relationships: {relationship_count}\",\n",
     "]\n",
     "\n",
     "memory_ids = []\n",
@@ -885,7 +1055,7 @@
     "print(\"Agent memory configured\")\n",
     "print(\"Memories stored:\", len(memory_ids))\n",
     "print(\"Total memories:\", memory_stats.get(\"total_memories\", 0))\n",
-    "print(\"Retrieved memories:\", len(financial_memories))\n"
+    "print(\"Retrieved memories:\", len(financial_memories))"
    ]
   },
   {
@@ -936,7 +1106,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "metadata": {},
    "outputs": [
     {
@@ -964,7 +1134,7 @@
     ")\n",
     "\n",
     "memory_id = agent_context.store(\n",
-    "    content=parsed_doc[\"full_text\"][:1000],\n",
+    "    content=chunks,\n",
     "    metadata={\"source\": \"earnings_call\", \"date\": \"2024-Q1\"},\n",
     "    extract_entities=True,\n",
     "    extract_relationships=True,\n",
@@ -1008,32 +1178,71 @@
     "\n",
     "generated_answers = []\n",
     "\n",
+    "print(\"--- Generating Enhanced Answers ---\\n\")\n",
+    "\n",
+    "def format_context(retrieved_contexts):\n",
+    "    \"\"\"Formats retrieved context with graph information.\"\"\"\n",
+    "    formatted_parts = []\n",
+    "    \n",
+    "    for i, ctx in enumerate(retrieved_contexts):\n",
+    "        content = getattr(ctx, \"content\", \"\")\n",
+    "        source = getattr(ctx, \"source\", \"unknown\")\n",
+    "        \n",
+    "        # Format related entities from the graph\n",
+    "        related_entities = getattr(ctx, \"related_entities\", [])\n",
+    "        entities_str = \", \".join([\n",
+    "            f\"{e.get('name', 'Unknown')} ({e.get('type', 'Entity')})\" \n",
+    "            for e in related_entities[:5]  # Limit to top 5 per chunk\n",
+    "        ])\n",
+    "        \n",
+    "        # Format related relationships\n",
+    "        related_rels = getattr(ctx, \"related_relationships\", [])\n",
+    "        rels_str = \"; \".join([\n",
+    "            f\"{r.get('source', '')} -> {r.get('type', '')} -> {r.get('target', '')}\"\n",
+    "            for r in related_rels[:3]  # Limit to top 3 per chunk\n",
+    "        ])\n",
+    "        \n",
+    "        part = f\"Source {i+1} ({source}):\\n{content}\\n\"\n",
+    "        if entities_str:\n",
+    "            part += f\"Related Entities: {entities_str}\\n\"\n",
+    "        if rels_str:\n",
+    "            part += f\"Graph Connections: {rels_str}\\n\"\n",
+    "            \n",
+    "        formatted_parts.append(part)\n",
+    "        \n",
+    "    return \"\\n---\\n\".join(formatted_parts)\n",
+    "\n",
     "for question in financial_questions:\n",
+    "    print(f\"Question: {question}\")\n",
+    "    \n",
+    "    # Retrieve with graph expansion enabled and higher limits\n",
     "    retrieved_contexts = context_retriever.retrieve(\n",
     "        query=question,\n",
-    "        max_results=3,\n",
+    "        max_results=10,          # Increased from 3\n",
     "        min_relevance_score=0.2,\n",
+    "        use_graph_expansion=True, # Explicitly enable graph expansion\n",
+    "        max_hops=2               # Traverse up to 2 hops in the graph\n",
     "    )\n",
     "\n",
-    "    context_text = \"\\n\\n\".join(\n",
-    "        ctx.get(\"content\", ctx.get(\"text\", \"\"))\n",
-    "        for ctx in retrieved_contexts\n",
-    "    )[:1000]\n",
+    "    # Use the rich formatter\n",
+    "    context_text = format_context(retrieved_contexts)\n",
     "\n",
-    "    entity_names = [\n",
-    "        entity.get(\"name\", \"\")\n",
-    "        for entity in knowledge_graph.get(\"entities\", [])[:5]\n",
+    "    # Get global key entities (optional, but good for high-level context)\n",
+    "    global_entities = [\n",
+    "        f\"{e.get('name', '')} ({e.get('type', '')})\"\n",
+    "        for e in knowledge_graph.get(\"entities\", [])[:10]\n",
     "    ]\n",
-    "    entities_text = \", \".join(entity_names) or \"N/A\"\n",
+    "    global_entities_text = \", \".join(global_entities)\n",
     "\n",
     "    prompt = f\"\"\"\n",
-    "Answer the question using only the context below.\n",
+    "Answer the question comprehensively using the provided context.\n",
+    "The context includes text chunks and knowledge graph connections (entities and relationships).\n",
     "If the answer is not present, say so.\n",
     "\n",
     "Context:\n",
     "{context_text}\n",
     "\n",
-    "Key entities: {entities_text}\n",
+    "Global Key Entities: {global_entities_text}\n",
     "\n",
     "Question:\n",
     "{question}\n",
@@ -1044,16 +1253,18 @@
     "    try:\n",
     "        answer = groq_llm.generate(\n",
     "            prompt,\n",
-    "            temperature=0.7,\n",
-    "            max_tokens=400,\n",
+    "            temperature=0.3, # Lower temperature for more factual answers\n",
+    "            max_tokens=1000, # Allow longer answers\n",
     "        )\n",
     "    except Exception as error:\n",
     "        answer = f\"Answer generation failed: {error}\"\n",
     "\n",
     "    generated_answers.append(answer)\n",
+    "    print(f\"Answer: {answer}\\n\")\n",
+    "    print(\"-\" * 50 + \"\\n\")\n",
     "\n",
     "print(\"Answer generation completed\")\n",
-    "print(\"Questions answered:\", len(generated_answers))\n"
+    "print(\"Questions answered:\", len(generated_answers))"
    ]
   },
   {
@@ -1072,28 +1283,44 @@
    "outputs": [],
    "source": [
     "from semantica.export import JSONExporter, RDFExporter\n",
+    "import json\n",
     "\n",
+    "# Initialize exporters\n",
     "json_exporter = JSONExporter()\n",
     "rdf_exporter = RDFExporter()\n",
     "\n",
-    "kg_json = json_exporter.export(knowledge_graph, format=\"json\")\n",
-    "kg_rdf = rdf_exporter.export_to_rdf(knowledge_graph, format=\"turtle\")\n",
+    "# Define output file paths\n",
+    "json_output_path = \"knowledge_graph.json\"\n",
+    "rdf_output_path = \"knowledge_graph.ttl\"\n",
     "\n",
+    "# Export to files (required by the API)\n",
+    "json_exporter.export(knowledge_graph, file_path=json_output_path, format=\"json\")\n",
+    "\n",
+    "# FIXED: Use .export() instead of .export_to_rdf() to write to disk\n",
+    "rdf_exporter.export(knowledge_graph, file_path=rdf_output_path, format=\"turtle\")\n",
+    "\n",
+    "# Load the RDF file content to check its size\n",
+    "with open(rdf_output_path, \"r\", encoding=\"utf-8\") as f:\n",
+    "    kg_rdf_content = f.read()\n",
+    "\n",
+    "# Create analysis summary\n",
     "analysis_summary = {\n",
     "    \"entities\": len(knowledge_graph.get(\"entities\", [])),\n",
     "    \"relationships\": len(knowledge_graph.get(\"relationships\", [])),\n",
-    "    \"conflicts_resolved\": len(resolved_conflicts),\n",
-    "    \"merged_entities\": len(merged_entities),\n",
-    "    \"communities\": num_communities,\n",
+    "    \"entity_conflicts_resolved\": len(locals().get(\"resolved_entity_value_conflicts\", [])),\n",
+    "    \"relationship_conflicts_resolved\": len(locals().get(\"resolved_relationship_conflicts\", [])),\n",
+    "    \"deduplicated_entities\": len(locals().get(\"deduplicated_entities\", [])),\n",
+    "    \"communities\": locals().get(\"num_communities\", 0),\n",
     "    \"questions_answered\": len(generated_answers),\n",
-    "    \"llm_model\": groq_llm.model,\n",
+    "    \"llm_model\": getattr(groq_llm, \"model\", \"unknown\"),\n",
     "}\n",
     "\n",
     "print(\"Export completed\")\n",
     "print(\"KG JSON entities:\", analysis_summary[\"entities\"])\n",
-    "print(\"KG RDF size (chars):\", len(kg_rdf))\n",
+    "print(\"KG RDF size (chars):\", len(kg_rdf_content))\n",
     "print(\"Questions answered:\", analysis_summary[\"questions_answered\"])\n",
-    "print(\"LLM model:\", analysis_summary[\"llm_model\"])\n"
+    "print(\"LLM model:\", analysis_summary[\"llm_model\"])\n",
+    "print(\"Conflicts resolved:\", analysis_summary[\"entity_conflicts_resolved\"] + analysis_summary[\"relationship_conflicts_resolved\"])"
    ]
   }
  ],

docs/citation.md

@@ -12,22 +12,22 @@ How to cite Semantica in academic papers and research.
   author = {Hawksight AI},
   year = {2026},
   url = {https://github.com/Hawksight-AI/semantica},
-  version = {0.2.2},
+  version = {0.2.5},
   doi = {10.5281/zenodo.XXXXXXX}
 }
 ```
 
 ### APA
-Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.2) [Computer software]. https://github.com/Hawksight-AI/semantica
+Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.5) [Computer software]. https://github.com/Hawksight-AI/semantica
 
 ### MLA
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.2, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.5, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
 
 ### Chicago
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.2. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.5. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
 
 ### IEEE
-Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.2, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
+Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.5, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
 
 ---
 

docs/index.md

@@ -17,7 +17,7 @@
   
   <p><em>The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.</em></p>
   
-  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.1.1</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
+  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.2.3</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
   
   <p>
     <a href="getting-started/" class="md-button md-button--primary">Get Started</a>

docs/js/version-selector.js

@@ -9,7 +9,12 @@ document.addEventListener("DOMContentLoaded", function () {
 
         // Define versions
         var versions = [
-            { name: "0.1.1", url: "#", current: true },
+            { name: "0.2.4", url: "#", current: true },
+            { name: "0.2.3", url: "#", current: false },
+            { name: "0.2.2", url: "#", current: false },
+            { name: "0.2.1", url: "#", current: false },
+            { name: "0.2.0", url: "#", current: false },
+            { name: "0.1.1", url: "#", current: false },
             { name: "0.1.0", url: "#", current: false }
         ];
 

docs/reference/ontology.md

@@ -63,6 +63,29 @@ The module uses several inference algorithms:
 
 ---
 
+## Ontology Ingestion
+
+Ingest existing ontology files directly into usable data structures using `OntologyIngestor`.
+
+**Function:** `ingest_ontology(source, method="file")`
+
+| Argument | Description |
+|----------|-------------|
+| `source` | File path, directory path, or list of paths |
+| `method` | Ingestion method (default: "file") |
+
+**Example:**
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest file
+data = ingest_ontology("ontology.ttl")
+
+# Ingest directory
+dataset = ingest_ontology("ontologies/")
+```
+
 ## Main Classes
 
 ### OntologyEngine
@@ -170,6 +193,17 @@ Manages external dependencies.
 | `import_external_ontology(uri, ontology)` | Load and merge external ontology |
 | `evaluate_alignment(uri, ontology)` | Assess alignment and compatibility |
 
+### OntologyIngestor
+
+Handles ingestion of existing ontologies from files and directories.
+
+**Methods:**
+
+| Method | Description |
+|--------|-------------|
+| `ingest_ontology(file_path)` | Ingest a single ontology file |
+| `ingest_directory(directory_path)` | Recursively ingest ontology files from a directory |
+
 ---
 
 ## Unified Engine Examples

docs/reference/semantic_extract.md

@@ -184,18 +184,15 @@ Core entity extraction implementation used by notebooks and lower-level integrat
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `method` | str or list | `"ml"` | Method(s): "ml", "llm", "pattern", "regex", "huggingface" |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `64000` | Max text length for auto-chunking (LLM only) |
-| `max_tokens` | int | `None` | Max output tokens for LLM generation |
-| `max_workers` | int | `1` | Threads for parallel batch processing |
-| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `provider`) |
+| `entity_types` | list | `None` | Filter for specific entity types |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `aggregation_strategy`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text)` | Alias for `extract_entities`. Get list of entities. |
-| `extract_entities(text)` | Get list of entities |
+| `extract(text, pipeline_id=None, **kwargs)` | Alias for `extract_entities`. Supports `max_workers`. |
+| `extract_entities(text, pipeline_id=None, **kwargs)` | Get list of entities. Supports `max_workers`. |
 
 **Example:**
 
@@ -231,18 +228,19 @@ Extracts relationships between entities.
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"dependency"` | Method: "dependency", "pattern", "cooccurrence", "huggingface", "llm" |
 | `relation_types` | list | `None` | Specific relation types to extract |
 | `bidirectional` | bool | `False` | Extract bidirectional relations |
 | `confidence_threshold` | float | `0.6` | Minimum confidence score |
 | `max_distance` | int | `50` | Max token distance between entities |
-| `max_workers` | int | `1` | Threads for parallel batch processing |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `device` for HuggingFace) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text, entities)` | Alias for `extract_relations`. Find links. |
-| `extract_relations(text, entities)` | Find links |
+| `extract(text, entities, pipeline_id=None, **kwargs)` | Alias for `extract_relations`. Supports `max_workers`. |
+| `extract_relations(text, entities, pipeline_id=None, **kwargs)` | Find links. Supports `max_workers`. |
 
 **Example:**
 
@@ -341,19 +339,18 @@ Extracts RDF triplets (Subject-Predicate-Object).
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
+| `triplet_types` | list | `None` | Specific triplet types/predicates to extract |
 | `include_temporal` | bool | `False` | Include time information |
 | `include_provenance` | bool | `False` | Track source sentences |
-| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `64000` | Max text length for auto-chunking (LLM only) |
-| `max_tokens` | int | `None` | Max output tokens for LLM generation |
-| `max_workers` | int | `1` | Threads for parallel batch processing |
+| `**kwargs` | dict | `{}` | Configuration options (e.g., `model`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract_triplets(text)` | Get (S, P, O) tuples |
+| `extract(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Alias for `extract_triplets`. Supports `max_workers`. |
+| `extract_triplets(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Get (S, P, O) tuples. Supports `max_workers`. |
 
 **Example:**
 

docs/reference/vector_store.md

@@ -1,12 +1,12 @@
 # Vector Store
 
-> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, and Milvus with Hybrid Search.**
+> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, Pinecone, and Milvus with Hybrid Search.**
 
 ---
 
 ## 🎯 Overview
 
-The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Milvus) and enables semantic search, RAG, and similarity matching.
+The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Pinecone, Milvus) and enables semantic search, RAG, and similarity matching.
 
 ### What is a Vector Store?
 
@@ -18,7 +18,7 @@ A **vector store** is a database optimized for storing and searching high-dimens
 
 ### Why Use the Vector Store Module?
 
-- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, and Milvus
+- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, Pinecone, and Milvus
 - **Unified Interface**: Same API regardless of backend
 - **Hybrid Search**: Combine vector similarity with metadata filtering
 - **Performance**: Optimized for high-throughput search operations
@@ -38,8 +38,8 @@ A **vector store** is a database optimized for storing and searching high-dimens
 -   :material-database:{ .lg .middle } **Multi-Backend Support**
 
     ---
-
-    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, and Milvus
+    
+    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, Pinecone, and Milvus
 
 -   :material-magnify-plus:{ .lg .middle } **Hybrid Search**
 
@@ -112,6 +112,8 @@ The main facade for all vector operations.
 | Method | Description |
 |--------|-------------|
 | `store_vectors(vectors, metadata)` | Store embeddings |
+| `add_documents(documents, metadata, batch_size, parallel)` | **(New)** Store documents with automatic embedding generation and parallelization |
+| `embed_batch(texts)` | **(New)** Generate embeddings for a batch of texts |
 | `search(query, k)` | Semantic search |
 | `delete(ids)` | Remove vectors |
 
@@ -120,15 +122,24 @@ The main facade for all vector operations.
 ```python
 from semantica.vector_store import VectorStore
 
-# Initialize (defaults to FAISS)
+# Initialize (defaults to FAISS, parallel enabled by default with 6 workers)
 store = VectorStore(backend="faiss", dimension=1536)
 
-# Store
+# 1. Store pre-computed vectors
 ids = store.store_vectors(
     vectors=[[0.1, 0.2, ...], ...],
     metadata=[{"text": "Hello"}, ...]
 )
 
+# 2. Store raw documents (High Performance)
+# Automatically handles embedding generation in parallel batches (uses default 6 workers)
+ids = store.add_documents(
+    documents=["Doc 1", "Doc 2", ...],
+    metadata=[{"id": 1}, {"id": 2}, ...],
+    batch_size=32,
+    parallel=True
+)
+
 # Search
 results = store.search(query_vector=[0.1, 0.2, ...], k=5)
 ```
@@ -771,6 +782,7 @@ print(f"Context: {context}")
 ---
 
 ## See Also
+- [High-Performance Usage Guide](../vector_store_usage.md) - **(New)** Parallel ingestion and batching guide
 - [Embeddings Module](embeddings.md) - Generates the vectors
 - [Context Module](context.md) - Uses vector store for memory
 - [Ingest Module](ingest.md) - Source of data

docs/vector_store_usage.md

@@ -0,0 +1,101 @@
+# High-Performance Vector Store Usage
+
+This guide demonstrates how to leverage the new high-performance features of the Semantica Vector Store, specifically designed for efficient batch processing and parallel ingestion of large document sets.
+
+## 🚀 Key Features
+
+- **Parallel Ingestion**: Utilize multi-threading to embed and store documents concurrently.
+- **Batch Processing**: Automatically group documents into batches to minimize overhead.
+- **Unified API**: A single `add_documents` method handles embedding generation and storage.
+
+---
+
+## ⚡ Quick Start: Parallel Ingestion
+
+The fastest way to ingest documents is using the `add_documents` method. Parallelization is enabled by default with optimized settings (6 workers).
+
+```python
+from semantica.vector_store import VectorStore
+import time
+
+store = VectorStore(
+    backend="faiss",
+    dimension=768,
+)
+
+documents = [f"This is document number {i} with some content." for i in range(1000)]
+metadata = [{"source": "generated", "id": i} for i in range(1000)]
+
+start_time = time.time()
+ids = store.add_documents(
+    documents=documents,
+    metadata=metadata,
+    batch_size=64,
+    parallel=True,
+)
+print(f"Ingested {len(ids)} documents in {time.time() - start_time:.2f}s")
+```
+
+---
+
+## 📊 Performance Comparison
+
+### Old Method (Sequential Loop)
+*Slower due to sequential processing and overhead per single item.*
+
+```python
+for doc in documents:
+    emb = embedder.generate(doc)
+    store.store_vectors([emb], [{"text": doc}])
+```
+
+### New Method (Parallel Batching)
+*Significantly faster (3x-10x) by utilizing thread pools and batch operations.*
+
+```python
+store.add_documents(documents, parallel=True)
+```
+
+---
+
+## 🛠 Configuration & Tuning
+
+### `max_workers`
+Controls the number of concurrent threads used for embedding generation.
+- **Default**: 6 (Optimized for most systems)
+- **Recommendation**: You generally don't need to change this. If you have very high core counts or specific throughput needs, you can override it.
+
+```python
+store = VectorStore(max_workers=16)
+```
+
+### `batch_size`
+Controls how many documents are processed in a single chunk.
+- **Default**: 32
+- **Recommendation**: 
+    - **Local Models**: 32-64 usually works well.
+    - **API Models (OpenAI, etc.)**: Larger batches (e.g., 100-200) can reduce network latency overhead.
+
+```python
+store.add_documents(documents, batch_size=100)
+```
+
+---
+
+## 🧩 Advanced: Manual Batch Embedding
+
+If you need the embeddings without storing them immediately, use `embed_batch`.
+
+```python
+vectors = store.embed_batch(
+    texts=documents[:100],
+)
+
+print(f"Generated {len(vectors)} vectors")
+```
+
+## ⚠️ Best Practices
+
+1.  **Metadata Consistency**: Ensure your `metadata` list has the same length as your `documents` list.
+2.  **Error Handling**: The `add_documents` method will propagate exceptions if embedding fails. Ensure your data is clean.
+3.  **Memory Usage**: Very large `batch_size` combined with high `max_workers` can increase memory usage. Monitor your system resources.

examples/huggingface_demo.py

@@ -0,0 +1,147 @@
+"""
+HuggingFace Local Model Usage Demo (Bring Your Own Model)
+
+This script demonstrates how to use the 'semantica' library with local HuggingFace models
+for Named Entity Recognition (NER), Relation Extraction (RE), and Triplet Extraction.
+
+Prerequisites:
+    pip install transformers torch
+
+Usage:
+    python examples/huggingface_demo.py
+"""
+
+import sys
+import os
+
+# Add project root to path (for running from this dir)
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor, Entity
+
+def demo_ner():
+    print("\n" + "="*50)
+    print("NER Demo: Bring Your Own Model (BYOM)")
+    print("="*50)
+    
+    # 1. Initialize NERExtractor with HuggingFace method and a specific model
+    # Common models: "dslim/bert-base-NER", "dbmdz/bert-large-cased-finetuned-conll03-english"
+    model_name = "dslim/bert-base-NER" 
+    print(f"Initializing NERExtractor with model: {model_name}...")
+    
+    extractor = NERExtractor(
+        method="huggingface", 
+        model=model_name,
+        device="cpu"  # Use "cuda" for GPU
+    )
+    
+    text = "Steve Jobs founded Apple Inc. in Cupertino, California on April 1, 1976."
+    print(f"\nInput text: {text}")
+    
+    try:
+        # Note: This will download the model if not cached (approx 400MB)
+        print("Extracting entities (this may take a moment on first run)...")
+        entities = extractor.extract_entities(text)
+        
+        print(f"\nExtracted {len(entities)} entities:")
+        for ent in entities:
+            print(f"  - {ent.text:20} | Type: {ent.label:10} | Conf: {ent.confidence:.2f}")
+            
+    except Exception as e:
+        print(f"Extraction failed (missing dependencies?): {e}")
+
+
+def demo_relation():
+    print("\n" + "="*50)
+    print("Relation Extraction Demo: Local Model")
+    print("="*50)
+    
+    # 1. Initialize RelationExtractor
+    # Note: Relation extraction usually requires a SequenceClassification model 
+    # trained on relation datasets (e.g., TACRED, SemEval).
+    # For demo purposes, we'll use a generic placeholder or a widely used one.
+    model_name = "semantica/relation-model-v1" # This is hypothetical; replace with real model
+    print(f"Initializing RelationExtractor with method='huggingface'...")
+    
+    extractor = RelationExtractor(
+        method="huggingface",
+        model=model_name,
+        device="cpu"
+    )
+    
+    text = "Steve Jobs founded Apple Inc."
+    # Pre-defined entities are usually required for relation extraction
+    entities = [
+        Entity(text="Steve Jobs", label="PERSON", start_char=0, end_char=10), 
+        Entity(text="Apple Inc.", label="ORG", start_char=19, end_char=29)
+    ]
+    
+    print(f"\nInput text: {text}")
+    print(f"Entities: {[e.text for e in entities]}")
+    
+    try:
+        print("Extracting relations...")
+        # Note: This will fail if the model doesn't exist on HF Hub.
+        # In a real scenario, use a valid model ID like "some-user/bert-relation-extraction"
+        # For this demo, we just show the call structure.
+        relations = extractor.extract_relations(text, entities)
+        
+        print(f"\nExtracted {len(relations)} relations:")
+        for rel in relations:
+            print(f"  - {rel.subject.text} --[{rel.predicate}]--> {rel.object.text} (Conf: {rel.confidence:.2f})")
+            
+    except Exception as e:
+        print(f"Note: Relation extraction mock run (model download might fail or be skipped): {e}")
+
+
+def demo_triplet():
+    print("\n" + "="*50)
+    print("Triplet Extraction Demo: REBEL (Seq2Seq)")
+    print("="*50)
+    
+    # 1. Initialize TripletExtractor with REBEL model
+    # REBEL is a popular model for end-to-end triplet extraction
+    model_name = "Babelscape/rebel-large"
+    print(f"Initializing TripletExtractor with model: {model_name}...")
+    
+    extractor = TripletExtractor(
+        method="huggingface",
+        model=model_name,
+        device="cpu"
+    )
+    
+    text = "Apple was founded by Steve Jobs in 1976."
+    print(f"\nInput text: {text}")
+    
+    try:
+        print("Extracting triplets (this may take a moment)...")
+        triplets = extractor.extract_triplets(text)
+        
+        print(f"\nExtracted {len(triplets)} triplets:")
+        for triplet in triplets:
+            print(f"  - ({triplet.subject}, {triplet.predicate}, {triplet.object})")
+            
+    except Exception as e:
+        print(f"Extraction failed (missing dependencies?): {e}")
+
+if __name__ == "__main__":
+    print("Starting Semantica HuggingFace Usage Demo...")
+    print("Note: This script attempts to download models from Hugging Face Hub.")
+    print("Ensure you have an internet connection and 'transformers' installed.")
+    
+    # Run demos
+    # We wrap in try-except to ensure the script doesn't crash the whole session if one fails
+    try:
+        demo_ner()
+    except Exception as e:
+        print(f"NER Demo Error: {e}")
+        
+    try:
+        demo_relation()
+    except Exception as e:
+        print(f"Relation Demo Error: {e}")
+        
+    try:
+        demo_triplet()
+    except Exception as e:
+        print(f"Triplet Demo Error: {e}")

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantica"
-version = "0.2.2"
+version = "0.2.5"
 description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering"
 readme = "README.md"
 license = { text = "MIT" }
@@ -114,6 +114,16 @@ graph-all = [
   "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune]"
 ]
 
+# ---- Vector Store Backends ----
+vectorstore-qdrant = ["qdrant-client>=1.0.0"]
+vectorstore-weaviate = ["weaviate-client>=4.0.0"]
+vectorstore-pinecone = ["pinecone-client>=3.0.0"]
+vectorstore-milvus = ["pymilvus>=2.0.0"]
+
+vectorstore-all = [
+  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus]"
+]
+
 # ---- Infra / Queues / Workers ----
 infra = [
   "redis>=4.3.0",
@@ -177,7 +187,7 @@ dev = [
 
 # ---- Everything ----
 all = [
-  "semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,parse-docling]"
+  "semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling]"
 ]
 
 # ---------------- ENTRYPOINTS ----------------

semantica/__init__.py

@@ -10,7 +10,7 @@ Main exports:
     - Config: Configuration management
 """
 
-__version__ = "0.2.2"
+__version__ = "0.2.5"
 __author__ = "Semantica Contributors"
 __license__ = "MIT"
 

semantica/ingest/__init__.py

@@ -86,6 +86,7 @@ Main Classes:
     - RepoIngestor: Git repository processing
     - EmailIngestor: Email protocol handling
     - DBIngestor: Database export handling
+    - OntologyIngestor: Ontology file processing
     - MethodRegistry: Registry for custom ingestion methods
     - IngestConfig: Configuration manager for ingest module
 
@@ -98,6 +99,7 @@ Convenience Functions:
     - ingest_repository: Repository ingestion wrapper
     - ingest_email: Email ingestion wrapper
     - ingest_database: Database ingestion wrapper
+    - ingest_ontology: Ontology ingestion wrapper
 
 
 Example Usage:
@@ -134,6 +136,7 @@ from .methods import (
     ingest_feed,
     ingest_file,
     ingest_mcp,
+    ingest_ontology,
     ingest_repository,
     ingest_stream,
     ingest_web,
@@ -166,6 +169,8 @@ from .web_ingestor import (
     WebIngestor,
 )
 
+from .ontology_ingestor import OntologyData, OntologyIngestor
+
 __all__ = [
     # File ingestion
     "FileIngestor",
@@ -216,6 +221,9 @@ __all__ = [
     "MCPClient",
     "MCPResource",
     "MCPTool",
+    # Ontology ingestion
+    "OntologyIngestor",
+    "OntologyData",
     # Registry and Methods
     "MethodRegistry",
     "method_registry",
@@ -227,6 +235,7 @@ __all__ = [
     "ingest_repository",
     "ingest_email",
     "ingest_database",
+    "ingest_ontology",
     "ingest_mcp",
     "get_ingest_method",
     "list_available_methods",

semantica/ingest/methods.py

@@ -150,6 +150,7 @@ from .email_ingestor import EmailData, EmailIngestor
 from .feed_ingestor import FeedData, FeedIngestor
 from .file_ingestor import FileIngestor, FileObject
 from .mcp_ingestor import MCPData, MCPIngestor
+from .ontology_ingestor import OntologyData, OntologyIngestor
 from .registry import method_registry
 from .repo_ingestor import RepoIngestor
 from .stream_ingestor import StreamIngestor, StreamProcessor
@@ -537,6 +538,66 @@ def ingest_email(
         raise
 
 
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], method: str = "file", **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source (convenience function).
+
+    This is a user-friendly wrapper that ingests ontologies using the specified method.
+
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+            - "file": Single file ingestion
+            - "directory": Directory ingestion with recursive scanning
+        **kwargs: Additional options passed to OntologyIngestor
+
+    Returns:
+        OntologyData, List[OntologyData] with ingestion results
+
+    Examples:
+        >>> from semantica.ingest.methods import ingest_ontology
+        >>> ontology = ingest_ontology("ontology.ttl")
+        >>> ontologies = ingest_ontology("./ontologies", method="directory")
+    """
+    # Check for custom method in registry
+    custom_method = method_registry.get("ontology", method)
+    if custom_method and custom_method != ingest_ontology:
+        try:
+            return custom_method(source, **kwargs)
+        except Exception as e:
+            logger.warning(
+                f"Custom method {method} failed: {e}, falling back to default"
+            )
+
+    try:
+        # Get config
+        config = ingest_config.get_method_config("ontology")
+        config.update(kwargs)
+
+        ingestor = OntologyIngestor(**config)
+
+        source_path = str(source) if isinstance(source, (str, Path)) else None
+        
+        if method == "file" and source_path:
+            if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+            return ingestor.ingest_ontology(source_path, **kwargs)
+        elif method == "directory" and source_path:
+             recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
+             return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
+        else:
+             # Default: try as file
+             if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+             return ingestor.ingest_ontology(str(source), **kwargs)
+
+    except Exception as e:
+        logger.error(f"Failed to ingest ontology: {e}")
+        raise
+
+
 def ingest_database(
     source: Union[str, Dict[str, Any]], method: Optional[str] = None, **kwargs
 ) -> Union[TableData, List[TableData], Dict[str, Any]]:
@@ -769,6 +830,7 @@ def ingest(
             - "repo": Repository ingestion
             - "email": Email ingestion
             - "db": Database ingestion
+            - "ontology": Ontology ingestion
         method: Optional specific ingestion method
         **kwargs: Additional options passed to ingestor
 
@@ -802,6 +864,8 @@ def ingest(
                 ("git@", "https://github.com", "https://gitlab.com")
             ):
                 source_type = "repo"
+            elif source_str.endswith((".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")):
+                source_type = "ontology"
             else:
                 source_type = "file"
         else:
@@ -830,6 +894,8 @@ def ingest(
             raise ProcessingError("Email ingestion requires configuration dictionary")
     elif source_type == "db":
         return {"data": ingest_database(sources, method=method, **kwargs)}
+    elif source_type == "ontology":
+        return {"ontology": ingest_ontology(sources, method=method or "file", **kwargs)}
     elif source_type == "mcp":
         return {"data": ingest_mcp(sources, method=method or "resources", **kwargs)}
     else:
@@ -909,5 +975,8 @@ method_registry.register("mcp", "default", ingest_mcp)
 method_registry.register("mcp", "resources", ingest_mcp)
 method_registry.register("mcp", "tools", ingest_mcp)
 method_registry.register("mcp", "all", ingest_mcp)
+method_registry.register("ontology", "default", ingest_ontology)
+method_registry.register("ontology", "file", ingest_ontology)
+method_registry.register("ontology", "directory", ingest_ontology)
 method_registry.register("ingest", "default", ingest)
 method_registry.register("ingest", "unified", ingest)

semantica/ingest/ontology_ingestor.py

@@ -0,0 +1,392 @@
+"""
+Ontology Ingestion Module
+
+This module provides capabilities to ingest external ontologies from files (OWL, RDF, TTL, etc.)
+and convert them into Semantica's internal ontology dictionary format.
+
+Supported Formats:
+    - Turtle (.ttl): Terse RDF Triple Language. A concise, human-readable
+      format for representing RDF graphs. Commonly used for writing
+      ontologies by hand.
+    - RDF/XML (.rdf, .owl): The XML serialization of RDF. The standard
+      format for OWL (Web Ontology Language) ontologies and often used
+      for data interchange.
+    - JSON-LD (.jsonld): JSON for Linked Data. A lightweight Linked Data
+      format that is easy for humans to read and for machines to parse
+      and generate. Ideal for web-based applications.
+    - N-Triples (.nt): A line-based, plain text format for encoding an
+      RDF graph. Each line represents a single triple. Very simple to
+      parse but verbose.
+    - Notation3 (.n3): A superset of Turtle that adds features like logic
+      and rules.
+
+Key Features:
+    - Support for multiple RDF formats (Turtle, RDF/XML, JSON-LD, N3, NT)
+    - Automatic parsing using rdflib
+    - Conversion to Semantica ontology structure
+    - Batch processing of ontology files
+    - Extraction of classes, properties, and metadata
+
+Example Usage:
+    >>> from semantica.ingest import OntologyIngestor
+    >>> ingestor = OntologyIngestor()
+    >>> ontology = ingestor.ingest_ontology("my_ontology.ttl")
+"""
+
+import os
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Union
+
+import rdflib
+from rdflib import RDF, RDFS, OWL, Graph
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from ..utils.progress_tracker import get_progress_tracker
+
+
+@dataclass
+class OntologyData:
+    """Ontology data representation."""
+
+    data: Dict[str, Any]
+    source_path: str
+    format: str
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    ingested_at: datetime = field(default_factory=datetime.now)
+
+
+class OntologyIngestor:
+    """
+    Ontology ingestion handler.
+
+    This class parses OWL/RDF files and converts them to Semantica's ontology dictionary format.
+    """
+
+    def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
+        """
+        Initialize ontology ingestor.
+
+        Args:
+            config: Optional configuration dictionary
+            **kwargs: Additional configuration parameters
+        """
+        self.logger = get_logger("ontology_ingestor")
+        self.progress = get_progress_tracker()
+        self.config = config or {}
+        self.config.update(kwargs)
+
+    def ingest_ontology(self, file_path: Union[str, Path], format: Optional[str] = None, **kwargs) -> OntologyData:
+        """
+        Ingest an ontology file.
+
+        Args:
+            file_path: Path to the ontology file (string or Path object)
+            format: Optional format hint (e.g., 'turtle', 'xml'). If None, rdflib guesses.
+            **kwargs: Additional arguments for rdflib parsing
+
+        Returns:
+            OntologyData object containing the parsed ontology and metadata
+        """
+        file_path = Path(file_path)
+
+        # Track file ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(file_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Ontology: {file_path.name}",
+        )
+
+        try:
+            # Validate file exists
+            if not file_path.exists():
+                raise ValidationError(f"File not found: {file_path}")
+
+            self.progress.update_tracking(tracking_id, message="Parsing RDF graph...")
+            g = Graph()
+            
+            # Use provided format or let rdflib guess based on extension
+            parse_kwargs = kwargs.copy()
+            if format:
+                parse_kwargs['format'] = format
+            
+            try:
+                g.parse(file_path, **parse_kwargs)
+            except Exception as e:
+                # Fallback: try to guess format from extension if not provided and initial parse failed
+                if not format:
+                    ext = os.path.splitext(file_path)[1].lower()
+                    fmt_map = {
+                        '.ttl': 'turtle',
+                        '.owl': 'xml', # OWL is often XML
+                        '.rdf': 'xml',
+                        '.jsonld': 'json-ld',
+                        '.n3': 'n3',
+                        '.nt': 'nt'
+                    }
+                    guessed_fmt = fmt_map.get(ext)
+                    if guessed_fmt:
+                        self.logger.info(f"Retrying with guessed format: {guessed_fmt}")
+                        g.parse(file_path, format=guessed_fmt, **kwargs)
+                    else:
+                        raise e
+                else:
+                    raise e
+
+            self.progress.update_tracking(tracking_id, message="Converting to internal format...")
+            
+            # Determine format for metadata
+            used_format = format
+            if not used_format:
+                ext = os.path.splitext(file_path)[1].lower()
+                fmt_map = {
+                    '.ttl': 'turtle',
+                    '.owl': 'xml',
+                    '.rdf': 'xml',
+                    '.jsonld': 'json-ld',
+                    '.n3': 'n3',
+                    '.nt': 'nt'
+                }
+                used_format = fmt_map.get(ext, 'unknown')
+
+            ontology_dict = self._convert_to_dict(g, source_path=str(file_path), format=used_format)
+
+            ontology_data = OntologyData(
+                data=ontology_dict,
+                source_path=str(file_path),
+                format=used_format,
+                metadata=ontology_dict.get("metadata", {}).copy()
+            )
+
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Successfully ingested ontology from {file_path}",
+            )
+            
+            return ontology_data
+
+        except Exception as e:
+            self.logger.error(f"Failed to ingest ontology: {str(e)}")
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise ProcessingError(f"Failed to ingest ontology: {str(e)}") from e
+
+    def ingest_directory(self, directory_path: Union[str, Path], recursive: bool = True, **kwargs) -> List[OntologyData]:
+        """
+        Ingest all ontology files in a directory.
+
+        Args:
+            directory_path: Path to the directory (string or Path object)
+            recursive: Whether to search recursively
+            **kwargs: Additional arguments
+
+        Returns:
+            List of OntologyData objects
+        """
+        directory_path = Path(directory_path)
+        ontologies = []
+        extensions = {'.ttl', '.owl', '.rdf', '.jsonld', '.n3', '.nt'}
+        
+        # Track directory ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(directory_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Directory: {directory_path.name}",
+        )
+
+        try:
+            if not directory_path.exists():
+                raise ValidationError(f"Directory not found: {directory_path}")
+                
+            if not directory_path.is_dir():
+                raise ValidationError(f"Path is not a directory: {directory_path}")
+
+            files_to_process = []
+            for root, _, files in os.walk(directory_path):
+                for file in files:
+                    ext = os.path.splitext(file)[1].lower()
+                    if ext in extensions:
+                        files_to_process.append(os.path.join(root, file))
+                
+                if not recursive:
+                    break
+            
+            total_files = len(files_to_process)
+            self.progress.update_tracking(
+                tracking_id, message=f"Processing {total_files} ontology files"
+            )
+
+            for idx, file_path in enumerate(files_to_process, 1):
+                try:
+                    ont_data = self.ingest_ontology(file_path, **kwargs)
+                    ontologies.append(ont_data)
+                    
+                    self.progress.update_progress(
+                        tracking_id,
+                        processed=idx,
+                        total=total_files,
+                        message=f"Processing {idx}/{total_files}: {Path(file_path).name}"
+                    )
+                except Exception as e:
+                    self.logger.warning(f"Skipping {file_path}: {e}")
+            
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Ingested {len(ontologies)} ontologies",
+            )
+            return ontologies
+
+        except Exception as e:
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
+    def _convert_to_dict(self, graph: Graph, source_path: str, format: str = "unknown") -> Dict[str, Any]:
+        """
+        Convert rdflib Graph to Semantica ontology dictionary.
+
+        Args:
+            graph: Parsed rdflib Graph
+            source_path: Source file path
+            format: Format of the ontology file
+
+        Returns:
+            Ontology dictionary
+        """
+        ontology = {
+            "uri": "",
+            "name": os.path.basename(source_path),
+            "version": "1.0",
+            "classes": [],
+            "properties": [],
+            "metadata": {
+                "source_path": source_path,
+                "ingested_at": datetime.now().isoformat(),
+                "format": format
+            }
+        }
+
+        # 1. Extract Ontology Metadata
+        for s, p, o in graph.triples((None, RDF.type, OWL.Ontology)):
+            ontology["uri"] = str(s)
+            
+            # Try to find label/comment/versionInfo
+            for _, _, label in graph.triples((s, RDFS.label, None)):
+                ontology["name"] = str(label)
+                
+            for _, _, comment in graph.triples((s, RDFS.comment, None)):
+                ontology["description"] = str(comment)
+                
+            for _, _, version in graph.triples((s, OWL.versionInfo, None)):
+                ontology["version"] = str(version)
+            
+            # Break after first ontology definition found (usually only one per file)
+            break
+            
+        # 2. Extract Classes
+        classes = {}
+        # Union of owl:Class and rdfs:Class
+        class_types = [OWL.Class, RDFS.Class]
+        for c_type in class_types:
+            for s, p, o in graph.triples((None, RDF.type, c_type)):
+                if isinstance(s, rdflib.BNode):
+                    continue # Skip blank nodes for now
+                    
+                uri = str(s)
+                if uri not in classes:
+                    cls_def = {
+                        "uri": uri,
+                        "name": self._get_local_name(uri),
+                        "type": "class"
+                    }
+                    
+                    # Add label/comment
+                    label = graph.value(s, RDFS.label)
+                    if label:
+                        cls_def["label"] = str(label)
+                        cls_def["name"] = str(label) # Prefer label as name if available? Or keep URI fragment? 
+                        # Keeping local name from URI is safer for internal IDs, label for display. 
+                        # But Semantica seems to use "name" for the identifier in some examples.
+                        # Let's keep name as local name or label if simple.
+                    
+                    comment = graph.value(s, RDFS.comment)
+                    if comment:
+                        cls_def["description"] = str(comment)
+                        
+                    # Superclasses
+                    parents = []
+                    for _, _, parent in graph.triples((s, RDFS.subClassOf, None)):
+                        if isinstance(parent, rdflib.URIRef):
+                            parents.append(str(parent))
+                    if parents:
+                        cls_def["parents"] = parents
+                        
+                    classes[uri] = cls_def
+        
+        ontology["classes"] = list(classes.values())
+
+        # 3. Extract Properties
+        properties = {}
+        # Object Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.ObjectProperty)):
+            self._add_property(graph, s, "object", properties)
+            
+        # Datatype Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.DatatypeProperty)):
+            self._add_property(graph, s, "data", properties)
+            
+        # RDF Properties (generic)
+        for s, p, o in graph.triples((None, RDF.type, RDF.Property)):
+            if str(s) not in properties: # Don't overwrite if already found as specific type
+                self._add_property(graph, s, "annotation", properties) # Default to annotation or generic
+        
+        ontology["properties"] = list(properties.values())
+        
+        return ontology
+
+    def _add_property(self, graph: Graph, subject: rdflib.term.Node, prop_type: str, properties_dict: Dict):
+        if isinstance(subject, rdflib.BNode):
+            return
+
+        uri = str(subject)
+        if uri in properties_dict:
+            return
+            
+        prop_def = {
+            "uri": uri,
+            "name": self._get_local_name(uri),
+            "type": prop_type
+        }
+        
+        label = graph.value(subject, RDFS.label)
+        if label:
+            prop_def["label"] = str(label)
+            
+        comment = graph.value(subject, RDFS.comment)
+        if comment:
+            prop_def["description"] = str(comment)
+            
+        # Domain and Range
+        domain = graph.value(subject, RDFS.domain)
+        if domain and isinstance(domain, rdflib.URIRef):
+            prop_def["domain"] = str(domain)
+            
+        range_val = graph.value(subject, RDFS.range)
+        if range_val and isinstance(range_val, rdflib.URIRef):
+            prop_def["range"] = str(range_val)
+            
+        properties_dict[uri] = prop_def
+
+    def _get_local_name(self, uri: str) -> str:
+        """Extract local name from URI."""
+        if '#' in uri:
+            return uri.split('#')[-1]
+        return uri.split('/')[-1]

semantica/kg/graph_builder.py

@@ -161,7 +161,14 @@ class GraphBuilder:
             }
             all_relationships.append(rel_dict)
         elif isinstance(item, dict):
-            # Detect and normalize Entity objects inside dict
+            if "source_id" in item and "source" not in item:
+                item["source"] = item["source_id"]
+            if "target_id" in item and "target" not in item:
+                item["target"] = item["target_id"]
+            if "subject" in item and "source" not in item:
+                item["source"] = item["subject"]
+            if "object" in item and "target" not in item:
+                item["target"] = item["object"]
             if "source" in item and not isinstance(item["source"], str):
                 src = item["source"]
                 item["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -347,6 +354,21 @@ class GraphBuilder:
         elif not isinstance(sources, list):
             sources = [sources]
 
+        # Count input relationships for warning if all are dropped
+        input_relationships_count = 0
+        if isinstance(source_dict, dict):
+            rels = source_dict.get("relationships", [])
+            if isinstance(rels, list):
+                input_relationships_count += len(rels)
+            elif rels is not None:
+                input_relationships_count += 1
+        if explicit_relationships:
+            for rel_item in explicit_relationships:
+                if isinstance(rel_item, list):
+                    input_relationships_count += len(rel_item)
+                else:
+                    input_relationships_count += 1
+
         # Track graph building
         build_start_time = time.time()
         
@@ -468,11 +490,12 @@ class GraphBuilder:
                         pipeline_id=pipeline_id,
                     )
                     
-                    # Check if relationships are already in dictionary format
                     sample_rel = relationships_list[0] if relationships_list else None
                     is_dict_format = isinstance(sample_rel, dict) and (
-                        "source" in sample_rel and "target" in sample_rel
-                    ) and not hasattr(sample_rel, "__dict__") # Ensure it's not a class instance
+                        ("source" in sample_rel and "target" in sample_rel)
+                        or ("source_id" in sample_rel and "target_id" in sample_rel)
+                        or ("subject" in sample_rel and "object" in sample_rel)
+                    ) and not hasattr(sample_rel, "__dict__")
                     
                     if is_dict_format:
                         # Fast path: directly append dictionaries after normalizing source/target
@@ -481,8 +504,15 @@ class GraphBuilder:
                             batch = relationships_list[i:i + batch_size]
                             for item in batch:
                                 if isinstance(item, dict):
-                                    # Normalize source/target if they are objects
                                     rel_dict = item.copy()
+                                    if "source_id" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["source_id"]
+                                    if "target_id" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["target_id"]
+                                    if "subject" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["subject"]
+                                    if "object" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["object"]
                                     if "source" in rel_dict and not isinstance(rel_dict["source"], str):
                                         src = rel_dict["source"]
                                         rel_dict["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -571,6 +601,14 @@ class GraphBuilder:
                     f"Entity resolution complete: {len(all_entities)} -> {len(resolved_entities)} unique entities"
                 )
 
+            if input_relationships_count > 0 and len(all_relationships) == 0:
+                warning_msg = (
+                    f"All relationships were dropped during graph building: "
+                    f"{input_relationships_count} input relationships, 0 in final graph"
+                )
+                self.logger.warning(warning_msg)
+                print(f"Warning: {warning_msg}")
+
             # Build graph structure
             print("Building graph structure...")
             structure_start = time.time()
@@ -682,6 +720,14 @@ class GraphBuilder:
             )
             raise
 
+    def build_single_source(
+        self,
+        kg_data: Dict[str, Any],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Dict[str, Any]:
+        return self.build(kg_data, pipeline_id=pipeline_id, **options)
+
     def add_temporal_edge(
         self,
         graph,

semantica/ontology/__init__.py

@@ -109,12 +109,14 @@ Convenience Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory
 
 Example Usage:
-    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator
+    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator, ingest_ontology
     >>> # Using convenience functions
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
     >>> # Using classes directly
     >>> from semantica.ontology import OntologyGenerator, ClassInferrer, PropertyGenerator
     >>> generator = OntologyGenerator(base_uri="https://example.org/ontology/")
@@ -155,6 +157,8 @@ from .registry import MethodRegistry, method_registry
 from .requirements_spec import RequirementsSpec, RequirementsSpecManager
 from .reuse_manager import ReuseDecision, ReuseManager
 from .version_manager import OntologyVersion, VersionManager
+from semantica.ingest import OntologyData, OntologyIngestor
+from .methods import ingest_ontology
 
 __all__ = [
     # Main generators
@@ -200,4 +204,7 @@ __all__ = [
     # Configuration
     "OntologyConfig",
     "ontology_config",
+    "ingest_ontology",
+    "OntologyData",
+    "OntologyIngestor",
 ]

semantica/ontology/methods.py

@@ -111,15 +111,19 @@ Main Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory (via semantica.ingest)
 
 Example Usage:
-    >>> from semantica.ontology.methods import generate_ontology, infer_classes
+    >>> from semantica.ontology.methods import generate_ontology, infer_classes, ingest_ontology
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
 """
 
-from typing import Any, Callable, Dict, List, Optional
+from typing import Any, Callable, Dict, List, Optional, Union
+from pathlib import Path
 
+from semantica.ingest import ingest_ontology as _ingest_ontology, OntologyData
 from .registry import method_registry
 
 
@@ -172,4 +176,23 @@ def list_available_methods(task: Optional[str] = None) -> Dict[str, List[str]]:
     return method_registry.list_all(task)
 
 
-pass
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], 
+    method: str = "file", 
+    **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source.
+    
+    This is a convenience wrapper around semantica.ingest.ingest_ontology.
+    
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+        **kwargs: Additional options
+        
+    Returns:
+        OntologyData or List[OntologyData]
+    """
+    return _ingest_ontology(source, method=method, **kwargs)
+

semantica/ontology/ontology_usage.md

@@ -42,6 +42,18 @@ classes = inferrer.infer_classes(entities, build_hierarchy=True)
 properties = prop_gen.infer_properties(entities, relationships, classes)
 ```
 
+### Ingesting Ontologies
+
+```python
+from semantica.ingest import OntologyIngestor
+
+# Create ingestor
+ingestor = OntologyIngestor()
+
+# Ingest ontology
+ontology_data = ingestor.ingest_ontology("ontology.ttl")
+```
+
 ## Ontology Generation
 
 ### Basic Ontology Generation
@@ -115,6 +127,49 @@ ontology = engine.from_data(
 )
 ```
 
+## Ontology Ingestion
+
+### Basic Ingestion
+
+Ingest existing ontologies from files (Turtle, RDF/XML, JSON-LD, etc.) into `OntologyData` objects.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a single file
+ontology_data = ingest_ontology("path/to/ontology.ttl")
+
+print(f"Source: {ontology_data.source_path}")
+print(f"Format: {ontology_data.format}")
+print(f"Data keys: {ontology_data.data.keys()}")
+```
+
+### Ingesting Directories
+
+Ingest all ontology files in a directory recursively.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a directory
+ontologies = ingest_ontology("path/to/ontologies_dir/")
+
+for ont in ontologies:
+    print(f"Ingested: {ont.source_path} ({ont.format})")
+```
+
+### Unified Ingestion Interface
+
+You can also use the unified `semantica.ingest` interface.
+
+```python
+from semantica.ingest import ingest
+
+# Ingest as "ontology" source type
+result = ingest("path/to/ontology.ttl", source_type="ontology")
+ontology_data = result["ontology"]
+```
+
 ## Class Inference
 
 ### Basic Class Inference

semantica/ontology/reuse_manager.py

@@ -363,3 +363,90 @@ class ReuseManager:
     def list_known_ontologies(self) -> List[str]:
         """List known ontology URIs."""
         return list(self.known_ontologies.keys())
+
+    def merge_ontology_data(
+        self, target: Dict[str, Any], source: Dict[str, Any], **options
+    ) -> Dict[str, Any]:
+        """
+        Merge source ontology data into target ontology.
+
+        Merges classes, properties, and metadata from source to target.
+        Handles deduplication based on URI and name.
+
+        Args:
+            target: Target ontology dictionary (modified in-place)
+            source: Source ontology dictionary
+            **options: Merge options:
+                - overwrite: Whether to overwrite existing elements (default: False)
+                - merge_metadata: Whether to merge metadata (default: True)
+
+        Returns:
+            Merged target ontology
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="ontology",
+            submodule="ReuseManager",
+            message=f"Merging ontology {source.get('name', 'unknown')} into {target.get('name', 'unknown')}",
+        )
+
+        try:
+            overwrite = options.get("overwrite", False)
+            
+            # Helper to merge lists of dicts (classes/properties)
+            def merge_lists(target_list, source_list, key_field="uri"):
+                existing_keys = {item.get(key_field): i for i, item in enumerate(target_list) if item.get(key_field)}
+                
+                for item in source_list:
+                    key = item.get(key_field)
+                    if not key:
+                        # Fallback to name if URI missing
+                        key = item.get("name")
+                        
+                    if key in existing_keys:
+                        if overwrite:
+                            target_list[existing_keys[key]] = item
+                    else:
+                        target_list.append(item)
+                        if key:
+                            existing_keys[key] = len(target_list) - 1
+
+            # Merge Classes
+            if "classes" in source:
+                if "classes" not in target:
+                    target["classes"] = []
+                merge_lists(target["classes"], source["classes"])
+
+            # Merge Properties
+            if "properties" in source:
+                if "properties" not in target:
+                    target["properties"] = []
+                merge_lists(target["properties"], source["properties"])
+
+            # Merge Metadata
+            if options.get("merge_metadata", True) and "metadata" in source:
+                if "metadata" not in target:
+                    target["metadata"] = {}
+                # Update with source metadata, preserving target's specific fields if needed
+                # Here we just update
+                target["metadata"].update(source["metadata"])
+                
+            # Merge Imports
+            if "imports" in source:
+                if "imports" not in target:
+                    target["imports"] = []
+                for imp in source["imports"]:
+                    if imp not in target["imports"]:
+                        target["imports"].append(imp)
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Merged ontology data successfully",
+            )
+            return target
+
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise

semantica/pipeline/pipeline_builder.py

@@ -32,12 +32,14 @@ License: MIT
 
 from dataclasses import dataclass, field
 from enum import Enum
-from typing import Any, Callable, Dict, List, Optional, Union
+from typing import Any, Callable, Dict, List, Optional, Union, TYPE_CHECKING
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
-from .pipeline_validator import PipelineValidator
+
+if TYPE_CHECKING:
+    from .pipeline_validator import PipelineValidator
 
 
 class StepStatus(Enum):
@@ -104,6 +106,8 @@ class PipelineBuilder:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        from .pipeline_validator import PipelineValidator
+
         self.validator = PipelineValidator(**self.config)
         self.steps: List[PipelineStep] = []
         self.step_registry: Dict[str, Callable] = {}

semantica/semantic_extract/coreference_resolver.py

@@ -240,6 +240,12 @@ class CoreferenceResolver:
             self.progress_tracker.stop_tracking(
                 tracking_id, status="failed", message=str(e)
             )
+            verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+            if verbose_mode:
+                import sys
+                print(f"    [CoreferenceResolver] ERROR: Resolution failed: {e}", flush=True, file=sys.stderr)
+                import traceback
+                traceback.print_exc(file=sys.stderr)
             raise
 
     def resolve(

semantica/semantic_extract/llm_extraction.py

@@ -108,7 +108,12 @@ class LLMExtraction:
 
         # Initialize provider using new system
         try:
-            self.provider = create_provider(provider, **config)
+            # Sanitize config: remove api_key if it's None/empty to allow fallback
+            provider_config = config.copy()
+            if "api_key" in provider_config and not provider_config["api_key"]:
+                del provider_config["api_key"]
+                
+            self.provider = create_provider(provider, **provider_config)
         except Exception as e:
             self.logger.warning(f"Failed to initialize {provider} provider: {e}")
             self.provider = None

semantica/semantic_extract/methods.py

@@ -19,6 +19,7 @@ Relation Extraction:
     - "pattern": Pattern-based relation extraction
     - "regex": Advanced regex-based relation extraction
     - "cooccurrence": Co-occurrence based relation detection
+    - "similarity": Similarity-based relation extraction
     - "dependency": Dependency parsing-based relation extraction
     - "huggingface": HuggingFace relation extraction models
     - "llm": LLM-based relation extraction
@@ -717,24 +718,148 @@ def extract_entities_huggingface(
     device: Optional[str] = None,
     **kwargs,
 ) -> List[Entity]:
-    """HuggingFace entity extraction."""
+    """
+    Extract entities using HuggingFace transformers.
+    
+    Args:
+        text: Input text
+        model: Model name or path
+        device: Device to use (cpu, cuda, mps)
+        **kwargs: Additional arguments passed to the pipeline (e.g., aggregation_strategy)
+    """
     loader = HuggingFaceModelLoader(device=device)
-    model_obj = loader.load_ner_model(model)
+    # Pass kwargs (like aggregation_strategy) to load_ner_model
+    model_obj = loader.load_ner_model(model, **kwargs)
     results = loader.extract_entities(model_obj, text)
 
     entities = []
-    for result in results:
-        if isinstance(result, dict):
-            entities.append(
-                Entity(
-                    text=result.get("word", result.get("entity", "")),
-                    label=result.get("entity_group", result.get("label", "UNKNOWN")),
-                    start_char=result.get("start", 0),
-                    end_char=result.get("end", 0),
-                    confidence=result.get("score", 1.0),
-                    metadata={"model": model, "extraction_method": "huggingface"},
+    
+    # Check if manual aggregation is needed (raw IOB tags detected)
+    needs_manual_aggregation = False
+    if results and isinstance(results[0], dict):
+        first_label = results[0].get("label", "")
+        # If we see B- tags and no entity_group (which implies aggregation wasn't done), we aggregate manually
+        if (first_label.startswith("B-") or first_label.startswith("I-")) and "entity_group" not in results[0]:
+            needs_manual_aggregation = True
+
+    if needs_manual_aggregation:
+        current_entity = None
+        for result in results:
+            label = result.get("label", "")
+            word = result.get("word", result.get("entity", ""))
+            score = result.get("score", 1.0)
+            start = result.get("start", 0)
+            end = result.get("end", 0)
+            
+            # Clean word (handle BERT ## and RoBERTa Ġ)
+            clean_word = word.replace("##", "").replace("Ġ", "")
+            if not clean_word:
+                continue
+
+            # Determine tag type and entity type
+            tag_prefix = label[:2] if len(label) > 2 else ""
+            entity_type = label[2:] if len(label) > 2 else label
+
+            if tag_prefix == "B-":
+                # Save previous entity
+                if current_entity:
+                    entities.append(current_entity)
+                
+                # Start new entity
+                current_entity = Entity(
+                    text=clean_word,
+                    label=entity_type,
+                    start_char=start,
+                    end_char=end,
+                    confidence=score,
+                    metadata={
+                        "model": model, 
+                        "extraction_method": "huggingface",
+                        "source": "huggingface",
+                        "raw_iob": True
+                    },
                 )
-            )
+            
+            elif tag_prefix == "I-" and current_entity:
+                # Check if type matches (loose check allows for some noise, strict check enforces type)
+                # We'll be lenient and allow continuation if it makes sense contextually, 
+                # but ideally types should match.
+                if current_entity.label == entity_type:
+                    # Append text
+                    # Use offsets to determine spacing
+                    if start > current_entity.end_char:
+                        # If there's a gap, add space (unless it was a subword that got split but has gap? Unlikely)
+                        # Usually gap means space.
+                        # However, for ## subwords, start usually equals end.
+                        # For Ġ, it implies space.
+                        current_entity.text += " " + clean_word
+                    else:
+                        current_entity.text += clean_word
+                    
+                    current_entity.end_char = end
+                    # Update confidence (average)
+                    current_entity.confidence = (current_entity.confidence + score) / 2
+                else:
+                    # Type mismatch - treat as new entity or ignore?
+                    # Treating as new B- is safer to avoid losing data
+                    if current_entity:
+                        entities.append(current_entity)
+                    
+                    current_entity = Entity(
+                        text=clean_word,
+                        label=entity_type,
+                        start_char=start,
+                        end_char=end,
+                        confidence=score,
+                        metadata={
+                            "model": model, 
+                            "extraction_method": "huggingface",
+                            "source": "huggingface",
+                            "raw_iob": True
+                        },
+                    )
+            
+            else:
+                # O tag or I- without B or other cases
+                if current_entity:
+                    entities.append(current_entity)
+                    current_entity = None
+        
+        # Append last entity
+        if current_entity:
+            entities.append(current_entity)
+
+    else:
+        # Standard processing for aggregated results or simple output
+        for result in results:
+            if isinstance(result, dict):
+                # Handle different output formats based on aggregation strategy
+                label = result.get("entity_group", result.get("label", "UNKNOWN"))
+                text_content = result.get("word", result.get("entity", ""))
+                
+                # Clean up text content (remove ## for subwords if raw)
+                if "##" in text_content and "aggregation_strategy" not in kwargs:
+                     text_content = text_content.replace("##", "")
+                if "Ġ" in text_content: # RoBERTa
+                     text_content = text_content.replace("Ġ", " ").strip()
+                     
+                entities.append(
+                    Entity(
+                        text=text_content,
+                        label=label,
+                        start_char=result.get("start", 0),
+                        end_char=result.get("end", 0),
+                        confidence=result.get("score", 1.0),
+                        metadata={
+                            "model": model, 
+                            "extraction_method": "huggingface",
+                            "source": "huggingface"
+                        },
+                    )
+                )
+            elif isinstance(result, list):
+                 # Handle list of lists (sometimes returned by pipeline)
+                 pass
 
     return entities
 
@@ -786,7 +911,7 @@ def extract_entities_llm(
 
     # Pass api_key if provided in kwargs (needed for all providers)
     provider_kwargs = kwargs.copy()
-    if "api_key" not in provider_kwargs:
+    if "api_key" not in provider_kwargs or not provider_kwargs["api_key"]:
         # Try to get from environment as fallback for all providers
         import os
         env_key = f"{provider.upper()}_API_KEY"
@@ -1494,14 +1619,26 @@ def extract_relations_huggingface(
 ) -> List[Relation]:
     """HuggingFace relation extraction."""
     loader = HuggingFaceModelLoader(device=device)
-    model_obj = loader.load_relation_model(model)
+    model_obj = loader.load_relation_model(model, **kwargs)
 
-    # This is simplified - actual implementation would depend on model architecture
-    results = loader.extract_relations(model_obj, text, entities)
+    # Pass kwargs (e.g. threshold)
+    results = loader.extract_relations(model_obj, text, entities, **kwargs)
 
     relations = []
-    # Parse results based on model output format
-    # This is a placeholder - actual parsing would depend on the model
+    for result in results:
+        relations.append(
+            Relation(
+                subject=result["subject"],
+                predicate=result["relation"],
+                object=result["object"],
+                confidence=result.get("score", 1.0),
+                context=text,
+                metadata={
+                    "model": model,
+                    "extraction_method": "huggingface"
+                }
+            )
+        )
     return relations
 
 
@@ -1513,6 +1650,7 @@ def extract_relations_llm(
     silent_fail: bool = False,
     max_text_length: Optional[int] = None,
     structured_output_mode: str = "typed",
+    max_retries: int = 3,
     **kwargs,
 ) -> List[Relation]:
     """
@@ -1525,6 +1663,7 @@ def extract_relations_llm(
         model: LLM model
         silent_fail: If True, return empty list on error. If False (default), raise exception.
         max_text_length: Maximum text length before auto-chunking. None = provider default.
+        max_retries: Maximum number of retries for LLM calls (default: 3)
         **kwargs: Additional options
     """
     # Support llm_model parameter to disambiguate from ML model
@@ -1537,6 +1676,7 @@ def extract_relations_llm(
         "model": model,
         "max_text_length": max_text_length,
         "structured_output_mode": structured_output_mode,
+        "max_retries": max_retries,
         "relation_types": kwargs.get("relation_types"),
         # Include entities hash/str in cache key implicitly via **cache_params
         "entities_hash": hash(tuple(sorted([e.text for e in entities]))) if entities else 0
@@ -1563,18 +1703,24 @@ def extract_relations_llm(
 
     # Pass api_key if provided in kwargs
     provider_kwargs = kwargs.copy()
-    if "api_key" not in provider_kwargs:
+    
+    # Check if api_key is provided but empty, or not provided at all
+    if "api_key" not in provider_kwargs or not provider_kwargs["api_key"]:
         import os
         env_key = f"{provider.upper()}_API_KEY"
         api_key = os.getenv(env_key)
         if api_key:
             provider_kwargs["api_key"] = api_key
+            
+    # Remove None/empty API key if still present to avoid provider errors
+    if "api_key" in provider_kwargs and not provider_kwargs["api_key"]:
+        del provider_kwargs["api_key"]
 
     # 2. PROVIDER VALIDATION
     try:
         llm = create_provider(provider, model=model, **provider_kwargs)
         if not llm.is_available():
-            error_msg = f"{provider} provider not available for relation extraction."
+            error_msg = f"{provider} provider not available for relation extraction (key missing?)."
             logger.error(error_msg)
             if not silent_fail:
                 raise ProcessingError(error_msg)
@@ -1602,16 +1748,13 @@ def extract_relations_llm(
         return _extract_relations_chunked(
             text, entities, provider=provider, model=model, 
             silent_fail=silent_fail, max_text_length=max_text_length, 
+            max_retries=max_retries,
             **kwargs
         )
 
     original_entities = entities
-    max_entities_prompt = kwargs.get("max_entities_prompt", kwargs.get("max_entities", 80))
-    try:
-        max_entities_prompt = int(max_entities_prompt)
-    except Exception:
-        max_entities_prompt = 80
-
+    # Use a fixed internal default for prompt entity cap (do not accept overrides from kwargs)
+    max_entities_prompt = 80
     prompt_entities = original_entities
     if max_entities_prompt > 0 and len(original_entities) > max_entities_prompt:
         prompt_entities = filter_entities_for_text(
@@ -1635,6 +1778,11 @@ If a relation doesn't fit any of the preferred types, use the most appropriate t
 Extract meaningful relationships between entities. Use appropriate relation types that accurately describe how entities are connected.
 Common relation types include: related_to, part_of, located_in, created_by, uses, depends_on, interacts_with, and similar variations."""
     
+    verbose_mode = kwargs.get("verbose", False)
+    if verbose_mode:
+        import sys
+        print(f"    [methods.extract_relations_llm] Constructing prompt for {len(prompt_entities)} entities...", flush=True, file=sys.stdout)
+    
     if not SCHEMAS_AVAILABLE:
         raise ImportError("Pydantic schemas not available. Install pydantic/instructor to use LLM extraction.")
 
@@ -1663,29 +1811,67 @@ Entities found in text: {entities_str}"""
     try:
         # Use typed generation with Pydantic schema
         # Pass kwargs to allow max_tokens and other parameters to be used
-        result_obj = llm.generate_typed(prompt, schema=RelationsResponse, **kwargs)
+        if verbose_mode:
+             import sys
+             print(f"    [methods.extract_relations_llm] Calling llm.generate_typed ({provider}/{model})...", flush=True, file=sys.stdout)
+        # Only forward minimal, safe parameters to provider calls
+        call_kwargs = {}
+        if "temperature" in kwargs:
+            call_kwargs["temperature"] = kwargs["temperature"]
+        if "verbose" in kwargs:
+            call_kwargs["verbose"] = kwargs["verbose"]
         
-        # Convert back to internal Relation format
-        relations = []
-        for r_out in result_obj.relations:
-            # Find matching entities using hybrid similarity
-            subject_entity = match_entity(r_out.subject, original_entities)
-            object_entity = match_entity(r_out.object, original_entities)
-            
-            if subject_entity and object_entity:
-                relations.append(Relation(
-                    subject=subject_entity,
-                    predicate=r_out.predicate,
-                    object=object_entity,
-                    confidence=r_out.confidence,
-                    context=text, # Simplified context
-                    metadata={
-                        "provider": provider, 
-                        "model": model, 
-                        "extraction_method": "llm_typed"
-                    }
-                ))
+        call_kwargs["max_retries"] = max_retries
+
+        result_obj = llm.generate_typed(prompt, schema=RelationsResponse, **call_kwargs)
+        if verbose_mode:
+             import sys
+             print(f"    [methods.extract_relations_llm] Received response from {provider}.", flush=True, file=sys.stdout)
         
+        # Convert back to internal Relation format (robust across providers)
+        # Normalize typed result to a plain dict compatible with _parse_relation_result
+        try:
+            if hasattr(result_obj, "model_dump"):
+                parsed = result_obj.model_dump()
+            elif isinstance(result_obj, dict):
+                parsed = result_obj
+            elif hasattr(result_obj, "relations"):
+                # Instructor may return objects for each relation; convert where possible
+                rel_items = []
+                for r in getattr(result_obj, "relations", []):
+                    if hasattr(r, "model_dump"):
+                        rel_items.append(r.model_dump())
+                    elif isinstance(r, dict):
+                        rel_items.append(r)
+                    else:
+                        # Best-effort attribute access
+                        rel_items.append({
+                            "subject": getattr(r, "subject", ""),
+                            "object": getattr(r, "object", ""),
+                            "predicate": getattr(r, "predicate", "related_to"),
+                            "confidence": getattr(r, "confidence", 0.9),
+                        })
+                parsed = {"relations": rel_items}
+            else:
+                parsed = result_obj
+        except Exception:
+            parsed = result_obj
+
+        # Use common parser to build internal Relation objects
+        relations = _parse_relation_result(parsed, original_entities, text, provider, model)
+        
+        # If typed path returned no relations, attempt a structured JSON fallback
+        if not relations:
+            try:
+                if verbose_mode:
+                    import sys
+                    print("    [methods.extract_relations_llm] Typed result empty, attempting structured JSON fallback...", flush=True, file=sys.stdout)
+                raw_json = llm.generate_structured(prompt, **call_kwargs)
+                relations = _parse_relation_result(raw_json, original_entities, text, provider, model)
+            except Exception as _e:
+                # Keep relations as empty if fallback fails
+                pass
+
         logger.info(f"Successfully extracted {len(relations)} relations using {provider}/{model} (typed)")
         _result_cache.set("relations", text, relations, **cache_params)
         return relations
@@ -1782,6 +1968,7 @@ def _extract_relations_chunked(
     silent_fail: bool,
     max_text_length: int,
     structured_output_mode: str = "typed",
+    max_retries: int = 3,
     **kwargs
 ) -> List[Relation]:
     """Internal helper to extract relations from long text by chunking."""
@@ -1813,6 +2000,8 @@ def _extract_relations_chunked(
                 
             logger.debug(f"Scheduling relation extraction for chunk {i+1}/{len(chunks)} with {len(chunk_entities)} entities")
             
+            # Only pass minimal kwargs downstream
+            limited_kwargs = {k: kwargs[k] for k in ("relation_types", "temperature", "verbose") if k in kwargs}
             future = executor.submit(
                 extract_relations_llm,
                 chunk.text,
@@ -1822,7 +2011,8 @@ def _extract_relations_chunked(
                 silent_fail=False,
                 max_text_length=len(chunk.text) + 1,
                 structured_output_mode=structured_output_mode,
-                **kwargs
+                max_retries=max_retries,
+                **limited_kwargs
             )
             future_to_chunk[future] = i
             
@@ -1941,16 +2131,46 @@ def extract_triplets_huggingface(
 ) -> List[Triplet]:
     """HuggingFace triplet extraction."""
     loader = HuggingFaceModelLoader(device=device)
-    model_obj = loader.load_triplet_model(model)
+    model_obj = loader.load_triplet_model(model, **kwargs)
+    
+    # REBEL needs special tokens to be preserved
+    if "skip_special_tokens" not in kwargs:
+        kwargs["skip_special_tokens"] = False
+        
     results = loader.extract_triplets(model_obj, text, **kwargs)
 
     triplets = []
     for result in results:
-        # Parse result based on model output format
-        # This is a placeholder - actual parsing would depend on the model
         if "triplet" in result:
-            # Parse triplet string (format depends on model)
-            pass
+            decoded_text = result["triplet"]
+            
+            # Clean up common special tokens that might interfere or are noise
+            decoded_text = decoded_text.replace("<s>", "").replace("</s>", "").replace("<pad>", "")
+            
+            # Parse REBEL format: <triplet> subject <subj> predicate <obj> object
+            # We use a non-greedy match and lookahead for next triplet or end of string
+            import re
+            pattern = r"<triplet>(?P<head>.*?)<subj>(?P<relation>.*?)<obj>(?P<tail>.*?)(?=<triplet>|$)"
+            
+            matches = re.finditer(pattern, decoded_text)
+            for match in matches:
+                head = match.group("head").strip()
+                relation = match.group("relation").strip()
+                tail = match.group("tail").strip()
+                
+                if head and relation and tail:
+                    triplets.append(
+                        Triplet(
+                            subject=head,
+                            predicate=relation,
+                            object=tail,
+                            confidence=0.9, # Model generation doesn't provide per-triplet confidence
+                            metadata={
+                                "model": model,
+                                "extraction_method": "huggingface_rebel"
+                            }
+                        )
+                    )
 
     return triplets
 
@@ -1964,6 +2184,7 @@ def extract_triplets_llm(
     silent_fail: bool = False,
     max_text_length: Optional[int] = None,
     structured_output_mode: str = "typed",
+    max_retries: int = 3,
     **kwargs,
 ) -> List[Triplet]:
     """
@@ -1977,6 +2198,7 @@ def extract_triplets_llm(
         model: LLM model
         silent_fail: If True, return empty list on error. If False (default), raise exception.
         max_text_length: Maximum text length before auto-chunking. None = provider default.
+        max_retries: Maximum number of retries for LLM calls (default: 3)
         **kwargs: Additional options
     """
     # Support llm_model parameter to disambiguate from ML model
@@ -1989,6 +2211,7 @@ def extract_triplets_llm(
         "model": model,
         "max_text_length": max_text_length,
         "structured_output_mode": structured_output_mode,
+        "max_retries": max_retries,
         "triplet_types": kwargs.get("triplet_types"),
         # Include entities/relations hash in cache key implicitly via **cache_params
         "entities_hash": hash(tuple(sorted([e.text for e in entities]))) if entities else 0,
@@ -2009,12 +2232,18 @@ def extract_triplets_llm(
 
     # Pass api_key if provided in kwargs
     provider_kwargs = kwargs.copy()
-    if "api_key" not in provider_kwargs:
+    
+    # Check if api_key is provided but empty, or not provided at all
+    if "api_key" not in provider_kwargs or not provider_kwargs["api_key"]:
         import os
         env_key = f"{provider.upper()}_API_KEY"
         api_key = os.getenv(env_key)
         if api_key:
             provider_kwargs["api_key"] = api_key
+            
+    # Remove None/empty API key if still present to avoid provider errors
+    if "api_key" in provider_kwargs and not provider_kwargs["api_key"]:
+        del provider_kwargs["api_key"]
 
     # 2. PROVIDER VALIDATION
     try:
@@ -2048,6 +2277,7 @@ def extract_triplets_llm(
         return _extract_triplets_chunked(
             text, provider=provider, model=model, 
             silent_fail=silent_fail, max_text_length=max_text_length, 
+            max_retries=max_retries,
             **kwargs
         )
     
@@ -2090,7 +2320,9 @@ Text to extract from:
 
     try:
         # Use typed generation with Pydantic schema
-        result_obj = llm.generate_typed(prompt, schema=TripletsResponse, **kwargs)
+        call_kwargs = kwargs.copy()
+        call_kwargs["max_retries"] = max_retries
+        result_obj = llm.generate_typed(prompt, schema=TripletsResponse, **call_kwargs)
         
         # Convert back to internal Triplet format
         triplets = []

semantica/semantic_extract/ner_extractor.py

@@ -364,8 +364,11 @@ class NERExtractor:
                     # Prepare method-specific options
                     method_options = all_options.copy()
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", self.huggingface_model
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model") 
+                            or self.huggingface_model
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -375,14 +378,13 @@ class NERExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key

semantica/semantic_extract/providers.py

@@ -243,68 +243,136 @@ class BaseProvider:
                 mode = instructor.Mode.TOOLS  # Default mode
                 
                 if provider_name == "OpenAIProvider" and self.client:
-                    client = instructor.from_openai(self.client)
-                elif provider_name == "AnthropicProvider" and self.client:
-                    client = instructor.from_anthropic(self.client)
-                elif provider_name == "GeminiProvider" and self.client:
-                    client = instructor.from_gemini(
-                        self.client, 
-                        mode=instructor.Mode.GEMINI_JSON
-                    )
-                elif provider_name == "GroqProvider" and self.client:
-                    # Try using from_groq if available (newer instructor versions)
-                    if hasattr(instructor, "from_groq"):
-                        client = instructor.from_groq(self.client, mode=instructor.Mode.JSON)
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"openai/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_openai(self.client)
                     else:
-                        # Fallback: Create OpenAI client pointing to Groq
-                        # This avoids the "Client should be an instance of openai.OpenAI" warning
+                        client = instructor.from_openai(self.client)
+                elif provider_name == "AnthropicProvider" and self.client:
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"anthropic/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_anthropic(self.client)
+                    else:
+                        client = instructor.from_anthropic(self.client)
+                elif provider_name == "GeminiProvider" and self.client:
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"gemini/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_gemini(
+                                self.client, 
+                                mode=instructor.Mode.GEMINI_JSON
+                            )
+                    else:
+                        client = instructor.from_gemini(
+                            self.client, 
+                            mode=instructor.Mode.GEMINI_JSON
+                        )
+                elif provider_name == "GroqProvider" and self.client:
+                    # Try using from_provider which is recommended for Groq in latest instructor
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"groq/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = None
+
+                    if not client:
+                        # Try using from_groq if available (newer instructor versions)
+                        if hasattr(instructor, "from_groq"):
+                            client = instructor.from_groq(self.client, mode=instructor.Mode.JSON)
+                        else:
+                            # Fallback: Create OpenAI client pointing to Groq
+                            # This avoids the "Client should be an instance of openai.OpenAI" warning
+                            try:
+                                from openai import OpenAI
+                                # Fix: Use self.api_key instead of self.client.api_key
+                                groq_client = OpenAI(
+                                    base_url="https://api.groq.com/openai/v1",
+                                    api_key=self.api_key,
+                                )
+                                client = instructor.from_openai(groq_client, mode=instructor.Mode.JSON)
+                            except Exception:
+                                # Last resort: try passing the groq client directly
+                                client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
+                elif provider_name == "OllamaProvider":
+                    # Try from_provider for Ollama if available
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"ollama/{kwargs.get('model', self.model)}",
+                            )
+                        except Exception:
+                            client = None
+                    
+                    if not client:
+                        # Create OpenAI-compatible client for Ollama
                         try:
                             from openai import OpenAI
-                            groq_client = OpenAI(
-                                base_url="https://api.groq.com/openai/v1",
-                                api_key=self.client.api_key,
+                            # Ollama typically runs on localhost:11434/v1
+                            base_url = getattr(self, "base_url", "http://localhost:11434")
+                            if not base_url.endswith("/v1"):
+                                base_url = f"{base_url.rstrip('/')}/v1"
+                            
+                            ollama_client = OpenAI(
+                                base_url=base_url,
+                                api_key="ollama", # required but unused
                             )
-                            client = instructor.from_openai(groq_client, mode=instructor.Mode.JSON)
-                        except Exception:
-                            # Last resort: try passing the groq client directly
-                            client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
-                elif provider_name == "OllamaProvider":
-                    # Create OpenAI-compatible client for Ollama
-                    try:
-                        from openai import OpenAI
-                        # Ollama typically runs on localhost:11434/v1
-                        base_url = getattr(self, "base_url", "http://localhost:11434")
-                        if not base_url.endswith("/v1"):
-                            base_url = f"{base_url.rstrip('/')}/v1"
-                        
-                        ollama_client = OpenAI(
-                            base_url=base_url,
-                            api_key="ollama", # required but unused
-                        )
-                        client = instructor.from_openai(ollama_client, mode=instructor.Mode.JSON)
-                    except ImportError:
-                        pass
+                            client = instructor.from_openai(ollama_client, mode=instructor.Mode.JSON)
+                        except ImportError:
+                            pass
                 elif provider_name == "DeepSeekProvider" and self.client:
-                    # DeepSeek is OpenAI compatible
-                    # We need to wrap the underlying client if it exposes the OpenAI interface
-                    # or create a new OpenAI client if self.client is a deepseek.Client (which might be just a wrapper)
-                    # Assuming deepseek.Client is compatible or we can use OpenAI client
-                    try:
-                        # DeepSeek usually works with standard OpenAI client
-                        # If self.client is deepseek.Client, check if we can wrap it
-                        # Otherwise create a new OpenAI client
-                        from openai import OpenAI
-                        if isinstance(self.client, OpenAI):
-                             client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
-                        else:
-                             # Try creating fresh client
-                             ds_client = OpenAI(
-                                 api_key=self.api_key, 
-                                 base_url="https://api.deepseek.com"
-                             )
-                             client = instructor.from_openai(ds_client, mode=instructor.Mode.JSON)
-                    except Exception:
-                        pass
+                    # Try from_provider for DeepSeek
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"deepseek/{kwargs.get('model', self.model)}",
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = None
+
+                    if not client:
+                        # DeepSeek is OpenAI compatible
+                        try:
+                            from openai import OpenAI
+                            if isinstance(self.client, OpenAI):
+                                 client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
+                            else:
+                                 # Try creating fresh client
+                                 ds_client = OpenAI(
+                                     api_key=self.api_key, 
+                                     base_url="https://api.deepseek.com"
+                                 )
+                                 client = instructor.from_openai(ds_client, mode=instructor.Mode.JSON)
+                        except Exception:
+                            pass
+
+                # Global LiteLLM support - if litellm is passed in kwargs or config
+                if not client and (kwargs.get("litellm") or self.config.get("litellm")):
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            # Format for litellm in instructor is litellm/model_name
+                            provider_model = kwargs.get("model", self.model)
+                            litellm_provider = f"litellm/{provider_model}"
+                            client = instructor.from_provider(litellm_provider, api_key=self.api_key)
+                        except Exception:
+                            pass
 
                 if client:
                     # Map generate arguments to client arguments
@@ -318,6 +386,11 @@ class BaseProvider:
                         "temperature": kwargs.get("temperature", 0.1), # Low temp for structured
                     }
                     
+                    verbose_mode = kwargs.get("verbose", False)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [BaseProvider.generate_typed] Using instructor via {provider_name}. Client: {type(client)}", flush=True, file=sys.stdout)
+
                     # Pass through other common parameters
                     for param in ["max_tokens", "max_completion_tokens", "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user", "top_k"]:
                         if param in kwargs:
@@ -328,6 +401,9 @@ class BaseProvider:
                         create_kwargs["response_format"] = {"type": "json_object"}
                     
                     response = client.chat.completions.create(**create_kwargs)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [BaseProvider.generate_typed] Typed response received via instructor ({provider_name}).", flush=True, file=sys.stdout)
                     return response
             except Exception as e:
                 self.logger.warning(f"Instructor generation failed ({e}), falling back to manual repair loop.")
@@ -706,7 +782,15 @@ class GroqProvider(BaseProvider):
             if param in kwargs:
                 create_kwargs[param] = kwargs[param]
 
+        verbose_mode = kwargs.get("verbose", False)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate] Sending request to Groq API (model: {create_kwargs['model']})...", flush=True, file=sys.stdout)
+
         response = self.client.chat.completions.create(**create_kwargs)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate] Response received from Groq.", flush=True, file=sys.stdout)
         return response.choices[0].message.content
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -737,7 +821,15 @@ class GroqProvider(BaseProvider):
             if param in kwargs:
                 create_kwargs[param] = kwargs[param]
 
+        verbose_mode = kwargs.get("verbose", False)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate_structured] Sending structured request to Groq API (model: {create_kwargs['model']})...", flush=True, file=sys.stdout)
+
         response = self.client.chat.completions.create(**create_kwargs)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate_structured] Structured response received from Groq.", flush=True, file=sys.stdout)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -1094,25 +1186,32 @@ class HuggingFaceModelLoader:
         # Import torch at method level to ensure it's available
         import torch
         
-        cache_key = f"{model_name}_ner"
+        # Include aggregation_strategy in cache key
+        agg_strategy = kwargs.get("aggregation_strategy", "simple")
+        cache_key = f"{model_name}_ner_{agg_strategy}"
         if cache_key in self._cache:
             return self._cache[cache_key]
 
         try:
             from transformers import pipeline
+        except ImportError:
+            raise ImportError(
+                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
+            )
 
+        try:
             nlp = pipeline(
                 "ner",
                 model=model_name,
                 device=self.device if torch.cuda.is_available() else -1,
-                aggregation_strategy="simple",
+                aggregation_strategy=agg_strategy,
+                tokenizer=kwargs.get("tokenizer") # Allow custom tokenizer
             )
             self._cache[cache_key] = nlp
             return nlp
-        except (ImportError, OSError):
-            raise ImportError(
-                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
-            )
+        except OSError as e:
+            self.logger.error(f"Failed to load NER model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load NER model {model_name}: {e}")
             raise
@@ -1127,19 +1226,31 @@ class HuggingFaceModelLoader:
             return self._cache[cache_key]
 
         try:
-            from transformers import pipeline
-
-            nlp = pipeline(
-                "text-classification",
-                model=model_name,
-                device=self.device if torch.cuda.is_available() else -1,
-            )
-            self._cache[cache_key] = nlp
-            return nlp
-        except (ImportError, OSError):
+            from transformers import pipeline, AutoTokenizer
+        except ImportError:
             raise ImportError(
                 "transformers library not installed. Install with: pip install semantica[models-huggingface]"
             )
+
+        try:
+            # Allow custom tokenizer
+            tokenizer = kwargs.get("tokenizer")
+            if not tokenizer and kwargs.get("tokenizer_name"):
+                tokenizer = AutoTokenizer.from_pretrained(kwargs.get("tokenizer_name"))
+
+            pipeline_kwargs = {
+                "model": model_name,
+                "device": self.device if torch.cuda.is_available() else -1,
+            }
+            if tokenizer:
+                pipeline_kwargs["tokenizer"] = tokenizer
+
+            nlp = pipeline("text-classification", **pipeline_kwargs)
+            self._cache[cache_key] = nlp
+            return nlp
+        except OSError as e:
+            self.logger.error(f"Failed to load relation model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load relation model {model_name}: {e}")
             raise
@@ -1152,18 +1263,27 @@ class HuggingFaceModelLoader:
 
         try:
             from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline
+        except ImportError:
+            raise ImportError(
+                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
+            )
 
-            tokenizer = AutoTokenizer.from_pretrained(model_name)
+        try:
+            # Allow custom tokenizer
+            tokenizer = kwargs.get("tokenizer")
+            if not tokenizer:
+                tokenizer_name = kwargs.get("tokenizer_name", model_name)
+                tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
+            
             model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
             model.to(self.device)
 
             nlp = {"tokenizer": tokenizer, "model": model, "device": self.device}
             self._cache[cache_key] = nlp
             return nlp
-        except (ImportError, OSError):
-            raise ImportError(
-                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
-            )
+        except OSError as e:
+            self.logger.error(f"Failed to load triplet model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load triplet model {model_name}: {e}")
             raise
@@ -1172,10 +1292,84 @@ class HuggingFaceModelLoader:
         """Extract entities using loaded model."""
         return model(text)
 
-    def extract_relations(self, model, text: str, entities: List) -> List[Dict]:
-        """Extract relations using loaded model."""
-        # This would need to be customized based on the model architecture
-        return model(text)
+    def extract_relations(self, model, text: str, entities: List, **kwargs) -> List[Dict]:
+        """
+        Extract relations using loaded model.
+        Iterates through entity pairs and classifies the relationship.
+        """
+        results = []
+        
+        # Sort entities by position
+        sorted_entities = sorted(entities, key=lambda e: e.start_char)
+        
+        # Marker configuration
+        subj_start = kwargs.get("subj_start_marker", "<subj>")
+        subj_end = kwargs.get("subj_end_marker", "</subj>")
+        obj_start = kwargs.get("obj_start_marker", "<obj>")
+        obj_end = kwargs.get("obj_end_marker", "</obj>")
+        
+        # Iterate through all pairs
+        import itertools
+        for i, e1 in enumerate(sorted_entities):
+            for e2 in sorted_entities:
+                if e1 == e2:
+                    continue
+                
+                # Check distance (optional optimization)
+                # if abs(e1.start_char - e2.start_char) > 200: continue
+                
+                # Format text with markers
+                # Strategy: [CLS] text with <subj>...</subj> and <obj>...</obj> [SEP]
+                # We need to insert markers into the original text
+                
+                # Create a copy of text with markers inserted
+                # We need to handle offsets correctly. 
+                # Simplest way: reconstruct string pieces
+                
+                p1_start, p1_end = e1.start_char, e1.end_char
+                p2_start, p2_end = e2.start_char, e2.end_char
+                
+                if p1_start < p2_start:
+                    formatted_text = (
+                        text[:p1_start] + 
+                        f"{subj_start} " + text[p1_start:p1_end] + f" {subj_end}" + 
+                        text[p1_end:p2_start] + 
+                        f"{obj_start} " + text[p2_start:p2_end] + f" {obj_end}" + 
+                        text[p2_end:]
+                    )
+                else:
+                     formatted_text = (
+                        text[:p2_start] + 
+                        f"{obj_start} " + text[p2_start:p2_end] + f" {obj_end}" + 
+                        text[p2_end:p1_start] + 
+                        f"{subj_start} " + text[p1_start:p1_end] + f" {subj_end}" + 
+                        text[p1_end:]
+                    )
+                
+                # Predict
+                try:
+                    # Pipeline returns [{'label': 'LABEL', 'score': 0.99}]
+                    prediction = model(formatted_text, top_k=1)
+                    
+                    if prediction:
+                        res = prediction[0] if isinstance(prediction, list) else prediction
+                        if isinstance(res, list): res = res[0] # top_k=1 returns list of dicts
+                        
+                        label = res.get("label")
+                        score = res.get("score")
+                        
+                        # Filter "no_relation" or low confidence
+                        if label != "no_relation" and score > kwargs.get("threshold", 0.5):
+                            results.append({
+                                "subject": e1,
+                                "object": e2,
+                                "relation": label,
+                                "score": score
+                            })
+                except Exception as e:
+                    self.logger.warning(f"Relation prediction failed for pair {e1.text}-{e2.text}: {e}")
+                    
+        return results
 
     def extract_triplets(self, model, text: str, **kwargs) -> List[Dict]:
         """Extract triplets using loaded model."""
@@ -1187,15 +1381,13 @@ class HuggingFaceModelLoader:
         max_input_length = kwargs.get("max_input_length", 512)
         max_length = kwargs.get("max_length", 128)
         
-        # Allow max_new_tokens as well
         generate_kwargs = {"max_length": max_length}
         if "max_new_tokens" in kwargs:
             generate_kwargs["max_new_tokens"] = kwargs["max_new_tokens"]
-            # If max_new_tokens is set, we might want to remove max_length or ensure they don't conflict
-            # For Seq2Seq, max_length usually refers to the total length of the target sequence
             
-        # Pass other generation args
-        for param in ["num_beams", "temperature", "top_p", "top_k", "do_sample"]:
+        # Pass other generation args including beams and penalties
+        for param in ["num_beams", "temperature", "top_p", "top_k", "do_sample", 
+                      "length_penalty", "repetition_penalty"]:
             if param in kwargs:
                 generate_kwargs[param] = kwargs[param]
 
@@ -1204,10 +1396,10 @@ class HuggingFaceModelLoader:
         ).to(device)
         
         outputs = model_obj.generate(**inputs, **generate_kwargs)
-        decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+        # Allow controlling skip_special_tokens (important for REBEL which uses special tokens for delimiters)
+        skip_special_tokens = kwargs.get("skip_special_tokens", True)
+        decoded = tokenizer.decode(outputs[0], skip_special_tokens=skip_special_tokens)
 
-        # Parse decoded output (format depends on model)
-        # This is a placeholder - actual parsing would depend on model output format
         return [{"triplet": decoded}]
 
 

semantica/semantic_extract/relation_extractor.py

@@ -408,8 +408,12 @@ class RelationExtractor:
                         method_options["relation_types"] = relation_types
 
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", all_options.get("model")
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model")
+                            or self.config.get("huggingface_model") 
+                            or self.config.get("model")
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -419,14 +423,13 @@ class RelationExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key
@@ -440,6 +443,12 @@ class RelationExtractor:
                     if verbose_mode and method_name == "llm":
                         import sys
                         print(f"    [RelationExtractor] Processing with {method_name}...", flush=True, file=sys.stdout)
+                        print(f"    [RelationExtractor Debug] method_options keys: {list(method_options.keys())}", flush=True, file=sys.stdout)
+                        if "api_key" in method_options:
+                            masked = method_options["api_key"][:4] + "..." if method_options["api_key"] else "None"
+                            print(f"    [RelationExtractor Debug] api_key present: {masked}", flush=True, file=sys.stdout)
+                        else:
+                            print(f"    [RelationExtractor Debug] api_key NOT present", flush=True, file=sys.stdout)
                     
                     relations = method_func(text, entities, **method_options)
                     
@@ -482,6 +491,11 @@ class RelationExtractor:
 
                 except Exception as e:
                     self.logger.warning(f"Method {method_name} failed: {e}")
+                    if verbose_mode:
+                        import sys
+                        print(f"    [RelationExtractor] ERROR: Method {method_name} failed: {e}", flush=True, file=sys.stderr)
+                        import traceback
+                        traceback.print_exc(file=sys.stderr)
                     continue
 
             # Use first successful method or combine

semantica/semantic_extract/semantic_extract_usage.md

@@ -113,9 +113,17 @@ extractor = NERExtractor(method="ml")
 entities = extractor.extract(text)
 print(f"ML method: {len(entities)} entities")
 
-# HuggingFace model extraction
+# HuggingFace model extraction (Bring Your Own Model)
 extractor = NERExtractor(method="huggingface")
-entities = extractor.extract(text, model="dslim/bert-base-NER")
+
+# Use a specific model and aggregation strategy at runtime
+# Runtime options override configuration defaults
+entities = extractor.extract(
+    text, 
+    model="dslim/bert-base-NER", 
+    aggregation_strategy="max", # Options: "simple", "first", "average", "max"
+    device="cpu" # or "cuda"
+)
 print(f"HuggingFace method: {len(entities)} entities")
 
 # LLM-based extraction with advanced options
@@ -229,9 +237,18 @@ relations = extractor.extract(text, entities=entities)
 extractor = RelationExtractor(method="cooccurrence")
 relations = extractor.extract(text, entities=entities)
 
-# HuggingFace model
+# HuggingFace model (Bring Your Own Model)
 extractor = RelationExtractor(method="huggingface")
-relations = extractor.extract(text, entities=entities, model="microsoft/DialoGPT-medium")
+
+# Use a sequence classification model trained for relations
+# The extractor automatically formats input with entity markers: 
+# "Steve Jobs founded Apple" -> "<subj> Steve Jobs </subj> founded <obj> Apple </obj>"
+relations = extractor.extract(
+    text, 
+    entities=entities, 
+    model="semantica/relation-model-v1", # Replace with your model ID
+    device="cpu"
+)
 
 # LLM-based relation extraction
 extractor = RelationExtractor(method="llm")
@@ -294,9 +311,16 @@ triplets = extractor.extract_triplets(text)
 extractor = TripletExtractor(method="rules")
 triplets = extractor.extract_triplets(text)
 
-# HuggingFace model
+# HuggingFace model (Seq2Seq / REBEL)
 extractor = TripletExtractor(method="huggingface")
-triplets = extractor.extract_triplets(text, model="t5-base")
+
+# Use a Seq2Seq model like REBEL for end-to-end triplet extraction
+# This method generates triplets directly from text without needing separate NER/RE steps
+triplets = extractor.extract_triplets(
+    text, 
+    model="Babelscape/rebel-large",
+    device="cpu"
+)
 
 # LLM-based triplet extraction
 extractor = TripletExtractor(method="llm")

semantica/semantic_extract/semantic_network_extractor.py

@@ -239,6 +239,10 @@ class SemanticNetworkExtractor:
                         return idx, network
                     except Exception as e:
                         self.logger.warning(f"Failed to process item {idx}: {e}")
+                        verbose_mode = kwargs.get("verbose", False) or self.config.get("verbose", False)
+                        if verbose_mode:
+                            import sys
+                            print(f"    [SemanticNetworkExtractor] ERROR: Batch item {idx} failed: {e}", flush=True, file=sys.stderr)
                         return idx, None
 
                 if max_workers > 1:
@@ -434,6 +438,12 @@ class SemanticNetworkExtractor:
             self.progress_tracker.stop_tracking(
                 tracking_id, status="failed", message=str(e)
             )
+            verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+            if verbose_mode:
+                import sys
+                print(f"    [SemanticNetworkExtractor] ERROR: Extraction failed: {e}", flush=True, file=sys.stderr)
+                import traceback
+                traceback.print_exc(file=sys.stderr)
             raise
 
     def _build_network(

semantica/semantic_extract/triplet_extractor.py

@@ -366,8 +366,17 @@ class TripletExtractor:
             from .ner_extractor import NERExtractor
             from .relation_extractor import RelationExtractor
 
+            # Use method-based extraction
+            methods = options.get("method", self.method)
+            if isinstance(methods, str):
+                methods = [methods]
+
+            # Determine if we need to extract entities/relations based on method
+            # HuggingFace (Seq2Seq) does not need pre-extracted entities/relations
+            needs_entities_relations = any(m not in ["huggingface"] for m in methods)
+
             # Extract entities if not provided
-            if entities is None:
+            if entities is None and needs_entities_relations:
                 self.progress_tracker.update_tracking(
                     tracking_id, message="Extracting entities..."
                 )
@@ -375,18 +384,23 @@ class TripletExtractor:
                     ner_config = self.config.get("ner", {})
                     if "ner_method" in self.config:
                         ner_config = {**ner_config, "method": self.config["ner_method"]}
+                    
+                    # Filter out 'model' and 'huggingface_model' from shared config
+                    # to prevent passing triplet model to NER extractor
+                    shared_config = {
+                        k: v
+                        for k, v in self.config.items()
+                        if k not in ["ner", "relation", "validator", "serializer", "quality", "model", "huggingface_model"]
+                    }
+
                     self._ner_extractor = NERExtractor(
                         **ner_config,
-                        **{
-                            k: v
-                            for k, v in self.config.items()
-                            if k not in ["ner", "relation", "validator", "serializer", "quality"]
-                        },
+                        **shared_config,
                     )
                 entities = self._ner_extractor.extract_entities(text)
 
             # Extract relations if not provided
-            if relations is None:
+            if relations is None and needs_entities_relations:
                 self.progress_tracker.update_tracking(
                     tracking_id, message="Extracting relations..."
                 )
@@ -394,21 +408,20 @@ class TripletExtractor:
                     rel_config = self.config.get("relation", {})
                     if "relation_method" in self.config:
                         rel_config = {**rel_config, "method": self.config["relation_method"]}
+                    
+                    # Filter out 'model' and 'huggingface_model' from shared config
+                    shared_config = {
+                        k: v
+                        for k, v in self.config.items()
+                        if k not in ["ner", "relation", "validator", "serializer", "quality", "model", "huggingface_model"]
+                    }
+
                     self._relation_extractor = RelationExtractor(
                         **rel_config,
-                        **{
-                            k: v
-                            for k, v in self.config.items()
-                            if k not in ["ner", "relation", "validator", "serializer", "quality"]
-                        },
+                        **shared_config,
                     )
                 relations = self._relation_extractor.extract_relations(text, entities)
 
-            # Use method-based extraction
-            methods = options.get("method", self.method)
-            if isinstance(methods, str):
-                methods = [methods]
-
             triplet_types = options.get("triplet_types", self.triplet_types)
             
             # Merge config with options
@@ -450,8 +463,12 @@ class TripletExtractor:
                         method_options["triplet_types"] = triplet_types
 
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", all_options.get("model")
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model")
+                            or self.config.get("huggingface_model") 
+                            or self.config.get("model")
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -461,18 +478,28 @@ class TripletExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key
 
+                    # Print progress if verbose mode is enabled (only for LLM method to avoid spam)
+                    verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+                    if verbose_mode and method_name == "llm":
+                        import sys
+                        print(f"    [TripletExtractor] Processing with {method_name}...", flush=True, file=sys.stdout)
+                        if "api_key" in method_options:
+                             masked = method_options["api_key"][:4] + "..." if method_options["api_key"] else "None"
+                             print(f"    [TripletExtractor Debug] api_key present: {masked}", flush=True, file=sys.stdout)
+                        else:
+                             print(f"    [TripletExtractor Debug] api_key NOT present", flush=True, file=sys.stdout)
+
                     triplets = method_func(
                         text,
                         entities=entities,
@@ -480,6 +507,11 @@ class TripletExtractor:
                         **method_options,
                     )
 
+                    # Print result count if verbose (only for LLM method)
+                    if verbose_mode and method_name == "llm" and len(triplets) > 0:
+                        import sys
+                        print(f"    [TripletExtractor] Extracted {len(triplets)} triplets", flush=True, file=sys.stdout)
+
                     # Apply weighted scoring if triplet_types are provided
                     if triplet_types:
                         try:
@@ -515,6 +547,12 @@ class TripletExtractor:
 
                 except Exception as e:
                     self.logger.warning(f"Method {method_name} failed: {e}")
+                    verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [TripletExtractor] ERROR: Method {method_name} failed: {e}", flush=True, file=sys.stderr)
+                        import traceback
+                        traceback.print_exc(file=sys.stderr)
                     continue
 
             # Use first successful method or fallback to relation conversion

semantica/utils/progress_tracker.py

@@ -34,6 +34,7 @@ License: MIT
 """
 
 import inspect
+import os
 import sys
 import threading
 import time
@@ -46,6 +47,13 @@ from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
 from .logging import get_logger
 
+DISABLE_JUPYTER_PROGRESS = os.getenv("SEMANTICA_DISABLE_JUPYTER_PROGRESS", "").strip().lower() in (
+    "1",
+    "true",
+    "yes",
+    "on",
+)
+
 # Try to import IPython for Jupyter support
 try:
     from IPython import get_ipython
@@ -1017,6 +1025,7 @@ class ProgressTracker:
 
         # Detect environment - will be checked dynamically
         self.is_jupyter = self._detect_jupyter()
+        self.disable_jupyter_progress = DISABLE_JUPYTER_PROGRESS
 
         # Create displays
         self.displays: List[ProgressDisplay] = []
@@ -1024,7 +1033,7 @@ class ProgressTracker:
         # Always try Jupyter first if available, fallback to console
         if IPYTHON_AVAILABLE:
             # Try to detect Jupyter - if available, use it
-            if self.is_jupyter:
+            if self.is_jupyter and not self.disable_jupyter_progress:
                 self.displays.append(JupyterProgressDisplay(use_emoji=use_emoji))
             # Also add console as fallback for immediate feedback
             self.displays.append(
@@ -1213,7 +1222,11 @@ class ProgressTracker:
         if IPYTHON_AVAILABLE and not self.is_jupyter:
             self.is_jupyter = self._detect_jupyter()
             # If Jupyter is now detected and we don't have a Jupyter display, add it
-            if self.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays):
+            if (
+                self.is_jupyter
+                and not self.disable_jupyter_progress
+                and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays)
+            ):
                 # Insert Jupyter display at the beginning for priority
                 self.displays.insert(0, JupyterProgressDisplay(use_emoji=self.use_emoji))
 
@@ -1341,7 +1354,11 @@ class ProgressTracker:
         if IPYTHON_AVAILABLE and not self.is_jupyter:
             self.is_jupyter = self._detect_jupyter()
             # If Jupyter is now detected and we don't have a Jupyter display, add it
-            if self.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays):
+            if (
+                self.is_jupyter
+                and not self.disable_jupyter_progress
+                and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays)
+            ):
                 # Insert Jupyter display at the beginning for priority
                 self.displays.insert(0, JupyterProgressDisplay(use_emoji=self.use_emoji))
 
@@ -1526,7 +1543,11 @@ def get_progress_tracker() -> ProgressTracker:
     if IPYTHON_AVAILABLE and not _global_tracker.is_jupyter:
         _global_tracker.is_jupyter = _global_tracker._detect_jupyter()
         # If Jupyter is now detected and we don't have a Jupyter display, add it
-        if _global_tracker.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in _global_tracker.displays):
+        if (
+            _global_tracker.is_jupyter
+            and not _global_tracker.disable_jupyter_progress
+            and not any(isinstance(d, JupyterProgressDisplay) for d in _global_tracker.displays)
+        ):
             # Insert Jupyter display at the beginning for priority
             _global_tracker.displays.insert(0, JupyterProgressDisplay(use_emoji=_global_tracker.use_emoji))
     

semantica/vector_store/__init__.py

@@ -3,7 +3,7 @@ Vector Store Management Module
 
 This module provides comprehensive vector storage and retrieval capabilities for the
 Semantica framework, including support for multiple vector store backends (FAISS,
-Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and
+Weaviate, Qdrant, Pinecone, Milvus), hybrid search combining vector similarity and
 metadata filtering, metadata management, and namespace isolation.
 
 Algorithms Used:
@@ -58,6 +58,7 @@ Supported Backends:
     - FAISS: In-memory/local disk (Facebook AI Similarity Search)
     - Weaviate: Cloud/Self-hosted (Schema-aware vector database)
     - Qdrant: Cloud/Self-hosted (Vector database for the next generation of AI)
+    - Pinecone: Cloud-managed (Managed vector database service)
     - Milvus: Cloud/Self-hosted (Highly scalable vector database)
     - InMemory: Simple list-based storage for testing/small datasets
 
@@ -73,7 +74,7 @@ Dependencies:
     - pymilvus
 
 Key Features:
-    - Multi-backend vector store support (FAISS, Weaviate, Qdrant, Milvus)
+    - Multi-backend vector store support (FAISS, Weaviate, Qdrant, Pinecone, Milvus)
     - Vector indexing and similarity search
     - Metadata indexing and filtering
     - Hybrid search combining vector and metadata queries
@@ -91,6 +92,7 @@ Main Classes:
     - FAISSStore: FAISS integration for local vector storage
     - WeaviateStore: Weaviate vector database integration
     - QdrantStore: Qdrant vector database integration
+    - PineconeStore: Pinecone vector database integration
     - MilvusStore: Milvus vector database integration
     - HybridSearch: Hybrid vector and metadata search
     - MetadataStore: Metadata indexing and management
@@ -145,6 +147,7 @@ from .methods import (
 )
 from .milvus_store import MilvusStore, MilvusClient, MilvusCollection, MilvusSearch
 from .namespace_manager import Namespace, NamespaceManager
+from .pinecone_store import PineconeStore, PineconeClient, PineconeIndex, PineconeSearch
 from .qdrant_store import QdrantStore, QdrantClient, QdrantCollection, QdrantSearch
 from .registry import MethodRegistry, method_registry
 from .vector_store import VectorIndexer, VectorManager, VectorRetriever, VectorStore
@@ -181,6 +184,11 @@ __all__ = [
     "MilvusClient",
     "MilvusCollection",
     "MilvusSearch",
+    # Pinecone
+    "PineconeStore",
+    "PineconeClient",
+    "PineconeIndex",
+    "PineconeSearch",
     # Hybrid search
     "HybridSearch",
     "MetadataFilter",

semantica/vector_store/pinecone_store.py

@@ -0,0 +1,639 @@
+"""
+Pinecone Store Module
+
+This module provides Pinecone vector database integration for vector storage and
+similarity search in the Semantica framework, supporting managed vector database
+service with serverless and pod-based indexes, namespace isolation, and efficient
+vector operations with metadata filtering.
+
+Key Features:
+ - Serverless and Pod-based index management
+ - Namespace isolation for multi-tenant support
+ - Metadata filtering during search
+ - Batch operations for efficient data loading
+ - Index creation, deletion, and listing
+ - Optional dependency handling
+
+Main Classes:
+ - PineconeStore: Main Pinecone store for vector operations
+ - PineconeClient: Pinecone client wrapper
+ - PineconeIndex: Index wrapper with operations
+ - PineconeSearch: Search operations and filtering
+
+Example Usage:
+ >>> from semantica.vector_store import PineconeStore
+ >>> store = PineconeStore(api_key="your-api-key")
+ >>> store.connect()
+ >>> store.create_index("my-index", dimension=768)
+ >>> store.upsert_vectors(vectors, ids, metadata=metadata)
+ >>> results = store.search_vectors(query_vector, k=10, filter={"category": "science"})
+ >>> stats = store.get_stats()
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any, Dict, List, Optional, Union
+
+import numpy as np
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from ..utils.progress_tracker import get_progress_tracker
+
+# Optional Pinecone import
+try:
+    from pinecone import Pinecone as PineconeClientLib, ServerlessSpec, PodSpec
+
+    PINECONE_AVAILABLE = True
+except (ImportError, OSError):
+    PINECONE_AVAILABLE = False
+    PineconeClientLib = None
+    ServerlessSpec = None
+    PodSpec = None
+
+
+class PineconeClient:
+    """Pinecone client wrapper."""
+
+    def __init__(self, client: Any):
+        """Initialize Pinecone client wrapper."""
+        self.client = client
+        self.logger = get_logger("pinecone_client")
+
+    def create_index(
+        self,
+        index_name: str,
+        dimension: int,
+        metric: str = "cosine",
+        spec: Optional[Dict[str, Any]] = None,
+        **options,
+    ) -> bool:
+        """Create an index in Pinecone."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            # Default to serverless spec if not provided
+            if spec is None:
+                spec = ServerlessSpec(cloud="aws", region="us-east-1")
+
+            # Map metric names
+            metric_map = {
+                "cosine": "cosine",
+                "euclidean": "euclidean_distance",
+                "dot": "dotproduct",
+            }
+            pinecone_metric = metric_map.get(metric.lower(), "cosine")
+
+            self.client.create_index(
+                name=index_name,
+                dimension=dimension,
+                metric=pinecone_metric,
+                spec=spec,
+                **options,
+            )
+            return True
+        except Exception as e:
+            raise ProcessingError(f"Failed to create index: {str(e)}")
+
+    def delete_index(self, index_name: str) -> bool:
+        """Delete an index from Pinecone."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            self.client.delete_index(index_name)
+            return True
+        except Exception as e:
+            raise ProcessingError(f"Failed to delete index: {str(e)}")
+
+    def list_indexes(self) -> List[str]:
+        """List available indexes."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            indexes = self.client.list_indexes()
+            return [index.name for index in indexes]
+        except Exception as e:
+            raise ProcessingError(f"Failed to list indexes: {str(e)}")
+
+    def get_index(self, index_name: str) -> Any:
+        """Get index object."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            return self.client.Index(index_name)
+        except Exception as e:
+            raise ProcessingError(f"Failed to get index: {str(e)}")
+
+
+class PineconeIndex:
+    """Pinecone index wrapper."""
+
+    def __init__(self, index: Any):
+        """Initialize Pinecone index wrapper."""
+        self.index = index
+        self.logger = get_logger("pinecone_index")
+
+    def upsert_vectors(
+        self,
+        vectors: List[List[float]],
+        ids: List[str],
+        metadata: Optional[List[Dict[str, Any]]] = None,
+        namespace: str = "",
+        **options,
+    ) -> Dict[str, Any]:
+        """Upsert vectors to index."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            # Prepare vectors for upsert
+            upsert_data = []
+            for i, (vector, vector_id) in enumerate(zip(vectors, ids)):
+                vector_dict = {"id": vector_id, "values": vector}
+                if metadata and i < len(metadata):
+                    vector_dict["metadata"] = metadata[i]
+                upsert_data.append(vector_dict)
+
+            response = self.index.upsert(
+                vectors=upsert_data, namespace=namespace, **options
+            )
+            return {"upserted_count": response.upserted_count}
+        except Exception as e:
+            raise ProcessingError(f"Failed to upsert vectors: {str(e)}")
+
+    def search_vectors(
+        self,
+        query_vector: List[float],
+        k: int = 10,
+        filter: Optional[Dict[str, Any]] = None,
+        namespace: str = "",
+        **options,
+    ) -> List[Dict[str, Any]]:
+        """Search for similar vectors."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            response = self.index.query(
+                vector=query_vector,
+                top_k=k,
+                filter=filter,
+                namespace=namespace,
+                include_metadata=True,
+                include_values=False,
+                **options,
+            )
+
+            results = []
+            for match in response.matches:
+                results.append(
+                    {
+                        "id": match.id,
+                        "score": match.score,
+                        "metadata": match.metadata or {},
+                    }
+                )
+
+            return results
+        except Exception as e:
+            raise ProcessingError(f"Failed to search vectors: {str(e)}")
+
+    def delete_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """Delete vectors from index."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            response = self.index.delete(ids=vector_ids, namespace=namespace, **options)
+            return {"deleted": True}
+        except Exception as e:
+            raise ProcessingError(f"Failed to delete vectors: {str(e)}")
+
+    def fetch_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """Fetch vectors by ID."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            response = self.index.fetch(ids=vector_ids, namespace=namespace, **options)
+            return {
+                "vectors": {
+                    vector_id: {
+                        "values": vector.values,
+                        "metadata": vector.metadata or {},
+                    }
+                    for vector_id, vector in response.vectors.items()
+                }
+            }
+        except Exception as e:
+            raise ProcessingError(f"Failed to fetch vectors: {str(e)}")
+
+    def describe_index_stats(self, **options) -> Dict[str, Any]:
+        """Get index statistics."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            stats = self.index.describe_index_stats(**options)
+            return {
+                "dimension": stats.dimension,
+                "index_fullness": stats.index_fullness,
+                "total_vector_count": stats.total_vector_count,
+                "namespaces": stats.namespaces,
+            }
+        except Exception as e:
+            raise ProcessingError(f"Failed to get index stats: {str(e)}")
+
+
+class PineconeSearch:
+    """Pinecone search operations."""
+
+    def __init__(self, index: PineconeIndex):
+        """Initialize Pinecone search."""
+        self.index = index
+        self.logger = get_logger("pinecone_search")
+
+    def similarity_search(
+        self,
+        query_vector: np.ndarray,
+        limit: int = 10,
+        filter: Optional[Dict[str, Any]] = None,
+        namespace: str = "",
+        **options,
+    ) -> List[Dict[str, Any]]:
+        """
+        Perform similarity search.
+
+        Args:
+            query_vector: Query vector
+            limit: Number of results
+            filter: Metadata filter
+            namespace: Namespace to search in
+            **options: Additional options
+
+        Returns:
+            List of search results
+        """
+        return self.index.search_vectors(
+            query_vector.tolist(), limit, filter, namespace, **options
+        )
+
+
+class PineconeStore:
+    """
+    Pinecone store for vector storage and similarity search.
+
+    • Pinecone connection and authentication
+    • Index and namespace management
+    • Vector storage and retrieval
+    • Similarity search and filtering
+    • Performance optimization
+    • Error handling and recovery
+    """
+
+    def __init__(
+        self,
+        api_key: Optional[str] = None,
+        environment: Optional[str] = None,
+        **config,
+    ):
+        """Initialize Pinecone store."""
+        self.logger = get_logger("pinecone_store")
+        self.config = config
+        self.progress_tracker = get_progress_tracker()
+        # Ensure progress tracker is enabled
+        if not self.progress_tracker.enabled:
+            self.progress_tracker.enabled = True
+
+        self.api_key = api_key or config.get("api_key")
+        self.environment = environment or config.get("environment")
+
+        self.client: Optional[PineconeClient] = None
+        self.index: Optional[PineconeIndex] = None
+        self.search_engine: Optional[PineconeSearch] = None
+
+        # Check Pinecone availability
+        if not PINECONE_AVAILABLE:
+            self.logger.warning(
+                "Pinecone not available. Install with: pip install pinecone-client"
+            )
+
+    def connect(self, **kwargs) -> bool:
+        """
+        Connect to Pinecone service.
+
+        Args:
+            **kwargs: Connection options
+
+        Returns:
+            True if connected successfully
+        """
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError(
+                "Pinecone is not available. Install it with: pip install pinecone-client"
+            )
+
+        api_key = kwargs.get("api_key") or self.api_key
+        if not api_key:
+            raise ValidationError("Pinecone API key is required")
+
+        try:
+            pinecone_client = PineconeClientLib(api_key=api_key, **kwargs)
+            self.client = PineconeClient(pinecone_client)
+
+            self.logger.info("Connected to Pinecone")
+            return True
+        except Exception as e:
+            raise ProcessingError(f"Failed to connect to Pinecone: {str(e)}")
+
+    def create_index(
+        self,
+        index_name: str,
+        dimension: int,
+        metric: str = "cosine",
+        spec: Optional[Dict[str, Any]] = None,
+        **kwargs,
+    ):
+        """
+        Create a Pinecone index.
+
+        Args:
+            index_name: Name of the index
+            dimension: Vector dimension
+            metric: Distance metric ("cosine", "euclidean", "dot")
+            spec: Index specification (ServerlessSpec or PodSpec)
+            **kwargs: Additional options
+
+        Returns:
+            PineconeIndex instance
+        """
+        if self.client is None:
+            self.connect()
+
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            # Create index spec if not provided
+            if spec is None:
+                spec = ServerlessSpec(cloud="aws", region="us-east-1")
+
+            self.client.create_index(index_name, dimension, metric, spec, **kwargs)
+
+            # Get the index
+            pinecone_index = self.client.get_index(index_name)
+            self.index = PineconeIndex(pinecone_index)
+            self.search_engine = PineconeSearch(self.index)
+
+            self.logger.info(f"Created Pinecone index: {index_name}")
+            return self.index
+
+        except Exception as e:
+            raise ProcessingError(f"Failed to create index: {str(e)}")
+
+    def get_index(self, index_name: str) -> PineconeIndex:
+        """
+        Get existing index.
+
+        Args:
+            index_name: Name of the index
+
+        Returns:
+            PineconeIndex instance
+        """
+        if self.client is None:
+            self.connect()
+
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            pinecone_index = self.client.get_index(index_name)
+            self.index = PineconeIndex(pinecone_index)
+            self.search_engine = PineconeSearch(self.index)
+            return self.index
+        except Exception as e:
+            raise ProcessingError(f"Failed to get index: {str(e)}")
+
+    def delete_index(self, index_name: str) -> bool:
+        """
+        Delete an index.
+
+        Args:
+            index_name: Name of the index to delete
+
+        Returns:
+            True if deleted successfully
+        """
+        if self.client is None:
+            self.connect()
+
+        return self.client.delete_index(index_name)
+
+    def list_indexes(self) -> List[str]:
+        """
+        List available indexes.
+
+        Returns:
+            List of index names
+        """
+        if self.client is None:
+            self.connect()
+
+        return self.client.list_indexes()
+
+    def upsert_vectors(
+        self,
+        vectors: List[Any],
+        ids: List[str],
+        metadata: Optional[List[Dict[str, Any]]] = None,
+        namespace: str = "",
+        **options,
+    ) -> Dict[str, Any]:
+        """
+        Upsert vectors to index.
+
+        Args:
+            vectors: List of vectors
+            ids: Vector IDs
+            metadata: Optional metadata for each vector
+            namespace: Namespace to upsert into
+            **options: Additional options
+
+        Returns:
+            Upsert response
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="vector_store",
+            submodule="PineconeStore",
+            message=f"Upserting {len(vectors)} vectors to Pinecone index",
+        )
+
+        try:
+            if self.index is None:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message="Index not initialized"
+                )
+                raise ProcessingError(
+                    "Index not initialized. Call create_index() or get_index() first."
+                )
+
+            if not PINECONE_AVAILABLE:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message="Pinecone not available"
+                )
+                raise ProcessingError("Pinecone not available")
+
+            self.progress_tracker.update_tracking(
+                tracking_id, message="Preparing vectors..."
+            )
+
+            # Convert vectors to list format
+            vector_list = []
+            for vector in vectors:
+                if isinstance(vector, np.ndarray):
+                    vector_list.append(vector.tolist())
+                else:
+                    vector_list.append(list(vector))
+
+            self.progress_tracker.update_tracking(
+                tracking_id, message="Upserting vectors to index..."
+            )
+            result = self.index.upsert_vectors(
+                vector_list, ids, metadata, namespace, **options
+            )
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Upserted {len(vectors)} vectors",
+            )
+            return result
+
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise ProcessingError(f"Failed to upsert vectors: {str(e)}")
+
+    def search_vectors(
+        self,
+        query_vector: Any,
+        k: int = 10,
+        filter: Optional[Dict[str, Any]] = None,
+        namespace: str = "",
+        **options,
+    ) -> List[Dict[str, Any]]:
+        """
+        Search vectors in index.
+
+        Args:
+            query_vector: Query vector
+            k: Number of results
+            filter: Metadata filter
+            namespace: Namespace to search in
+            **options: Additional options
+
+        Returns:
+            List of search results
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="vector_store",
+            submodule="PineconeStore",
+            message=f"Searching for {k} similar vectors in Pinecone",
+        )
+
+        try:
+            if self.search_engine is None:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message="Index not initialized"
+                )
+                raise ProcessingError(
+                    "Index not initialized. Call create_index() or get_index() first."
+                )
+
+            self.progress_tracker.update_tracking(
+                tracking_id, message="Performing similarity search..."
+            )
+
+            # Convert query vector to list
+            if isinstance(query_vector, np.ndarray):
+                query_vector = query_vector.tolist()
+            else:
+                query_vector = list(query_vector)
+
+            results = self.search_engine.similarity_search(
+                np.array(query_vector), k, filter, namespace, **options
+            )
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Found {len(results)} similar vectors",
+            )
+            return results
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
+    def delete_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """
+        Delete vectors from index.
+
+        Args:
+            vector_ids: Vector IDs to delete
+            namespace: Namespace to delete from
+            **options: Additional options
+
+        Returns:
+            Delete response
+        """
+        if self.index is None:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() or get_index() first."
+            )
+
+        return self.index.delete_vectors(vector_ids, namespace, **options)
+
+    def fetch_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """
+        Fetch vectors by ID.
+
+        Args:
+            vector_ids: Vector IDs to fetch
+            namespace: Namespace to fetch from
+            **options: Additional options
+
+        Returns:
+            Fetch response
+        """
+        if self.index is None:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() or get_index() first."
+            )
+
+        return self.index.fetch_vectors(vector_ids, namespace, **options)
+
+    def get_stats(self, **options) -> Dict[str, Any]:
+        """Get index statistics."""
+        if self.index is None:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() or get_index() first."
+            )
+
+        return self.index.describe_index_stats(**options)

semantica/vector_store/vector_store.py

@@ -38,6 +38,7 @@ License: MIT
 """
 
 from typing import Any, Dict, List, Optional, Tuple, Union
+import concurrent.futures
 
 import numpy as np
 
@@ -59,9 +60,9 @@ class VectorStore:
     • Provides vector store operations
     """
 
-    SUPPORTED_BACKENDS = {"faiss", "weaviate", "qdrant", "milvus", "inmemory"}
+    SUPPORTED_BACKENDS = {"faiss", "weaviate", "qdrant", "milvus", "pinecone", "inmemory"}
 
-    def __init__(self, backend="faiss", config=None, **kwargs):
+    def __init__(self, backend="faiss", config=None, max_workers: int = 6, **kwargs):
         """Initialize vector store."""
         if backend.lower() not in self.SUPPORTED_BACKENDS:
             raise ValueError(
@@ -72,6 +73,7 @@ class VectorStore:
         self.logger = get_logger("vector_store")
         self.config = config or {}
         self.config.update(kwargs)
+        self.max_workers = max_workers
         self.progress_tracker = get_progress_tracker()
         # Ensure progress tracker is enabled
         if not self.progress_tracker.enabled:
@@ -127,6 +129,141 @@ class VectorStore:
         self.logger.warning("Using random fallback embedding")
         return np.random.rand(self.dimension).astype(np.float32)
 
+    def embed_batch(self, texts: List[str]) -> List[np.ndarray]:
+        """
+        Generate embeddings for a list of texts using the internal embedder.
+        
+        Args:
+            texts: List of texts to embed
+            
+        Returns:
+            List of numpy arrays
+        """
+        if self.embedder:
+            try:
+                # generate_embeddings handles list input
+                embeddings = self.embedder.generate_embeddings(texts)
+                # Ensure it returns a list of arrays (it returns 2D array or list)
+                if isinstance(embeddings, np.ndarray):
+                    return list(embeddings)
+                return embeddings
+            except Exception as e:
+                self.logger.warning(f"Batch embedding generation failed: {e}")
+        
+        # Fallback
+        self.logger.warning("Using random fallback embeddings for batch")
+        return [np.random.rand(self.dimension).astype(np.float32) for _ in texts]
+
+    def add_documents(
+        self,
+        documents: List[str],
+        metadata: Optional[List[Dict[str, Any]]] = None,
+        batch_size: int = 32,
+        parallel: bool = True,
+        **options,
+    ) -> List[str]:
+        """
+        Add multiple documents to the store with parallel embedding generation.
+        
+        Args:
+            documents: List of document texts
+            metadata: List of metadata dictionaries
+            batch_size: Number of documents to process in one batch
+            parallel: Whether to use parallel processing for embeddings
+            **options: Additional options
+            
+        Returns:
+            List[str]: Vector IDs
+        """
+        if not documents:
+            return []
+            
+        num_docs = len(documents)
+        metadata = metadata or [{} for _ in range(num_docs)]
+        
+        if len(metadata) != num_docs:
+            raise ValueError("Metadata list length must match documents length")
+            
+        all_vectors = [None] * num_docs
+        
+        # Helper for processing a batch
+        def process_batch(start_idx: int, end_idx: int):
+            batch_texts = documents[start_idx:end_idx]
+            batch_embeddings = self.embed_batch(batch_texts)
+            return start_idx, batch_embeddings
+
+        # Calculate batches
+        batches = []
+        for i in range(0, num_docs, batch_size):
+            batches.append((i, min(i + batch_size, num_docs)))
+            
+        tracking_id = self.progress_tracker.start_tracking(
+            module="vector_store",
+            submodule="VectorStore",
+            message=f"Processing {num_docs} documents (parallel={parallel})",
+        )
+
+        try:
+            if parallel and self.max_workers > 1:
+                self.progress_tracker.update_tracking(
+                    tracking_id, message=f"Embedding with {self.max_workers} workers..."
+                )
+                with concurrent.futures.ThreadPoolExecutor(max_workers=self.max_workers) as executor:
+                    futures = [
+                        executor.submit(process_batch, start, end)
+                        for start, end in batches
+                    ]
+                    
+                    completed = 0
+                    for future in concurrent.futures.as_completed(futures):
+                        start_idx, embeddings = future.result()
+                        # Place results in correct order
+                        for i, emb in enumerate(embeddings):
+                            all_vectors[start_idx + i] = emb
+                        
+                        completed += 1
+                        if completed % 5 == 0:  # Update progress periodically
+                            self.progress_tracker.update_tracking(
+                                tracking_id, 
+                                message=f"Embedded batch {completed}/{len(batches)}"
+                            )
+            else:
+                # Sequential processing
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Embedding sequentially..."
+                )
+                for i, (start, end) in enumerate(batches):
+                    _, embeddings = process_batch(start, end)
+                    for j, emb in enumerate(embeddings):
+                        all_vectors[start + j] = emb
+                    
+                    if i % 5 == 0:
+                        self.progress_tracker.update_tracking(
+                            tracking_id, 
+                            message=f"Embedded batch {i+1}/{len(batches)}"
+                        )
+                        
+            # Verify all embeddings generated
+            if any(v is None for v in all_vectors):
+                raise ProcessingError("Failed to generate all embeddings")
+                
+            # Store all vectors in one go
+            self.progress_tracker.update_tracking(tracking_id, message="Storing vectors...")
+            vector_ids = self.store_vectors(all_vectors, metadata=metadata, **options)
+            
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Added {len(vector_ids)} documents",
+            )
+            return vector_ids
+            
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
     def store(
         self,
         vectors: List[np.ndarray],

semantica/vector_store/vector_store_usage.md

@@ -1,6 +1,6 @@
 # Vector Store Module Usage Guide
 
-This comprehensive guide demonstrates how to use the vector store module for vector storage and retrieval, supporting multiple vector store backends (FAISS, Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and metadata filtering, metadata management, and namespace isolation.
+This comprehensive guide demonstrates how to use the vector store module for vector storage and retrieval, supporting multiple vector store backends (FAISS, Weaviate, Qdrant, Pinecone, Milvus), hybrid search combining vector similarity and metadata filtering, metadata management, and namespace isolation.
 
 ## Table of Contents
 
@@ -754,6 +754,39 @@ results = store.search(
 print(f"Found {len(results)} results")
 ```
 
+### Pinecone Store
+
+```python
+from semantica.vector_store import PineconeStore
+import numpy as np
+
+# Create Pinecone store
+store = PineconeStore(api_key="your-api-key")
+
+# Connect
+store.connect()
+
+# Create index
+store.create_index("my-index", dimension=768, metric="cosine")
+
+# Upsert vectors
+vectors = [np.random.rand(768).tolist() for _ in range(100)]
+ids = [f"vec_{i}" for i in range(100)]
+metadata = [{"category": "science"} for _ in range(100)]
+store.upsert_vectors(vectors, ids, metadata=metadata, namespace="my-namespace")
+
+# Search
+query_vector = np.random.rand(768).tolist()
+results = store.search_vectors(
+    query_vector,
+    k=10,
+    filter={"category": {"$eq": "science"}},
+    namespace="my-namespace"
+)
+
+print(f"Found {len(results)} results")
+```
+
 ### Milvus Store
 
 ```python

tests/ingest/test_ontology_ingestor.py

@@ -0,0 +1,214 @@
+import os
+import shutil
+import tempfile
+import pytest
+from pathlib import Path
+from semantica.ingest import OntologyIngestor, ingest, ingest_ontology, OntologyData
+
+class TestOntologyIngestor:
+    @pytest.fixture
+    def sample_ttl_content(self):
+        return """
+        @prefix : <http://example.org/ontology/> .
+        @prefix owl: <http://www.w3.org/2002/07/owl#> .
+        @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+        @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+        @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+        <http://example.org/ontology/> rdf:type owl:Ontology ;
+            rdfs:label "Test Ontology" .
+
+        :Person rdf:type owl:Class ;
+            rdfs:label "Person" .
+
+        :hasName rdf:type owl:DatatypeProperty ;
+            rdfs:domain :Person ;
+            rdfs:range xsd:string .
+        """
+
+    def test_ingest_single_file(self, sample_ttl_content):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".ttl", mode="w") as tmp:
+            tmp.write(sample_ttl_content)
+            tmp_path = tmp.name
+
+        try:
+            ingestor = OntologyIngestor()
+            result = ingestor.ingest_ontology(tmp_path)
+            
+            assert isinstance(result, OntologyData)
+            assert result.data["name"] == "Test Ontology" or result.data["name"] == os.path.basename(tmp_path)
+            assert any(cls["name"] == "Person" for cls in result.data["classes"])
+            assert any(prop["name"] == "hasName" for prop in result.data["properties"])
+            assert result.metadata["format"] == "ttl" or result.metadata["format"] == "turtle"
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_ingest_directory(self, sample_ttl_content):
+        with tempfile.TemporaryDirectory() as tmp_dir:
+            # Create two ontology files
+            file1 = os.path.join(tmp_dir, "ont1.ttl")
+            file2 = os.path.join(tmp_dir, "ont2.rdf")
+            
+            with open(file1, "w") as f:
+                f.write(sample_ttl_content)
+            
+            # Simple RDF/XML content for the second file
+            rdf_content = """
+            <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+                     xmlns:owl="http://www.w3.org/2002/07/owl#">
+              <owl:Ontology rdf:about="http://example.org/ont2"/>
+              <owl:Class rdf:about="http://example.org/ont2/Animal"/>
+            </rdf:RDF>
+            """
+            with open(file2, "w") as f:
+                f.write(rdf_content)
+            
+            ingestor = OntologyIngestor()
+            results = ingestor.ingest_directory(tmp_dir)
+            
+            assert len(results) == 2
+            assert all(isinstance(r, OntologyData) for r in results)
+            
+            # Verify results contain expected classes
+            classes = [cls["name"] for res in results for cls in res.data["classes"]]
+            assert "Person" in classes
+            assert "Animal" in classes
+
+    def test_unified_ingest_function(self, sample_ttl_content):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".ttl", mode="w") as tmp:
+            tmp.write(sample_ttl_content)
+            tmp_path = tmp.name
+            
+        try:
+            # Test auto-detection via unified ingest
+            result = ingest(tmp_path)
+            assert "ontology" in result
+            assert isinstance(result["ontology"], OntologyData)
+            assert len(result["ontology"].data["classes"]) > 0
+            
+            # Test explicit source type
+            result_explicit = ingest(tmp_path, source_type="ontology")
+            assert "ontology" in result_explicit
+            assert result_explicit["ontology"].metadata["source_path"] == tmp_path
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_convenience_function(self, sample_ttl_content):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".n3", mode="w") as tmp:
+            tmp.write(sample_ttl_content)
+            tmp_path = tmp.name
+            
+        try:
+            result = ingest_ontology(tmp_path)
+            assert isinstance(result, OntologyData)
+            assert len(result.data["classes"]) > 0
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_ingest_formats(self):
+        """Test ingestion of all supported formats."""
+        ingestor = OntologyIngestor()
+        
+        # 1. JSON-LD
+        jsonld_content = """
+        {
+          "@context": {
+            "owl": "http://www.w3.org/2002/07/owl#",
+            "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
+            "rdfs": "http://www.w3.org/2000/01/rdf-schema#"
+          },
+          "@id": "http://example.org/jsonld",
+          "@type": "owl:Ontology",
+          "rdfs:label": "JSON-LD Ontology",
+          "owl:versionInfo": "1.0"
+        }
+        """
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".jsonld", mode="w") as tmp:
+            tmp.write(jsonld_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            assert result.data["name"] == "JSON-LD Ontology"
+            assert result.metadata["format"] == "json-ld"
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+        # 2. N-Triples
+        nt_content = '<http://example.org/nt/Class> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2002/07/owl#Class> .\n'
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".nt", mode="w") as tmp:
+            tmp.write(nt_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            # N-Triples often doesn't have ontology metadata, so name might default to basename
+            assert result.data["name"] == os.path.basename(tmp_path)
+            assert any(cls["uri"] == "http://example.org/nt/Class" for cls in result.data["classes"])
+            assert result.metadata["format"] == "nt"
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+        # 3. Notation3
+        n3_content = """
+        @prefix : <http://example.org/n3/> .
+        @prefix owl: <http://www.w3.org/2002/07/owl#> .
+        :N3Class a owl:Class .
+        """
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".n3", mode="w") as tmp:
+            tmp.write(n3_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            assert any(cls["uri"] == "http://example.org/n3/N3Class" for cls in result.data["classes"])
+            # format might be 'n3' or 'turtle' depending on rdflib detection as they are similar
+            assert result.metadata["format"] in ["n3", "turtle"] 
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+        # 4. RDF/XML (.owl)
+        owl_content = """
+        <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+                 xmlns:owl="http://www.w3.org/2002/07/owl#"
+                 xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
+          <owl:Ontology rdf:about="http://example.org/owl"/>
+          <owl:Class rdf:about="http://example.org/owl/OwlClass">
+            <rdfs:label>OwlClass</rdfs:label>
+          </owl:Class>
+        </rdf:RDF>
+        """
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".owl", mode="w") as tmp:
+            tmp.write(owl_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            assert any(cls["name"] == "OwlClass" for cls in result.data["classes"])
+            assert result.metadata["format"] in ["xml", "rdf", "owl"]
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_error_handling(self):
+        ingestor = OntologyIngestor()
+        with pytest.raises(Exception): # Specific exception type depends on implementation, likely ValidationError or FileNotFoundError
+            ingestor.ingest_ontology("non_existent_file.ttl")
+
+    def test_invalid_content(self):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".ttl", mode="w") as tmp:
+            tmp.write("This is not valid turtle content")
+            tmp_path = tmp.name
+            
+        try:
+            ingestor = OntologyIngestor()
+            # Depending on implementation, this might raise an exception or return partial/empty result with error in metadata
+            # Given current implementation uses g.parse(), it likely raises an exception which is caught or propagated
+            # If propagated:
+            with pytest.raises(Exception):
+                ingestor.ingest_ontology(tmp_path)
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)

tests/integration/test_relations_groq.py

@@ -0,0 +1,98 @@
+import os
+import unittest
+import time
+
+from semantica.semantic_extract import NERExtractor, RelationExtractor
+
+# Use environment variable for API key
+_GROQ_KEY = os.getenv("GROQ_API_KEY") or os.getenv("GROQ_TEST_API_KEY")
+@unittest.skipUnless(_GROQ_KEY, "Groq key not set; skipping live integration test")
+class TestGroqRelationsIntegration(unittest.TestCase):
+    def setUp(self):
+        self.api_key = _GROQ_KEY
+        self.model = "llama-3.1-8b-instant"
+        # Short, unambiguous finance snippet
+        self.text_short = (
+            "Apple reported revenue of $4.4 billion in Q1 2024 and provided guidance for FY 2025."
+        )
+        # Longer text to exercise chunking and ensure no hang
+        self.text_long = (
+            "Apple reported revenue of $4.4 billion in Q1 2024. "
+            "The company also reported growth of 12% year-over-year and provided guidance for FY 2025. "
+            "Microsoft reported revenue of $6.1 billion in Q2 2024 and expects sequential growth. "
+            "NVIDIA reported record revenue in 2024 Q1 and guided for higher revenue in Q2 2024. "
+        ) * 20  # expand length
+
+    def _extract_entities(self, text):
+        ner = NERExtractor(
+            method="llm",
+            provider="groq",
+            llm_model=self.model,
+            api_key=self.api_key,
+            temperature=0.0,
+        )
+        entities = ner.extract_entities(text, entity_types=["ORGANIZATION", "MONEY", "DATE", "EVENT", "PERCENT"])
+        self.assertIsInstance(entities, list)
+        return entities
+
+    def test_relations_short_text(self):
+        entities = self._extract_entities(self.text_short)
+        self.assertGreater(len(entities), 0, "NER should extract entities for short text")
+
+        relation_extractor = RelationExtractor(
+            method="llm",
+            relation_types=[
+                "HAS_REVENUE",
+                "HAS_GROWTH",
+                "PROVIDES_GUIDANCE",
+                "IN_QUARTER",
+                "FOR_PERIOD",
+                "RELATED_TO",
+            ],
+            provider="groq",
+            llm_model=self.model,
+            api_key=self.api_key,
+            temperature=0.0,
+            verbose=True,
+        )
+
+        start = time.time()
+        relations = relation_extractor.extract_relations(text=self.text_short, entities=entities)
+        elapsed = time.time() - start
+
+        self.assertIsInstance(relations, list)
+        # Ensure call completes reasonably fast (network dependent; allow generous bound)
+        self.assertLess(elapsed, 60, f"Extraction took too long: {elapsed:.2f}s")
+        # Do not strictly assert >0 as model output may vary, but log for diagnostics
+        if relations:
+            sample = relations[0]
+            self.assertTrue(hasattr(sample, "subject") and hasattr(sample, "predicate") and hasattr(sample, "object"))
+
+    def test_relations_long_text_chunking(self):
+        entities = self._extract_entities(self.text_long)
+        self.assertGreater(len(entities), 0, "NER should extract entities for long text")
+
+        relation_extractor = RelationExtractor(
+            method="llm",
+            relation_types=["RELATED_TO", "HAS_REVENUE", "IN_QUARTER"],
+            provider="groq",
+            llm_model=self.model,
+            api_key=self.api_key,
+            temperature=0.0,
+            verbose=True,
+        )
+
+        start = time.time()
+        relations = relation_extractor.extract_relations(text=self.text_long, entities=entities)
+        elapsed = time.time() - start
+
+        self.assertIsInstance(relations, list)
+        # Ensure completion (chunked path) and no hang
+        self.assertLess(elapsed, 120, f"Chunked extraction took too long: {elapsed:.2f}s")
+        if relations:
+            for r in relations[:3]:
+                self.assertTrue(hasattr(r, "subject") and hasattr(r, "predicate") and hasattr(r, "object"))
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/kg/test_graph_builder_external.py

@@ -0,0 +1,232 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+from semantica.kg.graph_builder import GraphBuilder
+
+
+class TestGraphBuilderExternal(unittest.TestCase):
+    def setUp(self):
+        self.mock_tracker_patcher = patch("semantica.utils.progress_tracker.get_progress_tracker")
+        self.mock_get_tracker = self.mock_tracker_patcher.start()
+        self.mock_tracker = MagicMock()
+        self.mock_get_tracker.return_value = self.mock_tracker
+
+        self.mock_resolver_patcher = patch("semantica.kg.entity_resolver.EntityResolver")
+        self.mock_resolver_cls = self.mock_resolver_patcher.start()
+
+        self.mock_conflict_patcher = patch("semantica.conflicts.conflict_detector.ConflictDetector")
+        self.mock_conflict_cls = self.mock_conflict_patcher.start()
+
+    def tearDown(self):
+        self.mock_tracker_patcher.stop()
+        self.mock_resolver_patcher.stop()
+        self.mock_conflict_patcher.stop()
+
+    def test_single_source_dict_with_source_id_target_id(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "drug:1", "name": "Aspirin", "type": "Drug"},
+            {"id": "disease:1", "name": "Myocardial infarction", "type": "Disease"},
+        ]
+        relationships = [
+            {"source_id": "drug:1", "target_id": "disease:1", "type": "TREATS"},
+        ]
+
+        source = {"entities": entities, "relationships": relationships}
+
+        kg = builder.build(source)
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "drug:1")
+        self.assertEqual(rel.get("target"), "disease:1")
+        self.assertEqual(kg["metadata"]["num_relationships"], 1)
+
+    def test_sources_list_merge_with_external_relationships(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source1 = {
+            "entities": [{"id": "1", "name": "A"}],
+            "relationships": [{"source_id": "1", "target_id": "2", "type": "REL_1"}],
+        }
+        source2 = {
+            "entities": [{"id": "2", "name": "B"}],
+            "relationships": [{"source_id": "2", "target_id": "1", "type": "REL_2"}],
+        }
+
+        kg = builder.build([source1, source2])
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 2)
+        sources = {r["source"] for r in kg["relationships"]}
+        targets = {r["target"] for r in kg["relationships"]}
+        self.assertEqual(sources, {"1", "2"})
+        self.assertEqual(targets, {"1", "2"})
+
+    def test_build_with_explicit_relationships_argument_external_ids(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "REL"},
+        ]
+
+        kg = builder.build(entities, relationships=relationships)
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "2")
+
+    def test_build_single_source_external_graph(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source = {
+            "entities": [{"id": "1", "name": "A"}],
+            "relationships": [{"source_id": "1", "target_id": "1", "type": "SELF"}],
+        }
+
+        kg = builder.build_single_source(source)
+
+        self.assertEqual(len(kg["entities"]), 1)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "1")
+
+    def test_relationship_key_variants_normalized(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+            {"id": "3", "name": "C"},
+            {"id": "4", "name": "D"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "R1"},
+            {"source": "2", "target": "3", "type": "R2"},
+            {"subject": "3", "object": "4", "type": "R3"},
+        ]
+
+        kg = builder.build({"entities": entities, "relationships": relationships})
+
+        self.assertEqual(len(kg["relationships"]), 3)
+        ids = {(r["source"], r["target"]) for r in kg["relationships"]}
+        self.assertIn(("1", "2"), ids)
+        self.assertIn(("2", "3"), ids)
+        self.assertIn(("3", "4"), ids)
+
+    def test_warning_when_all_relationships_dropped(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source = {
+            "entities": [],
+            "relationships": [{"foo": "x"}, {"bar": "y"}],
+        }
+
+        with patch.object(builder.logger, "warning") as mock_warning:
+            kg = builder.build(source)
+
+        self.assertEqual(len(kg["relationships"]), 0)
+        mock_warning.assert_called()
+        args, _ = mock_warning.call_args
+        self.assertIn("All relationships were dropped", args[0])
+
+    def test_no_warning_when_some_relationships_kept(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source = {
+            "entities": [{"id": "1"}, {"id": "2"}],
+            "relationships": [
+                {"source_id": "1", "target_id": "2", "type": "REL"},
+                {"foo": "x"},
+            ],
+        }
+
+        with patch.object(builder.logger, "warning") as mock_warning:
+            kg = builder.build(source)
+
+        self.assertEqual(len(kg["relationships"]), 2)
+        mock_warning.assert_not_called()
+
+    def test_issue_208_minimal_reproduction_shape(self):
+        builder = GraphBuilder(
+            merge_entities=False,
+            entity_resolution_strategy="none",
+            resolve_conflicts=False,
+        )
+
+        entities = [
+            {"id": "e1", "name": "Entity 1"},
+            {"id": "e2", "name": "Entity 2"},
+            {"id": "e3", "name": "Entity 3"},
+        ]
+        relationships = [
+            {"source_id": "e1", "target_id": "e2", "type": "REL_1"},
+            {"source_id": "e2", "target_id": "e3", "type": "REL_2"},
+        ]
+
+        entity_ids = {e["id"] for e in entities}
+        for r in relationships:
+            self.assertIn(r["source_id"], entity_ids)
+            self.assertIn(r["target_id"], entity_ids)
+
+        kg = builder.build(
+            sources=[{"entities": entities, "relationships": relationships}],
+            merge_entities=False,
+        )
+
+        self.assertEqual(len(kg["entities"]), 3)
+        self.assertEqual(len(kg["relationships"]), 2)
+        pairs = {(r["source"], r["target"]) for r in kg["relationships"]}
+        self.assertIn(("e1", "e2"), pairs)
+        self.assertIn(("e2", "e3"), pairs)
+
+    def test_issue_206_earnings_call_shape(self):
+        builder = GraphBuilder(
+            merge_entities=False,
+            entity_resolution_strategy="none",
+            resolve_conflicts=False,
+        )
+
+        entities = [
+            {
+                "id": "entity_446_MDA Space Ltd.",
+                "name": "MDA Space Ltd.",
+                "type": "ORGANIZATION",
+            },
+            {
+                "id": "entity_500_$409.8 million",
+                "name": "$409.8 million",
+                "type": "MONEY",
+            },
+        ]
+
+        relationships = [
+            {
+                "id": None,
+                "source_id": "MDA Space Ltd.",
+                "target_id": "$409.8 million",
+                "type": "HAS_REVENUE",
+                "confidence": 0.975,
+                "metadata": {},
+            }
+        ]
+
+        kg = builder.build(
+            sources=[{"entities": entities, "relationships": relationships}],
+            merge_entities=False,
+        )
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "MDA Space Ltd.")
+        self.assertEqual(rel.get("target"), "$409.8 million")

tests/kg/test_kg.py

@@ -91,6 +91,30 @@ class TestGraphBuilder(unittest.TestCase):
         graph2 = builder.build(source_list)
         self.assertEqual(len(graph2["entities"]), 2)
 
+    def test_build_with_external_relationship_ids(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "rel"},
+        ]
+
+        source = {
+            "entities": entities,
+            "relationships": relationships,
+        }
+
+        graph = builder.build(source)
+
+        self.assertEqual(len(graph["entities"]), 2)
+        self.assertEqual(len(graph["relationships"]), 1)
+        rel = graph["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "2")
+
     def test_build_with_conflict_resolution(self):
         """Test building with conflict resolution enabled"""
         builder = GraphBuilder(resolve_conflicts=True)
@@ -106,6 +130,50 @@ class TestGraphBuilder(unittest.TestCase):
         self.mock_conflict_cls.return_value.detect_conflicts.assert_called_once()
         self.mock_conflict_cls.return_value.resolve_conflicts.assert_called_once()
 
+    def test_build_single_source(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+        source = {
+            "entities": [{"id": "1", "name": "A"}],
+            "relationships": [{"source_id": "1", "target_id": "1", "type": "self"}],
+        }
+        graph = builder.build_single_source(source)
+        self.assertEqual(len(graph["entities"]), 1)
+        self.assertEqual(len(graph["relationships"]), 1)
+
+    def test_build_with_explicit_relationships_argument(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "rel"},
+        ]
+
+        graph = builder.build(entities, relationships=relationships)
+
+        self.assertEqual(len(graph["entities"]), 2)
+        self.assertEqual(len(graph["relationships"]), 1)
+        rel = graph["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "2")
+
+    def test_build_warns_when_all_relationships_dropped(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+        source = {
+            "entities": [],
+            "relationships": [{"foo": "x"}, {"bar": "y"}],
+        }
+
+        with patch.object(builder.logger, "warning") as mock_warning:
+            graph = builder.build(source)
+
+        self.assertEqual(len(graph["relationships"]), 0)
+        mock_warning.assert_called()
+        args, _ = mock_warning.call_args
+        self.assertIn("All relationships were dropped", args[0])
+
 class TestGraphAnalyzer(unittest.TestCase):
     def setUp(self):
         self.mock_tracker_patcher = patch("semantica.kg.graph_analyzer.get_progress_tracker")

tests/pipeline/test_pipeline.py

@@ -107,5 +107,25 @@ class TestPipelineModule(unittest.TestCase):
         
         self.assertEqual(execution_order, ["A", "B", "C"])
 
+    def test_imports_no_circular_dependencies(self):
+        import semantica
+
+        _ = semantica.pipeline
+
+        from semantica.pipeline import PipelineBuilder, PipelineValidator
+        from semantica.deduplication import DuplicateDetector
+
+        builder = PipelineBuilder()
+        builder.add_step("step1", "dummy")
+        pipeline = builder.build("import_test_pipeline")
+
+        validator = PipelineValidator()
+        result = validator.validate_pipeline(pipeline)
+
+        self.assertTrue(result.valid)
+
+        detector = DuplicateDetector()
+        self.assertIsNotNone(detector)
+
 if __name__ == "__main__":
     unittest.main()

tests/semantic_extract/test_retry_logic.py

@@ -0,0 +1,264 @@
+
+import unittest
+from unittest.mock import MagicMock, patch, ANY
+import sys
+import os
+from typing import List, Optional
+from pydantic import BaseModel
+import importlib.util
+
+# Add project root to path
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
+
+# Mock external dependencies with __spec__ for importlib checks
+mock_spacy = MagicMock()
+mock_spacy.__spec__ = MagicMock()
+sys.modules["spacy"] = mock_spacy
+
+sys.modules["instructor"] = MagicMock()
+sys.modules["groq"] = MagicMock()
+
+# Better mock for openai
+mock_openai = MagicMock()
+mock_openai.__spec__ = MagicMock()
+sys.modules["openai"] = mock_openai
+
+# Mock sentence_transformers and transformers to avoid heavy imports and dependency checks
+sys.modules["sentence_transformers"] = MagicMock()
+mock_transformers = MagicMock()
+mock_transformers.__spec__ = MagicMock()
+sys.modules["transformers"] = mock_transformers
+
+from semantica.semantic_extract import NERExtractor
+from semantica.semantic_extract.methods import extract_entities_llm, _extract_entities_chunked, extract_relations_llm, extract_triplets_llm
+from semantica.semantic_extract.providers import BaseProvider
+
+class EntitiesResponse(BaseModel):
+    entities: List[dict]
+
+class TestRetryLogic(unittest.TestCase):
+    
+    def setUp(self):
+        self.mock_provider = MagicMock()
+        self.mock_provider.is_available.return_value = True
+        self.mock_provider.generate_typed.return_value = MagicMock(entities=[])
+        
+    def test_ner_extractor_init_default(self):
+        """Test default max_retries in NERExtractor"""
+        ner = NERExtractor(method="llm", provider="test")
+        # Check internal config, max_retries not in config means default behavior downstream
+        self.assertIsNone(ner.config.get("max_retries"))
+
+    def test_ner_extractor_init_custom(self):
+        """Test custom max_retries in NERExtractor init"""
+        ner = NERExtractor(method="llm", provider="test", max_retries=5)
+        self.assertEqual(ner.config.get("max_retries"), 5)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_extract_entities_uses_init_value(self, mock_create_provider):
+        """Test extract_entities uses initialized max_retries"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        ner = NERExtractor(method="llm", provider="test", max_retries=5)
+        ner.extract_entities("test text")
+        
+        # Verify generate_typed called with max_retries=5
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 5)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_extract_entities_override(self, mock_create_provider):
+        """Test extract_entities override max_retries"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        ner = NERExtractor(method="llm", provider="test", max_retries=5)
+        # Override with 1
+        ner.extract_entities("test text", max_retries=1)
+        
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 1)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_chunked_extraction_propagation(self, mock_create_provider):
+        """Test max_retries propagation in chunked extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        # Patch TextSplitter where it lives
+        with patch('semantica.split.TextSplitter') as MockSplitter:
+            mock_splitter_instance = MockSplitter.return_value
+            # Mock split to return 2 chunks
+            mock_chunk1 = MagicMock()
+            mock_chunk1.text = "chunk1"
+            mock_chunk2 = MagicMock()
+            mock_chunk2.text = "chunk2"
+            mock_splitter_instance.split.return_value = [mock_chunk1, mock_chunk2]
+            
+            # Force chunking by setting max_text_length small
+            extract_entities_llm(
+                "very long text",
+                provider="test",
+                model="test-model",
+                max_text_length=10, # Force chunking
+                max_retries=7,
+                structured_output_mode="typed"
+            )
+            
+            # Check if generate_typed was called with max_retries=7 for chunks
+            # It should be called twice (once for each chunk)
+            self.assertEqual(self.mock_provider.generate_typed.call_count, 2)
+            
+            # Check arguments of the calls
+            call_args_list = self.mock_provider.generate_typed.call_args_list
+            for args, kwargs in call_args_list:
+                self.assertEqual(kwargs.get("max_retries"), 7)
+
+    def test_provider_base_logic(self):
+        """Test BaseProvider logic for max_retries with manual loop"""
+        provider = BaseProvider()
+        provider.client = MagicMock()
+        provider.logger = MagicMock()
+        provider.generate_structured = MagicMock(side_effect=Exception("Fail"))
+        
+        # Mock instructor failing
+        with patch('semantica.semantic_extract.providers.instructor') as mock_instructor:
+             # Make instructor client fail
+            mock_client = MagicMock()
+            mock_client.chat.completions.create.side_effect = Exception("Instructor Fail")
+            mock_instructor.from_provider.return_value = mock_client
+            mock_instructor.from_openai.return_value = mock_client
+            
+            # Run with max_retries=2
+            try:
+                provider.generate_typed("prompt", EntitiesResponse, max_retries=2)
+            except Exception:
+                pass
+            
+            # Should try manual generation exactly 2 times
+            self.assertEqual(provider.generate_structured.call_count, 2)
+
+    def test_provider_zero_retries(self):
+        """Test BaseProvider with max_retries=0"""
+        provider = BaseProvider()
+        provider.client = MagicMock()
+        provider.logger = MagicMock()
+        provider.generate_structured = MagicMock(side_effect=Exception("Fail"))
+        
+        # Mock instructor failing
+        with patch('semantica.semantic_extract.providers.instructor') as mock_instructor:
+            mock_client = MagicMock()
+            mock_client.chat.completions.create.side_effect = Exception("Instructor Fail")
+            mock_instructor.from_provider.return_value = mock_client
+            mock_instructor.from_openai.return_value = mock_client
+            
+            try:
+                provider.generate_typed("prompt", EntitiesResponse, max_retries=0)
+            except Exception:
+                pass
+            
+            # Should NOT try manual generation loop (range(0) is empty)
+            self.assertEqual(provider.generate_structured.call_count, 0)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_relations_retry_propagation(self, mock_create_provider):
+        """Test max_retries propagation in relation extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        # Create a mock entity
+        mock_entity = MagicMock()
+        mock_entity.text = "entity"
+        mock_entity.start_char = 0
+        mock_entity.end_char = 5
+        
+        extract_relations_llm(
+            "test text",
+            entities=[mock_entity],
+            provider="test",
+            max_retries=4
+        )
+        
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 4)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_relations_chunked_propagation(self, mock_create_provider):
+        """Test max_retries propagation in chunked relation extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        with patch('semantica.split.TextSplitter') as MockSplitter:
+            mock_splitter_instance = MockSplitter.return_value
+            mock_chunk1 = MagicMock()
+            mock_chunk1.text = "chunk1"
+            mock_chunk1.start_index = 0
+            mock_chunk1.end_index = 6
+            mock_splitter_instance.split.return_value = [mock_chunk1]
+            
+            # Create a mock entity
+            mock_entity = MagicMock()
+            mock_entity.text = "entity"
+            mock_entity.start_char = 0
+            mock_entity.end_char = 5
+            
+            extract_relations_llm(
+                "very long text",
+                entities=[mock_entity],
+                provider="test",
+                max_text_length=10,
+                max_retries=6
+            )
+            
+            # Check call count - should be called for the chunk
+            # Note: _extract_relations_chunked creates a new future for each chunk
+            # which calls extract_relations_llm, which calls generate_typed
+            self.assertEqual(self.mock_provider.generate_typed.call_count, 1)
+            
+            args, kwargs = self.mock_provider.generate_typed.call_args
+            self.assertEqual(kwargs.get("max_retries"), 6)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_triplets_retry_propagation(self, mock_create_provider):
+        """Test max_retries propagation in triplet extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        extract_triplets_llm(
+            "test text",
+            entities=[],
+            relations=[],
+            provider="test",
+            max_retries=7
+        )
+        
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 7)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_triplets_chunked_propagation(self, mock_create_provider):
+        """Test max_retries propagation in chunked triplet extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        with patch('semantica.split.TextSplitter') as MockSplitter:
+            mock_splitter_instance = MockSplitter.return_value
+            mock_chunk1 = MagicMock()
+            mock_chunk1.text = "chunk1"
+            mock_chunk1.start_index = 0
+            mock_chunk1.end_index = 6
+            mock_splitter_instance.split.return_value = [mock_chunk1]
+            
+            # Use max_text_length > 100 to pass the minimum viable chunk size check
+            # and make text longer than that
+            extract_triplets_llm(
+                "very long text " * 20, # length > 101
+                entities=[],
+                relations=[],
+                provider="test",
+                max_text_length=101,
+                max_retries=8
+            )
+            
+            # Check call count
+            self.assertEqual(self.mock_provider.generate_typed.call_count, 1)
+            
+            args, kwargs = self.mock_provider.generate_typed.call_args
+            self.assertEqual(kwargs.get("max_retries"), 8)
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_extractors_dispatch.py

@@ -0,0 +1,168 @@
+import unittest
+from unittest.mock import MagicMock, patch
+import sys
+import os
+
+# Add project root to path
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+# Mock dependencies to avoid import hangs and external calls
+sys.modules['spacy'] = MagicMock()
+sys.modules['semantica.semantic_extract.methods'] = MagicMock()
+sys.modules['semantica.utils.logging'] = MagicMock()
+sys.modules['semantica.utils.progress_tracker'] = MagicMock()
+sys.modules['semantica.semantic_extract.providers'] = MagicMock()
+
+# Mock get_logger and get_progress_tracker
+mock_logger = MagicMock()
+sys.modules['semantica.utils.logging'].get_logger.return_value = mock_logger
+
+mock_tracker = MagicMock()
+sys.modules['semantica.utils.progress_tracker'].get_progress_tracker.return_value = mock_tracker
+
+# Mock the methods module functions specifically
+mock_methods = sys.modules['semantica.semantic_extract.methods']
+mock_methods.get_entity_method = MagicMock()
+mock_methods.get_relation_method = MagicMock()
+mock_methods.get_triplet_method = MagicMock()
+
+# Mock specific extraction functions
+mock_extract_entities_hf = MagicMock()
+mock_extract_relations_hf = MagicMock()
+mock_extract_triplets_hf = MagicMock()
+
+# Setup the registry mocks to return our mock functions
+mock_methods.get_entity_method.return_value = mock_extract_entities_hf
+mock_methods.get_relation_method.return_value = mock_extract_relations_hf
+mock_methods.get_triplet_method.return_value = mock_extract_triplets_hf
+
+# Now import the classes under test
+# We need to patch where they import 'methods' locally if they do
+with patch.dict(sys.modules):
+    from semantica.semantic_extract.ner_extractor import NERExtractor
+    from semantica.semantic_extract.relation_extractor import RelationExtractor
+    from semantica.semantic_extract.triplet_extractor import TripletExtractor
+    from semantica.semantic_extract.ner_extractor import Entity
+    from semantica.semantic_extract.relation_extractor import Relation
+
+class TestExtractorsDispatch(unittest.TestCase):
+    def setUp(self):
+        self.mock_extract_entities_hf = mock_extract_entities_hf
+        self.mock_extract_relations_hf = mock_extract_relations_hf
+        self.mock_extract_triplets_hf = mock_extract_triplets_hf
+        
+        self.mock_extract_entities_hf.reset_mock()
+        self.mock_extract_relations_hf.reset_mock()
+        self.mock_extract_triplets_hf.reset_mock()
+        
+        # Configure mocks to return something iterable/valid
+        self.mock_extract_entities_hf.return_value = [MagicMock(spec=Entity, confidence=0.9, text="Test Entity")]
+        self.mock_extract_relations_hf.return_value = [MagicMock(spec=Relation, confidence=0.9)]
+        self.mock_extract_triplets_hf.return_value = [MagicMock(confidence=0.9)]
+
+    def test_ner_extractor_huggingface_dispatch(self):
+        print("\nTesting NERExtractor dispatch to HuggingFace...")
+        # Initialize with HuggingFace method
+        extractor = NERExtractor(method="huggingface")
+        
+        # Call extract_entities
+        text = "Steve Jobs founded Apple."
+        # Use a specific model via kwargs
+        extractor.extract_entities(text, model="my-custom-ner-model")
+        
+        # Verify get_entity_method was called with "huggingface"
+        mock_methods.get_entity_method.assert_called_with("huggingface")
+        
+        # Verify the extraction function was called with correct model
+        # We need to check the call args to see if 'model' was passed correctly
+        # The logic we implemented: method_options["model"] = all_options.get("huggingface_model") or all_options.get("model") or self.huggingface_model
+        
+        call_args = self.mock_extract_entities_hf.call_args
+        self.assertIsNotNone(call_args, "extract_entities_huggingface should have been called")
+        
+        _, kwargs = call_args
+        self.assertEqual(kwargs.get("model"), "my-custom-ner-model", "Should use model passed in kwargs")
+        
+        print("NERExtractor dispatch verified.")
+
+    def test_relation_extractor_huggingface_dispatch(self):
+        print("\nTesting RelationExtractor dispatch to HuggingFace...")
+        extractor = RelationExtractor(method="huggingface")
+        
+        text = "Steve Jobs founded Apple."
+        entities = [MagicMock(spec=Entity)]
+        
+        # Call extract_relations with explicit model
+        extractor.extract_relations(text, entities, model="my-relation-model")
+        
+        # Verify dispatch
+        mock_methods.get_relation_method.assert_called_with("huggingface")
+        
+        call_args = self.mock_extract_relations_hf.call_args
+        self.assertIsNotNone(call_args, "extract_relations_huggingface should have been called")
+        
+        _, kwargs = call_args
+        self.assertEqual(kwargs.get("model"), "my-relation-model", "Should use model passed in kwargs")
+        
+        print("RelationExtractor dispatch verified.")
+
+    def test_triplet_extractor_huggingface_dispatch(self):
+        print("\nTesting TripletExtractor dispatch to HuggingFace...")
+        extractor = TripletExtractor(method="huggingface")
+        
+        text = "Steve Jobs founded Apple."
+        
+        # Call extract_triplets with explicit model
+        extractor.extract_triplets(text, model="my-triplet-model")
+        
+        # Verify dispatch
+        mock_methods.get_triplet_method.assert_called_with("huggingface")
+        
+        call_args = self.mock_extract_triplets_hf.call_args
+        self.assertIsNotNone(call_args, "extract_triplets_huggingface should have been called")
+        
+        _, kwargs = call_args
+        self.assertEqual(kwargs.get("model"), "my-triplet-model", "Should use model passed in kwargs")
+        
+        print("TripletExtractor dispatch verified.")
+
+    def test_ner_extractor_huggingface_fallback(self):
+        print("\nTesting NERExtractor fallback logic...")
+        # Init with huggingface_model in config
+        extractor = NERExtractor(method="huggingface", huggingface_model="config-model")
+        
+        extractor.extract_entities("text")
+        
+        _, kwargs = self.mock_extract_entities_hf.call_args
+        self.assertEqual(kwargs.get("model"), "config-model", "Should prioritize huggingface_model from config")
+        
+        # Now override with kwargs model
+        extractor.extract_entities("text", model="kwargs-model")
+        _, kwargs = self.mock_extract_entities_hf.call_args
+        self.assertEqual(kwargs.get("model"), "kwargs-model", "Should allow overriding config huggingface_model via model kwarg")
+        
+        # Let's test passing 'huggingface_model' in kwargs
+        extractor.extract_entities("text", huggingface_model="override-model")
+        _, kwargs = self.mock_extract_entities_hf.call_args
+        self.assertEqual(kwargs.get("model"), "override-model", "Should allow overriding huggingface_model via kwargs")
+
+    def test_triplet_extractor_lazy_loading(self):
+        print("\nTesting TripletExtractor lazy loading for HuggingFace...")
+        # Initialize with HuggingFace method
+        extractor = TripletExtractor(method="huggingface")
+        
+        # Check initial state
+        self.assertIsNone(extractor._ner_extractor)
+        self.assertIsNone(extractor._relation_extractor)
+        
+        # Run extraction
+        extractor.extract_triplets("Steve Jobs founded Apple.")
+        
+        # Check state AFTER extraction - should STILL be None because huggingface (REBEL) doesn't need them
+        self.assertIsNone(extractor._ner_extractor, "NERExtractor should not be initialized for HuggingFace method")
+        self.assertIsNone(extractor._relation_extractor, "RelationExtractor should not be initialized for HuggingFace method")
+        
+        print("TripletExtractor lazy loading verified.")
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_hf_deep_verify.py

@@ -0,0 +1,167 @@
+import unittest
+from unittest.mock import MagicMock, patch
+import sys
+import os
+
+# Ensure project root is in path
+sys.path.append(os.getcwd())
+
+from semantica.semantic_extract.ner_extractor import NERExtractor, Entity
+from semantica.semantic_extract.relation_extractor import RelationExtractor
+from semantica.semantic_extract.triplet_extractor import TripletExtractor
+from semantica.utils.exceptions import ProcessingError
+
+class TestHuggingFaceDeepIntegration(unittest.TestCase):
+    """
+    Comprehensive test suite for Hugging Face models integration
+    in NER, Relation, and Triplet extraction modules.
+    """
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_ner_extraction_flow(self, MockLoaderClass):
+        """Test NER extraction with detailed IOB parsing and aggregation."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Simulate IOB output (Raw token classification)
+        mock_loader.extract_entities.return_value = [
+            {"entity": "B-PER", "score": 0.99, "index": 1, "word": "John", "start": 0, "end": 4, "label": "B-PER"},
+            {"entity": "I-PER", "score": 0.98, "index": 2, "word": "Doe", "start": 5, "end": 8, "label": "I-PER"},
+            {"entity": "O", "score": 0.99, "index": 3, "word": "lives", "start": 9, "end": 14, "label": "O"},
+            {"entity": "B-LOC", "score": 0.95, "index": 4, "word": "New", "start": 18, "end": 21, "label": "B-LOC"},
+            {"entity": "I-LOC", "score": 0.96, "index": 5, "word": "York", "start": 22, "end": 26, "label": "I-LOC"},
+        ]
+        
+        extractor = NERExtractor(method="huggingface", huggingface_model="dslim/bert-base-NER")
+        entities = extractor.extract_entities("John Doe lives in New York")
+        
+        # Verify aggregation worked (John Doe should be one entity)
+        # Note: The logic in extract_entities_huggingface handles manual aggregation
+        # if "entity_group" is missing and labels start with B-/I-
+        
+        # Let's debug what we expect. 
+        # "John" (B-PER) -> current_entity="John"
+        # "Doe" (I-PER) -> match! -> current_entity="John Doe"
+        # "lives" (O) -> append John Doe, current=None
+        # "New" (B-LOC) -> current="New"
+        # "York" (I-LOC) -> match! -> current="New York"
+        # End -> append New York
+        
+        self.assertEqual(len(entities), 2)
+        
+        person = next((e for e in entities if e.label == "PER"), None)
+        self.assertIsNotNone(person)
+        self.assertEqual(person.text, "John Doe")
+        
+        loc = next((e for e in entities if e.label == "LOC"), None)
+        self.assertIsNotNone(loc)
+        self.assertEqual(loc.text, "New York")
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_ner_aggregation_strategy_simple(self, MockLoaderClass):
+        """Test NER extraction when the pipeline handles aggregation (strategy='simple')."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Simulate Aggregated output
+        mock_loader.extract_entities.return_value = [
+            {"entity_group": "PER", "score": 0.99, "word": "John Doe", "start": 0, "end": 8},
+            {"entity_group": "LOC", "score": 0.95, "word": "New York", "start": 18, "end": 26},
+        ]
+        
+        extractor = NERExtractor(
+            method="huggingface", 
+            huggingface_model="dslim/bert-base-NER",
+            aggregation_strategy="simple" # Explicitly requesting simple
+        )
+        entities = extractor.extract_entities("John Doe lives in New York")
+        
+        self.assertEqual(len(entities), 2)
+        self.assertEqual(entities[0].text, "John Doe")
+        self.assertEqual(entities[0].label, "PER")
+        
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_relation_extraction_flow(self, MockLoaderClass):
+        """Test Relation extraction with Hugging Face model."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Mock extract_relations output
+        mock_loader.extract_relations.return_value = [{
+            "subject": Entity(text="Apple", label="ORG", start_char=0, end_char=5),
+            "object": Entity(text="Steve Jobs", label="PERSON", start_char=21, end_char=31),
+            "relation": "founded_by",
+            "score": 0.9
+        }]
+        
+        # We need to provide entities for relation extraction usually
+        entities = [
+            Entity(text="Apple", label="ORG", start_char=0, end_char=5),
+            Entity(text="Steve Jobs", label="PERSON", start_char=21, end_char=31)
+        ]
+        
+        extractor = RelationExtractor(method="huggingface", huggingface_model="facebook/bart-large-mnli")
+        relations = extractor.extract_relations("Apple was founded by Steve Jobs", entities=entities)
+        
+        # Check if relation is found
+        self.assertEqual(len(relations), 1)
+        self.assertEqual(relations[0].predicate, "founded_by")
+        self.assertEqual(relations[0].subject.text, "Apple")
+        self.assertEqual(relations[0].object.text, "Steve Jobs")
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_triplet_extraction_rebel(self, MockLoaderClass):
+        """Test Triplet extraction using REBEL parsing logic."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Mock extract_triplets output
+        # The extract_triplets method in Loader returns [{"triplet": decoded_text}]
+        # But wait, methods.py extract_triplets_huggingface handles parsing?
+        # No, let's check methods.py again.
+        
+        # Actually, methods.py for triplets calls loader.extract_triplets and then parses the result?
+        # Or does loader.extract_triplets return the raw generation?
+        # Let's check the code I read earlier.
+        # loader.extract_triplets returns [{"triplet": decoded}]
+        
+        # But methods.py `extract_triplets_huggingface` logic needs to be verified.
+        # I didn't read extract_triplets_huggingface in methods.py yet (I read entities).
+        # Assuming standard behavior, let's return what loader returns.
+        
+        mock_loader.extract_triplets.return_value = [{"triplet": "<triplet> Apple <subj> founded by <obj> Steve Jobs"}]
+        
+        # Wait, if methods.py expects raw text and parses it, then I need to know IF methods.py does the parsing or if it expects pre-parsed.
+        # Usually, if it's REBEL, the parsing happens after generation.
+        # Let's assume methods.py parses the REBEL format.
+        
+        extractor = TripletExtractor(method="huggingface", huggingface_model="Babelscape/rebel-large")
+        
+        # If the extractor relies on methods.py to parse, and methods.py relies on REBEL format:
+        triplets = extractor.extract_triplets("Apple was founded by Steve Jobs")
+        
+        # Note: If this fails, it might be because I need to check how extract_triplets_huggingface is implemented.
+        # But let's try.
+        if not triplets:
+             # Fallback: maybe methods.py expects the model to return parsed triplets?
+             pass
+             
+        self.assertTrue(len(triplets) > 0)
+        self.assertEqual(triplets[0].subject, "Apple")
+        self.assertEqual(triplets[0].object, "Steve Jobs")
+        self.assertEqual(triplets[0].predicate, "founded by")
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_byom_override(self, MockLoaderClass):
+        """Verify Bring Your Own Model (runtime override) works for all extractors."""
+        mock_loader = MockLoaderClass.return_value
+        mock_loader.extract_entities.return_value = []
+        
+        # NER
+        ner = NERExtractor(method="huggingface", huggingface_model="default-ner")
+        ner.extract_entities("test", huggingface_model="runtime-ner")
+        
+        # Check if load_ner_model was called with runtime model
+        # mock_loader.load_ner_model.assert_called_with("runtime-ner", ...)
+        # args[0] should be "runtime-ner"
+        call_args = mock_loader.load_ner_model.call_args
+        self.assertEqual(call_args[0][0], "runtime-ner")
+            
+if __name__ == "__main__":
+    unittest.main()

tests/test_huggingface_impl.py

@@ -0,0 +1,178 @@
+
+import sys
+import os
+import traceback
+from unittest.mock import MagicMock
+
+print("Starting test script...", flush=True)
+
+# Mock transformers and torch BEFORE any project imports
+try:
+    mock_transformers = MagicMock()
+    mock_pipeline = MagicMock()
+    mock_transformers.pipeline = mock_pipeline
+    sys.modules["transformers"] = mock_transformers
+    sys.modules["torch"] = MagicMock()
+    sys.modules["torch"].cuda.is_available.return_value = False
+    
+    # Mock spacy
+    mock_spacy = MagicMock()
+    sys.modules["spacy"] = mock_spacy
+    
+    # Mock instructor
+    sys.modules["instructor"] = MagicMock()
+    
+    # Also mock semantica.semantic_extract.config to avoid initialization issues
+    mock_config_module = MagicMock()
+    mock_config_instance = MagicMock()
+    # Setup default return values for config
+    mock_config_instance.get.return_value = {}
+    mock_config_instance.get_optimization_config.return_value = {"enable_cache": False}
+    
+    mock_config_module.config = mock_config_instance
+    mock_config_module.Config = MagicMock(return_value=mock_config_instance)
+    sys.modules["semantica.semantic_extract.config"] = mock_config_module
+    
+    print("Mocks setup complete.", flush=True)
+except Exception as e:
+    print(f"Error setting up mocks: {e}", flush=True)
+    sys.exit(1)
+
+# Add project root
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+print(f"Added to path: {sys.path[0]}", flush=True)
+
+try:
+    print("Importing methods...", flush=True)
+    from semantica.semantic_extract.methods import extract_entities_huggingface, extract_relations_huggingface, extract_triplets_huggingface
+    print("Importing Entity class...", flush=True)
+    from semantica.semantic_extract.ner_extractor import Entity
+    print("Imports successful.", flush=True)
+except Exception as e:
+    print(f"Import failed: {e}", flush=True)
+    traceback.print_exc()
+    sys.exit(1)
+
+def test_enhanced_impl():
+    print("Testing enhanced implementation...", flush=True)
+
+    try:
+        # 1. Test NER with aggregation strategy
+        print("\n--- Testing NER ---", flush=True)
+        
+        # Setup mock pipeline return value
+        mock_ner_pipeline = MagicMock()
+        mock_ner_pipeline.return_value = [
+            {"entity_group": "PERSON", "score": 0.99, "word": "Elon Musk", "start": 0, "end": 9},
+        ]
+        
+        # Configure pipeline side effect
+        def pipeline_side_effect(task, **kwargs):
+            if task == "ner": return mock_ner_pipeline
+            return MagicMock()
+            
+        mock_pipeline.side_effect = pipeline_side_effect
+        
+        # Test calling with aggregation_strategy
+        entities = extract_entities_huggingface(
+            "Elon Musk founded SpaceX.", 
+            model="dslim/bert-base-NER", 
+            aggregation_strategy="max"
+        )
+        print(f"Entities: {entities}", flush=True)
+        
+        # Verify aggregation_strategy was passed
+        mock_pipeline.assert_any_call(
+            "ner", 
+            model="dslim/bert-base-NER", 
+            device=-1, 
+            aggregation_strategy="max",
+            tokenizer=None
+        )
+
+        # 2. Test Relations with Input Formatting
+        print("\n--- Testing Relations ---", flush=True)
+        e1 = Entity(text="Elon Musk", label="PERSON", start_char=0, end_char=9)
+        e2 = Entity(text="SpaceX", label="ORG", start_char=18, end_char=24)
+        
+        mock_rel_pipeline = MagicMock()
+        mock_rel_pipeline.return_value = [{"label": "founded", "score": 0.9}]
+        
+        # Update pipeline mock to return rel pipeline
+        def pipeline_side_effect_rel(task, **kwargs):
+            if task == "ner": return mock_ner_pipeline
+            if task == "text-classification": return mock_rel_pipeline
+            return MagicMock()
+            
+        mock_pipeline.side_effect = pipeline_side_effect_rel
+        
+        relations = extract_relations_huggingface(
+            "Elon Musk founded SpaceX.", 
+            entities=[e1, e2], 
+            model="some-relation-model"
+        )
+        print(f"Relations: {relations}", flush=True)
+        
+        # Verify input formatting
+        # Check if ANY call contained the correct formatting
+        found_match = False
+        for call in mock_rel_pipeline.call_args_list:
+            args, _ = call
+            if "<subj> Elon Musk </subj>" in args[0] and "<obj> SpaceX </obj>" in args[0]:
+                found_match = True
+                break
+        
+        if not found_match:
+            print("Failed to find expected call args in:", flush=True)
+            for call in mock_rel_pipeline.call_args_list:
+                print(f"  {call[0]}", flush=True)
+                
+        assert found_match, "Did not find relation call with Elon Musk as subject"
+
+        # 3. Test Triplets with REBEL parsing
+        print("\n--- Testing Triplets ---", flush=True)
+        
+        # Mock Tokenizer and Model
+        mock_tokenizer_instance = MagicMock()
+        mock_transformers.AutoTokenizer.from_pretrained.return_value = mock_tokenizer_instance
+        mock_tokenizer_instance.encode.return_value = MagicMock()
+        # Mock decode to return REBEL format
+        mock_tokenizer_instance.decode.return_value = "<s><triplet> Elon Musk <subj> founded <obj> SpaceX <triplet> SpaceX <subj> created <obj> Starship</s>"
+        
+        mock_model_instance = MagicMock()
+        mock_transformers.AutoModelForSeq2SeqLM.from_pretrained.return_value = mock_model_instance
+        mock_model_instance.generate.return_value = [MagicMock()]
+        
+        triplets = extract_triplets_huggingface(
+            "Elon Musk founded SpaceX and created Starship.",
+            model="Babelscape/rebel-large"
+        )
+        print(f"Triplets: {triplets}", flush=True)
+        
+        # Verify parsing
+        assert len(triplets) == 2
+        assert triplets[0].subject == "Elon Musk"
+        assert triplets[0].predicate == "founded"
+        assert triplets[0].object == "SpaceX"
+        assert triplets[1].subject == "SpaceX"
+        assert triplets[1].predicate == "created"
+        assert triplets[1].object == "Starship"
+        
+        # Verify skip_special_tokens=False was passed
+        mock_tokenizer_instance.decode.assert_called_with(
+            mock_model_instance.generate.return_value[0], 
+            skip_special_tokens=False
+        )
+
+    except Exception as e:
+        print(f"Error during test execution: {e}", flush=True)
+        traceback.print_exc()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    try:
+        test_enhanced_impl()
+        print("\nAll tests passed!", flush=True)
+    except Exception as e:
+        print(f"\nTest failed: {e}", flush=True)
+        traceback.print_exc()

tests/test_relations_llm.py

@@ -0,0 +1,118 @@
+import unittest
+
+from semantica.semantic_extract.methods import extract_relations_llm
+from semantica.semantic_extract.ner_extractor import Entity
+
+
+class FakeProvider:
+    def __init__(self, typed_payload=None, structured_payload=None):
+        self._typed_payload = typed_payload
+        self._structured_payload = structured_payload
+
+    def is_available(self):
+        return True
+
+    # Simulate typed output return: can be dict or an object with relations
+    def generate_typed(self, prompt, schema, **kwargs):
+        return self._typed_payload if self._typed_payload is not None else {"relations": []}
+
+    def generate_structured(self, prompt, **kwargs):
+        return self._structured_payload if self._structured_payload is not None else {"relations": []}
+
+
+class TestLLMRelationExtraction(unittest.TestCase):
+    def setUp(self):
+        # Minimal realistic text and entities
+        self.text = "Apple reported revenue of $4.4 billion in Q1 2024."
+        self.entities = [
+            Entity(text="Apple", label="ORGANIZATION", start_char=0, end_char=5, confidence=0.99),
+            Entity(text="$4.4 billion", label="MONEY", start_char=26, end_char=39, confidence=0.99),
+            Entity(text="Q1 2024", label="DATE", start_char=43, end_char=51, confidence=0.99),
+        ]
+
+    def _monkeypatch_provider(self, provider_instance):
+        # Monkeypatch create_provider used by extract_relations_llm
+        import semantica.semantic_extract.methods as methods
+        self._orig_create_provider = methods.create_provider
+
+        def _fake_create_provider(provider, model=None, **kwargs):
+            return provider_instance
+
+        methods.create_provider = _fake_create_provider
+
+    def tearDown(self):
+        # Restore original create_provider if patched
+        try:
+            import semantica.semantic_extract.methods as methods
+            if hasattr(self, "_orig_create_provider"):
+                methods.create_provider = self._orig_create_provider
+        except Exception:
+            pass
+
+    def test_typed_relations_parsed(self):
+        # Typed returns a dict compatible with parser
+        typed_payload = {
+            "relations": [
+                {
+                    "subject": "Apple",
+                    "predicate": "HAS_REVENUE",
+                    "object": "$4.4 billion",
+                    "confidence": 0.92,
+                },
+                {
+                    "subject": "Apple",
+                    "predicate": "IN_QUARTER",
+                    "object": "Q1 2024",
+                    "confidence": 0.9,
+                },
+            ]
+        }
+        fake = FakeProvider(typed_payload=typed_payload)
+        self._monkeypatch_provider(fake)
+
+        rels = extract_relations_llm(
+            text=self.text,
+            entities=self.entities,
+            provider="groq",
+            model="llama-3.1-8b-instant",
+            relation_types=["HAS_REVENUE", "IN_QUARTER"],
+            verbose=True,
+        )
+        self.assertGreaterEqual(len(rels), 2, "Expected at least two relations from typed payload")
+        preds = {(r.subject.text, r.predicate, r.object.text) for r in rels}
+        self.assertIn(("Apple", "HAS_REVENUE", "$4.4 billion"), preds)
+        self.assertIn(("Apple", "IN_QUARTER", "Q1 2024"), preds)
+
+    def test_structured_fallback_used(self):
+        # Typed returns zero, structured has content
+        typed_payload = {"relations": []}
+        structured_payload = {
+            "relations": [
+                {
+                    "subject": "Apple",
+                    "predicate": "HAS_REVENUE",
+                    "object": "$4.4 billion",
+                    "confidence": 0.88,
+                }
+            ]
+        }
+        fake = FakeProvider(typed_payload=typed_payload, structured_payload=structured_payload)
+        self._monkeypatch_provider(fake)
+
+        rels = extract_relations_llm(
+            text=self.text,
+            entities=self.entities,
+            provider="groq",
+            model="llama-3.1-8b-instant",
+            relation_types=["HAS_REVENUE"],
+            verbose=True,
+        )
+        self.assertEqual(len(rels), 1, "Expected fallback to structured JSON to yield one relation")
+        r = rels[0]
+        self.assertEqual(r.subject.text, "Apple")
+        self.assertEqual(r.object.text, "$4.4 billion")
+        self.assertEqual(r.predicate, "HAS_REVENUE")
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/vector_store/test_pinecone_removal.py

@@ -1,62 +1,3 @@
-import unittest
-from unittest.mock import MagicMock, patch
-import os
-import sys
-
-# Ensure semantica is in path
-sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
-
-from semantica.vector_store.vector_store import VectorStore
-from semantica.vector_store.registry import method_registry
-from semantica.vector_store.config import vector_store_config
-
-class TestPineconeRemoval(unittest.TestCase):
-    """Verify that Pinecone has been completely removed from the system."""
-
-    def test_pinecone_backend_rejected(self):
-        """Test that initializing VectorStore with backend='pinecone' raises an error."""
-        with self.assertRaises(ValueError) as context:
-            VectorStore(backend="pinecone")
-        
-        # The error message might be generic "Unknown backend" or specific.
-        # We just want to ensure it fails.
-        self.assertTrue("pinecone" in str(context.exception).lower() or "unknown" in str(context.exception).lower())
-
-    def test_registry_clean(self):
-        """Test that no Pinecone methods are registered."""
-        # Check all task types
-        task_types = ["store", "search", "index", "hybrid_search", "metadata", "namespace"]
-        
-        for task in task_types:
-            methods = method_registry.list_all(task)
-            # Flatten if it's a dict
-            if isinstance(methods, dict):
-                method_names = methods.get(task, [])
-            else:
-                method_names = methods
-                
-            for name in method_names:
-                self.assertNotIn("pinecone", name.lower(), f"Found pinecone reference in registry task {task}: {name}")
-
-    def test_config_clean(self):
-        """Test that configuration does not contain Pinecone keys."""
-        config = vector_store_config.get_all()
-        
-        for key in config.keys():
-            self.assertNotIn("pinecone", key.lower(), f"Found pinecone key in config: {key}")
-
-    def test_stores_existence(self):
-        """Verify that other stores exist but PineconeStore does not."""
-        try:
-            from semantica.vector_store import faiss_store
-            from semantica.vector_store import weaviate_store
-            from semantica.vector_store import qdrant_store
-            from semantica.vector_store import milvus_store
-        except ImportError as e:
-            self.fail(f"Failed to import a required store: {e}")
-
-        with self.assertRaises(ImportError):
-            from semantica.vector_store import pinecone_store
-
-if __name__ == '__main__':
-    unittest.main()
+# This test file has been updated as Pinecone support has been re-added to Semantica.
+# Pinecone is now a supported vector store backend (PR #220).
+# See test_pinecone_store.py for Pinecone-specific tests.

tests/vector_store/test_pinecone_store.py

@@ -0,0 +1,250 @@
+import unittest
+from unittest.mock import MagicMock, patch
+import numpy as np
+import sys
+import os
+
+# Ensure semantica is in path if running directly
+if __name__ == "__main__":
+    sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
+
+try:
+    from semantica.vector_store.pinecone_store import (
+        PineconeStore,
+        PineconeClient,
+        PineconeIndex,
+        PineconeSearch,
+        PINECONE_AVAILABLE
+    )
+    from semantica.utils.exceptions import ProcessingError
+except ImportError:
+    # If we can't import, we can't run these tests
+    # But we should not crash silently.
+    # We will define dummy classes if needed or fail loudly.
+    raise
+
+class TestPineconeStore(unittest.TestCase):
+    """Test Pinecone store functionality."""
+
+    def setUp(self):
+        self.mock_logger = MagicMock()
+        self.mock_tracker = MagicMock()
+
+        self.logger_patcher = patch('semantica.vector_store.pinecone_store.get_logger', return_value=self.mock_logger)
+        self.tracker_patcher = patch('semantica.vector_store.pinecone_store.get_progress_tracker', return_value=self.mock_tracker)
+        self.mock_logger_instance = self.logger_patcher.start()
+        self.mock_tracker_instance = self.tracker_patcher.start()
+
+    def tearDown(self):
+        self.logger_patcher.stop()
+        self.tracker_patcher.stop()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_connect(self, mock_pinecone_client):
+        """Test connecting to Pinecone."""
+        mock_client_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        self.assertIsNotNone(store.client)
+        mock_pinecone_client.assert_called_once_with(api_key="test-key")
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', False)
+    def test_connect_unavailable(self):
+        """Test connecting when Pinecone is not available."""
+        store = PineconeStore(api_key="test-key")
+        with self.assertRaises(ProcessingError):
+            store.connect()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_create_index(self, mock_pinecone_client):
+        """Test creating an index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+        mock_client_instance.Index.return_value = mock_index_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Mock the client's create_index method
+        store.client.create_index = MagicMock()
+        store.client.get_index = MagicMock(return_value=mock_index_instance)
+
+        result = store.create_index("test-index", dimension=768, metric="cosine")
+
+        self.assertIsInstance(result, PineconeIndex)
+        self.assertIsInstance(store.index, PineconeIndex)
+        self.assertIsInstance(store.search_engine, PineconeSearch)
+        store.client.create_index.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_upsert_vectors(self, mock_pinecone_client):
+        """Test upserting vectors to Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up index
+        store.index = PineconeIndex(mock_index_instance)
+        store.index.upsert_vectors = MagicMock(return_value={"upserted_count": 2})
+
+        vectors = [np.array([0.1, 0.2, 0.3]), np.array([0.4, 0.5, 0.6])]
+        ids = ["id1", "id2"]
+        metadata = [{"key": "value1"}, {"key": "value2"}]
+
+        result = store.upsert_vectors(vectors, ids, metadata)
+
+        self.assertEqual(result["upserted_count"], 2)
+        store.index.upsert_vectors.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_search_vectors(self, mock_pinecone_client):
+        """Test searching vectors in Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up search engine
+        store.search_engine = PineconeSearch(PineconeIndex(mock_index_instance))
+        store.search_engine.similarity_search = MagicMock(return_value=[
+            {"id": "id1", "score": 0.9, "metadata": {"key": "value1"}}
+        ])
+
+        query_vector = np.array([0.1, 0.2, 0.3])
+        results = store.search_vectors(query_vector, k=5)
+
+        self.assertEqual(len(results), 1)
+        self.assertEqual(results[0]["id"], "id1")
+        store.search_engine.similarity_search.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_delete_vectors(self, mock_pinecone_client):
+        """Test deleting vectors from Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up index
+        store.index = PineconeIndex(mock_index_instance)
+        store.index.delete_vectors = MagicMock(return_value={"deleted": True})
+
+        result = store.delete_vectors(["id1", "id2"])
+
+        self.assertEqual(result["deleted"], True)
+        # Fix: assert called without the empty dict
+        store.index.delete_vectors.assert_called_once_with(["id1", "id2"], "")
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_fetch_vectors(self, mock_pinecone_client):
+        """Test fetching vectors from Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up index
+        store.index = PineconeIndex(mock_index_instance)
+        store.index.fetch_vectors = MagicMock(return_value={
+            "vectors": {
+                "id1": {"values": [0.1, 0.2], "metadata": {"key": "value1"}}
+            }
+        })
+
+        result = store.fetch_vectors(["id1"])
+
+        self.assertIn("vectors", result)
+        # Fix: assert called without the empty dict
+        store.index.fetch_vectors.assert_called_once_with(["id1"], "")
+
+
+class TestPineconeClient(unittest.TestCase):
+    """Test PineconeClient wrapper."""
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_create_index(self, mock_pinecone_client):
+        """Test creating an index via PineconeClient."""
+        mock_client_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        client = PineconeClient(mock_client_instance)
+        client.create_index("test-index", 768, "cosine")
+
+        mock_client_instance.create_index.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_list_indexes(self, mock_pinecone_client):
+        """Test listing indexes via PineconeClient."""
+        mock_client_instance = MagicMock()
+        mock_index_obj = MagicMock()
+        mock_index_obj.name = "test-index"
+        mock_client_instance.list_indexes.return_value = [mock_index_obj]
+        mock_pinecone_client.return_value = mock_client_instance
+
+        client = PineconeClient(mock_client_instance)
+        result = client.list_indexes()
+
+        self.assertEqual(result, ["test-index"])
+
+
+class TestPineconeIndex(unittest.TestCase):
+    """Test PineconeIndex wrapper."""
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_upsert_vectors(self):
+        """Test upserting vectors via PineconeIndex."""
+        mock_index = MagicMock()
+        mock_response = MagicMock()
+        mock_response.upserted_count = 2
+        mock_index.upsert.return_value = mock_response
+
+        index = PineconeIndex(mock_index)
+        result = index.upsert_vectors(
+            [[0.1, 0.2], [0.3, 0.4]],
+            ["id1", "id2"],
+            [{"key": "value1"}]
+        )
+
+        self.assertEqual(result["upserted_count"], 2)
+        mock_index.upsert.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_search_vectors(self):
+        """Test searching vectors via PineconeIndex."""
+        mock_index = MagicMock()
+        mock_match = MagicMock()
+        mock_match.id = "id1"
+        mock_match.score = 0.9
+        mock_match.metadata = {"key": "value1"}
+        mock_response = MagicMock()
+        mock_response.matches = [mock_match]
+        mock_index.query.return_value = mock_response
+
+        index = PineconeIndex(mock_index)
+        result = index.search_vectors([0.1, 0.2], k=5)
+
+        self.assertEqual(len(result), 1)
+        self.assertEqual(result[0]["id"], "id1")
+        mock_index.query.assert_called_once()
+
+
+if __name__ == '__main__':
+    print("DEBUG: Starting unittest.main()")
+    unittest.main()

tests/vector_store/test_vector_store_parallel.py

@@ -0,0 +1,149 @@
+
+import unittest
+import numpy as np
+import time
+import sys
+import os
+import logging
+from unittest.mock import MagicMock, patch
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
+
+from semantica.vector_store import VectorStore
+from semantica.utils.exceptions import ProcessingError
+
+
+class TestVectorStoreParallel(unittest.TestCase):
+    def setUp(self):
+        logging.getLogger("vector_store").setLevel(logging.ERROR)
+
+        self.dimension = 4
+        self.store = VectorStore(
+            backend="inmemory",
+            dimension=self.dimension,
+        )
+
+        self.store.embedder = MagicMock()
+
+    def test_embed_batch_success(self):
+        texts = ["a", "b", "c"]
+        expected_embeddings = [
+            np.array([0.1] * 4, dtype=np.float32),
+            np.array([0.2] * 4, dtype=np.float32),
+            np.array([0.3] * 4, dtype=np.float32),
+        ]
+
+        self.store.embedder.generate_embeddings.return_value = expected_embeddings
+
+        results = self.store.embed_batch(texts)
+
+        self.assertEqual(len(results), 3)
+        self.assertTrue(np.allclose(results[0], expected_embeddings[0]))
+        self.store.embedder.generate_embeddings.assert_called_once_with(texts)
+
+    def test_embed_batch_fallback(self):
+        texts = ["a", "b"]
+
+        self.store.embedder.generate_embeddings.side_effect = Exception("Model error")
+
+        results = self.store.embed_batch(texts)
+
+        self.assertEqual(len(results), 2)
+        self.assertEqual(results[0].shape, (self.dimension,))
+        self.assertTrue(isinstance(results[0], np.ndarray))
+
+    def test_add_documents_empty(self):
+        ids = self.store.add_documents([])
+        self.assertEqual(ids, [])
+
+    def test_add_documents_metadata_mismatch(self):
+        with self.assertRaises(ValueError):
+            self.store.add_documents(["doc1"], metadata=[{}, {}])
+
+    def test_add_documents_parallel_success(self):
+        num_docs = 10
+        documents = [f"doc_{i}" for i in range(num_docs)]
+        metadata = [{"id": i} for i in range(num_docs)]
+
+        def mock_embed_batch(texts):
+            return [np.full(self.dimension, float(i)) for i, _ in enumerate(texts)]
+
+        with patch.object(self.store, "embed_batch", side_effect=mock_embed_batch):
+            ids = self.store.add_documents(
+                documents,
+                metadata,
+                batch_size=2,
+                parallel=True,
+            )
+
+        self.assertEqual(len(ids), num_docs)
+        self.assertEqual(len(self.store.vectors), num_docs)
+
+        for i, vec_id in enumerate(ids):
+            stored_meta = self.store.get_metadata(vec_id)
+            self.assertEqual(stored_meta["id"], i)
+
+    def test_add_documents_sequential_success(self):
+        num_docs = 5
+        documents = [f"doc_{i}" for i in range(num_docs)]
+
+        with patch.object(self.store, "embed_batch") as mock_batch:
+            mock_batch.return_value = [np.zeros(self.dimension) for _ in range(num_docs)]
+
+            ids = self.store.add_documents(documents, parallel=False)
+
+            self.assertEqual(len(ids), num_docs)
+            self.assertEqual(mock_batch.call_count, 1)
+
+    def test_add_documents_error_propagation(self):
+        documents = ["doc1", "doc2"]
+
+        with patch.object(self.store, "embed_batch", side_effect=ValueError("Embedding Error")):
+            with self.assertRaises(Exception):
+                self.store.add_documents(documents, parallel=True)
+
+    def test_performance_simulation(self):
+        num_batches = 4
+        batch_delay = 0.1
+        batch_size = 1
+        documents = [f"doc_{i}" for i in range(num_batches)]
+
+        def slow_embed(texts):
+            time.sleep(batch_delay)
+            return [np.zeros(self.dimension) for _ in texts]
+
+        with patch.object(self.store, "embed_batch", side_effect=slow_embed):
+            start_seq = time.time()
+            self.store.add_documents(documents, batch_size=batch_size, parallel=False)
+            dur_seq = time.time() - start_seq
+
+            self.store.vectors = {}
+
+            start_par = time.time()
+            self.store.add_documents(documents, batch_size=batch_size, parallel=True)
+            dur_par = time.time() - start_par
+
+            print(f"\nPerformance Test:")
+            print(f"Sequential Duration: {dur_seq:.4f}s")
+            print(f"Parallel Duration:   {dur_par:.4f}s")
+            print(f"Speedup:             {dur_seq / dur_par:.2f}x")
+
+            self.assertLess(dur_par, dur_seq * 0.7)
+
+    def test_add_documents_batch_size_edge_cases(self):
+        documents = ["a", "b", "c"]
+
+        with patch.object(self.store, "embed_batch") as mock_batch:
+            mock_batch.side_effect = lambda texts: [np.zeros(4) for _ in texts]
+
+            self.store.add_documents(documents, batch_size=100)
+            self.assertEqual(mock_batch.call_count, 1)
+
+            mock_batch.reset_mock()
+
+            self.store.add_documents(documents, batch_size=1)
+            self.assertEqual(mock_batch.call_count, 3)
+
+
+if __name__ == "__main__":
+    unittest.main()