chore: release v0.2.5

docs: update CONTRIBUTING.md and CONTRIBUTORS.md with improved format…

.github/FUNDING.yml

@@ -1,8 +1,3 @@
 # Funding options for Semantica
-# Uncomment and add your usernames/links below
-
-# github: [username]
-# patreon: username
-# ko_fi: username
-# custom: ["https://your-funding-page.com"]
+github: Hawksight-AI
 

.github/SUPPORT.md

@@ -32,6 +32,8 @@ For enterprise support, custom development, or consulting services:
 
 ## Sponsorship
 
+### Sponsor this project
+
 Support Semantica development:
 - [GitHub Sponsors](https://github.com/sponsors/Hawksight-AI)
 

.pre-commit-config.yaml

@@ -5,6 +5,7 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-yaml
+        exclude: 'neptune-setup\.yaml$'
       - id: check-json
       - id: check-toml
       - id: check-added-large-files
@@ -49,9 +50,15 @@ repos:
     hooks:
       - id: yamllint
         args: ['-d', '{extends: default, rules: {line-length: {max: 120}}}']
+        exclude: 'neptune-setup\.yaml$'
+
+  - repo: https://github.com/aws-cloudformation/cfn-lint
+    rev: v1.43.3
+    hooks:
+      - id: cfn-lint
+        files: 'neptune-setup\.yaml$'
 
   # Removed slow hooks for faster development:
   # - mypy: Type checking (can be run manually or in CI)
   # - bandit: Security scanning (can be run separately)
   # - pytest: Testing (should be run manually, not on every commit)
-

CHANGELOG.md

@@ -7,6 +7,180 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.5] - 2026-01-27
+
+### Added
+- **Pinecone Vector Store Support**:
+    - Implemented native Pinecone support (`PineconeStore`) with full CRUD capabilities.
+    - Added support for serverless and pod-based indexes, namespaces, and metadata filtering.
+    - Integrated with `VectorStore` unified interface and registry.
+    - (Closes #219, Resolves #220)
+- **Configurable LLM Retry Logic**:
+    - Exposed `max_retries` parameter in `NERExtractor`, `RelationExtractor`, `TripletExtractor` and low-level extraction methods (`extract_entities_llm`, `extract_relations_llm`, `extract_triplets_llm`).
+    - Defaults to 3 retries to prevent infinite loops during JSON validation failures or API timeouts.
+    - Propagated retry configuration through chunked processing helpers to ensure consistent behavior for long documents.
+    - Updated `03_Earnings_Call_Analysis.ipynb` to use `max_retries=3` by default.
+
+### Added
+- **Bring Your Own Model (BYOM) Support**:
+    - Enabled full support for custom Hugging Face models in `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Added support for custom tokenizers in `HuggingFaceModelLoader` to handle models with non-standard tokenization requirements.
+    - Implemented robust fallback logic for model selection: runtime options (`extract(model=...)`) now correctly override configuration defaults.
+- **Enhanced NER Implementation**:
+    - Added configurable aggregation strategies (`simple`, `first`, `average`, `max`) to `extract_entities_huggingface` for better sub-word token handling.
+    - Implemented robust IOB/BILOU parsing to reconstruct entities from raw model outputs when structured output is unavailable.
+    - Added confidence scoring for aggregated entities.
+- **Relation Extraction Improvements**:
+    - Implemented standard entity marker technique (wrapping subject/object with `<subj>`, `<obj>` tags) in `extract_relations_huggingface` for compatibility with sequence classification models.
+    - Added structured output parsing to convert raw model predictions into validated `Relation` objects.
+- **Triplet Extraction Completion**:
+    - Added specialized parsing for Seq2Seq models (e.g., REBEL) in `extract_triplets_huggingface` to generate structured triplets directly from text.
+    - Implemented post-processing logic to clean and validate generated triplets.
+
+### Fixed
+- **LLM Extraction Stability**:
+    - Fixed infinite retry loops in `BaseProvider` by strictly enforcing `max_retries` limit during structured output generation.
+    - Resolved stuck execution in earnings call analysis notebooks when using smaller models (e.g., Llama 3 8B) that frequently produce invalid JSON.
+- **Model Parameter Precedence**:
+    - Fixed issue where configuration defaults took precedence over runtime arguments in Hugging Face extractors. Runtime options now correctly override config values.
+- **Import Handling**:
+    - Fixed circular import issues in test suites by implementing robust mocking strategies.
+
+## [0.2.4] - 2026-01-22
+
+### Added
+- **Ontology Ingestion Module**:
+    - Implemented `OntologyIngestor` in `semantica.ingest` for parsing RDF/OWL files (Turtle, RDF/XML, JSON-LD, N3) into standardized `OntologyData` objects.
+    - Added `ingest_ontology` convenience function and integrated it into the unified `ingest(source_type="ontology")` interface.
+    - Added recursive directory scanning support for batch ontology ingestion.
+    - Exposed ingestion tools in `semantica.ontology` for better discoverability.
+    - Added `OntologyData` dataclass for consistent metadata handling (source path, format, timestamps).
+- **Documentation**:
+    - **Ontology Usage Guide**: Updated `ontology_usage.md` with comprehensive examples for single-file and directory ingestion.
+    - **API Reference**: Updated `ontology.md` with `OntologyIngestor` class documentation and method details.
+- **Tests**:
+    - **Comprehensive Test Suite**: Added `tests/ingest/test_ontology_ingestor.py` covering all supported formats, error handling, and unified interface integration.
+    - **Demo Script**: Added `examples/demo_ontology_ingest.py` for end-to-end usage demonstration.
+
+## [0.2.3] - 2026-01-20
+
+### Fixed
+- **LLM Relation Extraction Parsing**:
+    - Fixed relation extraction returning zero relations despite successful API calls to Groq and other providers
+    - Normalized typed responses from instructor/OpenAI/Groq to consistent dict format before parsing
+    - Added structured JSON fallback when typed generation yields zero relations to avoid silent empty outputs
+    - Removed acceptance of extra kwargs (`max_tokens`, `max_entities_prompt`) from relation extraction internals
+    - Filtered kwargs passed to provider LLM calls to only `temperature` and `verbose`
+- **API Parameter Handling**:
+    - Limited kwargs forwarded in chunked extraction helper to prevent parameter leakage
+    - Ensured minimal, safe parameters are passed to provider calls
+- **Pipeline Circular Import (Issues #192, #193)**:
+    - Fixed circular import between `pipeline_builder` and `pipeline_validator` triggered during `semantica.pipeline` import
+    - Lazy-loaded `PipelineValidator` inside `PipelineBuilder.__init__` and guarded type hints with `TYPE_CHECKING`
+    - Ensured `from semantica.deduplication import DuplicateDetector` no longer fails even when pipeline module is imported
+- **JupyterLab Progress Output (Issue #181)**:
+    - Added `SEMANTICA_DISABLE_JUPYTER_PROGRESS` environment variable to disable rich Jupyter/Colab progress tables
+    - When enabled, progress falls back to console-style output, preventing infinite scrolling and JupyterLab out-of-memory errors
+
+### Added
+- **Comprehensive Test Suite**:
+-    - Added unit tests (`tests/test_relations_llm.py`) with mocked LLM provider covering both typed and structured response paths
+-    - Added integration tests (`tests/integration/test_relations_groq.py`) for real Groq API calls with environment variable API key
+-    - Tests validate relation extraction completion and result parsing across different response formats
+- **Amazon Neptune Dev Environment**:
+-    - Added CloudFormation template (`cookbook/introduction/neptune-setup.yaml`) to provision a dev Neptune cluster with public endpoint and IAM auth enabled
+-    - Documented deployment, cost estimates, and IAM User vs IAM Role best practices in `cookbook/introduction/21_Amazon_Neptune_Store.ipynb`
+-    - Added `cfn-lint` to `.pre-commit-config.yaml` for validating CloudFormation templates while excluding `neptune-setup.yaml` from generic YAML linters
+- **Vector Store High-Performance Ingestion**:
+-    - Added `VectorStore.add_documents` for high-throughput ingestion with automatic embedding generation, batching, and parallel processing
+-    - Added `VectorStore.embed_batch` helper for generating embeddings for lists of texts without immediately storing them
+-    - Enabled default parallel ingestion in `VectorStore` with `max_workers=6` for common workloads
+-    - Added dedicated documentation page `docs/vector_store_usage.md` describing high-performance vector store usage and configuration
+-    - Added `tests/vector_store/test_vector_store_parallel.py` covering parallel vs sequential performance, error handling, and edge cases for `add_documents` and `embed_batch`
+
+### Changed
+- **Relation Extraction API**:
+-    - Simplified parameter interface by removing unused kwargs that were previously ignored
+-    - Improved error handling and verbose logging for debugging relation extraction issues
+-    - Enhanced robustness of post-response parsing across different LLM providers
+- **Vector Store Defaults and Examples**:
+-    - Standardized `VectorStore` default concurrency to `max_workers=6` for parallel ingestion
+-    - Updated vector store reference documentation and usage guides to rely on implicit defaults instead of requiring manual `max_workers` configuration in examples
+
+
+## [0.2.2] - 2026-01-15
+
+### Added
+- **Parallel Extraction Engine**:
+    - Implemented high-throughput parallel batch processing across all core extractors (`NERExtractor`, `RelationExtractor`, `TripletExtractor`, `EventDetector`, `SemanticNetworkExtractor`) using `concurrent.futures.ThreadPoolExecutor`.
+    - Added `max_workers` configuration parameter (default: 1) to all extractor `extract()` methods, allowing users to tune concurrency based on available CPU cores or API rate limits.
+    - **Parallel Chunking**: Implemented parallel processing for large document chunking in `_extract_entities_chunked` and `_extract_relations_chunked`, significantly reducing latency for long-form text analysis.
+    - **Thread-Safe Progress Tracking**: Enhanced `ProgressTracker` to handle concurrent updates from multiple threads without race conditions during batch processing.
+- **Semantic Extract Performance & Regression**:
+    - Added edge-case regression suite covering max worker defaults, LLM prompt entity filtering, and extractor reuse.
+    - Added a runnable real-use-case benchmark script for batch latency across `NERExtractor`, `RelationExtractor`, `TripletExtractor`, `EventDetector`, `SemanticAnalyzer`, and `SemanticNetworkExtractor`.
+    - Added Groq LLM smoke tests that exercise LLM-based entities/relations/triplets when `GROQ_API_KEY` is available via environment configuration.
+
+### Security
+- **Credential Sanitization**:
+    - Removed hardcoded API keys from 8 cookbook notebooks to prevent secret leakage.
+    - Enforced environment variable usage for `GROQ_API_KEY` across all examples.
+- **Secure Caching**:
+    - Updated `ExtractionCache` to exclude sensitive parameters (e.g., `api_key`, `token`, `password`) from cache key generation, preventing secret leakage and enabling safe cache sharing.
+    - Upgraded cache key hashing algorithm from MD5 to **SHA-256** for enhanced collision resistance and security.
+
+### Changed
+- **Gemini SDK Migration**:
+    - Migrated `GeminiProvider` to use the new `google-genai` SDK (v0.1.0+) to address deprecation warnings.
+    - Implemented graceful fallback to `google.generativeai` for backward compatibility.
+- **Dependency Resolution**:
+    - Pinned `opentelemetry-api` and `opentelemetry-sdk` to `1.37.0` to resolve pip conflicts.
+    - Updated `protobuf` and `grpcio` constraints for better stability.
+- **Entity Filtering Scope**:
+    - Removed entity filtering from non-LLM extraction flows to avoid accuracy regressions.
+    - Applied entity downselection only to LLM relation prompt construction, while matching returned entities against the full original entity list.
+- **Batch Concurrency Defaults**:
+    - Standardized `max_workers` defaulting across `semantic_extract` and tuned for low-latency: ML-backed methods default to single-worker, while pattern/regex/rules/LLM/huggingface methods use a higher parallelism default capped by CPU.
+    - Raised the global `optimization.max_workers` default to 8 for better throughput on batch workloads.
+
+### Performance
+- **Bottleneck Optimization (GitHub Issue #186)**:
+    - **Resolved Bottleneck #1 (Sequential Processing)**: Replaced sequential `for` loops with parallel execution for both document-level batches and intra-document chunks.
+    - **Performance Gains**: Achieved **~1.89x speedup** in real-world extraction scenarios (tested with Groq `llama-3.3-70b-versatile` on standard datasets).
+    - **Initialization Optimization**: Refactored test suite to use class-level `setUpClass` for LLM provider initialization, eliminating redundant API client creation overhead.
+- **Low-Latency Entity Matching**:
+    - Avoided heavyweight embedding stack imports on common matches by improving fast matching heuristics and short-circuiting before embedding similarity.
+    - Optimized entity matching to prioritize exact/substring/word-boundary matches and only fall back to embedding similarity when needed, reducing CPU overhead in LLM relation/triplet mapping.
+
+
+## [0.2.1] - 2026-01-12
+
+### Fixed
+- **LLM Output Stability (Bug #176)**:
+    - Fixed incomplete JSON output issues by correctly propagating `max_tokens` parameter in `extract_relations_llm`.
+    - Implemented automatic error handling that halves chunk sizes and retries when LLM context or output limits are exceeded.
+    - Fixed `AttributeError` in provider integration by ensuring consistent parameter passing via `**kwargs`.
+- **Constraint Relaxations**:
+    - Removed hardcoded `max_length` constraints from `Entity`, `Relation`, and `Triplet` classes to support long-form semantic extraction (e.g., long descriptions or names).
+- Fixed orchestrator lazy property initialization and configuration normalization logic in `Orchestrator`.
+- Resolved `AssertionError` in orchestrator tests by aligning test mocks with production component usage.
+- Fixed dependency compatibility issues by pinning `protobuf>=5.29.1,<7.0` and `grpcio>=1.71.2`.
+- Added missing dependencies `GitPython` and `chardet` to `pyproject.toml`.
+- Verified and aligned `FileObject.text` property usage in GraphRAG notebooks for consistent content decoding.
+
+### Changed
+- **Chunking Defaults**:
+    - Increased default `max_text_length` for auto-chunking to **64,000 characters** (from 32k/16k) for OpenAI, Anthropic, Gemini, Groq, and DeepSeek providers.
+    - Unified chunking logic across `extract_entities_llm`, `extract_relations_llm`, and `extract_triplets_llm`.
+- **Groq Support**:
+    - Standardized Groq provider defaults to use `llama-3.3-70b-versatile` with a 64k context window.
+    - Added native support for `max_tokens` and `max_completion_tokens` to prevent output truncation.
+
+### Added
+- **Testing**:
+    - Added `tests/reproduce_issue_176.py` to validate `max_tokens` propagation and chunking behavior across all extractors.
+
+
 ## [0.2.0] - 2026-01-10
 
 ### Added

CONTRIBUTING.md

@@ -1,306 +1,266 @@
 # Contributing to Semantica
 
-Thank you for your interest in contributing to Semantica! This document provides guidelines and instructions for contributing to the project.
+Thank you for your interest in contributing! Every contribution, no matter how small, is valuable. 🎉
 
-## Table of Contents
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-- [Code of Conduct](#code-of-conduct)
-- [Getting Started](#getting-started)
-- [Development Setup](#development-setup)
-- [Code Style Guidelines](#code-style-guidelines)
-- [Testing Requirements](#testing-requirements)
-- [Commit Message Conventions](#commit-message-conventions)
-- [Pull Request Process](#pull-request-process)
-- [Documentation Standards](#documentation-standards)
-- [Types of Contributions](#types-of-contributions)
-- [Getting Help](#getting-help)
+> **New to contributing?** Start with a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/vqRt2qbx) community.
 
-## Code of Conduct
+---
 
-This project adheres to a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the maintainers.
+## 🚀 Quick Start
 
-## Getting Started
+1. Find a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue)
+2. [Fork Semantica](https://github.com/Hawksight-AI/semantica/fork) & clone the repository
+3. Make your changes
+4. Submit a pull request!
 
-1. **Fork the repository** on GitHub
-2. **Clone your fork** locally:
-   ```bash
-   git clone https://github.com/your-username/semantica.git
-   cd semantica
-   ```
-3. **Add the upstream remote**:
-   ```bash
-   git remote add upstream https://github.com/Hawksight-AI/semantica.git
-   ```
+**Need help?** Join [Discord](https://discord.gg/vqRt2qbx) or [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
 
-## Development Setup
+---
 
-### Prerequisites
+## 🎯 Ways to Contribute
 
-- Python 3.8 or higher (3.9+ recommended)
-- pip package manager
-- Git
+### 💻 Code
 
-### Installation
+**What you can do:**
+- Fix bugs
+- Add new features
+- Improve code quality (add type hints, docstrings, improve error messages)
+- Optimize performance
 
-1. **Create a virtual environment** (recommended):
-   ```bash
-   python -m venv venv
-   source venv/bin/activate  # On Windows: venv\Scripts\activate
-   ```
+**Where:** `semantica/` directory
 
-2. **Install the project in editable mode with dev dependencies**:
-   ```bash
-   pip install -e ".[dev]"
-   ```
+**Good first issues:** Add docstrings, type hints, or improve error messages
 
-3. **Install pre-commit hooks**:
-   ```bash
-   pre-commit install
-   ```
+---
 
-### Verify Installation
+### 📝 Documentation
+
+**What you can do:**
+- Fix typos and grammar errors
+- Improve clarity and readability
+- Add code examples and tutorials
+- Create new cookbook notebooks
+- Improve API documentation (docstrings)
+- Create troubleshooting guides
+- Update installation instructions
+- Add missing documentation
+
+**Where:** `README.md`, `docs/`, `cookbook/`, docstrings in code
+
+**Good first issues:** Fix typos, add examples, create cookbook tutorials, improve docstrings
+
+**Documentation formatting:**
+- Use clear, concise language
+- Include code examples where helpful
+- Follow markdown best practices
+- Use proper headings hierarchy
+- Add links to related sections
+- Include screenshots for UI-related docs
+
+---
+
+### 🧪 Testing
+
+**What you can do:**
+- Add unit tests
+- Improve test coverage
+- Add integration tests
+
+**Where:** `tests/` directory
+
+**Good first issues:** Add tests for specific functions or classes
+
+---
+
+### 🐛 Bug Reports
+
+**What:** Report bugs you find
+
+**How:** Use the [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
+
+**Include:** Description, steps to reproduce, expected vs actual behavior, environment details
+
+---
+
+### 💡 Feature Requests
+
+**What:** Suggest new features or improvements
+
+**How:** Use the [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
+
+**Include:** Problem statement, proposed solution, use cases
+
+---
+
+### 🎨 Cookbook & Examples
+
+**What:** Create tutorials and examples
+
+**Where:** `cookbook/` directory
+
+**Examples:** Create new notebooks, add examples, improve existing tutorials
+
+---
+
+### 💬 Community Support
+
+**What:** Help others in the community
+
+**Where:** [Discord](https://discord.gg/vqRt2qbx), [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
+
+**Examples:** Answer questions, review PRs, share your projects
+
+---
+
+### 🎓 Educational Content
+
+**What:** Create educational materials
+
+**Examples:** Blog posts, video tutorials, talks, workshops, case studies
+
+---
+
+### 🔧 Other Contributions
+
+- **Design & Graphics:** Logos, diagrams, visualizations
+- **Tools & Integrations:** CLI tools, integrations with other frameworks
+- **Infrastructure:** CI/CD improvements, Docker optimization
+- **Security:** Report security vulnerabilities (privately)
+
+---
+
+## 📋 Getting Started
+
+### 1. Fork & Clone
+
+First, [fork Semantica](https://github.com/Hawksight-AI/semantica/fork) on GitHub, then:
 
 ```bash
-python -c "import semantica; print(semantica.__version__)"
-pytest --version
-black --version
+git clone https://github.com/your-username/semantica.git
+cd semantica
+git remote add upstream https://github.com/Hawksight-AI/semantica.git
 ```
 
-## Code Style Guidelines
-
-We use several tools to maintain code quality and consistency:
-
-### Formatting
-
-- **Black**: Code formatting (line length: 88)
-  ```bash
-  black semantica/
-  ```
-
-- **isort**: Import sorting
-  ```bash
-  isort semantica/
-  ```
-
-### Linting
-
-- **flake8**: Style guide enforcement
-  ```bash
-  flake8 semantica/
-  ```
-
-- **mypy**: Static type checking
-  ```bash
-  mypy semantica/
-  ```
-
-### Running All Checks
+### 2. Set Up Environment
 
 ```bash
-# Format code
-black semantica/ tests/
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # Windows: venv\Scripts\activate
 
-# Sort imports
-isort semantica/ tests/
+# Install dev dependencies
+pip install -e ".[dev]"
 
-# Lint
-flake8 semantica/ tests/
-
-# Type check
-mypy semantica/
+# Install pre-commit hooks (optional)
+pre-commit install
 ```
 
-Or use pre-commit hooks (automatically runs on commit):
-```bash
-pre-commit run --all-files
-```
-
-## Testing Requirements
-
-### Running Tests
+### 3. Create Branch
 
 ```bash
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=semantica --cov-report=html
-
-# Run specific test file
-pytest tests/test_specific.py
-
-# Run with verbose output
-pytest -v
+git checkout -b feature/your-feature-name
+# or
+git checkout -b fix/bug-description
 ```
 
-### Test Coverage
+### 4. Make Changes
 
-- Minimum coverage: **80%**
-- Critical modules: **90%+**
-- Coverage reports are generated in `htmlcov/`
+- Follow code style (see below)
+- Add tests for new features
+- Update documentation
 
-### Writing Tests
+### 5. Run Checks
 
-- Follow pytest conventions
-- Use descriptive test names
-- Include docstrings for complex tests
-- Test both success and failure cases
-- Use fixtures for common setup
-
-Example:
-```python
-def test_entity_extraction():
-    """Test basic entity extraction functionality."""
-    from semantica.semantic_extract import NamedEntityRecognizer
-    
-    ner = NamedEntityRecognizer()
-    entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
-    
-    assert len(entities) > 0
-    assert any(e.text == "Apple Inc." for e in entities)
+```bash
+pytest                          # Run tests
+black semantica/ tests/        # Format code
+isort semantica/ tests/         # Sort imports
+flake8 semantica/ tests/        # Lint
 ```
 
-## Commit Message Conventions
+Or use pre-commit hooks: `pre-commit run --all-files`
 
-We follow [Conventional Commits](https://www.conventionalcommits.org/) specification:
+### 6. Commit & Push
 
-### Format
-
-```
-<type>(<scope>): <subject>
-
-<body>
-
-<footer>
+```bash
+git commit -m "feat(module): add new feature"
+git push origin feature/your-feature-name
 ```
 
-### Types
+Then create a pull request on GitHub!
 
-- `feat`: New feature
-- `fix`: Bug fix
-- `docs`: Documentation changes
-- `style`: Code style changes (formatting, etc.)
-- `refactor`: Code refactoring
-- `test`: Adding or updating tests
-- `chore`: Maintenance tasks
-- `perf`: Performance improvements
-- `ci`: CI/CD changes
+---
 
-### Examples
+## 📐 Code Style
+
+We use automated tools:
+
+| Tool     | Purpose                    | Command                    |
+|----------|----------------------------|----------------------------|
+| **Black** | Code formatting            | `black semantica/ tests/` |
+| **isort** | Import sorting             | `isort semantica/ tests/` |
+| **flake8** | Style enforcement          | `flake8 semantica/ tests/` |
+| **mypy** | Type checking              | `mypy semantica/`          |
+
+**Run all:** `black semantica/ tests/ && isort semantica/ tests/ && flake8 semantica/ tests/ && mypy semantica/`
+
+---
+
+## 🧪 Testing
+
+```bash
+pytest                          # Run all tests
+pytest --cov=semantica         # With coverage
+pytest tests/test_file.py      # Specific file
+```
+
+**Coverage goal:** 80% minimum, 90%+ for critical modules
+
+---
+
+## 📝 Commit Messages
+
+Use [Conventional Commits](https://www.conventionalcommits.org/):
 
 ```
 feat(kg): add temporal graph support
-
-Add support for temporal knowledge graphs with version tracking
-and time-based queries.
-
-Closes #123
+fix(parse): handle empty PDF files
+docs(readme): add installation guide
+test(extract): add unit tests
 ```
 
-```
-fix(parse): handle empty PDF files gracefully
+**Types:** `feat`, `fix`, `docs`, `test`, `refactor`, `perf`, `style`, `chore`
 
-Previously, empty PDF files would cause a crash. Now they return
-an empty document with appropriate warnings.
+---
 
-Fixes #456
-```
+## ✅ PR Checklist
 
-## Pull Request Process
-
-### Before Submitting
-
-1. **Update your fork**:
-   ```bash
-   git fetch upstream
-   git checkout main
-   git merge upstream/main
-   ```
-
-2. **Create a feature branch**:
-   ```bash
-   git checkout -b feature/your-feature-name
-   # or
-   git checkout -b fix/bug-description
-   ```
-
-3. **Make your changes** and commit following our conventions
-
-4. **Run all checks**:
-   ```bash
-   pytest
-   black semantica/ tests/
-   isort semantica/ tests/
-   flake8 semantica/ tests/
-   mypy semantica/
-   ```
-
-5. **Push to your fork**:
-   ```bash
-   git push origin feature/your-feature-name
-   ```
-
-### PR Checklist
+Before submitting:
 
 - [ ] Code follows style guidelines
 - [ ] Tests pass locally
-- [ ] New tests added for new features
+- [ ] New tests added (if applicable)
 - [ ] Documentation updated
 - [ ] Commit messages follow conventions
 - [ ] No merge conflicts
-- [ ] PR description is clear and complete
 
-### PR Description Template
+---
 
-```markdown
-## Description
-Brief description of changes
+## 📖 Documentation Standards
 
-## Type of Change
-- [ ] Bug fix
-- [ ] New feature
-- [ ] Breaking change
-- [ ] Documentation update
+### Code Documentation (Docstrings)
 
-## Related Issues
-Closes #123
-Related to #456
+**Format:** Use Google-style docstrings
 
-## Testing
-- [ ] Tests pass locally
-- [ ] Added new tests
-- [ ] Updated existing tests
-
-## Checklist
-- [ ] Code follows style guidelines
-- [ ] Self-review completed
-- [ ] Comments added for complex code
-- [ ] Documentation updated
-- [ ] No new warnings generated
-```
-
-## Documentation Standards
-
-### Code Documentation
-
-- Use Google-style docstrings
-- Include type hints
-- Document all public functions and classes
-- Include examples for complex functions
-
-Example:
 ```python
-def extract_entities(
-    text: str,
-    model: str = "transformer",
-    confidence_threshold: float = 0.7
-) -> List[Entity]:
+def extract_entities(text: str, model: str = "transformer") -> List[Entity]:
     """Extract named entities from text.
     
     Args:
         text: Input text to process
         model: NER model to use (default: "transformer")
-        confidence_threshold: Minimum confidence score (default: 0.7)
     
     Returns:
         List of extracted Entity objects
@@ -309,92 +269,98 @@ def extract_entities(
         ValueError: If text is empty or model is invalid
     
     Example:
-        >>> ner = NamedEntityRecognizer()
+        >>> from semantica.semantic_extract import NERExtractor
+        >>> ner = NERExtractor(method="ml", model="en_core_web_sm")
         >>> entities = ner.extract("Apple Inc. was founded in 1976.")
         >>> len(entities)
         2
     """
-    ...
 ```
 
-### Documentation Files
+### Markdown Documentation Formatting
 
-- Update relevant documentation in `docs/`
-- Add examples to cookbook if applicable
-- Update API reference if adding new public APIs
-- Keep README.md up to date
+**General Guidelines:**
+- Use clear headings (H1 for title, H2 for main sections, H3 for subsections)
+- Keep paragraphs short and focused
+- Use bullet points for lists
+- Add code blocks with syntax highlighting
+- Include links to related documentation
 
-## Types of Contributions
+**Code Blocks:**
+- Use triple backticks with language identifier: ` ```python `, ` ```bash `
+- Include comments in code examples
+- Show expected output when helpful
 
-### 💻 Code Contributions
+**Examples:**
 
-- **Bug Fixes**: Resolving issues reported in the issue tracker.
-- **New Features**: Implementing new capabilities (please discuss via an issue first!).
-- **Refactoring**: Improving code structure and maintainability without changing behavior.
-- **Algorithm Optimization**: Improving the efficiency of graph algorithms and vector search.
+```markdown
+## Section Title
 
-#### ⚡ Performance and Latency
-We deeply value efficiency. Contributions that make Semantica faster and lighter are highly appreciated!
+Brief introduction paragraph.
 
-- **Latency Reduction**: Optimize critical paths and RAG pipeline response times.
-- **Memory Optimization**: Reduce graph/vector processing memory footprint.
-- **Throughput**: Improve operations per second (bulk ingestion, parallel queries).
-- **Benchmarks**: Add performance benchmarks to track regressions.
-- **Async/Concurrency**: Enhance asynchronous execution and concurrency.
+### Subsection
 
-### 📚 Documentation Contributions
+- Bullet point 1
+- Bullet point 2
 
-- Fix typos and grammar
-- Improve clarity
-- Add examples
-- Create tutorials
-- Translate documentation
+**Code example:**
 
-### Testing Contributions
+```python
+from semantica import SomeClass
 
-- Add test coverage
-- Improve test quality
-- Add integration tests
-- Performance benchmarks
+instance = SomeClass()
+result = instance.method()
+```
 
-### Other Contributions
+**Note:** Additional context or warnings.
+```
 
-- Answer questions in discussions
-- Help with issues
-- Review pull requests
-- Share use cases
-- Report bugs
-- Suggest features
+**Best Practices:**
+- Start with an overview/introduction
+- Use consistent terminology
+- Include "See also" links
+- Add examples for complex concepts
+- Keep formatting consistent across docs
 
-## Getting Help
+---
 
-### Communication Channels
+## 🆘 Getting Help
 
-- **GitHub Discussions**: General questions and discussions
-- **GitHub Issues**: Bug reports and feature requests
-- **Discord**: Real-time chat and community support
+- 💬 [Discord](https://discord.gg/vqRt2qbx) - Real-time chat
+- 💭 [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) - Q&A
+- 🐛 [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) - Bug reports
 
-### Before Asking for Help
+**Before asking:** Check existing documentation, search issues/discussions, review cookbook examples
 
-1. Check existing documentation
-2. Search GitHub issues and discussions
-3. Review code examples in cookbook
-4. Check FAQ in documentation
+---
 
-### Asking Good Questions
+## 🏆 Recognition
 
-- Provide context and environment details
-- Include code examples
-- Show what you've tried
-- Include error messages and logs
-- Be specific about what you need
-
-## Recognition
-
-Contributors are recognized in:
+All contributors are recognized in:
 - [CONTRIBUTORS.md](CONTRIBUTORS.md)
 - GitHub contributors page
-- Release notes for significant contributions
+- Release notes
 
-Thank you for contributing to Semantica! 🎉
+We follow the [all-contributors](https://allcontributors.org) specification!
 
+---
+
+## 📜 Code of Conduct
+
+This project follows a [Code of Conduct](CODE_OF_CONDUCT.md). Be respectful and inclusive.
+
+---
+
+## 📚 Resources
+
+- [README.md](README.md) - Project overview
+- [Cookbook](cookbook/) - Tutorials and examples
+- [Documentation](docs/) - Comprehensive guides
+
+---
+
+**Thank you for contributing!** 🚀
+
+Every contribution matters - whether it's a single line of code, a typo fix, a helpful answer, or a bug report. We appreciate you! 🙏
+
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**

CONTRIBUTORS.md

@@ -4,44 +4,31 @@ Thank you to all the people who have contributed to Semantica! 🎉
 
 This project follows the [all-contributors](https://allcontributors.org) specification. Contributions of any kind are welcome!
 
-## How to Contribute
+⭐ **Give us a Star** • 🍴 **Fork us** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-We welcome contributions of all kinds! Whether you're:
-- Writing code
-- Improving documentation
-- Reporting bugs
-- Suggesting features
-- Answering questions
-- Reviewing pull requests
-- Sharing use cases
-- Creating examples
-
-All contributions are valuable and appreciated!
+---
 
 ## Contribution Types
 
+We recognize all types of contributions:
+
 - 💻 **Code**: Writing code, fixing bugs, implementing features
 - 📝 **Documentation**: Writing docs, tutorials, examples
+- 🧪 **Testing**: Writing tests, improving test coverage
 - 🐛 **Bug Reports**: Finding and reporting bugs
 - 💡 **Ideas**: Suggesting new features or improvements
 - 🎨 **Design**: UI/UX improvements, graphics, branding
-- 📖 **Examples**: Creating code examples and tutorials
-- 🔍 **Testing**: Writing tests, improving test coverage
-- 💬 **Answering Questions**: Helping others in discussions
-- 📢 **Talks**: Giving talks, presentations, workshops
-- 🌍 **Translation**: Translating documentation
+- 🎨 **Cookbook**: Creating tutorials and examples
+- 💬 **Community**: Answering questions, reviewing PRs
+- 🎓 **Education**: Blog posts, video tutorials, talks, workshops
 - 🔧 **Tools**: Creating tools, scripts, integrations
 - 📦 **Packaging**: Improving build, release, distribution
 - ⚠️ **Security**: Reporting security vulnerabilities
-- 🎓 **Education**: Teaching, mentoring, tutorials
-- 📹 **Video**: Creating video content, tutorials
-- 🎵 **Audio**: Podcasts, audio content
-- 📸 **Photography**: Screenshots, images
-- 🔬 **Research**: Research, analysis, studies
-- 💰 **Financial**: Sponsoring, funding
 - 🏗️ **Infrastructure**: CI/CD, hosting, infrastructure
 - 🚇 **Maintenance**: Maintenance, triage, project management
 
+---
+
 ## Contributors
 
 <!-- ALL-CONTRIBUTORS-LIST:START -->
@@ -50,48 +37,78 @@ All contributions are valuable and appreciated!
 
 <!-- ALL-CONTRIBUTORS-LIST:END -->
 
+---
+
 ## Recognition
 
-### Top Contributors
+All contributors are recognized in:
 
-Contributors are recognized based on their contributions to the project. Recognition includes:
+- This contributors list
+- [GitHub contributors page](https://github.com/Hawksight-AI/semantica/graphs/contributors)
+- Release notes for significant contributions
+- Community appreciation
 
-- Listing in this file
-- GitHub contributor statistics
-- Special mentions in release notes
-- Featured showcases for significant contributions
-
-### Hall of Fame
-
-Special recognition for exceptional contributions:
-
-- **Coming soon** - We'll feature outstanding contributors here!
+---
 
 ## How to Add Yourself
 
-If you've contributed to Semantica and want to be added to this list:
+### Automatic Recognition
 
-1. **Automatic**: If you've made a commit, you'll appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors)
-2. **Manual**: Open a PR adding yourself to this file, or use the [@all-contributors bot](https://allcontributors.org/docs/en/bot/usage)
+If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors).
 
-Example:
-```markdown
-- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
-```
+### Using All-Contributors Bot
 
-## All Contributors Bot
-
-We use the [all-contributors](https://allcontributors.org) bot to automatically recognize contributors. To add a contributor, comment on an issue or PR:
+Comment on any issue or PR with:
 
 ```
 @all-contributors please add @username for code, docs, bug
 ```
 
-## Thank You!
+**Examples:**
 
-Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community!
+```
+@all-contributors please add @johndoe for code
+@all-contributors please add @janedoe for docs, bug
+@all-contributors please add @devuser for code, test, maintenance
+```
+
+### Manual Addition
+
+Open a PR adding yourself to this file:
+
+```markdown
+- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
+```
 
 ---
 
-**Want to contribute?** Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
+## Contribution Type Codes
 
+When using the all-contributors bot, use these codes:
+
+- `code` - Code contributions
+- `doc` - Documentation
+- `test` - Testing
+- `bug` - Bug reports
+- `ideas` - Feature requests/ideas
+- `design` - Design work
+- `example` - Cookbook/examples
+- `question` - Answering questions
+- `talk` - Talks/presentations
+- `tool` - Tools/integrations
+- `packaging` - Packaging/distribution
+- `security` - Security reports
+- `infra` - Infrastructure
+- `maintenance` - Maintenance
+
+See [all-contributors specification](https://allcontributors.org/docs/en/emoji-key) for complete list.
+
+---
+
+## Thank You!
+
+Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community! 🙏
+
+**Want to contribute?** 
+
+⭐ Give us a Star • 🍴 [Fork us](https://github.com/Hawksight-AI/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!

README.md

@@ -1,204 +1,236 @@
 <div align="center">
 
-<img src="semantica_logo.png" alt="Semantica Logo" width="450" height="auto">
+<img src="semantica_logo.png" alt="Semantica Logo" width="460"/>
 
 # 🧠 Semantica
+### Open-Source Semantic Layer & Knowledge Engineering Framework
 
-[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/)
+[![PyPI](https://img.shields.io/pypi/v/semantica.svg)](https://pypi.org/project/semantica/)
 [![Monthly Downloads](https://img.shields.io/pypi/dm/semantica)](https://pypi.org/project/semantica/)
 [![Total Downloads](https://static.pepy.tech/badge/semantica)](https://pepy.tech/project/semantica)
-[![Discord](https://img.shields.io/badge/Discord-Join%20Us-7289da?style=flat&logo=discord&logoColor=white)](https://discord.gg/pMHguUzG)
 [![CI](https://github.com/Hawksight-AI/semantica/workflows/CI/badge.svg)](https://github.com/Hawksight-AI/semantica/actions)
+[![Discord](https://img.shields.io/badge/Discord-Join-7289da?logo=discord&logoColor=white)](https://discord.gg/RgaGTj9J)
 
-<p align="center">
-    <a href="https://github.com/Hawksight-AI/semantica/stargazers">
-        <img src="https://img.shields.io/badge/Give%20a%20Star-%E2%AD%90-yellow?style=for-the-badge&labelColor=555555" alt="Give a Star">
-    </a>
-    &nbsp;&nbsp;
-    <a href="https://github.com/Hawksight-AI/semantica/fork">
-        <img src="https://img.shields.io/badge/Support%20Project-Fork%20Us-blue?style=for-the-badge&labelColor=555555" alt="Support Project">
-    </a>
-</p>
+### ⭐ Give us a Star • 🍴 Fork us • 💬 Join our Discord
 
-**Open Source Framework for Semantic Layer & Knowledge Engineering**
-
-> **Transform chaotic data into intelligent knowledge.**
-
-*The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.*
-
-**100% Open Source** • **MIT Licensed** • **Latest Version: 0.2.0** • **Production Ready** • **Community Driven**
-
-[**Discord**](https://discord.gg/pMHguUzG)
+> **Transform Choas into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.**
 
 </div>
 
-## What is Semantica?
-
-Semantica bridges the gap between raw data chaos and AI-ready knowledge. It's a **semantic intelligence platform** that transforms unstructured data into structured, queryable knowledge graphs powering GraphRAG, AI agents, and multi-agent systems.
-
-### What Makes Semantica Different?
-
-Unlike traditional approaches that process isolated documents and extract text into vectors, Semantica understands **semantic relationships across all content**, provides **automated ontology generation**, and builds a **unified semantic layer** with **production-grade QA**.
-
-| **Traditional Approaches** | **Semantica's Approach** |
-|:---------------------------|:-------------------------|
-| Process data as isolated documents | Understands semantic relationships across all content |
-| Extract text and store vectors | Builds knowledge graphs with meaningful connections |
-| Generic entity recognition | General-purpose ontology generation and validation |
-| Manual schema definition | Automatic semantic modeling from content patterns |
-| Disconnected data silos | Unified semantic layer across all data sources |
-| Basic quality checks | Production-grade QA with conflict detection & resolution |
 
 ---
 
-## 🎯 The Problem We Solve
+## 🚀 Why Semantica?
 
-### The Semantic Gap
+**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and compliant.
 
-Organizations today face a **fundamental mismatch** between how data exists and how AI systems need it.
-
-#### The Semantic Gap: Problem vs. Solution
-
-Organizations have **unstructured data** (PDFs, emails, logs), **messy data** (inconsistent formats, duplicates, conflicts), and **disconnected silos** (no shared context, missing relationships). AI systems need **clear rules** (formal ontologies), **structured entities** (validated, consistent), and **relationships** (semantic connections, context-aware reasoning).
-
-| **What Organizations Have** | **What AI Systems Require** |
-|:------------------------------|:------------------------------|
-| **Unstructured Data** | **Clear Rules** |
-| PDFs, emails, logs | Formal ontologies |
-| Mixed schemas | Graphs & Networks |
-| Conflicting facts | |
-| **Messy, Noisy Data** | **Structured Entities** |
-| Inconsistent formats | Validated entities |
-| Duplicate records | Domain Knowledge |
-| Missing relationships | |
-| **Disconnected, Siloed Data** | **Relationships** |
-| Data in separate systems | Semantic connections |
-| No shared context | Context-Aware Reasoning |
-| Isolated knowledge | |
-
-### **SEMANTICA FRAMEWORK**
-
-Semantica operates through three integrated layers that transform raw data into AI-ready knowledge:
-
-**Input Layer** — Universal ingestion from multiple data formats (PDFs, DOCX, HTML, JSON, CSV, databases, live feeds, APIs, streams, archives, multi-modal content) into a unified pipeline.
-
-**Semantic Layer** — Core intelligence engine performing entity extraction, relationship mapping, ontology generation, context engineering, and quality assurance. Includes **advanced entity deduplication** (Jaro-Winkler, disjoint property handling) to ensure a clean single source of truth.
-
-**Output Layer** — Production-ready knowledge graphs, vector embeddings, and validated ontologies that power GraphRAG systems, AI agents, and multi-agent systems.
-
-**Powers: GraphRAG, AI Agents, Multi-Agent Systems**
-
-
-
-
-
-### What Happens Without Semantics?
-
-**They Break** — Systems crash due to inconsistent formats and missing structure.
-
-**They Hallucinate** — AI models generate false information without semantic context to validate outputs.
-
-**They Fail Silently** — Systems return wrong answers without warnings, leading to bad decisions.
-
-**Why?** Systems have data — not semantics. They can't connect concepts, understand relationships, validate against domain rules, or detect conflicts.
+Perfect for **high-stakes domains** where mistakes have real consequences.
 
 ---
 
-## 💡 The Semantica Solution
+### ⚡ Get Started in 30 Seconds
 
-**Semantica** is an **open-source framework** that closes the semantic gap between real-world messy data and the structured semantic layers required by advanced AI systems — GraphRAG, agents, multi-agent systems, reasoning models, and more.
+```bash
+pip install semantica
+```
 
-### How Semantica Solves These Problems
+```python
+from semantica.semantic_extract import NERExtractor
+from semantica.kg import GraphBuilder
 
-**Efficient Embeddings** — Uses **FastEmbed** by default for high-performance, lightweight local embedding generation (faster than sentence-transformers).
+# Extract entities and build knowledge graph
+ner = NERExtractor(method="ml", model="en_core_web_sm")
+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
+kg = GraphBuilder().build({"entities": entities, "relationships": []})
 
-**Universal Data Ingestion** — Handles multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams) with unified pipeline, no custom parsers needed.
+print(f"Built KG with {len(kg.get('entities', []))} entities")
+```
 
-**Automated Semantic Extraction** — NER, relationship extraction, and triplet generation with LLM enhancement. Includes **auto-chunking** for long documents and **robust error handling** with automatic retry logic.
-
-**Knowledge Graph Construction** — Production-ready graphs with entity resolution, temporal support, and graph analytics. Queryable knowledge ready for AI applications.
-
-**GraphRAG Engine** — Hybrid vector + graph retrieval achieves 91% accuracy (30% improvement) via semantic search + graph traversal for multi-hop reasoning. Features LLM-generated responses grounded in knowledge graph context with reasoning traces. [See Comparison Benchmark](cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
-
-**AI Agent Context Engineering** — Persistent memory with RAG + knowledge graphs enables context maintenance, action validation, and structured knowledge access.
-
-**Automated Ontology Generation** — 6-stage LLM pipeline generates validated OWL ontologies with HermiT/Pellet validation, eliminating manual engineering.
-
-**Production-Grade QA** — Conflict detection, deduplication, quality scoring, and provenance tracking ensure trusted, production-ready knowledge graphs.
-
-**Pipeline Orchestration** — Flexible pipeline builder with parallel execution enables scalable processing via orchestrator-worker pattern.
-
-### Core Features at a Glance
-
-| **Feature Category** | **Capabilities** | **Key Benefits** |
-|:---------------------|:-----------------|:------------------|
-| **Data Ingestion** | Multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams, archives) | Universal ingestion, no custom parsers needed |
-| **Semantic Extraction** | NER, relations, triplets, LLM enhancement, **auto-chunking** | Automated discovery with robust error handling |
-| **Knowledge Graphs** | Entity resolution, temporal support, graph analytics, query interface | Production-ready, queryable knowledge structures |
-| **Ontology Generation** | 6-stage LLM pipeline, OWL generation, HermiT/Pellet validation | Automated ontology creation from documents |
-| **GraphRAG** | Hybrid vector + graph retrieval, multi-hop reasoning, LLM-generated responses | 91% accuracy, 30% improvement over vector-only, reasoning traces |
-| **LLM Providers** | Unified interface to 100+ LLMs (Groq, OpenAI, HuggingFace, LiteLLM) | Clean imports, multiple providers, structured output |
-| **Agent Memory** | Persistent memory (Save/Load), Hybrid Retrieval (Vector+Graph), FastEmbed support | Context-aware agents with semantic understanding |
-| **Pipeline Orchestration** | Parallel execution, custom steps, orchestrator-worker pattern | Scalable, flexible data processing |
-| **Quality Assurance** | Conflict detection, deduplication, quality scoring, provenance | Trusted knowledge graphs ready for production |
+**[📖 Full Quick Start](#-quick-start)** • **[🍳 Cookbook Examples](#-semantica-cookbook)** • **[💬 Join Discord](https://discord.gg/RgaGTj9J)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)**
 
 ---
 
-## 👥 Who Is This For?
+## Core Value Proposition
 
-Semantica is designed for **developers, data engineers, and organizations** building the next generation of AI applications that require semantic understanding and knowledge graphs.
+| **Trustworthy** | **Explainable** | **Auditable** |
+|:------------------:|:------------------:|:-----------------:|
+| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking |
+| Rule-based governance | Entity relationships & ontologies | Source-level provenance |
+| Production-grade QA | Multi-hop graph reasoning | Audit-ready compliance |
 
-### Who Uses Semantica
+---
 
-**AI/ML Engineers & Data Scientists** — Build GraphRAG systems, AI agents, and multi-agent systems.
+## Key Features & Benefits
 
-**Data Engineers** — Build scalable pipelines with semantic enrichment.
+### Not Just Another Agentic Framework
 
-**Knowledge Engineers & Ontologists** — Create knowledge graphs and ontologies with automated pipelines.
+**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with:
 
-**Enterprise Data Teams** — Unify semantic layers, improve data quality, resolve conflicts.
+| Feature | Benefit |
+|:--------|:--------|
+| **Auditable** | Complete provenance tracking with full audit trails |
+| **Explainable** | Transparent reasoning paths with entity relationships |
+| **Provenance-Aware** | Source-level provenance from documents to responses |
+| **Validated** | Built-in conflict detection, deduplication, QA |
+| **Governed** | Rule-based validation and semantic consistency |
 
-**Software & DevOps Engineers** — Build semantic APIs and infrastructure with production-ready SDK.
+### Perfect For High-Stakes Use Cases
 
-**Analysts & Researchers** — Transform data into queryable knowledge graphs for insights.
+| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** |
+|:-----------------:|:--------------:|:------------:|
+| Clinical decisions | Fraud detection | Evidence-backed research |
+| Drug interactions | Regulatory compliance | Contract analysis |
+| Patient safety | Risk assessment | Case law reasoning |
 
-**Security & Compliance Teams** — Threat intelligence, regulatory reporting, audit trails.
+| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** |
+|:-------------------:|:----------------:|:-------------------:|:-----------------:|
+| Threat attribution | Policy decisions | Power grids | Decision logs |
+| Incident response | Classified info | Transportation | Safety validation |
 
-**Product Teams & Startups** — Rapid prototyping of AI products and semantic features.
+### Powers Your AI Stack
+
+- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search
+- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory
+- **Reasoning Models** — Explainable AI decisions with reasoning paths
+- **Enterprise AI** — Governed, auditable platforms for compliance
+
+### Integrations
+
+- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)
+- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication
+- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)
+
+> **Built for environments where every answer must be explainable and governed.**
+
+
+---
+
+## 🚨 The Problem: The Semantic Gap
+
+### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.
+
+### Understanding the Semantic Gap
+
+The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).
+
+**Traditional AI approaches:**
+- Rely on statistical patterns and text similarity
+- Cannot understand relationships between entities
+- Cannot reason about domain-specific rules
+- Cannot explain why decisions were made
+- Cannot trace back to original sources with confidence
+
+**High-stakes AI requires:**
+- Semantic understanding of entities and their relationships
+- Domain knowledge encoded as formal rules (ontologies)
+- Explainable reasoning paths
+- Source-level provenance
+- Conflict detection and resolution
+
+**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.
+
+### What Organizations Have vs What They Need
+
+| **Current State** | **Required for High-Stakes AI** |
+|:---------------------|:-----------------------------------|
+| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |
+| APIs, databases, streams | Structured and validated entities |
+| Conflicting facts and duplicates | Explicit semantic relationships |
+| Siloed systems with no lineage | **Explainable reasoning paths** |
+| | **Source-level provenance** |
+| | **Audit-ready compliance** |
+
+### The Cost of Missing Semantics
+
+- **Decisions cannot be explained** — No transparency in AI reasoning
+- **Errors cannot be traced** — No way to debug or improve
+- **Conflicts go undetected** — Contradictory information causes failures
+- **Compliance becomes impossible** — No audit trails for regulations
+
+**Trustworthy AI requires semantic accountability.**
+
+---
+
+## 🆚 Semantica vs Traditional RAG
+
+| Feature | Traditional RAG | Semantica |
+|:--------|:----------------|:----------|
+| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths |
+| **Provenance** | ❌ No provenance | ✅ Source-level provenance |
+| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning |
+| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection |
+| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments |
+| **Compliance** | ❌ No audit trails | ✅ Audit-ready provenance |
+
+---
+
+## 🧩 Semantica Architecture
+
+### 1️⃣ Input Layer — Governed Ingestion
+- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX
+- 🔧 **Docling Support** — Docling parser for table extraction
+- 💾 **Data Sources** — Databases, APIs, streams, archives, web content
+- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction
+- 📊 **Single Pipeline** — Unified ingestion with metadata and source tracking
+
+### 2️⃣ Semantic Layer — Trust & Reasoning Engine
+- 🔍 **Entity Extraction** — NER, normalization, classification
+- 🔗 **Relationship Discovery** — Triplet generation, semantic links
+- 📐 **Ontology Induction** — Automated domain rule generation
+- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution
+- ✅ **Quality Assurance** — Conflict detection, validation
+- 📊 **Provenance Tracking** — Source, time, confidence metadata
+- 🧠 **Reasoning Traces** — Explainable inference paths
+
+### 3️⃣ Output Layer — Auditable Knowledge Assets
+- 📊 **Knowledge Graphs** — Queryable, temporal, explainable
+- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support
+- 🔢 **Vector Embeddings** — FastEmbed by default
+- ☁️ **AWS Neptune** — Amazon Neptune graph database support
+- 🔍 **Provenance** — Every AI response links back to:
+  - 📄 Source documents
+  - 🏷️ Extracted entities & relations
+  - 📐 Ontology rules applied
+  - 🧠 Reasoning steps used
+
+---
+
+## 🏥 Built for High-Stakes Domains
+
+Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:
+
+- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety compliance
+- **💰 Finance & Risk** — Fraud detection, regulatory compliance (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation
+- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change management, case law reasoning
+- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis
+- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence
+- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response
+- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation  
+
+---
+
+## 👥 Who Uses Semantica?
+
+- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents
+- **⚙️ Data Engineers** — Creating governed semantic pipelines
+- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale
+- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure
+- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems  
 
 ---
 
 ## 📦 Installation
 
-> **✅ Available on PyPI!** Semantica is now published on PyPI. Install it with a single command: `pip install semantica`
-
-**Prerequisites:** Python 3.8+ (3.9+ recommended) • pip (latest version)
-
 ### Install from PyPI (Recommended)
 
 ```bash
-# Install latest version from PyPI
 pip install semantica
-
-# Or install with optional dependencies
+# or
 pip install semantica[all]
-
-# GitHub Workaround (if PyPI version has issues)
-pip install git+https://github.com/Hawksight-AI/semantica.git@main
-
-# Verify installation
-python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')"
 ```
 
-**Current Version:** [![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/) • [View on PyPI](https://pypi.org/project/semantica/)
-
-!!! info "Windows PyTorch Note"
-    If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica.
-
-
-
 ### Install from Source (Development)
 
 ```bash
@@ -258,7 +290,7 @@ print(f" Ingested {len(sources)} sources")
 
 ### Document Parsing & Processing
 
-> **Multi-format parsing** • **Text normalization** • **Intelligent chunking**
+> **Multi-format parsing** • **Docling Support** • **Text normalization** • **Intelligent chunking**
 
 ```python
 from semantica.parse import DocumentParser, DoclingParser
@@ -269,7 +301,7 @@ from semantica.split import TextSplitter
 parser = DocumentParser()
 parsed = parser.parse("document.pdf", format="auto")
 
-# Enhanced parsing with Docling (recommended for complex layouts/tables)
+# Parsing with Docling (for complex layouts/tables)
 # Requires: pip install docling
 docling_parser = DoclingParser(enable_ocr=True)
 result = docling_parser.parse("complex_table.pdf")
@@ -360,7 +392,7 @@ results = vector_store.search(query="supply chain", top_k=5)
 
 ### Graph Store & Triplet Store
 
-> **Neo4j, FalkorDB, Amazon Neptune support** • **SPARQL queries** • **RDF triplets**
+> **Neo4j, FalkorDB, Amazon Neptune** • **SPARQL queries** • **RDF triplets**
 
 ```python
 from semantica.graph_store import GraphStore
@@ -398,15 +430,19 @@ results = triplet_store.execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT
 
 ### Ontology Generation & Management
 
-> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation
+> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation • **Custom Ontology Import** (OWL, RDF, Turtle, JSON-LD)
 
 ```python
 from semantica.ontology import OntologyGenerator
+from semantica.ingest import ingest_ontology
 
+# Generate ontology automatically
 generator = OntologyGenerator(llm_provider="openai", model="gpt-4")
 ontology = generator.generate_from_documents(sources=["domain_docs/"])
 
-print(f"Classes: {len(ontology.classes)}")
+# Or import your existing ontology
+custom_ontology = ingest_ontology("my_ontology.ttl")  # Supports OWL, RDF, Turtle, JSON-LD
+print(f"Classes: {len(custom_ontology.classes)}")
 ```
 
 [**Cookbook: Ontology**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/14_Ontology.ipynb)
@@ -476,7 +512,7 @@ reasoned_result = context.query_with_reasoning(
 
 ### Knowledge Graph-Powered RAG (GraphRAG)
 
-> **30% Accuracy Improvement** • Vector + Graph Hybrid Search • 91% Accuracy • **Multi-Hop Reasoning** • **LLM-Generated Responses**
+> **Vector + Graph Hybrid Search** • **Multi-Hop Reasoning** • **LLM-Generated Responses** • **Semantic Re-ranking**
 
 ```python
 from semantica.context import AgentContext
@@ -525,7 +561,7 @@ print(f"Confidence: {result['confidence']:.3f}")
 from semantica.llms import Groq, OpenAI, HuggingFaceLLM, LiteLLM
 import os
 
-# Groq - Fast inference
+# Groq
 groq = Groq(
     model="llama-3.1-8b-instant",
     api_key=os.getenv("GROQ_API_KEY")
@@ -555,7 +591,7 @@ structured = groq.generate_structured("Extract entities from: Apple Inc. was fou
 ```
 
 **Supported Providers:**
-- **Groq**: Fast inference with Llama models
+- **Groq**: Inference with Llama models
 - **OpenAI**: GPT-3.5, GPT-4, and other OpenAI models
 - **HuggingFace**: Local LLM inference with Transformers
 - **LiteLLM**: Unified interface to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, and more)
@@ -755,7 +791,7 @@ print(f"Found {len(results)} results")
 
 #### Cybersecurity
 - [**Real-Time Anomaly Detection**](cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) - CVE RSS, Kafka streams, temporal KGs, sentence chunking
-- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, enhanced GraphRAG, deduplication
+- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, GraphRAG, deduplication
 
 #### Intelligence & Law Enforcement
 - [**Criminal Network Analysis**](cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) - OSINT RSS, deduplication, network centrality, graph analytics
@@ -772,12 +808,16 @@ print(f"Found {len(results)} results")
 
 ## 🔬 Advanced Features
 
+**Docling Integration** — Document parsing with table extraction for PDFs, DOCX, PPTX, and XLSX files. Supports OCR and multiple export formats.
+
+**AWS Neptune Support** — Amazon Neptune graph database integration with IAM authentication and OpenCypher queries.
+
+**Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD, N3) and extend Schema.org, FOAF, Dublin Core, or custom ontologies.
+
 **Incremental Updates** — Real-time stream processing with Kafka, RabbitMQ, Kinesis for live updates.
 
 **Multi-Language Support** — Process multiple languages with automatic detection.
 
-**Custom Ontology Import** — Import and extend Schema.org and custom ontologies.
-
 **Advanced Reasoning** — Forward/backward chaining, Rete-based pattern matching, and automated explanation generation.
 
 **Graph Analytics** — Centrality, community detection, path finding, temporal analysis.
@@ -867,20 +907,11 @@ git push origin feature/your-feature
 4. **Feature Requests** - [Request feature](https://github.com/Hawksight-AI/semantica/issues/new)
 
 
-###  Contributors
-
-<a href="https://github.com/Hawksight-AI/semantica/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=Hawksight-AI/semantica" alt="Contributors" />
-</a>
-
 ## 📜 License
 
 Semantica is licensed under the **MIT License** - see the [LICENSE](https://github.com/Hawksight-AI/semantica/blob/main/LICENSE) file for details.
 
-<div align="center">
-
 **Built by the Semantica Community**
 
-[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/pMHguUzG)
 
-</div>
+[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/RgaGTj9J)

RELEASE.md

@@ -26,10 +26,10 @@ Before releasing, ensure:
 
 The project uses GitHub Actions for automated releases to PyPI.
 
-1. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.0`).
+1.29. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.3`).
    ```bash
-   git tag -a v0.2.0 -m "Release v0.2.0"
-   git push origin v0.2.0
+   git tag -a v0.2.3 -m "Release v0.2.3"
+   git push origin v0.2.3
    ```
 2. **GitHub Action**: The `Release` workflow will automatically trigger, build the package, create a GitHub Release, and publish to PyPI using Trusted Publishing.
 

SECURITY.md

@@ -6,6 +6,9 @@ We actively support the following versions of Semantica with security updates:
 
 | Version | Supported          |
 | ------- | ------------------ |
+| 0.2.3   | :white_check_mark: |
+| 0.2.2   | :white_check_mark: |
+| 0.2.1   | :white_check_mark: |
 | 0.2.0   | :white_check_mark: |
 | 0.1.1   | :white_check_mark: |
 | 0.1.0   | :white_check_mark: |

cookbook/introduction/21_Amazon_Neptune_Store.ipynb

@@ -25,6 +25,57 @@
     "- AWS credentials configured (boto3, environment variables, or IAM role)\n",
     "- Network access to your Neptune cluster (VPC, security groups)\n",
     "\n",
+    "#### Quick Setup with CloudFormation\n",
+    "\n",
+    "If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n",
+    "\n",
+    "```bash\n",
+    "# Deploy the Neptune stack (takes ~15-20 minutes)\n",
+    "aws cloudformation create-stack \\\n",
+    "  --stack-name semantica-neptune \\\n",
+    "  --template-body file://neptune-setup.yaml \\\n",
+    "  --capabilities CAPABILITY_NAMED_IAM\n",
+    "\n",
+    "# Wait for stack creation to complete\n",
+    "aws cloudformation wait stack-create-complete --stack-name semantica-neptune\n",
+    "\n",
+    "# Get the outputs (endpoint, port, credentials)\n",
+    "aws cloudformation describe-stacks --stack-name semantica-neptune \\\n",
+    "  --query 'Stacks[0].Outputs' --output table\n",
+    "```\n",
+    "\n",
+    "The template creates:\n",
+    "- VPC with public subnets and Internet Gateway\n",
+    "- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
+    "- IAM user with least-privilege access for OpenCypher queries\n",
+    "- Security group allowing Bolt protocol (port 8182) access\n",
+    "\n",
+    "> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
+    "\n",
+    "**Outputs:**\n",
+    "- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
+    "- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
+    "- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
+    "- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
+    "- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
+    "\n",
+    "**Cleanup:**\n",
+    "```bash\n",
+    "aws cloudformation delete-stack --stack-name semantica-neptune\n",
+    "```\n",
+    "\n",
+    "**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n",
+    "\n",
+    "| Resource | Cost (USD) |\n",
+    "| --- | --- |\n",
+    "| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n",
+    "| Storage (10 GB) | ~1/month |\n",
+    "| I/O requests | ~1-5/month |\n",
+    "| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
+    "| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
+    "\n",
+    "> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
+    "\n",
     "---"
    ]
   },
@@ -70,14 +121,21 @@
     "import os\n",
     "\n",
     "# Neptune cluster configuration - REPLACE WITH YOUR VALUES\n",
+    "# (Get these from CloudFormation stack outputs)\n",
     "os.environ[\"NEPTUNE_ENDPOINT\"] = \"your-cluster.us-east-1.neptune.amazonaws.com\"\n",
     "os.environ[\"NEPTUNE_PORT\"] = \"8182\"\n",
     "os.environ[\"AWS_REGION\"] = \"us-east-1\"\n",
     "\n",
-    "# AWS credentials (if using IAM Auth and not relying on IAM role or ~/.aws/credentials)\n",
-    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n",
-    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n",
-    "# os.environ[\"AWS_SESSION_TOKEN\"] = \"your-session-token\"\n",
+    "# AWS credentials for IAM Authentication\n",
+    "# Option 1: IAM User (static credentials from CloudFormation template)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"AKIA...\"  # From AwsAccessKeyId output\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # From AwsSecretAccessKey output\n",
+    "# Note: No AWS_SESSION_TOKEN needed for IAM users\n",
+    "\n",
+    "# Option 2: IAM Role / Temporary credentials (e.g., STS AssumeRole, EC2 instance role)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"ASIA...\"  # Temporary access key\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # Temporary secret key\n",
+    "# os.environ[\"AWS_SESSION_TOKEN\"] = \"...\"      # REQUIRED for temporary credentials\n",
     "\n",
     "print(f\"Neptune Endpoint: {os.environ.get('NEPTUNE_ENDPOINT')}\")\n",
     "print(f\"AWS Region: {os.environ.get('AWS_REGION')}\")"

cookbook/introduction/neptune-setup.yaml

@@ -0,0 +1,228 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: >
+  Amazon Neptune cluster with public endpoint, IAM authentication, and least-privilege
+  IAM user for Semantica cookbook. Uses db.t3.medium (most cost-effective Neptune instance type).
+
+Parameters:
+  EnvironmentName:
+    Type: String
+    Default: semantica-neptune
+    Description: Environment name prefix for resource naming
+
+Resources:
+  # =============================================================================
+  # VPC & NETWORKING
+  # =============================================================================
+
+  VPC:
+    Type: AWS::EC2::VPC
+    Properties:
+      CidrBlock: 10.0.0.0/16
+      EnableDnsHostnames: true
+      EnableDnsSupport: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-vpc
+
+  InternetGateway:
+    Type: AWS::EC2::InternetGateway
+    Properties:
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-igw
+
+  InternetGatewayAttachment:
+    Type: AWS::EC2::VPCGatewayAttachment
+    Properties:
+      InternetGatewayId: !Ref InternetGateway
+      VpcId: !Ref VPC
+
+  PublicSubnet1:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [0, !GetAZs '']
+      CidrBlock: 10.0.1.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-1
+
+  PublicSubnet2:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [1, !GetAZs '']
+      CidrBlock: 10.0.2.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-2
+
+  PublicRouteTable:
+    Type: AWS::EC2::RouteTable
+    Properties:
+      VpcId: !Ref VPC
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-rt
+
+  DefaultPublicRoute:
+    Type: AWS::EC2::Route
+    DependsOn: InternetGatewayAttachment
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      DestinationCidrBlock: 0.0.0.0/0
+      GatewayId: !Ref InternetGateway
+
+  PublicSubnet1RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet1
+
+  PublicSubnet2RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet2
+
+  # =============================================================================
+  # SECURITY GROUP
+  # =============================================================================
+
+  NeptuneSecurityGroup:
+    Type: AWS::EC2::SecurityGroup
+    Properties:
+      GroupName: !Sub ${EnvironmentName}-neptune-sg
+      GroupDescription: Security group for Neptune cluster - allows Bolt protocol access
+      VpcId: !Ref VPC
+      SecurityGroupIngress:
+        - IpProtocol: tcp
+          FromPort: 8182
+          ToPort: 8182
+          CidrIp: 0.0.0.0/0
+          Description: Allow Bolt protocol access from anywhere
+      SecurityGroupEgress:
+        - IpProtocol: -1
+          CidrIp: 0.0.0.0/0
+          Description: Allow all outbound traffic
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-neptune-sg
+
+  # =============================================================================
+  # NEPTUNE CLUSTER
+  # =============================================================================
+
+  NeptuneSubnetGroup:
+    Type: AWS::Neptune::DBSubnetGroup
+    Properties:
+      DBSubnetGroupDescription: Subnet group for Neptune cluster
+      DBSubnetGroupName: !Sub ${EnvironmentName}-subnet-group
+      SubnetIds:
+        - !Ref PublicSubnet1
+        - !Ref PublicSubnet2
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-subnet-group
+
+  NeptuneCluster:
+    Type: AWS::Neptune::DBCluster
+    Properties:
+      DBClusterIdentifier: !Sub ${EnvironmentName}-cluster
+      DBSubnetGroupName: !Ref NeptuneSubnetGroup
+      VpcSecurityGroupIds:
+        - !Ref NeptuneSecurityGroup
+      EngineVersion: '1.4.6.3'
+      IamAuthEnabled: true
+      StorageEncrypted: true
+      DeletionProtection: false
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-cluster
+
+  NeptuneInstance:
+    Type: AWS::Neptune::DBInstance
+    Properties:
+      DBInstanceIdentifier: !Sub ${EnvironmentName}-instance
+      DBInstanceClass: db.t3.medium
+      DBClusterIdentifier: !Ref NeptuneCluster
+      PubliclyAccessible: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-instance
+
+  # =============================================================================
+  # IAM USER WITH LEAST PRIVILEGES
+  # =============================================================================
+
+  NeptuneUser:
+    Type: AWS::IAM::User
+    Properties:
+      UserName: !Sub ${EnvironmentName}-user
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-user
+
+  NeptuneUserPolicy:
+    Type: AWS::IAM::Policy
+    Properties:
+      PolicyName: !Sub ${EnvironmentName}-neptune-access
+      Users:
+        - !Ref NeptuneUser
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Sid: NeptuneDataAccess
+            Effect: Allow
+            Action:
+              - neptune-db:connect
+              - neptune-db:ReadDataViaQuery
+              - neptune-db:WriteDataViaQuery
+              - neptune-db:DeleteDataViaQuery
+            Resource: !Sub
+              - arn:aws:neptune-db:${AWS::Region}:${AWS::AccountId}:${ClusterResourceId}/*
+              - ClusterResourceId: !GetAtt NeptuneCluster.ClusterResourceId
+
+  NeptuneUserAccessKey:
+    Type: AWS::IAM::AccessKey
+    Properties:
+      UserName: !Ref NeptuneUser
+
+# =============================================================================
+# OUTPUTS
+# =============================================================================
+
+Outputs:
+  NeptuneEndpoint:
+    Description: Neptune cluster endpoint (hostname only) - use as NEPTUNE_ENDPOINT
+    Value: !GetAtt NeptuneCluster.Endpoint
+
+  NeptunePort:
+    Description: Neptune cluster port - use as NEPTUNE_PORT
+    Value: !GetAtt NeptuneCluster.Port
+
+  AwsAccessKeyId:
+    Description: Access key ID for the Neptune IAM user - use as AWS_ACCESS_KEY_ID
+    Value: !Ref NeptuneUserAccessKey
+
+  AwsSecretAccessKey:
+    Description: Secret access key for the Neptune IAM user - use as AWS_SECRET_ACCESS_KEY
+    Value: !GetAtt NeptuneUserAccessKey.SecretAccessKey
+
+  AwsRegion:
+    Description: AWS region where Neptune is deployed - use as AWS_REGION
+    Value: !Ref AWS::Region
+
+  NeptuneClusterResourceId:
+    Description: Neptune cluster resource ID (for IAM policy reference)
+    Value: !GetAtt NeptuneCluster.ClusterResourceId
+
+  VpcId:
+    Description: VPC ID
+    Value: !Ref VPC
+
+  SecurityGroupId:
+    Description: Neptune security group ID
+    Value: !Ref NeptuneSecurityGroup

cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb

@@ -110,7 +110,7 @@
    "source": [
     "# Set up API keys\n",
     "# Note: In production, use environment variables: export GROQ_API_KEY=\"your-key\"\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"Your Groq API\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb

@@ -30,7 +30,7 @@
     "# Environment Setup\n",
     "import os\n",
     "\n",
-    "os.environ['GROQ_API_KEY'] = os.getenv('GROQ_API_KEY', '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')\n",
+    "os.environ['GROQ_API_KEY'] = os.getenv('GROQ_API_KEY', '')\n",
     "\n",
     "# Install Semantica and all required dependencies\n",
     "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers\n"
@@ -84,7 +84,7 @@
    "source": [
     "# Set up API keys\n",
     "# Note: In production, use environment variables: export GROQ_API_KEY=\"your-key\"\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-groq-api-key-here\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "print(\"API keys configured.\")\n"
    ]

cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb

@@ -109,7 +109,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb

@@ -85,7 +85,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb

@@ -81,7 +81,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/finance/02_Fraud_Detection.ipynb

@@ -98,7 +98,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb

@@ -83,7 +83,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb

@@ -80,7 +80,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

docs/citation.md

@@ -12,22 +12,22 @@ How to cite Semantica in academic papers and research.
   author = {Hawksight AI},
   year = {2026},
   url = {https://github.com/Hawksight-AI/semantica},
-  version = {0.2.0},
+  version = {0.2.5},
   doi = {10.5281/zenodo.XXXXXXX}
 }
 ```
 
 ### APA
-Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.0) [Computer software]. https://github.com/Hawksight-AI/semantica
+Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.5) [Computer software]. https://github.com/Hawksight-AI/semantica
 
 ### MLA
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.0, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.5, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
 
 ### Chicago
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.0. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.5. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
 
 ### IEEE
-Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.0, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
+Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.5, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
 
 ---
 

docs/index.md

@@ -17,7 +17,7 @@
   
   <p><em>The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.</em></p>
   
-  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.1.1</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
+  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.2.3</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
   
   <p>
     <a href="getting-started/" class="md-button md-button--primary">Get Started</a>

docs/js/version-selector.js

@@ -9,7 +9,12 @@ document.addEventListener("DOMContentLoaded", function () {
 
         // Define versions
         var versions = [
-            { name: "0.1.1", url: "#", current: true },
+            { name: "0.2.4", url: "#", current: true },
+            { name: "0.2.3", url: "#", current: false },
+            { name: "0.2.2", url: "#", current: false },
+            { name: "0.2.1", url: "#", current: false },
+            { name: "0.2.0", url: "#", current: false },
+            { name: "0.1.1", url: "#", current: false },
             { name: "0.1.0", url: "#", current: false }
         ];
 

docs/reference/ontology.md

@@ -63,6 +63,29 @@ The module uses several inference algorithms:
 
 ---
 
+## Ontology Ingestion
+
+Ingest existing ontology files directly into usable data structures using `OntologyIngestor`.
+
+**Function:** `ingest_ontology(source, method="file")`
+
+| Argument | Description |
+|----------|-------------|
+| `source` | File path, directory path, or list of paths |
+| `method` | Ingestion method (default: "file") |
+
+**Example:**
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest file
+data = ingest_ontology("ontology.ttl")
+
+# Ingest directory
+dataset = ingest_ontology("ontologies/")
+```
+
 ## Main Classes
 
 ### OntologyEngine
@@ -170,6 +193,17 @@ Manages external dependencies.
 | `import_external_ontology(uri, ontology)` | Load and merge external ontology |
 | `evaluate_alignment(uri, ontology)` | Assess alignment and compatibility |
 
+### OntologyIngestor
+
+Handles ingestion of existing ontologies from files and directories.
+
+**Methods:**
+
+| Method | Description |
+|--------|-------------|
+| `ingest_ontology(file_path)` | Ingest a single ontology file |
+| `ingest_directory(directory_path)` | Recursively ingest ontology files from a directory |
+
 ---
 
 ## Unified Engine Examples

docs/reference/semantic_extract.md

@@ -23,7 +23,7 @@ The **Semantic Extract Module** extracts structured information from unstructure
 - **High Accuracy**: LLM-based extraction for complex schemas
 - **Flexible Configuration**: Customize extraction for your domain
 - **Confidence Scores**: Get confidence scores for all extractions
-- **Batch Processing**: Efficient batch processing for large datasets
+- **Batch Processing**: Efficient parallel batch processing for large datasets
 - **Coreference Resolution**: Resolve pronouns to their entity references
 
 ### How It Works
@@ -184,16 +184,15 @@ Core entity extraction implementation used by notebooks and lower-level integrat
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `method` | str or list | `"ml"` | Method(s): "ml", "llm", "pattern", "regex", "huggingface" |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
-| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `provider`) |
+| `entity_types` | list | `None` | Filter for specific entity types |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `aggregation_strategy`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text)` | Alias for `extract_entities`. Get list of entities. |
-| `extract_entities(text)` | Get list of entities |
+| `extract(text, pipeline_id=None, **kwargs)` | Alias for `extract_entities`. Supports `max_workers`. |
+| `extract_entities(text, pipeline_id=None, **kwargs)` | Get list of entities. Supports `max_workers`. |
 
 **Example:**
 
@@ -204,11 +203,12 @@ from semantica.semantic_extract import NERExtractor
 extractor = NERExtractor(method="ml", model="en_core_web_trf")
 entities = extractor.extract("Elon Musk leads SpaceX.")
 
-# 2. LLM (OpenAI/Gemini/etc)
+# 2. LLM (OpenAI/Gemini/Groq/etc)
 extractor = NERExtractor(
     method="llm", 
-    provider="openai", 
-    model="gpt-4",
+    provider="groq", 
+    model="llama-3.3-70b-versatile",
+    max_tokens=2048, # Increased output limit
     temperature=0.0
 )
 
@@ -228,17 +228,19 @@ Extracts relationships between entities.
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"dependency"` | Method: "dependency", "pattern", "cooccurrence", "huggingface", "llm" |
 | `relation_types` | list | `None` | Specific relation types to extract |
 | `bidirectional` | bool | `False` | Extract bidirectional relations |
 | `confidence_threshold` | float | `0.6` | Minimum confidence score |
 | `max_distance` | int | `50` | Max token distance between entities |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `device` for HuggingFace) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text, entities)` | Alias for `extract_relations`. Find links. |
-| `extract_relations(text, entities)` | Find links |
+| `extract(text, entities, pipeline_id=None, **kwargs)` | Alias for `extract_relations`. Supports `max_workers`. |
+| `extract_relations(text, entities, pipeline_id=None, **kwargs)` | Find links. Supports `max_workers`. |
 
 **Example:**
 
@@ -308,6 +310,7 @@ Identifies events with temporal information and participants.
 | `extract_participants` | bool | `True` | Extract event participants |
 | `extract_location` | bool | `True` | Extract event locations |
 | `extract_time` | bool | `True` | Extract temporal information |
+| `max_workers` | int | `1` | Threads for parallel batch processing |
 
 **Methods:**
 
@@ -336,17 +339,18 @@ Extracts RDF triplets (Subject-Predicate-Object).
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
+| `triplet_types` | list | `None` | Specific triplet types/predicates to extract |
 | `include_temporal` | bool | `False` | Include time information |
 | `include_provenance` | bool | `False` | Track source sentences |
-| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
+| `**kwargs` | dict | `{}` | Configuration options (e.g., `model`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract_triplets(text)` | Get (S, P, O) tuples |
+| `extract(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Alias for `extract_triplets`. Supports `max_workers`. |
+| `extract_triplets(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Get (S, P, O) tuples. Supports `max_workers`. |
 
 **Example:**
 
@@ -371,6 +375,7 @@ Extracts structured semantic networks with nodes and edges.
 |-----------|------|---------|-------------|
 | `ner_method` | str | `None` | Method for node extraction |
 | `relation_method` | str | `None` | Method for edge extraction |
+| `max_workers` | int | `1` | Threads for parallel batch processing |
 | `**config` | dict | `{}` | Configuration for underlying extractors |
 
 **Methods:**

docs/reference/vector_store.md

@@ -1,12 +1,12 @@
 # Vector Store
 
-> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, and Milvus with Hybrid Search.**
+> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, Pinecone, and Milvus with Hybrid Search.**
 
 ---
 
 ## 🎯 Overview
 
-The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Milvus) and enables semantic search, RAG, and similarity matching.
+The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Pinecone, Milvus) and enables semantic search, RAG, and similarity matching.
 
 ### What is a Vector Store?
 
@@ -18,7 +18,7 @@ A **vector store** is a database optimized for storing and searching high-dimens
 
 ### Why Use the Vector Store Module?
 
-- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, and Milvus
+- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, Pinecone, and Milvus
 - **Unified Interface**: Same API regardless of backend
 - **Hybrid Search**: Combine vector similarity with metadata filtering
 - **Performance**: Optimized for high-throughput search operations
@@ -38,8 +38,8 @@ A **vector store** is a database optimized for storing and searching high-dimens
 -   :material-database:{ .lg .middle } **Multi-Backend Support**
 
     ---
-
-    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, and Milvus
+    
+    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, Pinecone, and Milvus
 
 -   :material-magnify-plus:{ .lg .middle } **Hybrid Search**
 
@@ -112,6 +112,8 @@ The main facade for all vector operations.
 | Method | Description |
 |--------|-------------|
 | `store_vectors(vectors, metadata)` | Store embeddings |
+| `add_documents(documents, metadata, batch_size, parallel)` | **(New)** Store documents with automatic embedding generation and parallelization |
+| `embed_batch(texts)` | **(New)** Generate embeddings for a batch of texts |
 | `search(query, k)` | Semantic search |
 | `delete(ids)` | Remove vectors |
 
@@ -120,15 +122,24 @@ The main facade for all vector operations.
 ```python
 from semantica.vector_store import VectorStore
 
-# Initialize (defaults to FAISS)
+# Initialize (defaults to FAISS, parallel enabled by default with 6 workers)
 store = VectorStore(backend="faiss", dimension=1536)
 
-# Store
+# 1. Store pre-computed vectors
 ids = store.store_vectors(
     vectors=[[0.1, 0.2, ...], ...],
     metadata=[{"text": "Hello"}, ...]
 )
 
+# 2. Store raw documents (High Performance)
+# Automatically handles embedding generation in parallel batches (uses default 6 workers)
+ids = store.add_documents(
+    documents=["Doc 1", "Doc 2", ...],
+    metadata=[{"id": 1}, {"id": 2}, ...],
+    batch_size=32,
+    parallel=True
+)
+
 # Search
 results = store.search(query_vector=[0.1, 0.2, ...], k=5)
 ```
@@ -771,6 +782,7 @@ print(f"Context: {context}")
 ---
 
 ## See Also
+- [High-Performance Usage Guide](../vector_store_usage.md) - **(New)** Parallel ingestion and batching guide
 - [Embeddings Module](embeddings.md) - Generates the vectors
 - [Context Module](context.md) - Uses vector store for memory
 - [Ingest Module](ingest.md) - Source of data

docs/vector_store_usage.md

@@ -0,0 +1,101 @@
+# High-Performance Vector Store Usage
+
+This guide demonstrates how to leverage the new high-performance features of the Semantica Vector Store, specifically designed for efficient batch processing and parallel ingestion of large document sets.
+
+## 🚀 Key Features
+
+- **Parallel Ingestion**: Utilize multi-threading to embed and store documents concurrently.
+- **Batch Processing**: Automatically group documents into batches to minimize overhead.
+- **Unified API**: A single `add_documents` method handles embedding generation and storage.
+
+---
+
+## ⚡ Quick Start: Parallel Ingestion
+
+The fastest way to ingest documents is using the `add_documents` method. Parallelization is enabled by default with optimized settings (6 workers).
+
+```python
+from semantica.vector_store import VectorStore
+import time
+
+store = VectorStore(
+    backend="faiss",
+    dimension=768,
+)
+
+documents = [f"This is document number {i} with some content." for i in range(1000)]
+metadata = [{"source": "generated", "id": i} for i in range(1000)]
+
+start_time = time.time()
+ids = store.add_documents(
+    documents=documents,
+    metadata=metadata,
+    batch_size=64,
+    parallel=True,
+)
+print(f"Ingested {len(ids)} documents in {time.time() - start_time:.2f}s")
+```
+
+---
+
+## 📊 Performance Comparison
+
+### Old Method (Sequential Loop)
+*Slower due to sequential processing and overhead per single item.*
+
+```python
+for doc in documents:
+    emb = embedder.generate(doc)
+    store.store_vectors([emb], [{"text": doc}])
+```
+
+### New Method (Parallel Batching)
+*Significantly faster (3x-10x) by utilizing thread pools and batch operations.*
+
+```python
+store.add_documents(documents, parallel=True)
+```
+
+---
+
+## 🛠 Configuration & Tuning
+
+### `max_workers`
+Controls the number of concurrent threads used for embedding generation.
+- **Default**: 6 (Optimized for most systems)
+- **Recommendation**: You generally don't need to change this. If you have very high core counts or specific throughput needs, you can override it.
+
+```python
+store = VectorStore(max_workers=16)
+```
+
+### `batch_size`
+Controls how many documents are processed in a single chunk.
+- **Default**: 32
+- **Recommendation**: 
+    - **Local Models**: 32-64 usually works well.
+    - **API Models (OpenAI, etc.)**: Larger batches (e.g., 100-200) can reduce network latency overhead.
+
+```python
+store.add_documents(documents, batch_size=100)
+```
+
+---
+
+## 🧩 Advanced: Manual Batch Embedding
+
+If you need the embeddings without storing them immediately, use `embed_batch`.
+
+```python
+vectors = store.embed_batch(
+    texts=documents[:100],
+)
+
+print(f"Generated {len(vectors)} vectors")
+```
+
+## ⚠️ Best Practices
+
+1.  **Metadata Consistency**: Ensure your `metadata` list has the same length as your `documents` list.
+2.  **Error Handling**: The `add_documents` method will propagate exceptions if embedding fails. Ensure your data is clean.
+3.  **Memory Usage**: Very large `batch_size` combined with high `max_workers` can increase memory usage. Monitor your system resources.

examples/huggingface_demo.py

@@ -0,0 +1,147 @@
+"""
+HuggingFace Local Model Usage Demo (Bring Your Own Model)
+
+This script demonstrates how to use the 'semantica' library with local HuggingFace models
+for Named Entity Recognition (NER), Relation Extraction (RE), and Triplet Extraction.
+
+Prerequisites:
+    pip install transformers torch
+
+Usage:
+    python examples/huggingface_demo.py
+"""
+
+import sys
+import os
+
+# Add project root to path (for running from this dir)
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor, Entity
+
+def demo_ner():
+    print("\n" + "="*50)
+    print("NER Demo: Bring Your Own Model (BYOM)")
+    print("="*50)
+    
+    # 1. Initialize NERExtractor with HuggingFace method and a specific model
+    # Common models: "dslim/bert-base-NER", "dbmdz/bert-large-cased-finetuned-conll03-english"
+    model_name = "dslim/bert-base-NER" 
+    print(f"Initializing NERExtractor with model: {model_name}...")
+    
+    extractor = NERExtractor(
+        method="huggingface", 
+        model=model_name,
+        device="cpu"  # Use "cuda" for GPU
+    )
+    
+    text = "Steve Jobs founded Apple Inc. in Cupertino, California on April 1, 1976."
+    print(f"\nInput text: {text}")
+    
+    try:
+        # Note: This will download the model if not cached (approx 400MB)
+        print("Extracting entities (this may take a moment on first run)...")
+        entities = extractor.extract_entities(text)
+        
+        print(f"\nExtracted {len(entities)} entities:")
+        for ent in entities:
+            print(f"  - {ent.text:20} | Type: {ent.label:10} | Conf: {ent.confidence:.2f}")
+            
+    except Exception as e:
+        print(f"Extraction failed (missing dependencies?): {e}")
+
+
+def demo_relation():
+    print("\n" + "="*50)
+    print("Relation Extraction Demo: Local Model")
+    print("="*50)
+    
+    # 1. Initialize RelationExtractor
+    # Note: Relation extraction usually requires a SequenceClassification model 
+    # trained on relation datasets (e.g., TACRED, SemEval).
+    # For demo purposes, we'll use a generic placeholder or a widely used one.
+    model_name = "semantica/relation-model-v1" # This is hypothetical; replace with real model
+    print(f"Initializing RelationExtractor with method='huggingface'...")
+    
+    extractor = RelationExtractor(
+        method="huggingface",
+        model=model_name,
+        device="cpu"
+    )
+    
+    text = "Steve Jobs founded Apple Inc."
+    # Pre-defined entities are usually required for relation extraction
+    entities = [
+        Entity(text="Steve Jobs", label="PERSON", start_char=0, end_char=10), 
+        Entity(text="Apple Inc.", label="ORG", start_char=19, end_char=29)
+    ]
+    
+    print(f"\nInput text: {text}")
+    print(f"Entities: {[e.text for e in entities]}")
+    
+    try:
+        print("Extracting relations...")
+        # Note: This will fail if the model doesn't exist on HF Hub.
+        # In a real scenario, use a valid model ID like "some-user/bert-relation-extraction"
+        # For this demo, we just show the call structure.
+        relations = extractor.extract_relations(text, entities)
+        
+        print(f"\nExtracted {len(relations)} relations:")
+        for rel in relations:
+            print(f"  - {rel.subject.text} --[{rel.predicate}]--> {rel.object.text} (Conf: {rel.confidence:.2f})")
+            
+    except Exception as e:
+        print(f"Note: Relation extraction mock run (model download might fail or be skipped): {e}")
+
+
+def demo_triplet():
+    print("\n" + "="*50)
+    print("Triplet Extraction Demo: REBEL (Seq2Seq)")
+    print("="*50)
+    
+    # 1. Initialize TripletExtractor with REBEL model
+    # REBEL is a popular model for end-to-end triplet extraction
+    model_name = "Babelscape/rebel-large"
+    print(f"Initializing TripletExtractor with model: {model_name}...")
+    
+    extractor = TripletExtractor(
+        method="huggingface",
+        model=model_name,
+        device="cpu"
+    )
+    
+    text = "Apple was founded by Steve Jobs in 1976."
+    print(f"\nInput text: {text}")
+    
+    try:
+        print("Extracting triplets (this may take a moment)...")
+        triplets = extractor.extract_triplets(text)
+        
+        print(f"\nExtracted {len(triplets)} triplets:")
+        for triplet in triplets:
+            print(f"  - ({triplet.subject}, {triplet.predicate}, {triplet.object})")
+            
+    except Exception as e:
+        print(f"Extraction failed (missing dependencies?): {e}")
+
+if __name__ == "__main__":
+    print("Starting Semantica HuggingFace Usage Demo...")
+    print("Note: This script attempts to download models from Hugging Face Hub.")
+    print("Ensure you have an internet connection and 'transformers' installed.")
+    
+    # Run demos
+    # We wrap in try-except to ensure the script doesn't crash the whole session if one fails
+    try:
+        demo_ner()
+    except Exception as e:
+        print(f"NER Demo Error: {e}")
+        
+    try:
+        demo_relation()
+    except Exception as e:
+        print(f"Relation Demo Error: {e}")
+        
+    try:
+        demo_triplet()
+    except Exception as e:
+        print(f"Triplet Demo Error: {e}")

pyproject.toml

@@ -4,317 +4,208 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantica"
-version = "0.2.0"
-description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering "
+version = "0.2.5"
+description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering"
 readme = "README.md"
-license = {text = "MIT"}
-authors = [
-    {name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com"}
-]
-maintainers = [
-    {name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com"}
-]
+license = { text = "MIT" }
+
+authors = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
+maintainers = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
+
 requires-python = ">=3.8"
+
 classifiers = [
-    "Development Status :: 3 - Alpha",
-    "Intended Audience :: Developers",
-    "Intended Audience :: Science/Research",
-    "License :: OSI Approved :: MIT License",
-    "Operating System :: OS Independent",
-    "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
-    "Programming Language :: Python :: 3.11",
-    "Programming Language :: Python :: 3.12",
-    "Topic :: Scientific/Engineering :: Artificial Intelligence",
-    "Topic :: Software Development :: Libraries :: Python Modules",
-    "Topic :: Text Processing :: Linguistic",
-    "Topic :: Database :: Database Engines/Servers",
-    "Topic :: Internet :: WWW/HTTP :: Indexing/Search"
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Software Development :: Libraries :: Python Modules"
 ]
+
 keywords = [
-    "semantic-layer", "knowledge-engineering", "nlp", "knowledge-graph",
-    "embeddings", "entity-extraction", "relationship-extraction", "rdf",
-    "ontology", "semantic-analysis", "ai", "machine-learning"
+  "semantic-layer", "knowledge-graph", "nlp", "embeddings",
+  "entity-extraction", "relationship-extraction", "rdf", "ontology"
 ]
+
+# ---------------- CORE DEPENDENCIES (SAFE DEFAULT) ----------------
 dependencies = [
-    "numpy>=1.21.0",
-    "pandas>=1.3.0",
-    "scikit-learn>=1.0.0",
-    "umap-learn>=0.5.0",
-    "spacy>=3.4.0",
-    "transformers>=4.20.0",
-    "torch>=1.12.0",
-    "sentence-transformers>=2.2.0",
-    "rdflib>=6.2.0",
-    "networkx>=2.8.0",
-    "matplotlib>=3.5.0",
-    "seaborn>=0.11.0",
-    "plotly>=5.10.0",
-    "ipywidgets>=8.0.0",
-    "requests>=2.28.0",
-    "GitPython>=3.1.30",
-    "chardet>=5.1.0",
-    "protobuf==4.25.8",
-    "grpcio==1.67.1",
-    "beautifulsoup4>=4.11.0",
-    "lxml>=4.9.0",
-    "pypdf2>=2.10.0",
-    "python-docx>=0.8.11",
-    "docling>=1.0.0",
-    "openpyxl>=3.0.10",
-    "pillow>=9.2.0",
-    "librosa>=0.9.0",
-    "opencv-python>=4.6.0",
-    "faiss-cpu>=1.7.0",
-    "fastembed>=0.2.0",
-    "onnxruntime>=1.17.0",
-    "tokenizers>=0.15.0",
-    "weaviate-client>=3.15.0",
-    "qdrant-client>=1.3.0",
-    "neo4j>=5.0.0",
-    "falkordb>=1.0.0",
-    "pymongo>=4.2.0",
-    "sqlalchemy>=1.4.0",
-    "psycopg2-binary>=2.9.0",
-    "pymysql>=1.0.0",
-    "redis>=4.3.0",
-    "celery>=5.2.0",
-    "kafka-python>=2.0.0",
-    "pulsar-client>=3.0.0",
-    "pika>=1.3.0",
-    "boto3>=1.24.0",
-    "azure-storage-blob>=12.12.0",
-    "google-cloud-storage>=2.5.0",
-    "pydantic>=2.0.0",
-    "fastmcp>=0.1.0",
-    "groq>=0.4.0",
-    "openai>=1.0.0",
-    "litellm>=1.0.0",
-    "instructor>=1.0.0",
-    "click>=8.1.0",
-    "rich>=12.5.0",
-    "tqdm>=4.64.0",
-    "pyyaml>=6.0",
-    "toml>=0.10.0",
-    "python-dotenv>=0.20.0",
-    "loguru>=0.6.0",
-    "structlog>=22.1.0",
-    "prometheus-client>=0.14.0",
-    "opentelemetry-api>=1.12.0",
-    "opentelemetry-sdk>=1.12.0",
-    "opentelemetry-instrumentation",
-    "fastapi>=0.78.0",
-    "uvicorn>=0.18.0",
-    "pytest>=7.1.0",
-    "pytest-cov>=3.0.0",
-    "pytest-asyncio>=0.19.0",
-    "black>=22.6.0",
-    "isort>=5.10.0",
-    "flake8>=4.0.0",
-    "mypy>=0.971",
-    "pre-commit>=2.19.0"
+  "numpy>=1.21.0",
+  "pandas>=1.3.0",
+  "scikit-learn>=1.0.0",
+  "umap-learn>=0.5.0",
+  "spacy>=3.4.0",
+  "transformers>=4.20.0",
+  "torch>=1.12.0",
+  "sentence-transformers>=2.2.0",
+  "rdflib>=6.2.0",
+  "networkx>=2.8.0",
+  "matplotlib>=3.5.0",
+  "seaborn>=0.11.0",
+  "plotly>=5.10.0",
+  "ipywidgets>=8.0.0",
+  "requests>=2.28.0",
+  "GitPython>=3.1.30",
+  "chardet>=5.1.0",
+  "protobuf>=5.29.1,<7.0",
+  "grpcio>=1.71.2",
+  "beautifulsoup4>=4.11.0",
+  "lxml>=4.9.0",
+  "pypdf2>=2.10.0",
+  "python-docx>=0.8.11",
+  "openpyxl>=3.0.10",
+  "pillow>=9.2.0",
+  "librosa>=0.9.0",
+  "opencv-python>=4.6.0",
+  "faiss-cpu>=1.7.0",
+  "fastembed>=0.2.0",
+  "onnxruntime>=1.17.0",
+  "tokenizers>=0.15.0",
+  "pydantic>=2.0.0",
+  "click>=8.1.0",
+  "rich>=12.5.0",
+  "tqdm>=4.64.0",
+  "pyyaml>=6.0",
+  "toml>=0.10.0",
+  "python-dotenv>=0.20.0",
+  "loguru>=0.6.0",
+  "structlog>=22.1.0"
 ]
-[project.urls]
-Homepage = "https://github.com/Hawksight-AI/semantica"
-Repository = "https://github.com/Hawksight-AI/semantica"
-"Bug Tracker" = "https://github.com/Hawksight-AI/semantica/issues"
-Discussions = "https://github.com/Hawksight-AI/semantica/discussions"
 
+# ---------------- OPTIONAL DEPENDENCIES ----------------
 [project.optional-dependencies]
-dev = [
-    "pytest>=7.1.0",
-    "pytest-cov>=3.0.0",
-    "pytest-asyncio>=0.19.0",
-    "black>=22.6.0",
-    "isort>=5.10.0",
-    "flake8>=4.0.0",
-    "mypy>=0.971",
-    "pre-commit>=2.19.0",
-    "jupyter>=1.0.0",
-    "ipykernel>=6.15.0",
-    "notebook>=6.4.0"
-]
-viz = [
-    "pyvis>=0.3.0",
-    "graphviz>=0.20.0",
-    "umap-learn>=0.5.0",
-    "d3blocks>=1.0.0"
-]
-gpu = [
-    "torch>=1.12.0",
-    "faiss-gpu>=1.7.0",
-    "cupy>=10.0.0"
-]
-cloud = [
-    "boto3>=1.24.0",
-    "azure-storage-blob>=12.12.0",
-    "google-cloud-storage>=2.5.0",
-    "kubernetes>=24.0.0",
-    "helm>=3.10.0"
-]
-monitoring = [
-    "prometheus-client>=0.14.0",
-    "opentelemetry-api>=1.12.0",
-    "opentelemetry-sdk>=1.12.0",
-    "opentelemetry-instrumentation>=0.32.0",
-    "grafana-api>=1.0.0",
-    "elasticsearch>=8.5.0"
-]
-llm-openai = [
-    "openai>=1.0.0"
-]
-llm-gemini = [
-    "google-generativeai>=0.3.0"
-]
-llm-groq = [
-    "groq>=0.4.0"
-]
-llm-anthropic = [
-    "anthropic>=0.18.0"
-]
-llm-ollama = [
-    "ollama>=0.1.0"
-]
-llm-deepseek = [
-    "deepseek>=0.1.0"
-]
-llm-litellm = [
-    "litellm>=1.0.0"
-]
-llm-instructor = [
-    "instructor>=1.0.0"
-]
+
+# ---- LLM Providers ----
+llm-openai = ["openai>=1.0.0"]
+llm-groq = ["groq>=0.4.0"]
+llm-gemini = ["google-genai>=0.1.0"]
+llm-anthropic = ["anthropic>=0.18.0"]
+llm-ollama = ["ollama>=0.1.0"]
+llm-deepseek = ["deepseek>=0.1.0"]
+llm-litellm = ["litellm>=1.0.0"]
+llm-instructor = ["instructor>=1.0.0"]
+
 llm-all = [
-    "semantica[llm-openai,llm-gemini,llm-groq,llm-anthropic,llm-ollama,llm-deepseek,llm-litellm,llm-instructor]"
-]
-models-huggingface = [
-    "transformers>=4.20.0",
-    "torch>=1.12.0"
-]
-split-tiktoken = [
-    "tiktoken>=0.5.0"
-]
-split-community = [
-    "python-louvain>=0.16"
-]
-split-topic = [
-    "bertopic>=0.15.0",
-    "gensim>=4.3.0"
-]
-split-all = [
-    "semantica[split-tiktoken,split-community,split-topic]"
-]
-graph-neo4j = [
-    "neo4j>=5.0.0"
-]
-graph-falkordb = [
-    "falkordb>=1.0.0",
-    "redis>=4.3.0"
-]
-graph-amazon-neptune = [
-    "boto3>=1.24.0",
-    "neo4j>=5.0.0"
-]
-graph-all = [
-    "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune]"
-]
-parse-docling = [
-    "docling>=1.0.0"
-]
-all = [
-    "semantica[dev,viz,gpu,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,parse-docling]"
+  "semantica[llm-openai,llm-groq,llm-gemini,llm-anthropic,llm-ollama,llm-deepseek,llm-litellm,llm-instructor]"
 ]
 
+# ---- Document Parsing ----
+parse-docling = ["docling>=1.0.0"]
+
+# ---- Embedding / Models ----
+models-huggingface = [
+  "transformers>=4.20.0",
+  "torch>=1.12.0"
+]
+
+# ---- Graph Backends ----
+graph-neo4j = ["neo4j>=5.0.0"]
+graph-falkordb = ["falkordb>=1.0.0", "redis>=4.3.0"]
+graph-amazon-neptune = ["boto3>=1.24.0", "neo4j>=5.0.0"]
+
+graph-all = [
+  "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune]"
+]
+
+# ---- Vector Store Backends ----
+vectorstore-qdrant = ["qdrant-client>=1.0.0"]
+vectorstore-weaviate = ["weaviate-client>=4.0.0"]
+vectorstore-pinecone = ["pinecone-client>=3.0.0"]
+vectorstore-milvus = ["pymilvus>=2.0.0"]
+
+vectorstore-all = [
+  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus]"
+]
+
+# ---- Infra / Queues / Workers ----
+infra = [
+  "redis>=4.3.0",
+  "celery>=5.2.0",
+  "kafka-python>=2.0.0",
+  "pulsar-client>=3.0.0",
+  "pika>=1.3.0"
+]
+
+# ---- Cloud Providers ----
+cloud = [
+  "boto3>=1.24.0",
+  "azure-storage-blob>=12.12.0",
+  "google-cloud-storage>=2.5.0"
+]
+
+# ---- Monitoring (FIXED) ----
+monitoring = [
+  "prometheus-client>=0.14.0",
+  "opentelemetry-api>=1.30.0,<2.0.0",
+  "opentelemetry-sdk>=1.30.0,<2.0.0",
+  "opentelemetry-semantic-conventions>=0.58b0,<0.61b0",
+  "opentelemetry-instrumentation>=0.58b0,<0.61b0"
+]
+
+# ---- Visualization ----
+viz = [
+  "pyvis>=0.3.0",
+  "graphviz>=0.20.0",
+  "d3blocks>=1.0.0"
+]
+
+# ---- GPU ----
+gpu = [
+  "faiss-gpu>=1.7.0",
+  "cupy>=10.0.0"
+]
+
+# ---- Splitting / Chunking ----
+split-tiktoken = ["tiktoken>=0.5.0"]
+split-community = ["python-louvain>=0.16"]
+split-topic = ["bertopic>=0.15.0", "gensim>=4.3.0"]
+
+split-all = [
+  "semantica[split-tiktoken,split-community,split-topic]"
+]
+
+# ---- Dev ----
+dev = [
+  "pytest>=7.1.0",
+  "pytest-cov>=3.0.0",
+  "pytest-asyncio>=0.19.0",
+  "black>=22.6.0",
+  "isort>=5.10.0",
+  "flake8>=4.0.0",
+  "mypy>=0.971",
+  "pre-commit>=2.19.0",
+  "jupyter>=1.0.0",
+  "ipykernel>=6.15.0"
+]
+
+# ---- Everything ----
+all = [
+  "semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling]"
+]
+
+# ---------------- ENTRYPOINTS ----------------
 [project.scripts]
 semantica = "semantica.cli:main"
 semantica-server = "semantica.server:main"
 semantica-worker = "semantica.worker:main"
 
+# ---------------- TOOLING ----------------
 [tool.setuptools.packages.find]
 where = ["."]
 include = ["semantica*"]
-exclude = ["tests*", "docs*", "examples*"]
-
-[tool.setuptools.package-data]
-semantica = ["*.yaml", "*.yml", "*.json", "*.toml", "*.txt", "*.md"]
 
 [tool.black]
 line-length = 88
-target-version = ['py38', 'py39', 'py310', 'py311', 'py312']
-include = '\.pyi?$'
-extend-exclude = '''
-/(
-  # directories
-  \.eggs
-  | \.git
-  | \.hg
-  | \.mypy_cache
-  | \.tox
-  | \.venv
-  | build
-  | dist
-)/
-'''
 
 [tool.isort]
 profile = "black"
-multi_line_output = 3
-line_length = 88
-known_first_party = ["semantica"]
-known_third_party = ["numpy", "pandas", "scikit-learn", "spacy", "transformers", "torch"]
-
-[tool.mypy]
-python_version = "3.9"
-warn_return_any = true
-warn_unused_configs = true
-disallow_untyped_defs = true
-disallow_incomplete_defs = true
-check_untyped_defs = true
-disallow_untyped_decorators = true
-no_implicit_optional = true
-warn_redundant_casts = true
-warn_unused_ignores = true
-warn_no_return = true
-warn_unreachable = true
-strict_equality = true
-show_error_codes = true
 
 [tool.pytest.ini_options]
-minversion = "7.0"
-addopts = "-ra -q --strict-markers --strict-config"
 testpaths = ["tests"]
-python_files = ["test_*.py", "*_test.py"]
-python_classes = ["Test*"]
-python_functions = ["test_*"]
-markers = [
-    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
-    "integration: marks tests as integration tests",
-    "unit: marks tests as unit tests",
-    "gpu: marks tests that require GPU",
-    "cloud: marks tests that require cloud services"
-]
-
-[tool.coverage.run]
-source = ["semantica"]
-omit = [
-    "*/tests/*",
-    "*/test_*",
-    "*/__pycache__/*",
-    "*/migrations/*"
-]
-
-[tool.coverage.report]
-exclude_lines = [
-    "pragma: no cover",
-    "def __repr__",
-    "if self.debug:",
-    "if settings.DEBUG",
-    "raise AssertionError",
-    "raise NotImplementedError",
-    "if 0:",
-    "if __name__ == .__main__.:",
-    "class .*\\bProtocol\\):",
-    "@(abc\\.)?abstractmethod"
-]

semantica/__init__.py

@@ -10,7 +10,7 @@ Main exports:
     - Config: Configuration management
 """
 
-__version__ = "0.2.0"
+__version__ = "0.2.5"
 __author__ = "Semantica Contributors"
 __license__ = "MIT"
 

semantica/ingest/__init__.py

@@ -86,6 +86,7 @@ Main Classes:
     - RepoIngestor: Git repository processing
     - EmailIngestor: Email protocol handling
     - DBIngestor: Database export handling
+    - OntologyIngestor: Ontology file processing
     - MethodRegistry: Registry for custom ingestion methods
     - IngestConfig: Configuration manager for ingest module
 
@@ -98,6 +99,7 @@ Convenience Functions:
     - ingest_repository: Repository ingestion wrapper
     - ingest_email: Email ingestion wrapper
     - ingest_database: Database ingestion wrapper
+    - ingest_ontology: Ontology ingestion wrapper
 
 
 Example Usage:
@@ -134,6 +136,7 @@ from .methods import (
     ingest_feed,
     ingest_file,
     ingest_mcp,
+    ingest_ontology,
     ingest_repository,
     ingest_stream,
     ingest_web,
@@ -166,6 +169,8 @@ from .web_ingestor import (
     WebIngestor,
 )
 
+from .ontology_ingestor import OntologyData, OntologyIngestor
+
 __all__ = [
     # File ingestion
     "FileIngestor",
@@ -216,6 +221,9 @@ __all__ = [
     "MCPClient",
     "MCPResource",
     "MCPTool",
+    # Ontology ingestion
+    "OntologyIngestor",
+    "OntologyData",
     # Registry and Methods
     "MethodRegistry",
     "method_registry",
@@ -227,6 +235,7 @@ __all__ = [
     "ingest_repository",
     "ingest_email",
     "ingest_database",
+    "ingest_ontology",
     "ingest_mcp",
     "get_ingest_method",
     "list_available_methods",

semantica/ingest/methods.py

@@ -150,6 +150,7 @@ from .email_ingestor import EmailData, EmailIngestor
 from .feed_ingestor import FeedData, FeedIngestor
 from .file_ingestor import FileIngestor, FileObject
 from .mcp_ingestor import MCPData, MCPIngestor
+from .ontology_ingestor import OntologyData, OntologyIngestor
 from .registry import method_registry
 from .repo_ingestor import RepoIngestor
 from .stream_ingestor import StreamIngestor, StreamProcessor
@@ -537,6 +538,66 @@ def ingest_email(
         raise
 
 
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], method: str = "file", **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source (convenience function).
+
+    This is a user-friendly wrapper that ingests ontologies using the specified method.
+
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+            - "file": Single file ingestion
+            - "directory": Directory ingestion with recursive scanning
+        **kwargs: Additional options passed to OntologyIngestor
+
+    Returns:
+        OntologyData, List[OntologyData] with ingestion results
+
+    Examples:
+        >>> from semantica.ingest.methods import ingest_ontology
+        >>> ontology = ingest_ontology("ontology.ttl")
+        >>> ontologies = ingest_ontology("./ontologies", method="directory")
+    """
+    # Check for custom method in registry
+    custom_method = method_registry.get("ontology", method)
+    if custom_method and custom_method != ingest_ontology:
+        try:
+            return custom_method(source, **kwargs)
+        except Exception as e:
+            logger.warning(
+                f"Custom method {method} failed: {e}, falling back to default"
+            )
+
+    try:
+        # Get config
+        config = ingest_config.get_method_config("ontology")
+        config.update(kwargs)
+
+        ingestor = OntologyIngestor(**config)
+
+        source_path = str(source) if isinstance(source, (str, Path)) else None
+        
+        if method == "file" and source_path:
+            if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+            return ingestor.ingest_ontology(source_path, **kwargs)
+        elif method == "directory" and source_path:
+             recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
+             return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
+        else:
+             # Default: try as file
+             if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+             return ingestor.ingest_ontology(str(source), **kwargs)
+
+    except Exception as e:
+        logger.error(f"Failed to ingest ontology: {e}")
+        raise
+
+
 def ingest_database(
     source: Union[str, Dict[str, Any]], method: Optional[str] = None, **kwargs
 ) -> Union[TableData, List[TableData], Dict[str, Any]]:
@@ -769,6 +830,7 @@ def ingest(
             - "repo": Repository ingestion
             - "email": Email ingestion
             - "db": Database ingestion
+            - "ontology": Ontology ingestion
         method: Optional specific ingestion method
         **kwargs: Additional options passed to ingestor
 
@@ -802,6 +864,8 @@ def ingest(
                 ("git@", "https://github.com", "https://gitlab.com")
             ):
                 source_type = "repo"
+            elif source_str.endswith((".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")):
+                source_type = "ontology"
             else:
                 source_type = "file"
         else:
@@ -830,6 +894,8 @@ def ingest(
             raise ProcessingError("Email ingestion requires configuration dictionary")
     elif source_type == "db":
         return {"data": ingest_database(sources, method=method, **kwargs)}
+    elif source_type == "ontology":
+        return {"ontology": ingest_ontology(sources, method=method or "file", **kwargs)}
     elif source_type == "mcp":
         return {"data": ingest_mcp(sources, method=method or "resources", **kwargs)}
     else:
@@ -909,5 +975,8 @@ method_registry.register("mcp", "default", ingest_mcp)
 method_registry.register("mcp", "resources", ingest_mcp)
 method_registry.register("mcp", "tools", ingest_mcp)
 method_registry.register("mcp", "all", ingest_mcp)
+method_registry.register("ontology", "default", ingest_ontology)
+method_registry.register("ontology", "file", ingest_ontology)
+method_registry.register("ontology", "directory", ingest_ontology)
 method_registry.register("ingest", "default", ingest)
 method_registry.register("ingest", "unified", ingest)

semantica/ingest/ontology_ingestor.py

@@ -0,0 +1,392 @@
+"""
+Ontology Ingestion Module
+
+This module provides capabilities to ingest external ontologies from files (OWL, RDF, TTL, etc.)
+and convert them into Semantica's internal ontology dictionary format.
+
+Supported Formats:
+    - Turtle (.ttl): Terse RDF Triple Language. A concise, human-readable
+      format for representing RDF graphs. Commonly used for writing
+      ontologies by hand.
+    - RDF/XML (.rdf, .owl): The XML serialization of RDF. The standard
+      format for OWL (Web Ontology Language) ontologies and often used
+      for data interchange.
+    - JSON-LD (.jsonld): JSON for Linked Data. A lightweight Linked Data
+      format that is easy for humans to read and for machines to parse
+      and generate. Ideal for web-based applications.
+    - N-Triples (.nt): A line-based, plain text format for encoding an
+      RDF graph. Each line represents a single triple. Very simple to
+      parse but verbose.
+    - Notation3 (.n3): A superset of Turtle that adds features like logic
+      and rules.
+
+Key Features:
+    - Support for multiple RDF formats (Turtle, RDF/XML, JSON-LD, N3, NT)
+    - Automatic parsing using rdflib
+    - Conversion to Semantica ontology structure
+    - Batch processing of ontology files
+    - Extraction of classes, properties, and metadata
+
+Example Usage:
+    >>> from semantica.ingest import OntologyIngestor
+    >>> ingestor = OntologyIngestor()
+    >>> ontology = ingestor.ingest_ontology("my_ontology.ttl")
+"""
+
+import os
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Union
+
+import rdflib
+from rdflib import RDF, RDFS, OWL, Graph
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from ..utils.progress_tracker import get_progress_tracker
+
+
+@dataclass
+class OntologyData:
+    """Ontology data representation."""
+
+    data: Dict[str, Any]
+    source_path: str
+    format: str
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    ingested_at: datetime = field(default_factory=datetime.now)
+
+
+class OntologyIngestor:
+    """
+    Ontology ingestion handler.
+
+    This class parses OWL/RDF files and converts them to Semantica's ontology dictionary format.
+    """
+
+    def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
+        """
+        Initialize ontology ingestor.
+
+        Args:
+            config: Optional configuration dictionary
+            **kwargs: Additional configuration parameters
+        """
+        self.logger = get_logger("ontology_ingestor")
+        self.progress = get_progress_tracker()
+        self.config = config or {}
+        self.config.update(kwargs)
+
+    def ingest_ontology(self, file_path: Union[str, Path], format: Optional[str] = None, **kwargs) -> OntologyData:
+        """
+        Ingest an ontology file.
+
+        Args:
+            file_path: Path to the ontology file (string or Path object)
+            format: Optional format hint (e.g., 'turtle', 'xml'). If None, rdflib guesses.
+            **kwargs: Additional arguments for rdflib parsing
+
+        Returns:
+            OntologyData object containing the parsed ontology and metadata
+        """
+        file_path = Path(file_path)
+
+        # Track file ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(file_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Ontology: {file_path.name}",
+        )
+
+        try:
+            # Validate file exists
+            if not file_path.exists():
+                raise ValidationError(f"File not found: {file_path}")
+
+            self.progress.update_tracking(tracking_id, message="Parsing RDF graph...")
+            g = Graph()
+            
+            # Use provided format or let rdflib guess based on extension
+            parse_kwargs = kwargs.copy()
+            if format:
+                parse_kwargs['format'] = format
+            
+            try:
+                g.parse(file_path, **parse_kwargs)
+            except Exception as e:
+                # Fallback: try to guess format from extension if not provided and initial parse failed
+                if not format:
+                    ext = os.path.splitext(file_path)[1].lower()
+                    fmt_map = {
+                        '.ttl': 'turtle',
+                        '.owl': 'xml', # OWL is often XML
+                        '.rdf': 'xml',
+                        '.jsonld': 'json-ld',
+                        '.n3': 'n3',
+                        '.nt': 'nt'
+                    }
+                    guessed_fmt = fmt_map.get(ext)
+                    if guessed_fmt:
+                        self.logger.info(f"Retrying with guessed format: {guessed_fmt}")
+                        g.parse(file_path, format=guessed_fmt, **kwargs)
+                    else:
+                        raise e
+                else:
+                    raise e
+
+            self.progress.update_tracking(tracking_id, message="Converting to internal format...")
+            
+            # Determine format for metadata
+            used_format = format
+            if not used_format:
+                ext = os.path.splitext(file_path)[1].lower()
+                fmt_map = {
+                    '.ttl': 'turtle',
+                    '.owl': 'xml',
+                    '.rdf': 'xml',
+                    '.jsonld': 'json-ld',
+                    '.n3': 'n3',
+                    '.nt': 'nt'
+                }
+                used_format = fmt_map.get(ext, 'unknown')
+
+            ontology_dict = self._convert_to_dict(g, source_path=str(file_path), format=used_format)
+
+            ontology_data = OntologyData(
+                data=ontology_dict,
+                source_path=str(file_path),
+                format=used_format,
+                metadata=ontology_dict.get("metadata", {}).copy()
+            )
+
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Successfully ingested ontology from {file_path}",
+            )
+            
+            return ontology_data
+
+        except Exception as e:
+            self.logger.error(f"Failed to ingest ontology: {str(e)}")
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise ProcessingError(f"Failed to ingest ontology: {str(e)}") from e
+
+    def ingest_directory(self, directory_path: Union[str, Path], recursive: bool = True, **kwargs) -> List[OntologyData]:
+        """
+        Ingest all ontology files in a directory.
+
+        Args:
+            directory_path: Path to the directory (string or Path object)
+            recursive: Whether to search recursively
+            **kwargs: Additional arguments
+
+        Returns:
+            List of OntologyData objects
+        """
+        directory_path = Path(directory_path)
+        ontologies = []
+        extensions = {'.ttl', '.owl', '.rdf', '.jsonld', '.n3', '.nt'}
+        
+        # Track directory ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(directory_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Directory: {directory_path.name}",
+        )
+
+        try:
+            if not directory_path.exists():
+                raise ValidationError(f"Directory not found: {directory_path}")
+                
+            if not directory_path.is_dir():
+                raise ValidationError(f"Path is not a directory: {directory_path}")
+
+            files_to_process = []
+            for root, _, files in os.walk(directory_path):
+                for file in files:
+                    ext = os.path.splitext(file)[1].lower()
+                    if ext in extensions:
+                        files_to_process.append(os.path.join(root, file))
+                
+                if not recursive:
+                    break
+            
+            total_files = len(files_to_process)
+            self.progress.update_tracking(
+                tracking_id, message=f"Processing {total_files} ontology files"
+            )
+
+            for idx, file_path in enumerate(files_to_process, 1):
+                try:
+                    ont_data = self.ingest_ontology(file_path, **kwargs)
+                    ontologies.append(ont_data)
+                    
+                    self.progress.update_progress(
+                        tracking_id,
+                        processed=idx,
+                        total=total_files,
+                        message=f"Processing {idx}/{total_files}: {Path(file_path).name}"
+                    )
+                except Exception as e:
+                    self.logger.warning(f"Skipping {file_path}: {e}")
+            
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Ingested {len(ontologies)} ontologies",
+            )
+            return ontologies
+
+        except Exception as e:
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
+    def _convert_to_dict(self, graph: Graph, source_path: str, format: str = "unknown") -> Dict[str, Any]:
+        """
+        Convert rdflib Graph to Semantica ontology dictionary.
+
+        Args:
+            graph: Parsed rdflib Graph
+            source_path: Source file path
+            format: Format of the ontology file
+
+        Returns:
+            Ontology dictionary
+        """
+        ontology = {
+            "uri": "",
+            "name": os.path.basename(source_path),
+            "version": "1.0",
+            "classes": [],
+            "properties": [],
+            "metadata": {
+                "source_path": source_path,
+                "ingested_at": datetime.now().isoformat(),
+                "format": format
+            }
+        }
+
+        # 1. Extract Ontology Metadata
+        for s, p, o in graph.triples((None, RDF.type, OWL.Ontology)):
+            ontology["uri"] = str(s)
+            
+            # Try to find label/comment/versionInfo
+            for _, _, label in graph.triples((s, RDFS.label, None)):
+                ontology["name"] = str(label)
+                
+            for _, _, comment in graph.triples((s, RDFS.comment, None)):
+                ontology["description"] = str(comment)
+                
+            for _, _, version in graph.triples((s, OWL.versionInfo, None)):
+                ontology["version"] = str(version)
+            
+            # Break after first ontology definition found (usually only one per file)
+            break
+            
+        # 2. Extract Classes
+        classes = {}
+        # Union of owl:Class and rdfs:Class
+        class_types = [OWL.Class, RDFS.Class]
+        for c_type in class_types:
+            for s, p, o in graph.triples((None, RDF.type, c_type)):
+                if isinstance(s, rdflib.BNode):
+                    continue # Skip blank nodes for now
+                    
+                uri = str(s)
+                if uri not in classes:
+                    cls_def = {
+                        "uri": uri,
+                        "name": self._get_local_name(uri),
+                        "type": "class"
+                    }
+                    
+                    # Add label/comment
+                    label = graph.value(s, RDFS.label)
+                    if label:
+                        cls_def["label"] = str(label)
+                        cls_def["name"] = str(label) # Prefer label as name if available? Or keep URI fragment? 
+                        # Keeping local name from URI is safer for internal IDs, label for display. 
+                        # But Semantica seems to use "name" for the identifier in some examples.
+                        # Let's keep name as local name or label if simple.
+                    
+                    comment = graph.value(s, RDFS.comment)
+                    if comment:
+                        cls_def["description"] = str(comment)
+                        
+                    # Superclasses
+                    parents = []
+                    for _, _, parent in graph.triples((s, RDFS.subClassOf, None)):
+                        if isinstance(parent, rdflib.URIRef):
+                            parents.append(str(parent))
+                    if parents:
+                        cls_def["parents"] = parents
+                        
+                    classes[uri] = cls_def
+        
+        ontology["classes"] = list(classes.values())
+
+        # 3. Extract Properties
+        properties = {}
+        # Object Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.ObjectProperty)):
+            self._add_property(graph, s, "object", properties)
+            
+        # Datatype Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.DatatypeProperty)):
+            self._add_property(graph, s, "data", properties)
+            
+        # RDF Properties (generic)
+        for s, p, o in graph.triples((None, RDF.type, RDF.Property)):
+            if str(s) not in properties: # Don't overwrite if already found as specific type
+                self._add_property(graph, s, "annotation", properties) # Default to annotation or generic
+        
+        ontology["properties"] = list(properties.values())
+        
+        return ontology
+
+    def _add_property(self, graph: Graph, subject: rdflib.term.Node, prop_type: str, properties_dict: Dict):
+        if isinstance(subject, rdflib.BNode):
+            return
+
+        uri = str(subject)
+        if uri in properties_dict:
+            return
+            
+        prop_def = {
+            "uri": uri,
+            "name": self._get_local_name(uri),
+            "type": prop_type
+        }
+        
+        label = graph.value(subject, RDFS.label)
+        if label:
+            prop_def["label"] = str(label)
+            
+        comment = graph.value(subject, RDFS.comment)
+        if comment:
+            prop_def["description"] = str(comment)
+            
+        # Domain and Range
+        domain = graph.value(subject, RDFS.domain)
+        if domain and isinstance(domain, rdflib.URIRef):
+            prop_def["domain"] = str(domain)
+            
+        range_val = graph.value(subject, RDFS.range)
+        if range_val and isinstance(range_val, rdflib.URIRef):
+            prop_def["range"] = str(range_val)
+            
+        properties_dict[uri] = prop_def
+
+    def _get_local_name(self, uri: str) -> str:
+        """Extract local name from URI."""
+        if '#' in uri:
+            return uri.split('#')[-1]
+        return uri.split('/')[-1]

semantica/kg/graph_builder.py

@@ -161,7 +161,14 @@ class GraphBuilder:
             }
             all_relationships.append(rel_dict)
         elif isinstance(item, dict):
-            # Detect and normalize Entity objects inside dict
+            if "source_id" in item and "source" not in item:
+                item["source"] = item["source_id"]
+            if "target_id" in item and "target" not in item:
+                item["target"] = item["target_id"]
+            if "subject" in item and "source" not in item:
+                item["source"] = item["subject"]
+            if "object" in item and "target" not in item:
+                item["target"] = item["object"]
             if "source" in item and not isinstance(item["source"], str):
                 src = item["source"]
                 item["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -347,6 +354,21 @@ class GraphBuilder:
         elif not isinstance(sources, list):
             sources = [sources]
 
+        # Count input relationships for warning if all are dropped
+        input_relationships_count = 0
+        if isinstance(source_dict, dict):
+            rels = source_dict.get("relationships", [])
+            if isinstance(rels, list):
+                input_relationships_count += len(rels)
+            elif rels is not None:
+                input_relationships_count += 1
+        if explicit_relationships:
+            for rel_item in explicit_relationships:
+                if isinstance(rel_item, list):
+                    input_relationships_count += len(rel_item)
+                else:
+                    input_relationships_count += 1
+
         # Track graph building
         build_start_time = time.time()
         
@@ -468,11 +490,12 @@ class GraphBuilder:
                         pipeline_id=pipeline_id,
                     )
                     
-                    # Check if relationships are already in dictionary format
                     sample_rel = relationships_list[0] if relationships_list else None
                     is_dict_format = isinstance(sample_rel, dict) and (
-                        "source" in sample_rel and "target" in sample_rel
-                    ) and not hasattr(sample_rel, "__dict__") # Ensure it's not a class instance
+                        ("source" in sample_rel and "target" in sample_rel)
+                        or ("source_id" in sample_rel and "target_id" in sample_rel)
+                        or ("subject" in sample_rel and "object" in sample_rel)
+                    ) and not hasattr(sample_rel, "__dict__")
                     
                     if is_dict_format:
                         # Fast path: directly append dictionaries after normalizing source/target
@@ -481,8 +504,15 @@ class GraphBuilder:
                             batch = relationships_list[i:i + batch_size]
                             for item in batch:
                                 if isinstance(item, dict):
-                                    # Normalize source/target if they are objects
                                     rel_dict = item.copy()
+                                    if "source_id" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["source_id"]
+                                    if "target_id" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["target_id"]
+                                    if "subject" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["subject"]
+                                    if "object" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["object"]
                                     if "source" in rel_dict and not isinstance(rel_dict["source"], str):
                                         src = rel_dict["source"]
                                         rel_dict["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -571,6 +601,14 @@ class GraphBuilder:
                     f"Entity resolution complete: {len(all_entities)} -> {len(resolved_entities)} unique entities"
                 )
 
+            if input_relationships_count > 0 and len(all_relationships) == 0:
+                warning_msg = (
+                    f"All relationships were dropped during graph building: "
+                    f"{input_relationships_count} input relationships, 0 in final graph"
+                )
+                self.logger.warning(warning_msg)
+                print(f"Warning: {warning_msg}")
+
             # Build graph structure
             print("Building graph structure...")
             structure_start = time.time()
@@ -682,6 +720,14 @@ class GraphBuilder:
             )
             raise
 
+    def build_single_source(
+        self,
+        kg_data: Dict[str, Any],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Dict[str, Any]:
+        return self.build(kg_data, pipeline_id=pipeline_id, **options)
+
     def add_temporal_edge(
         self,
         graph,

semantica/ontology/__init__.py

@@ -109,12 +109,14 @@ Convenience Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory
 
 Example Usage:
-    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator
+    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator, ingest_ontology
     >>> # Using convenience functions
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
     >>> # Using classes directly
     >>> from semantica.ontology import OntologyGenerator, ClassInferrer, PropertyGenerator
     >>> generator = OntologyGenerator(base_uri="https://example.org/ontology/")
@@ -155,6 +157,8 @@ from .registry import MethodRegistry, method_registry
 from .requirements_spec import RequirementsSpec, RequirementsSpecManager
 from .reuse_manager import ReuseDecision, ReuseManager
 from .version_manager import OntologyVersion, VersionManager
+from semantica.ingest import OntologyData, OntologyIngestor
+from .methods import ingest_ontology
 
 __all__ = [
     # Main generators
@@ -200,4 +204,7 @@ __all__ = [
     # Configuration
     "OntologyConfig",
     "ontology_config",
+    "ingest_ontology",
+    "OntologyData",
+    "OntologyIngestor",
 ]

semantica/ontology/methods.py

@@ -111,15 +111,19 @@ Main Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory (via semantica.ingest)
 
 Example Usage:
-    >>> from semantica.ontology.methods import generate_ontology, infer_classes
+    >>> from semantica.ontology.methods import generate_ontology, infer_classes, ingest_ontology
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
 """
 
-from typing import Any, Callable, Dict, List, Optional
+from typing import Any, Callable, Dict, List, Optional, Union
+from pathlib import Path
 
+from semantica.ingest import ingest_ontology as _ingest_ontology, OntologyData
 from .registry import method_registry
 
 
@@ -172,4 +176,23 @@ def list_available_methods(task: Optional[str] = None) -> Dict[str, List[str]]:
     return method_registry.list_all(task)
 
 
-pass
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], 
+    method: str = "file", 
+    **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source.
+    
+    This is a convenience wrapper around semantica.ingest.ingest_ontology.
+    
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+        **kwargs: Additional options
+        
+    Returns:
+        OntologyData or List[OntologyData]
+    """
+    return _ingest_ontology(source, method=method, **kwargs)
+

semantica/ontology/ontology_usage.md

@@ -42,6 +42,18 @@ classes = inferrer.infer_classes(entities, build_hierarchy=True)
 properties = prop_gen.infer_properties(entities, relationships, classes)
 ```
 
+### Ingesting Ontologies
+
+```python
+from semantica.ingest import OntologyIngestor
+
+# Create ingestor
+ingestor = OntologyIngestor()
+
+# Ingest ontology
+ontology_data = ingestor.ingest_ontology("ontology.ttl")
+```
+
 ## Ontology Generation
 
 ### Basic Ontology Generation
@@ -115,6 +127,49 @@ ontology = engine.from_data(
 )
 ```
 
+## Ontology Ingestion
+
+### Basic Ingestion
+
+Ingest existing ontologies from files (Turtle, RDF/XML, JSON-LD, etc.) into `OntologyData` objects.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a single file
+ontology_data = ingest_ontology("path/to/ontology.ttl")
+
+print(f"Source: {ontology_data.source_path}")
+print(f"Format: {ontology_data.format}")
+print(f"Data keys: {ontology_data.data.keys()}")
+```
+
+### Ingesting Directories
+
+Ingest all ontology files in a directory recursively.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a directory
+ontologies = ingest_ontology("path/to/ontologies_dir/")
+
+for ont in ontologies:
+    print(f"Ingested: {ont.source_path} ({ont.format})")
+```
+
+### Unified Ingestion Interface
+
+You can also use the unified `semantica.ingest` interface.
+
+```python
+from semantica.ingest import ingest
+
+# Ingest as "ontology" source type
+result = ingest("path/to/ontology.ttl", source_type="ontology")
+ontology_data = result["ontology"]
+```
+
 ## Class Inference
 
 ### Basic Class Inference

semantica/ontology/reuse_manager.py

@@ -363,3 +363,90 @@ class ReuseManager:
     def list_known_ontologies(self) -> List[str]:
         """List known ontology URIs."""
         return list(self.known_ontologies.keys())
+
+    def merge_ontology_data(
+        self, target: Dict[str, Any], source: Dict[str, Any], **options
+    ) -> Dict[str, Any]:
+        """
+        Merge source ontology data into target ontology.
+
+        Merges classes, properties, and metadata from source to target.
+        Handles deduplication based on URI and name.
+
+        Args:
+            target: Target ontology dictionary (modified in-place)
+            source: Source ontology dictionary
+            **options: Merge options:
+                - overwrite: Whether to overwrite existing elements (default: False)
+                - merge_metadata: Whether to merge metadata (default: True)
+
+        Returns:
+            Merged target ontology
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="ontology",
+            submodule="ReuseManager",
+            message=f"Merging ontology {source.get('name', 'unknown')} into {target.get('name', 'unknown')}",
+        )
+
+        try:
+            overwrite = options.get("overwrite", False)
+            
+            # Helper to merge lists of dicts (classes/properties)
+            def merge_lists(target_list, source_list, key_field="uri"):
+                existing_keys = {item.get(key_field): i for i, item in enumerate(target_list) if item.get(key_field)}
+                
+                for item in source_list:
+                    key = item.get(key_field)
+                    if not key:
+                        # Fallback to name if URI missing
+                        key = item.get("name")
+                        
+                    if key in existing_keys:
+                        if overwrite:
+                            target_list[existing_keys[key]] = item
+                    else:
+                        target_list.append(item)
+                        if key:
+                            existing_keys[key] = len(target_list) - 1
+
+            # Merge Classes
+            if "classes" in source:
+                if "classes" not in target:
+                    target["classes"] = []
+                merge_lists(target["classes"], source["classes"])
+
+            # Merge Properties
+            if "properties" in source:
+                if "properties" not in target:
+                    target["properties"] = []
+                merge_lists(target["properties"], source["properties"])
+
+            # Merge Metadata
+            if options.get("merge_metadata", True) and "metadata" in source:
+                if "metadata" not in target:
+                    target["metadata"] = {}
+                # Update with source metadata, preserving target's specific fields if needed
+                # Here we just update
+                target["metadata"].update(source["metadata"])
+                
+            # Merge Imports
+            if "imports" in source:
+                if "imports" not in target:
+                    target["imports"] = []
+                for imp in source["imports"]:
+                    if imp not in target["imports"]:
+                        target["imports"].append(imp)
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Merged ontology data successfully",
+            )
+            return target
+
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise

semantica/pipeline/pipeline_builder.py

@@ -32,12 +32,14 @@ License: MIT
 
 from dataclasses import dataclass, field
 from enum import Enum
-from typing import Any, Callable, Dict, List, Optional, Union
+from typing import Any, Callable, Dict, List, Optional, Union, TYPE_CHECKING
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
-from .pipeline_validator import PipelineValidator
+
+if TYPE_CHECKING:
+    from .pipeline_validator import PipelineValidator
 
 
 class StepStatus(Enum):
@@ -104,6 +106,8 @@ class PipelineBuilder:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        from .pipeline_validator import PipelineValidator
+
         self.validator = PipelineValidator(**self.config)
         self.steps: List[PipelineStep] = []
         self.step_registry: Dict[str, Callable] = {}

semantica/semantic_extract/cache.py

@@ -0,0 +1,184 @@
+"""
+Result Caching Module
+
+This module provides caching mechanisms for extraction results to avoid redundant
+computations and API calls. It implements an LRU (Least Recently Used) cache
+with Time-To-Live (TTL) support.
+
+Key Features:
+    - LRU Caching: Evicts least recently used items when cache is full
+    - TTL Support: Expires items after a configurable duration
+    - Namespaced Caching: Separate caches for entities, relations, and triplets
+    - Hash-based Keys: Uses stable hashing for text and parameters
+
+Classes:
+    - ExtractionCache: Main cache manager
+    - CacheItem: Container for cached data with metadata
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import time
+import hashlib
+import json
+from collections import OrderedDict
+from typing import Any, Dict, Optional, Union, List
+from threading import Lock
+
+from ..utils.logging import get_logger
+
+class CacheItem:
+    """Container for cached data."""
+    def __init__(self, value: Any, ttl: Optional[int] = None):
+        self.value = value
+        self.timestamp = time.time()
+        self.ttl = ttl
+
+    def is_expired(self) -> bool:
+        """Check if item has expired."""
+        if self.ttl is None:
+            return False
+        return time.time() - self.timestamp > self.ttl
+
+class ExtractionCache:
+    """
+    LRU Cache for extraction results.
+    Thread-safe implementation.
+    """
+    def __init__(self, max_size: int = 1000, ttl: int = 3600):
+        """
+        Initialize the cache.
+        
+        Args:
+            max_size: Maximum number of items to store per namespace
+            ttl: Time to live in seconds (default 1 hour)
+        """
+        self.max_size = max_size
+        self.ttl = ttl
+        self._caches: Dict[str, OrderedDict] = {
+            "entities": OrderedDict(),
+            "relations": OrderedDict(),
+            "triplets": OrderedDict()
+        }
+        self._locks: Dict[str, Lock] = {
+            "entities": Lock(),
+            "relations": Lock(),
+            "triplets": Lock()
+        }
+        self.logger = get_logger("extraction_cache")
+        self.enabled = True
+
+    def _generate_key(self, text: str, **params) -> str:
+        """
+        Generate a stable cache key based on text and parameters.
+        
+        Note: Sensitive parameters like 'api_key' are excluded from the cache key
+        to prevent security risks and ensure cache sharing where appropriate.
+        """
+        # Filter out sensitive keys
+        sensitive_keys = {'api_key', 'token', 'password', 'secret', 'auth', 'authorization'}
+        filtered_params = {k: v for k, v in params.items() if k.lower() not in sensitive_keys}
+
+        # Create a stable string representation of params
+        # Sort keys to ensure consistent ordering
+        param_str = json.dumps(filtered_params, sort_keys=True, default=str)
+        
+        # Combine text and params
+        content = f"{text}|{param_str}"
+        
+        # Return hash (SHA-256 for better security than MD5)
+        return hashlib.sha256(content.encode('utf-8')).hexdigest()
+
+    def get(self, namespace: str, text: str, **params) -> Optional[Any]:
+        """
+        Retrieve item from cache.
+        
+        Args:
+            namespace: Cache namespace ("entities", "relations", "triplets")
+            text: Input text used for extraction
+            **params: Extraction parameters used
+            
+        Returns:
+            Cached result or None if not found/expired
+        """
+        if not self.enabled:
+            return None
+
+        if namespace not in self._caches:
+            return None
+
+        key = self._generate_key(text, **params)
+        
+        with self._locks[namespace]:
+            cache = self._caches[namespace]
+            if key in cache:
+                item = cache[key]
+                
+                # Check expiration
+                if item.is_expired():
+                    del cache[key]
+                    return None
+                
+                # Move to end (mark as recently used)
+                cache.move_to_end(key)
+                return item.value
+        
+        return None
+
+    def set(self, namespace: str, text: str, value: Any, **params) -> None:
+        """
+        Add item to cache.
+        
+        Args:
+            namespace: Cache namespace
+            text: Input text
+            value: Result to cache
+            **params: Extraction parameters
+        """
+        if not self.enabled:
+            return
+
+        if namespace not in self._caches:
+            self.logger.warning(f"Unknown cache namespace: {namespace}")
+            return
+
+        key = self._generate_key(text, **params)
+        item = CacheItem(value, self.ttl)
+        
+        with self._locks[namespace]:
+            cache = self._caches[namespace]
+            
+            # If key exists, update and move to end
+            if key in cache:
+                cache.move_to_end(key)
+            
+            cache[key] = item
+            
+            # Evict if full
+            if len(cache) > self.max_size:
+                cache.popitem(last=False)  # Remove first (least recently used)
+
+    def clear(self, namespace: Optional[str] = None):
+        """Clear cache(s)."""
+        if namespace:
+            if namespace in self._caches:
+                with self._locks[namespace]:
+                    self._caches[namespace].clear()
+        else:
+            for ns in self._caches:
+                with self._locks[ns]:
+                    self._caches[ns].clear()
+
+    def get_stats(self) -> Dict[str, Dict[str, int]]:
+        """Get cache statistics."""
+        stats = {}
+        for ns, cache in self._caches.items():
+            stats[ns] = {
+                "size": len(cache),
+                "max_size": self.max_size
+            }
+        return stats
+
+# Global cache instance
+extraction_cache = ExtractionCache()

semantica/semantic_extract/config.py

@@ -40,8 +40,9 @@ License: MIT
 """
 
 import os
+import multiprocessing
 from pathlib import Path
-from typing import Dict, Optional
+from typing import Dict, Optional, Any
 
 from ..utils.logging import get_logger
 
@@ -53,9 +54,23 @@ class Config:
         """Initialize configuration manager."""
         self.logger = get_logger("config")
         self._configs: Dict[str, Dict] = {}
+        # Default optimization settings
+        self._configs["optimization"] = {
+            "enable_cache": True,
+            "cache_size": 1000,
+            "max_workers": 8,
+            "enable_batching": True,
+            "batch_size": 10,
+            "max_tokens_per_batch": 2000
+        }
         self._load_config_file(config_file)
         self._load_env_vars()
 
+    def get_optimization_config(self) -> Dict:
+        """Get optimization configuration."""
+        return self._configs.get("optimization", {})
+
+
     def _load_config_file(self, config_file: Optional[str]):
         """Load configuration from file."""
         if config_file and Path(config_file).exists():
@@ -114,6 +129,66 @@ class Config:
             return self._configs[provider].get("api_key")
         return os.getenv(f"{provider.upper()}_API_KEY")
 
+    def get(self, key: str, default: Any = None) -> Any:
+        """
+        Get configuration value by key.
+        Searches in top-level configs and optimization settings.
+        """
+        # 1. Check top-level keys
+        if key in self._configs:
+            return self._configs[key]
+            
+        # 2. Check optimization settings (common keys)
+        if "optimization" in self._configs and key in self._configs["optimization"]:
+            return self._configs["optimization"][key]
+            
+        # 3. Handle specific mapping for optimization keys
+        # Map cache_enabled -> enable_cache if needed
+        if key == "cache_enabled":
+            return self._configs.get("optimization", {}).get("enable_cache", default)
+            
+        return default
+
 
 # Global config instance
 config = Config()
+
+
+def resolve_max_workers(
+    explicit: Optional[int] = None,
+    local_config: Optional[Dict[str, Any]] = None,
+    methods: Optional[Any] = None,
+) -> int:
+    def to_int(val: Any, default: int) -> int:
+        try:
+            return int(val)
+        except Exception:
+            return default
+
+    if isinstance(methods, str):
+        normalized_methods = [methods]
+    elif isinstance(methods, (list, tuple, set)):
+        normalized_methods = [m for m in methods if isinstance(m, str)]
+    else:
+        normalized_methods = []
+
+    if explicit is not None:
+        value = to_int(explicit, 1)
+    elif local_config and "max_workers" in local_config:
+        value = to_int(local_config.get("max_workers", 1), 1)
+    else:
+        value = to_int(config.get("max_workers", 5), 5)
+
+    if "ml" in normalized_methods and explicit is None and not (local_config and "max_workers" in local_config):
+        value = 1
+
+    if value < 1:
+        value = 1
+
+    cpu_count = multiprocessing.cpu_count() or 1
+    if value > cpu_count:
+        value = cpu_count
+    if value > 32:
+        value = 32
+
+    return value

semantica/semantic_extract/coreference_resolver.py

@@ -240,6 +240,12 @@ class CoreferenceResolver:
             self.progress_tracker.stop_tracking(
                 tracking_id, status="failed", message=str(e)
             )
+            verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+            if verbose_mode:
+                import sys
+                print(f"    [CoreferenceResolver] ERROR: Resolution failed: {e}", flush=True, file=sys.stderr)
+                import traceback
+                traceback.print_exc(file=sys.stderr)
             raise
 
     def resolve(

semantica/semantic_extract/event_detector.py

@@ -85,68 +85,59 @@ class Event:
 class EventDetector:
     """Event detection and extraction handler."""
 
-    def __init__(
-        self,
-        event_types: Optional[List[str]] = None,
-        extract_participants: bool = True,
-        extract_location: bool = True,
-        extract_time: bool = True,
-        method: Union[str, List[str]] = None,
-        config=None,
-        **kwargs
-    ):
+    def __init__(self, method: str = "llm", **config):
         """
         Initialize event detector.
 
         Args:
-            event_types: Specific event types to detect (e.g., ["launch", "acquisition"])
-            extract_participants: Whether to extract event participants
-            extract_location: Whether to extract event locations
-            extract_time: Whether to extract temporal information
-            method: Extraction method(s) for underlying NER/relation extractors.
-                   Can be passed to ner_method and relation_method in config.
-            config: Legacy config dict (deprecated, use kwargs)
-            **kwargs: Configuration options:
-                - ner_method: Method for NER extraction (if entities need to be extracted)
-                - relation_method: Method for relation extraction (if relations need to be extracted)
-                - Other options passed to sub-components
+            method: Extraction method ("llm", "pattern")
+            **config: Configuration options
         """
         self.logger = get_logger("event_detector")
-        self.config = config or {}
-        self.config.update(kwargs)
+        self.config = config
+        self.method = method
         self.progress_tracker = get_progress_tracker()
+        
         # Ensure progress tracker is enabled
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
-        # Store parameters
-        self.event_types_filter = event_types
-        self.extract_participants = extract_participants
-        self.extract_location = extract_location
-        self.extract_time = extract_time
+        # Initialize components
+        self.event_classifier = EventClassifier(**config)
+        self.temporal_processor = TemporalEventProcessor(**config)
+
+        # Configure extraction options
+        self.extract_participants = config.get("extract_participants", True)
+        self.extract_location = config.get("extract_location", True)
+        self.extract_time = config.get("extract_time", True)
+        self.event_types_filter = config.get("event_types", [])
+
+        # Define event patterns
+        self.event_patterns = {
+            "acquisition": r"\b(acquired|acquisition|buying|bought|merger|merged)\b",
+            "partnership": r"\b(partnered|partnership|collaborate|collaboration)\b",
+            "launch": r"\b(launch|launched|releasing|released|unveil|unveiled)\b",
+            "investment": r"\b(invest|invested|investment|funding|raised)\b",
+            "legal": r"\b(sue|sued|lawsuit|litigation|legal action)\b",
+        }
+        
+        # Pre-compile location patterns
+        self.location_patterns = [
+            re.compile(r"in\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)"),
+            re.compile(r"at\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)"),
+        ]
+        
+        # Pre-compile time patterns
+        self.time_patterns = [
+            re.compile(r"on\s+([A-Z][a-z]+\s+\d{1,2},?\s+\d{4})"),
+            re.compile(r"in\s+(\d{4})"),
+            re.compile(r"(\d{1,2}[/-]\d{1,2}[/-]\d{2,4})"),
+        ]
 
-        # Store method for passing to extractors if needed
         if method is not None:
             self.config["ner_method"] = method
             self.config["relation_method"] = method
 
-        self.event_classifier = EventClassifier(**self.config.get("classifier", {}))
-        self.temporal_processor = TemporalEventProcessor(
-            **self.config.get("temporal", {})
-        )
-        self.relationship_extractor = EventRelationshipExtractor(
-            **self.config.get("relationship", {})
-        )
-
-        # Event patterns
-        self.event_patterns = {
-            "founded": r"founded|created|established",
-            "acquired": r"acquired|bought|purchased",
-            "launched": r"launched|released|introduced",
-            "announced": r"announced|declared|stated",
-            "meeting": r"met|meeting|conference|summit",
-        }
-
     def extract(
         self,
         text: Union[str, List[str], List[Dict[str, Any]]],
@@ -175,9 +166,10 @@ class EventDetector:
             )
 
             try:
-                results = []
+                results = [None] * len(text)  # Pre-allocate to maintain order
                 total_items = len(text)
                 total_events_count = 0
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -193,33 +185,77 @@ class EventDetector:
                     message=f"Starting batch detection... 0/{total_items} (remaining: {total_items})"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=[self.config.get("ner_method"), self.config.get("relation_method"), self.config.get("method")],
+                )
+
+                def process_item(idx, item):
+                    try:
+                        # Prepare arguments for single item
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                        
+                        # Detect
+                        events = self.detect_events(doc_text, **kwargs)
+
+                        # Add provenance metadata
+                        for event in events:
+                            if event.metadata is None:
+                                event.metadata = {}
+                            event.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                event.metadata["document_id"] = item["id"]
+                        
+                        return idx, events
+                    except Exception as e:
+                        self.logger.error(f"Error processing item {idx}: {e}")
+                        # Return empty list on failure to continue processing
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
                     
-                    # Detect
-                    events = self.detect_events(doc_text, **kwargs)
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {}
+                        for idx, item in enumerate(text):
+                            future = executor.submit(process_item, idx, item)
+                            future_to_idx[future] = idx
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, events = future.result()
+                            results[idx] = events
+                            total_events_count += len(events)
+                            processed_count += 1
+                            
+                            # Update progress
+                            if processed_count % update_interval == 0 or processed_count == total_items:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, events = process_item(idx, item)
+                        results[idx] = events
+                        total_events_count += len(events)
+                        processed_count += 1
 
-                    # Add provenance metadata
-                    for event in events:
-                        if event.metadata is None:
-                            event.metadata = {}
-                        event.metadata["batch_index"] = idx
-                        if isinstance(item, dict) and "id" in item:
-                            event.metadata["document_id"] = item["id"]
-
-                    results.append(events)
-                    total_events_count += len(events)
-
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
-                        )
+                        # Update progress
+                        if processed_count % update_interval == 0 or processed_count == total_items:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
+                            )
 
                 self.progress_tracker.stop_tracking(
                     tracking_id,
@@ -238,17 +274,26 @@ class EventDetector:
             # Single item
             return self.detect_events(text, **kwargs)
 
-    def detect_events(self, text: str, **options) -> List[Event]:
+    def detect_events(
+        self,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Event], List[List[Event]]]:
         """
         Detect events in text content.
 
         Args:
             text: Input text
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Detection options
 
         Returns:
             list: List of detected events
         """
+        if isinstance(text, list):
+            return self.extract(text, pipeline_id=pipeline_id, **options)
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="EventDetector",

semantica/semantic_extract/llm_extraction.py

@@ -108,7 +108,12 @@ class LLMExtraction:
 
         # Initialize provider using new system
         try:
-            self.provider = create_provider(provider, **config)
+            # Sanitize config: remove api_key if it's None/empty to allow fallback
+            provider_config = config.copy()
+            if "api_key" in provider_config and not provider_config["api_key"]:
+                del provider_config["api_key"]
+                
+            self.provider = create_provider(provider, **provider_config)
         except Exception as e:
             self.logger.warning(f"Failed to initialize {provider} provider: {e}")
             self.provider = None

semantica/semantic_extract/ner_extractor.py

@@ -178,9 +178,11 @@ class NERExtractor:
             )
             
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
                 total_entities_count = 0
+                processed_count = 0
+                
                 # Update more frequently: every 1% or at least every 10 items, but always update for small datasets
                 if total_items <= 10:
                     update_interval = 1  # Update every item for small datasets
@@ -188,15 +190,22 @@ class NERExtractor:
                     update_interval = max(1, min(10, total_items // 100))
                 
                 # Initial progress update - ALWAYS show this
-                remaining = total_items
                 self.progress_tracker.update_progress(
                     tracking_id,
                     processed=0,
                     total=total_items,
-                    message=f"Starting batch extraction... 0/{total_items} (remaining: {remaining})"
+                    message=f"Starting batch extraction... 0/{total_items}"
                 )
                 
-                for idx, item in enumerate(text, 1):
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=self.method,
+                )
+                
+                # Helper function for single item processing
+                def process_item(idx, item):
                     try:
                         current_entities = []
                         if isinstance(item, dict) and "content" in item:
@@ -214,30 +223,69 @@ class NERExtractor:
                         for ent in current_entities:
                             if ent.metadata is None:
                                 ent.metadata = {}
-                            ent.metadata["batch_index"] = idx - 1
+                            ent.metadata["batch_index"] = idx
                             if isinstance(item, dict) and "id" in item:
                                 ent.metadata["document_id"] = item["id"]
                         
-                        results.append(current_entities)
-                        total_entities_count += len(current_entities)
-                    except Exception:
-                        results.append([])
+                        return idx, current_entities
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {idx}: {e}")
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
                     
-                    remaining = total_items - idx
-                    # Update progress: always update for small datasets, or at intervals for large ones
-                    should_update = (
-                        idx % update_interval == 0 or 
-                        idx == total_items or 
-                        idx == 1 or
-                        total_items <= 10  # Always update for small datasets
-                    )
-                    if should_update:
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx,
-                            total=total_items,
-                            message=f"Processing documents... {idx}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit all tasks
+                        future_to_idx = {
+                            executor.submit(process_item, idx, item): idx 
+                            for idx, item in enumerate(text)
+                        }
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, entities = future.result()
+                            results[idx] = entities
+                            total_entities_count += len(entities)
+                            processed_count += 1
+                            
+                            # Update progress
+                            should_update = (
+                                processed_count % update_interval == 0 or 
+                                processed_count == total_items or 
+                                processed_count == 1 or
+                                total_items <= 10
+                            )
+                            if should_update:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing documents... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, entities = process_item(idx, item)
+                        results[idx] = entities
+                        total_entities_count += len(entities)
+                        processed_count += 1
+                        
+                        # Update progress
+                        should_update = (
+                            processed_count % update_interval == 0 or 
+                            processed_count == total_items or 
+                            processed_count == 1 or
+                            total_items <= 10
                         )
+                        if should_update:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing documents... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                            )
                 
                 self.progress_tracker.stop_tracking(
                     tracking_id,
@@ -253,12 +301,18 @@ class NERExtractor:
         else:
             return self.extract_entities(text, **kwargs)
 
-    def extract_entities(self, text: str, **options) -> List[Entity]:
+    def extract_entities(
+        self,
+        text: Union[str, List[Dict[str, Any]], List[str]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Entity], List[List[Entity]]]:
         """
         Extract named entities from text.
 
         Args:
             text: Input text
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Extraction options:
                 - entity_types: Filter by entity types (list)
                 - min_confidence: Minimum confidence threshold
@@ -267,6 +321,9 @@ class NERExtractor:
         Returns:
             list: List of extracted entities
         """
+        if isinstance(text, list):
+            return self.extract(text, pipeline_id=pipeline_id, **options)
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="NERExtractor",
@@ -307,8 +364,11 @@ class NERExtractor:
                     # Prepare method-specific options
                     method_options = all_options.copy()
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", self.huggingface_model
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model") 
+                            or self.huggingface_model
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -318,14 +378,13 @@ class NERExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key

semantica/semantic_extract/providers.py

@@ -243,68 +243,136 @@ class BaseProvider:
                 mode = instructor.Mode.TOOLS  # Default mode
                 
                 if provider_name == "OpenAIProvider" and self.client:
-                    client = instructor.from_openai(self.client)
-                elif provider_name == "AnthropicProvider" and self.client:
-                    client = instructor.from_anthropic(self.client)
-                elif provider_name == "GeminiProvider" and self.client:
-                    client = instructor.from_gemini(
-                        self.client, 
-                        mode=instructor.Mode.GEMINI_JSON
-                    )
-                elif provider_name == "GroqProvider" and self.client:
-                    # Try using from_groq if available (newer instructor versions)
-                    if hasattr(instructor, "from_groq"):
-                        client = instructor.from_groq(self.client, mode=instructor.Mode.JSON)
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"openai/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_openai(self.client)
                     else:
-                        # Fallback: Create OpenAI client pointing to Groq
-                        # This avoids the "Client should be an instance of openai.OpenAI" warning
+                        client = instructor.from_openai(self.client)
+                elif provider_name == "AnthropicProvider" and self.client:
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"anthropic/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_anthropic(self.client)
+                    else:
+                        client = instructor.from_anthropic(self.client)
+                elif provider_name == "GeminiProvider" and self.client:
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"gemini/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_gemini(
+                                self.client, 
+                                mode=instructor.Mode.GEMINI_JSON
+                            )
+                    else:
+                        client = instructor.from_gemini(
+                            self.client, 
+                            mode=instructor.Mode.GEMINI_JSON
+                        )
+                elif provider_name == "GroqProvider" and self.client:
+                    # Try using from_provider which is recommended for Groq in latest instructor
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"groq/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = None
+
+                    if not client:
+                        # Try using from_groq if available (newer instructor versions)
+                        if hasattr(instructor, "from_groq"):
+                            client = instructor.from_groq(self.client, mode=instructor.Mode.JSON)
+                        else:
+                            # Fallback: Create OpenAI client pointing to Groq
+                            # This avoids the "Client should be an instance of openai.OpenAI" warning
+                            try:
+                                from openai import OpenAI
+                                # Fix: Use self.api_key instead of self.client.api_key
+                                groq_client = OpenAI(
+                                    base_url="https://api.groq.com/openai/v1",
+                                    api_key=self.api_key,
+                                )
+                                client = instructor.from_openai(groq_client, mode=instructor.Mode.JSON)
+                            except Exception:
+                                # Last resort: try passing the groq client directly
+                                client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
+                elif provider_name == "OllamaProvider":
+                    # Try from_provider for Ollama if available
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"ollama/{kwargs.get('model', self.model)}",
+                            )
+                        except Exception:
+                            client = None
+                    
+                    if not client:
+                        # Create OpenAI-compatible client for Ollama
                         try:
                             from openai import OpenAI
-                            groq_client = OpenAI(
-                                base_url="https://api.groq.com/openai/v1",
-                                api_key=self.client.api_key,
+                            # Ollama typically runs on localhost:11434/v1
+                            base_url = getattr(self, "base_url", "http://localhost:11434")
+                            if not base_url.endswith("/v1"):
+                                base_url = f"{base_url.rstrip('/')}/v1"
+                            
+                            ollama_client = OpenAI(
+                                base_url=base_url,
+                                api_key="ollama", # required but unused
                             )
-                            client = instructor.from_openai(groq_client, mode=instructor.Mode.JSON)
-                        except Exception:
-                            # Last resort: try passing the groq client directly
-                            client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
-                elif provider_name == "OllamaProvider":
-                    # Create OpenAI-compatible client for Ollama
-                    try:
-                        from openai import OpenAI
-                        # Ollama typically runs on localhost:11434/v1
-                        base_url = getattr(self, "base_url", "http://localhost:11434")
-                        if not base_url.endswith("/v1"):
-                            base_url = f"{base_url.rstrip('/')}/v1"
-                        
-                        ollama_client = OpenAI(
-                            base_url=base_url,
-                            api_key="ollama", # required but unused
-                        )
-                        client = instructor.from_openai(ollama_client, mode=instructor.Mode.JSON)
-                    except ImportError:
-                        pass
+                            client = instructor.from_openai(ollama_client, mode=instructor.Mode.JSON)
+                        except ImportError:
+                            pass
                 elif provider_name == "DeepSeekProvider" and self.client:
-                    # DeepSeek is OpenAI compatible
-                    # We need to wrap the underlying client if it exposes the OpenAI interface
-                    # or create a new OpenAI client if self.client is a deepseek.Client (which might be just a wrapper)
-                    # Assuming deepseek.Client is compatible or we can use OpenAI client
-                    try:
-                        # DeepSeek usually works with standard OpenAI client
-                        # If self.client is deepseek.Client, check if we can wrap it
-                        # Otherwise create a new OpenAI client
-                        from openai import OpenAI
-                        if isinstance(self.client, OpenAI):
-                             client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
-                        else:
-                             # Try creating fresh client
-                             ds_client = OpenAI(
-                                 api_key=self.api_key, 
-                                 base_url="https://api.deepseek.com"
-                             )
-                             client = instructor.from_openai(ds_client, mode=instructor.Mode.JSON)
-                    except Exception:
-                        pass
+                    # Try from_provider for DeepSeek
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"deepseek/{kwargs.get('model', self.model)}",
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = None
+
+                    if not client:
+                        # DeepSeek is OpenAI compatible
+                        try:
+                            from openai import OpenAI
+                            if isinstance(self.client, OpenAI):
+                                 client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
+                            else:
+                                 # Try creating fresh client
+                                 ds_client = OpenAI(
+                                     api_key=self.api_key, 
+                                     base_url="https://api.deepseek.com"
+                                 )
+                                 client = instructor.from_openai(ds_client, mode=instructor.Mode.JSON)
+                        except Exception:
+                            pass
+
+                # Global LiteLLM support - if litellm is passed in kwargs or config
+                if not client and (kwargs.get("litellm") or self.config.get("litellm")):
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            # Format for litellm in instructor is litellm/model_name
+                            provider_model = kwargs.get("model", self.model)
+                            litellm_provider = f"litellm/{provider_model}"
+                            client = instructor.from_provider(litellm_provider, api_key=self.api_key)
+                        except Exception:
+                            pass
 
                 if client:
                     # Map generate arguments to client arguments
@@ -318,11 +386,24 @@ class BaseProvider:
                         "temperature": kwargs.get("temperature", 0.1), # Low temp for structured
                     }
                     
+                    verbose_mode = kwargs.get("verbose", False)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [BaseProvider.generate_typed] Using instructor via {provider_name}. Client: {type(client)}", flush=True, file=sys.stdout)
+
+                    # Pass through other common parameters
+                    for param in ["max_tokens", "max_completion_tokens", "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user", "top_k"]:
+                        if param in kwargs:
+                            create_kwargs[param] = kwargs[param]
+
                     # Add provider-specific params
                     if provider_name == "GroqProvider":
                         create_kwargs["response_format"] = {"type": "json_object"}
                     
                     response = client.chat.completions.create(**create_kwargs)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [BaseProvider.generate_typed] Typed response received via instructor ({provider_name}).", flush=True, file=sys.stdout)
                     return response
             except Exception as e:
                 self.logger.warning(f"Instructor generation failed ({e}), falling back to manual repair loop.")
@@ -464,11 +545,24 @@ class OpenAIProvider(BaseProvider):
                 "OpenAI client not initialized. Set OPENAI_API_KEY or pass api_key."
             )
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        
+        # Support max_tokens and max_completion_tokens (for o1 models)
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+            
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+            
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -476,12 +570,25 @@ class OpenAIProvider(BaseProvider):
         if not self.client:
             raise ProcessingError("OpenAI client not initialized.")
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            response_format={"type": "json_object"},
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "response_format": {"type": "json_object"},
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        
+        # Support max_tokens and max_completion_tokens
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        response = self.client.chat.completions.create(**create_kwargs)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -499,26 +606,40 @@ class GeminiProvider(BaseProvider):
         self.api_key = api_key or config.get_api_key("gemini")
         self.model = model
         self.client = None
+        self._use_new_genai = False
         self._init_client()
 
     def _init_client(self):
-        """Initialize Gemini client."""
         try:
-            import google.generativeai as genai
-
+            from google import genai as new_genai
             if self.api_key:
-                genai.configure(api_key=self.api_key)
-                self.client = genai.GenerativeModel(self.model)
-        except (ImportError, OSError):
+                self.client = new_genai.Client(api_key=self.api_key)
+                self._use_new_genai = True
+                return
+        except Exception:
+            pass
+        try:
+            import google.generativeai as old_genai
+            if self.api_key:
+                old_genai.configure(api_key=self.api_key)
+                self.client = old_genai.GenerativeModel(self.model)
+                self._use_new_genai = False
+        except Exception:
             self.client = None
-            self.logger.warning(
-                "google-generativeai library not installed. Install with: pip install semantica[llm-gemini]"
-            )
+            self.logger.warning("Gemini SDK not installed. Install with: pip install semantica[llm-gemini]")
 
     def is_available(self) -> bool:
         """Check if provider is available."""
         return self.client is not None
 
+    def _resp_text(self, resp: Any) -> str:
+        if hasattr(resp, "text"):
+            return getattr(resp, "text")
+        try:
+            return resp.candidates[0].content.parts[0].text
+        except Exception:
+            return str(resp)
+
     def generate(self, prompt: str, **kwargs) -> str:
         """Generate text from prompt."""
         if not self.client:
@@ -526,23 +647,46 @@ class GeminiProvider(BaseProvider):
                 "Gemini client not initialized. Set GEMINI_API_KEY or pass api_key."
             )
 
-        response = self.client.generate_content(
-            prompt, generation_config={"temperature": kwargs.get("temperature", 0.3)}
-        )
-        return response.text
+        if self._use_new_genai:
+            model = kwargs.get("model", self.model)
+            temperature = kwargs.get("temperature", 0.3)
+            create_kwargs = {"model": model, "contents": prompt, "config": {"temperature": temperature}}
+            if "max_tokens" in kwargs:
+                create_kwargs["config"]["max_output_tokens"] = kwargs["max_tokens"]
+            for p in ["top_p", "top_k", "stop_sequences", "candidate_count"]:
+                if p in kwargs:
+                    create_kwargs["config"][p] = kwargs[p]
+            resp = self.client.models.generate_content(**create_kwargs)
+            return self._resp_text(resp)
+        else:
+            generation_config = {"temperature": kwargs.get("temperature", 0.3)}
+            if "max_tokens" in kwargs:
+                generation_config["max_output_tokens"] = kwargs["max_tokens"]
+            for param in ["top_p", "top_k", "stop_sequences", "candidate_count"]:
+                if param in kwargs:
+                    generation_config[param] = kwargs[param]
+            response = self.client.generate_content(prompt, generation_config=generation_config)
+            return self._resp_text(response)
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
         """Generate structured output."""
         if not self.client:
             raise ProcessingError("Gemini client not initialized.")
 
-        # Add JSON format instruction to prompt
         json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
-        response = self.client.generate_content(json_prompt)
-        try:
-            return self._parse_json(response.text)
-        except Exception as e:
-            raise ProcessingError(f"Failed to parse JSON from Gemini response: {e}")
+        if self._use_new_genai:
+            model = kwargs.get("model", self.model)
+            resp = self.client.models.generate_content(model=model, contents=json_prompt)
+            try:
+                return self._parse_json(self._resp_text(resp))
+            except Exception as e:
+                raise ProcessingError(f"Failed to parse JSON from Gemini response: {e}")
+        else:
+            response = self.client.generate_content(json_prompt)
+            try:
+                return self._parse_json(self._resp_text(response))
+            except Exception as e:
+                raise ProcessingError(f"Failed to parse JSON from Gemini response: {e}")
 
 
 class GroqProvider(BaseProvider):
@@ -621,11 +765,32 @@ class GroqProvider(BaseProvider):
                 "Groq client not initialized. Set GROQ_API_KEY or pass api_key."
             )
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        
+        # Support max_tokens and max_completion_tokens
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+            
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        verbose_mode = kwargs.get("verbose", False)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate] Sending request to Groq API (model: {create_kwargs['model']})...", flush=True, file=sys.stdout)
+
+        response = self.client.chat.completions.create(**create_kwargs)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate] Response received from Groq.", flush=True, file=sys.stdout)
         return response.choices[0].message.content
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -638,12 +803,33 @@ class GroqProvider(BaseProvider):
         if "json" not in prompt.lower():
             json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": json_prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-            response_format={"type": "json_object"},
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": json_prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+            "response_format": {"type": "json_object"},
+        }
+        
+        # Support max_tokens and max_completion_tokens
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+            
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        verbose_mode = kwargs.get("verbose", False)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate_structured] Sending structured request to Groq API (model: {create_kwargs['model']})...", flush=True, file=sys.stdout)
+
+        response = self.client.chat.completions.create(**create_kwargs)
+        if verbose_mode:
+            import sys
+            print(f"    [GroqProvider.generate_structured] Structured response received from Groq.", flush=True, file=sys.stdout)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -690,11 +876,23 @@ class AnthropicProvider(BaseProvider):
                 "Anthropic client not initialized. Set ANTHROPIC_API_KEY or pass api_key."
             )
 
-        response = self.client.messages.create(
-            model=kwargs.get("model", self.model),
-            max_tokens=kwargs.get("max_tokens", 4096),
-            messages=[{"role": "user", "content": prompt}],
-        )
+        # Anthropic requires max_tokens. 
+        # We rely on kwargs, but fallback to 8192 (safe max for newer models) if not provided.
+        max_tokens = kwargs.get("max_tokens", 8192)
+        
+        # Prepare arguments
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "max_tokens": max_tokens,
+            "messages": [{"role": "user", "content": prompt}],
+        }
+        
+        # Pass through other common parameters
+        for param in ["temperature", "top_p", "top_k", "stop_sequences", "system", "metadata"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        response = self.client.messages.create(**create_kwargs)
         return response.content[0].text
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -703,11 +901,23 @@ class AnthropicProvider(BaseProvider):
             raise ProcessingError("Anthropic client not initialized.")
 
         json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
-        response = self.client.messages.create(
-            model=kwargs.get("model", self.model),
-            max_tokens=kwargs.get("max_tokens", 4096),
-            messages=[{"role": "user", "content": json_prompt}],
-        )
+        
+        # Anthropic requires max_tokens. 
+        max_tokens = kwargs.get("max_tokens", 8192)
+        
+        # Prepare arguments
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "max_tokens": max_tokens,
+            "messages": [{"role": "user", "content": json_prompt}],
+        }
+        
+        # Pass through other common parameters
+        for param in ["temperature", "top_p", "top_k", "stop_sequences", "system", "metadata"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+        
+        response = self.client.messages.create(**create_kwargs)
         try:
             return self._parse_json(response.content[0].text)
         except Exception as e:
@@ -758,10 +968,25 @@ class OllamaProvider(BaseProvider):
                 "Ollama client not initialized. Make sure Ollama is running."
             )
 
+        options = {"temperature": kwargs.get("temperature", 0.3)}
+        
+        if "max_tokens" in kwargs:
+            options["num_predict"] = kwargs["max_tokens"]
+            
+        if "num_ctx" in kwargs:
+            options["num_ctx"] = kwargs["num_ctx"]
+        elif "context_window" in kwargs:
+            options["num_ctx"] = kwargs["context_window"]
+        
+        # Pass through other common options
+        for param in ["top_p", "top_k", "repeat_penalty", "seed"]:
+            if param in kwargs:
+                options[param] = kwargs[param]
+
         response = self.client.generate(
             model=kwargs.get("model", self.model),
             prompt=prompt,
-            options={"temperature": kwargs.get("temperature", 0.3)},
+            options=options,
         )
         return response.get("response", "")
 
@@ -771,10 +996,26 @@ class OllamaProvider(BaseProvider):
             raise ProcessingError("Ollama client not initialized.")
 
         json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
+        
+        options = {"temperature": kwargs.get("temperature", 0.3)}
+        
+        if "max_tokens" in kwargs:
+            options["num_predict"] = kwargs["max_tokens"]
+            
+        if "num_ctx" in kwargs:
+            options["num_ctx"] = kwargs["num_ctx"]
+        elif "context_window" in kwargs:
+            options["num_ctx"] = kwargs["context_window"]
+
+        # Pass through other common options
+        for param in ["top_p", "top_k", "repeat_penalty", "seed"]:
+            if param in kwargs:
+                options[param] = kwargs[param]
+
         response = self.client.generate(
             model=kwargs.get("model", self.model),
             prompt=json_prompt,
-            options={"temperature": kwargs.get("temperature", 0.3)},
+            options=options,
         )
         try:
             return self._parse_json(response.get("response", "{}"))
@@ -808,11 +1049,16 @@ class DeepSeekProvider(BaseProvider):
     def generate(self, prompt: str, **kwargs) -> str:
         if not self.client:
             raise ProcessingError("DeepSeek client not initialized. Set DEEPSEEK_API_KEY or pass api_key.")
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+            
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        if "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
     def generate_structured(self, prompt: str, **kwargs) -> Union[dict, list]:
         """Generate structured output."""
@@ -879,11 +1125,27 @@ class HuggingFaceLLMProvider(BaseProvider):
             raise ProcessingError("HuggingFace model not initialized.")
 
         inputs = self.tokenizer.encode(prompt, return_tensors="pt").to(self.device)
+        
+        # Use max_new_tokens if available, otherwise fallback to max_length with a safe default
+        generate_kwargs = {
+            "temperature": kwargs.get("temperature", 0.7),
+            "do_sample": True,
+        }
+        
+        if "max_new_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_new_tokens"]
+        elif "max_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_tokens"]
+            
+        # Support legacy max_length if explicitly provided
+        if "max_length" in kwargs:
+            generate_kwargs["max_length"] = kwargs["max_length"]
+            # Remove max_new_tokens if max_length is set to avoid conflict
+            generate_kwargs.pop("max_new_tokens", None)
+
         outputs = self.model.generate(
             inputs,
-            max_length=kwargs.get("max_length", 100),
-            temperature=kwargs.get("temperature", 0.7),
-            do_sample=True,
+            **generate_kwargs
         )
         generated_text = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
         # Remove the original prompt from the response
@@ -924,25 +1186,32 @@ class HuggingFaceModelLoader:
         # Import torch at method level to ensure it's available
         import torch
         
-        cache_key = f"{model_name}_ner"
+        # Include aggregation_strategy in cache key
+        agg_strategy = kwargs.get("aggregation_strategy", "simple")
+        cache_key = f"{model_name}_ner_{agg_strategy}"
         if cache_key in self._cache:
             return self._cache[cache_key]
 
         try:
             from transformers import pipeline
+        except ImportError:
+            raise ImportError(
+                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
+            )
 
+        try:
             nlp = pipeline(
                 "ner",
                 model=model_name,
                 device=self.device if torch.cuda.is_available() else -1,
-                aggregation_strategy="simple",
+                aggregation_strategy=agg_strategy,
+                tokenizer=kwargs.get("tokenizer") # Allow custom tokenizer
             )
             self._cache[cache_key] = nlp
             return nlp
-        except (ImportError, OSError):
-            raise ImportError(
-                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
-            )
+        except OSError as e:
+            self.logger.error(f"Failed to load NER model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load NER model {model_name}: {e}")
             raise
@@ -957,19 +1226,31 @@ class HuggingFaceModelLoader:
             return self._cache[cache_key]
 
         try:
-            from transformers import pipeline
-
-            nlp = pipeline(
-                "text-classification",
-                model=model_name,
-                device=self.device if torch.cuda.is_available() else -1,
-            )
-            self._cache[cache_key] = nlp
-            return nlp
-        except (ImportError, OSError):
+            from transformers import pipeline, AutoTokenizer
+        except ImportError:
             raise ImportError(
                 "transformers library not installed. Install with: pip install semantica[models-huggingface]"
             )
+
+        try:
+            # Allow custom tokenizer
+            tokenizer = kwargs.get("tokenizer")
+            if not tokenizer and kwargs.get("tokenizer_name"):
+                tokenizer = AutoTokenizer.from_pretrained(kwargs.get("tokenizer_name"))
+
+            pipeline_kwargs = {
+                "model": model_name,
+                "device": self.device if torch.cuda.is_available() else -1,
+            }
+            if tokenizer:
+                pipeline_kwargs["tokenizer"] = tokenizer
+
+            nlp = pipeline("text-classification", **pipeline_kwargs)
+            self._cache[cache_key] = nlp
+            return nlp
+        except OSError as e:
+            self.logger.error(f"Failed to load relation model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load relation model {model_name}: {e}")
             raise
@@ -982,18 +1263,27 @@ class HuggingFaceModelLoader:
 
         try:
             from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline
+        except ImportError:
+            raise ImportError(
+                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
+            )
 
-            tokenizer = AutoTokenizer.from_pretrained(model_name)
+        try:
+            # Allow custom tokenizer
+            tokenizer = kwargs.get("tokenizer")
+            if not tokenizer:
+                tokenizer_name = kwargs.get("tokenizer_name", model_name)
+                tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
+            
             model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
             model.to(self.device)
 
             nlp = {"tokenizer": tokenizer, "model": model, "device": self.device}
             self._cache[cache_key] = nlp
             return nlp
-        except (ImportError, OSError):
-            raise ImportError(
-                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
-            )
+        except OSError as e:
+            self.logger.error(f"Failed to load triplet model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load triplet model {model_name}: {e}")
             raise
@@ -1002,50 +1292,199 @@ class HuggingFaceModelLoader:
         """Extract entities using loaded model."""
         return model(text)
 
-    def extract_relations(self, model, text: str, entities: List) -> List[Dict]:
-        """Extract relations using loaded model."""
-        # This would need to be customized based on the model architecture
-        return model(text)
+    def extract_relations(self, model, text: str, entities: List, **kwargs) -> List[Dict]:
+        """
+        Extract relations using loaded model.
+        Iterates through entity pairs and classifies the relationship.
+        """
+        results = []
+        
+        # Sort entities by position
+        sorted_entities = sorted(entities, key=lambda e: e.start_char)
+        
+        # Marker configuration
+        subj_start = kwargs.get("subj_start_marker", "<subj>")
+        subj_end = kwargs.get("subj_end_marker", "</subj>")
+        obj_start = kwargs.get("obj_start_marker", "<obj>")
+        obj_end = kwargs.get("obj_end_marker", "</obj>")
+        
+        # Iterate through all pairs
+        import itertools
+        for i, e1 in enumerate(sorted_entities):
+            for e2 in sorted_entities:
+                if e1 == e2:
+                    continue
+                
+                # Check distance (optional optimization)
+                # if abs(e1.start_char - e2.start_char) > 200: continue
+                
+                # Format text with markers
+                # Strategy: [CLS] text with <subj>...</subj> and <obj>...</obj> [SEP]
+                # We need to insert markers into the original text
+                
+                # Create a copy of text with markers inserted
+                # We need to handle offsets correctly. 
+                # Simplest way: reconstruct string pieces
+                
+                p1_start, p1_end = e1.start_char, e1.end_char
+                p2_start, p2_end = e2.start_char, e2.end_char
+                
+                if p1_start < p2_start:
+                    formatted_text = (
+                        text[:p1_start] + 
+                        f"{subj_start} " + text[p1_start:p1_end] + f" {subj_end}" + 
+                        text[p1_end:p2_start] + 
+                        f"{obj_start} " + text[p2_start:p2_end] + f" {obj_end}" + 
+                        text[p2_end:]
+                    )
+                else:
+                     formatted_text = (
+                        text[:p2_start] + 
+                        f"{obj_start} " + text[p2_start:p2_end] + f" {obj_end}" + 
+                        text[p2_end:p1_start] + 
+                        f"{subj_start} " + text[p1_start:p1_end] + f" {subj_end}" + 
+                        text[p1_end:]
+                    )
+                
+                # Predict
+                try:
+                    # Pipeline returns [{'label': 'LABEL', 'score': 0.99}]
+                    prediction = model(formatted_text, top_k=1)
+                    
+                    if prediction:
+                        res = prediction[0] if isinstance(prediction, list) else prediction
+                        if isinstance(res, list): res = res[0] # top_k=1 returns list of dicts
+                        
+                        label = res.get("label")
+                        score = res.get("score")
+                        
+                        # Filter "no_relation" or low confidence
+                        if label != "no_relation" and score > kwargs.get("threshold", 0.5):
+                            results.append({
+                                "subject": e1,
+                                "object": e2,
+                                "relation": label,
+                                "score": score
+                            })
+                except Exception as e:
+                    self.logger.warning(f"Relation prediction failed for pair {e1.text}-{e2.text}: {e}")
+                    
+        return results
 
-    def extract_triplets(self, model, text: str) -> List[Dict]:
+    def extract_triplets(self, model, text: str, **kwargs) -> List[Dict]:
         """Extract triplets using loaded model."""
         tokenizer = model["tokenizer"]
         model_obj = model["model"]
         device = model["device"]
 
+        # Use kwargs for max_length, default to 512 for input and 128 for output if not specified
+        max_input_length = kwargs.get("max_input_length", 512)
+        max_length = kwargs.get("max_length", 128)
+        
+        generate_kwargs = {"max_length": max_length}
+        if "max_new_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_new_tokens"]
+            
+        # Pass other generation args including beams and penalties
+        for param in ["num_beams", "temperature", "top_p", "top_k", "do_sample", 
+                      "length_penalty", "repetition_penalty"]:
+            if param in kwargs:
+                generate_kwargs[param] = kwargs[param]
+
         inputs = tokenizer(
-            text, return_tensors="pt", truncation=True, max_length=512
+            text, return_tensors="pt", truncation=True, max_length=max_input_length
         ).to(device)
-        outputs = model_obj.generate(**inputs, max_length=128)
-        decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+        
+        outputs = model_obj.generate(**inputs, **generate_kwargs)
+        # Allow controlling skip_special_tokens (important for REBEL which uses special tokens for delimiters)
+        skip_special_tokens = kwargs.get("skip_special_tokens", True)
+        decoded = tokenizer.decode(outputs[0], skip_special_tokens=skip_special_tokens)
 
-        # Parse decoded output (format depends on model)
-        # This is a placeholder - actual parsing would depend on model output format
         return [{"triplet": decoded}]
 
 
-def create_provider(name: str, **kwargs) -> BaseProvider:
-    """Create provider - checks registry for custom providers."""
-    # Check registry first
-    custom_provider = provider_registry.get(name)
-    if custom_provider:
-        return custom_provider(**kwargs)
+class ProviderPool:
+    """Pool for reusing provider instances."""
+    
+    def __init__(self):
+        self._providers: Dict[str, BaseProvider] = {}
+        self.logger = get_logger("provider_pool")
 
-    # Built-in providers
-    builtin = {
-        "openai": OpenAIProvider,
-        "gemini": GeminiProvider,
-        "groq": GroqProvider,
-        "anthropic": AnthropicProvider,
-        "ollama": OllamaProvider,
-        "huggingface_llm": HuggingFaceLLMProvider,
-        "deepseek": DeepSeekProvider,
-    }
+    def get(self, name: str, **kwargs) -> BaseProvider:
+        """Get or create a provider instance."""
+        # Create a cache key from name and kwargs
+        # Filter out non-hashable items or volatile args if any
+        # For now, we assume kwargs are configuration options that should match
+        
+        # Helper to make dict hashable
+        def make_hashable(value):
+            if isinstance(value, dict):
+                return tuple(sorted((k, make_hashable(v)) for k, v in value.items()))
+            elif isinstance(value, list):
+                return tuple(make_hashable(v) for v in value)
+            return value
 
-    provider_class = builtin.get(name.lower())
-    if not provider_class:
-        raise ValueError(
-            f"Unknown provider: {name}. Register custom provider or use built-in: {list(builtin.keys())}"
-        )
+        key_parts = [name]
+        for k, v in sorted(kwargs.items()):
+            # Skip some keys if they shouldn't affect pooling? 
+            # For now, all init args matter for the instance identity.
+            key_parts.append((k, make_hashable(v)))
+            
+        key = str(tuple(key_parts))
+        
+        if key in self._providers:
+            return self._providers[key]
+            
+        self.logger.debug(f"Creating new provider instance for {name}")
+        provider = self._create_provider(name, **kwargs)
+        self._providers[key] = provider
+        return provider
+    
+    def _create_provider(self, name: str, **kwargs) -> BaseProvider:
+        """Internal creation logic."""
+        # Check registry first
+        custom_provider = provider_registry.get(name)
+        if custom_provider:
+            return custom_provider(**kwargs)
 
-    return provider_class(**kwargs)
+        # Built-in providers
+        builtin = {
+            "openai": OpenAIProvider,
+            "gemini": GeminiProvider,
+            "groq": GroqProvider,
+            "anthropic": AnthropicProvider,
+            "ollama": OllamaProvider,
+            "huggingface_llm": HuggingFaceLLMProvider,
+            "deepseek": DeepSeekProvider,
+        }
+
+        provider_class = builtin.get(name.lower())
+        if not provider_class:
+            raise ValueError(
+                f"Unknown provider: {name}. Register custom provider or use built-in: {list(builtin.keys())}"
+            )
+
+        return provider_class(**kwargs)
+
+    def clear(self):
+        """Clear the provider pool."""
+        self._providers.clear()
+
+
+# Global provider pool
+_provider_pool = ProviderPool()
+
+
+def create_provider(name: str, use_pool: bool = True, **kwargs) -> BaseProvider:
+    """
+    Create provider - checks registry for custom providers.
+    
+    Args:
+        name: Provider name
+        use_pool: Whether to use the provider pool (default: True)
+        **kwargs: Provider arguments
+    """
+    if use_pool:
+        return _provider_pool.get(name, **kwargs)
+        
+    return _provider_pool._create_provider(name, **kwargs)

semantica/semantic_extract/relation_extractor.py

@@ -201,10 +201,12 @@ class RelationExtractor:
             )
             
             try:
-                results = []
                 # Ensure lists are same length
                 min_len = min(len(text), len(entities))
+                results = [None] * min_len
                 total_relations_count = 0
+                processed_count = 0
+                
                 # Update more frequently: every 1% or at least every 10 items, but always update for small datasets
                 if min_len <= 10:
                     update_interval = 1  # Update every item for small datasets
@@ -212,58 +214,101 @@ class RelationExtractor:
                     update_interval = max(1, min(10, min_len // 100))
                 
                 # Initial progress update - ALWAYS show this
-                remaining = min_len
                 self.progress_tracker.update_progress(
                     tracking_id,
                     processed=0,
                     total=min_len,
-                    message=f"Starting batch extraction... 0/{min_len} (remaining: {remaining})"
+                    message=f"Starting batch extraction... 0/{min_len}"
                 )
                 
-                for i in range(min_len):
-                    doc_item = text[i]
-                    ent_item = entities[i]
-                    
-                    doc_text = ""
-                    if isinstance(doc_item, dict) and "content" in doc_item:
-                        doc_text = doc_item["content"]
-                    elif isinstance(doc_item, str):
-                        doc_text = doc_item
-                    else:
-                        doc_text = str(doc_item)
-                    
-                    # Ensure ent_item is a list of entities
-                    if not isinstance(ent_item, list):
-                        ent_item = [] # Should not happen if entities is List[List[Entity]]
-                    
-                    current_relations = self.extract_relations(doc_text, ent_item, **kwargs)
-                    
-                    # Add provenance metadata
-                    for rel in current_relations:
-                        if rel.metadata is None:
-                            rel.metadata = {}
-                        rel.metadata["batch_index"] = i
-                        if isinstance(doc_item, dict) and "id" in doc_item:
-                            rel.metadata["document_id"] = doc_item["id"]
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=self.method,
+                )
 
-                    results.append(current_relations)
-                    total_relations_count += len(current_relations)
+                def process_item(i, doc_item, ent_item):
+                    try:
+                        doc_text = ""
+                        if isinstance(doc_item, dict) and "content" in doc_item:
+                            doc_text = doc_item["content"]
+                        elif isinstance(doc_item, str):
+                            doc_text = doc_item
+                        else:
+                            doc_text = str(doc_item)
+                        
+                        # Ensure ent_item is a list of entities
+                        if not isinstance(ent_item, list):
+                            ent_item = [] # Should not happen if entities is List[List[Entity]]
+                        
+                        current_relations = self.extract_relations(doc_text, ent_item, **kwargs)
+                        
+                        # Add provenance metadata
+                        for rel in current_relations:
+                            if rel.metadata is None:
+                                rel.metadata = {}
+                            rel.metadata["batch_index"] = i
+                            if isinstance(doc_item, dict) and "id" in doc_item:
+                                rel.metadata["document_id"] = doc_item["id"]
+                        
+                        return i, current_relations
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {i}: {e}")
+                        return i, []
+
+                if max_workers > 1:
+                    import concurrent.futures
                     
-                    remaining = min_len - (i + 1)
-                    # Update progress: always update for small datasets, or at intervals for large ones
-                    should_update = (
-                        (i + 1) % update_interval == 0 or 
-                        (i + 1) == min_len or 
-                        i == 0 or
-                        min_len <= 10  # Always update for small datasets
-                    )
-                    if should_update:
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=i + 1,
-                            total=min_len,
-                            message=f"Processing documents... {i + 1}/{min_len} (remaining: {remaining}) - Extracted {total_relations_count} relations so far"
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {
+                            executor.submit(process_item, i, text[i], entities[i]): i
+                            for i in range(min_len)
+                        }
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            i, relations = future.result()
+                            results[i] = relations
+                            total_relations_count += len(relations)
+                            processed_count += 1
+                            
+                            should_update = (
+                                processed_count % update_interval == 0 or 
+                                processed_count == min_len or 
+                                processed_count == 1 or
+                                min_len <= 10
+                            )
+                            if should_update:
+                                remaining = min_len - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=min_len,
+                                    message=f"Processing documents... {processed_count}/{min_len} (remaining: {remaining}) - Extracted {total_relations_count} relations so far"
+                                )
+                else:
+                    # Sequential processing
+                    for i in range(min_len):
+                        _, relations = process_item(i, text[i], entities[i])
+                        results[i] = relations
+                        total_relations_count += len(relations)
+                        processed_count += 1
+                        
+                        should_update = (
+                            processed_count % update_interval == 0 or 
+                            processed_count == min_len or 
+                            processed_count == 1 or
+                            min_len <= 10
                         )
+                        if should_update:
+                            remaining = min_len - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=min_len,
+                                message=f"Processing documents... {processed_count}/{min_len} (remaining: {remaining}) - Extracted {total_relations_count} relations so far"
+                            )
                 
                 self.progress_tracker.stop_tracking(
                     tracking_id,
@@ -284,14 +329,19 @@ class RelationExtractor:
             return []
 
     def extract_relations(
-        self, text: str, entities: List[Entity], **options
-    ) -> List[Relation]:
+        self,
+        text: Union[str, List[Dict[str, Any]], List[str]],
+        entities: Union[List[Entity], List[List[Entity]]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Relation], List[List[Relation]]]:
         """
         Extract relations between entities.
 
         Args:
             text: Input text
             entities: List of extracted entities
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Extraction options:
                 - method: Override method (if not set in __init__)
                 - min_confidence: Minimum confidence threshold
@@ -300,7 +350,18 @@ class RelationExtractor:
         Returns:
             list: List of extracted relations
         """
-        from .methods import get_relation_method
+        if isinstance(text, list):
+            if entities is None:
+                entities_batch = [[] for _ in range(len(text))]
+            elif isinstance(entities, list) and (not entities):
+                entities_batch = [[] for _ in range(len(text))]
+            elif isinstance(entities, list) and all(isinstance(e, Entity) for e in entities):
+                entities_batch = [entities for _ in range(len(text))]
+            else:
+                entities_batch = entities
+            return self.extract(text, entities_batch, pipeline_id=pipeline_id, **options)
+
+        from .methods import get_relation_method, match_entity
 
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
@@ -347,8 +408,12 @@ class RelationExtractor:
                         method_options["relation_types"] = relation_types
 
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", all_options.get("model")
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model")
+                            or self.config.get("huggingface_model") 
+                            or self.config.get("model")
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -358,14 +423,13 @@ class RelationExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key
@@ -379,6 +443,12 @@ class RelationExtractor:
                     if verbose_mode and method_name == "llm":
                         import sys
                         print(f"    [RelationExtractor] Processing with {method_name}...", flush=True, file=sys.stdout)
+                        print(f"    [RelationExtractor Debug] method_options keys: {list(method_options.keys())}", flush=True, file=sys.stdout)
+                        if "api_key" in method_options:
+                            masked = method_options["api_key"][:4] + "..." if method_options["api_key"] else "None"
+                            print(f"    [RelationExtractor Debug] api_key present: {masked}", flush=True, file=sys.stdout)
+                        else:
+                            print(f"    [RelationExtractor Debug] api_key NOT present", flush=True, file=sys.stdout)
                     
                     relations = method_func(text, entities, **method_options)
                     
@@ -421,6 +491,11 @@ class RelationExtractor:
 
                 except Exception as e:
                     self.logger.warning(f"Method {method_name} failed: {e}")
+                    if verbose_mode:
+                        import sys
+                        print(f"    [RelationExtractor] ERROR: Method {method_name} failed: {e}", flush=True, file=sys.stderr)
+                        import traceback
+                        traceback.print_exc(file=sys.stderr)
                     continue
 
             # Use first successful method or combine
@@ -487,11 +562,9 @@ class RelationExtractor:
         self, text: str, entities: List[Entity]
     ) -> List[Relation]:
         """Extract relations using pattern matching."""
+        from .methods import match_entity
         relations = []
 
-        # Create entity lookup by text
-        entity_map = {e.text.lower(): e for e in entities}
-
         # Check each relation pattern
         for relation_type, patterns in self.relation_patterns.items():
             for pattern in patterns:
@@ -499,8 +572,8 @@ class RelationExtractor:
                     subject_text = match.group("subject").strip()
                     object_text = match.group("object").strip()
 
-                    subject_entity = entity_map.get(subject_text.lower())
-                    object_entity = entity_map.get(object_text.lower())
+                    subject_entity = match_entity(subject_text, entities)
+                    object_entity = match_entity(object_text, entities)
 
                     if subject_entity and object_entity:
                         # Get context around the match

semantica/semantic_extract/semantic_analyzer.py

@@ -148,8 +148,9 @@ class SemanticAnalyzer:
             )
 
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -165,40 +166,83 @@ class SemanticAnalyzer:
                     message=f"Starting batch analysis... 0/{total_items} (remaining: {total_items})"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
-                    
-                    # Analyze
-                    analysis = self.analyze_semantics(doc_text, **kwargs)
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                )
 
-                    # Add provenance metadata
-                    analysis["batch_index"] = idx
-                    if isinstance(item, dict) and "id" in item:
-                        analysis["document_id"] = item["id"]
-                        
-                    # Also inject into semantic roles if present
-                    if "semantic_roles" in analysis:
-                        for role in analysis["semantic_roles"]:
-                            # role is a dict here because analyze_semantics converts it
-                            if "metadata" not in role:
-                                role["metadata"] = {}
-                            
-                            role["metadata"]["batch_index"] = idx
-                            if isinstance(item, dict) and "id" in item:
-                                role["metadata"]["document_id"] = item["id"]
+                def process_item(idx, item):
+                    try:
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                        analysis = self.analyze_semantics(doc_text, **kwargs)
 
-                    results.append(analysis)
+                        analysis["batch_index"] = idx
+                        if isinstance(item, dict) and "id" in item:
+                            analysis["document_id"] = item["id"]
 
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining})"
+                        if "semantic_roles" in analysis:
+                            for role in analysis["semantic_roles"]:
+                                if "metadata" not in role:
+                                    role["metadata"] = {}
+
+                                role["metadata"]["batch_index"] = idx
+                                if isinstance(item, dict) and "id" in item:
+                                    role["metadata"]["document_id"] = item["id"]
+
+                        return idx, analysis
+                    except Exception as e:
+                        self.logger.warning(f"Failed to analyze item {idx}: {e}")
+                        return idx, {"error": str(e), "batch_index": idx}
+
+                if max_workers > 1:
+                    import concurrent.futures
+
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        future_to_idx = {
+                            executor.submit(process_item, idx, item): idx
+                            for idx, item in enumerate(text)
+                        }
+
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, analysis = future.result()
+                            results[idx] = analysis
+                            processed_count += 1
+
+                            should_update = (
+                                processed_count % update_interval == 0
+                                or processed_count == total_items
+                                or processed_count == 1
+                                or total_items <= 10
+                            )
+                            if should_update:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                                )
+                else:
+                    for idx, item in enumerate(text):
+                        _, analysis = process_item(idx, item)
+                        results[idx] = analysis
+                        processed_count += 1
+
+                        should_update = (
+                            processed_count % update_interval == 0
+                            or processed_count == total_items
+                            or processed_count == 1
+                            or total_items <= 10
                         )
+                        if should_update:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                            )
 
                 self.progress_tracker.stop_tracking(
                     tracking_id,

semantica/semantic_extract/semantic_extract_usage.md

@@ -41,6 +41,7 @@ print(f"Extracted {len(entities)} entities and {len(relations)} relations")
 All extractors support batch processing for high-throughput extraction. You can pass a list of strings or a list of dictionaries (with `content` and `id` keys).
 
 **Features:**
+- **Parallel Processing**: Multi-threaded extraction for high throughput (control via `max_workers`).
 - **Progress Tracking**: Automatically shows a progress bar for large batches.
 - **Provenance Metadata**: Each extracted item includes `batch_index` and `document_id` in its `metadata`.
 
@@ -52,9 +53,13 @@ documents = [
     {"id": "doc_2", "content": "Microsoft Corporation was founded by Bill Gates."}
 ]
 
-extractor = NERExtractor()
+# Initialize with parallel processing enabled
+extractor = NERExtractor(max_workers=4)
 batch_results = extractor.extract(documents)
 
+# OR override during extraction call
+# batch_results = extractor.extract(documents, max_workers=8)
+
 for i, doc_entities in enumerate(batch_results):
     print(f"Document {i} entities:")
     for entity in doc_entities:
@@ -108,9 +113,17 @@ extractor = NERExtractor(method="ml")
 entities = extractor.extract(text)
 print(f"ML method: {len(entities)} entities")
 
-# HuggingFace model extraction
+# HuggingFace model extraction (Bring Your Own Model)
 extractor = NERExtractor(method="huggingface")
-entities = extractor.extract(text, model="dslim/bert-base-NER")
+
+# Use a specific model and aggregation strategy at runtime
+# Runtime options override configuration defaults
+entities = extractor.extract(
+    text, 
+    model="dslim/bert-base-NER", 
+    aggregation_strategy="max", # Options: "simple", "first", "average", "max"
+    device="cpu" # or "cuda"
+)
 print(f"HuggingFace method: {len(entities)} entities")
 
 # LLM-based extraction with advanced options
@@ -120,9 +133,22 @@ entities = extractor.extract(
     provider="openai", 
     model="gpt-4",
     silent_fail=False,      # Raise ProcessingError on failure (default)
-    max_text_length=4000     # Auto-chunking for long text
+    max_text_length=4000,   # Auto-chunking for long text (default: 64k for major providers)
+    max_tokens=4096,        # Explicitly control generation output length
+    temperature=0.0
 )
 print(f"LLM method: {len(entities)} entities")
+
+# Groq extraction with long context support
+# Groq defaults to 64k chunking limit for models like llama-3.3-70b
+groq_extractor = NERExtractor(method="llm")
+groq_entities = groq_extractor.extract(
+    text,
+    provider="groq",
+    model="llama-3.3-70b-versatile",
+    max_tokens=8000 # Passed directly to Groq API
+)
+print(f"Groq method: {len(groq_entities)} entities")
 ```
 
 ### Using NERExtractor Directly
@@ -211,9 +237,18 @@ relations = extractor.extract(text, entities=entities)
 extractor = RelationExtractor(method="cooccurrence")
 relations = extractor.extract(text, entities=entities)
 
-# HuggingFace model
+# HuggingFace model (Bring Your Own Model)
 extractor = RelationExtractor(method="huggingface")
-relations = extractor.extract(text, entities=entities, model="microsoft/DialoGPT-medium")
+
+# Use a sequence classification model trained for relations
+# The extractor automatically formats input with entity markers: 
+# "Steve Jobs founded Apple" -> "<subj> Steve Jobs </subj> founded <obj> Apple </obj>"
+relations = extractor.extract(
+    text, 
+    entities=entities, 
+    model="semantica/relation-model-v1", # Replace with your model ID
+    device="cpu"
+)
 
 # LLM-based relation extraction
 extractor = RelationExtractor(method="llm")
@@ -221,6 +256,8 @@ relations = extractor.extract(
     text, 
     entities=entities, 
     provider="openai",
+    model="gpt-4",
+    max_tokens=2048, # Increased output limit for many relations
     silent_fail=True  # Return empty list if extraction fails
 )
 ```
@@ -274,9 +311,16 @@ triplets = extractor.extract_triplets(text)
 extractor = TripletExtractor(method="rules")
 triplets = extractor.extract_triplets(text)
 
-# HuggingFace model
+# HuggingFace model (Seq2Seq / REBEL)
 extractor = TripletExtractor(method="huggingface")
-triplets = extractor.extract_triplets(text, model="t5-base")
+
+# Use a Seq2Seq model like REBEL for end-to-end triplet extraction
+# This method generates triplets directly from text without needing separate NER/RE steps
+triplets = extractor.extract_triplets(
+    text, 
+    model="Babelscape/rebel-large",
+    device="cpu"
+)
 
 # LLM-based triplet extraction
 extractor = TripletExtractor(method="llm")
@@ -284,7 +328,8 @@ triplets = extractor.extract_triplets(
     text, 
     provider="openai", 
     model="gpt-4",
-    max_text_length=2000  # Force chunking for long text
+    max_text_length=64000, # Large default chunk size supported
+    max_tokens=4096 # Ensure enough tokens for all triplets
 )
 ```
 

semantica/semantic_extract/semantic_network_extractor.py

@@ -149,6 +149,9 @@ class SemanticNetworkExtractor:
             self.config["ner_method"] = method
             self.config["relation_method"] = method
 
+        self._ner_extractor = None
+        self._relation_extractor = None
+
     def extract(
         self,
         text: Union[str, List[str], List[Dict[str, Any]]],
@@ -181,8 +184,9 @@ class SemanticNetworkExtractor:
             )
 
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -198,54 +202,113 @@ class SemanticNetworkExtractor:
                     message=f"Starting batch extraction... 0/{total_items} (remaining: {total_items})"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
-                    
-                    doc_entities = None
-                    if entities and isinstance(entities, list) and idx < len(entities):
-                        doc_entities = entities[idx]
-                    
-                    doc_relations = None
-                    if relations and isinstance(relations, list) and idx < len(relations):
-                        doc_relations = relations[idx]
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                )
 
-                    # Extract
-                    network = self.extract_network(
-                        doc_text, 
-                        entities=doc_entities, 
-                        relations=doc_relations, 
-                        **kwargs
-                    )
-
-                    # Add provenance metadata to nodes and edges
-                    batch_meta = {"batch_index": idx}
-                    if isinstance(item, dict) and "id" in item:
-                        batch_meta["document_id"] = item["id"]
-                    
-                    # Update network metadata
-                    network.metadata.update(batch_meta)
-                    
-                    # Update nodes metadata
-                    for node in network.nodes:
-                        node.metadata.update(batch_meta)
+                def process_item(idx, item, doc_entities, doc_relations):
+                    try:
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
                         
-                    # Update edges metadata
-                    for edge in network.edges:
-                        edge.metadata.update(batch_meta)
-
-                    results.append(network)
-
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining})"
+                        # Extract
+                        network = self.extract_network(
+                            doc_text, 
+                            entities=doc_entities, 
+                            relations=doc_relations, 
+                            **kwargs
                         )
 
+                        # Add provenance metadata to nodes and edges
+                        batch_meta = {"batch_index": idx}
+                        if isinstance(item, dict) and "id" in item:
+                            batch_meta["document_id"] = item["id"]
+                        
+                        # Update network metadata
+                        network.metadata.update(batch_meta)
+                        
+                        # Update nodes metadata
+                        for node in network.nodes:
+                            node.metadata.update(batch_meta)
+                            
+                        # Update edges metadata
+                        for edge in network.edges:
+                            edge.metadata.update(batch_meta)
+                            
+                        return idx, network
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {idx}: {e}")
+                        verbose_mode = kwargs.get("verbose", False) or self.config.get("verbose", False)
+                        if verbose_mode:
+                            import sys
+                            print(f"    [SemanticNetworkExtractor] ERROR: Batch item {idx} failed: {e}", flush=True, file=sys.stderr)
+                        return idx, None
+
+                if max_workers > 1:
+                    import concurrent.futures
+                    
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {}
+                        for idx, item in enumerate(text):
+                            doc_entities = None
+                            if entities and isinstance(entities, list) and idx < len(entities):
+                                doc_entities = entities[idx]
+                            
+                            doc_relations = None
+                            if relations and isinstance(relations, list) and idx < len(relations):
+                                doc_relations = relations[idx]
+                                
+                            future = executor.submit(process_item, idx, item, doc_entities, doc_relations)
+                            future_to_idx[future] = idx
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, network = future.result()
+                            if network:
+                                results[idx] = network
+                            
+                            processed_count += 1
+                            
+                            # Update progress
+                            if (processed_count) % update_interval == 0 or (processed_count) == total_items:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        doc_entities = None
+                        if entities and isinstance(entities, list) and idx < len(entities):
+                            doc_entities = entities[idx]
+                        
+                        doc_relations = None
+                        if relations and isinstance(relations, list) and idx < len(relations):
+                            doc_relations = relations[idx]
+
+                        _, network = process_item(idx, item, doc_entities, doc_relations)
+                        if network:
+                            results[idx] = network
+
+                        processed_count += 1
+                        
+                        # Update progress
+                        if (processed_count) % update_interval == 0 or (processed_count) == total_items:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                            )
+
+                # Filter out None results if any failed
+                results = [r for r in results if r is not None]
+
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
@@ -265,11 +328,12 @@ class SemanticNetworkExtractor:
 
     def extract_network(
         self,
-        text: str,
-        entities: Optional[List[Entity]] = None,
-        relations: Optional[List[Relation]] = None,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        entities: Optional[Union[List[Entity], List[List[Entity]]]] = None,
+        relations: Optional[Union[List[Relation], List[List[Relation]]]] = None,
+        pipeline_id: Optional[str] = None,
         **options,
-    ) -> SemanticNetwork:
+    ) -> Union[SemanticNetwork, List[SemanticNetwork]]:
         """
         Extract semantic network from text.
 
@@ -282,6 +346,23 @@ class SemanticNetworkExtractor:
         Returns:
             SemanticNetwork: Extracted semantic network
         """
+        if isinstance(text, list):
+            entities_batch = entities
+            if entities is not None and isinstance(entities, list) and (not entities or all(isinstance(e, Entity) for e in entities)):
+                entities_batch = [entities for _ in range(len(text))] if entities else [[] for _ in range(len(text))]
+
+            relations_batch = relations
+            if relations is not None and isinstance(relations, list) and (not relations or all(isinstance(r, Relation) for r in relations)):
+                relations_batch = [relations for _ in range(len(text))] if relations else [[] for _ in range(len(text))]
+
+            return self.extract(
+                text,
+                entities=entities_batch,
+                relations=relations_batch,
+                pipeline_id=pipeline_id,
+                **options,
+            )
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="SemanticNetworkExtractor",
@@ -301,15 +382,16 @@ class SemanticNetworkExtractor:
                 # Pass method if specified
                 if "ner_method" in self.config:
                     ner_config["method"] = self.config["ner_method"]
-                ner = NERExtractor(
-                    **ner_config,
-                    **{
-                        k: v
-                        for k, v in self.config.items()
-                        if k not in ["ner", "relation"]
-                    },
-                )
-                entities = ner.extract_entities(text, **options)
+                if self._ner_extractor is None:
+                    self._ner_extractor = NERExtractor(
+                        **ner_config,
+                        **{
+                            k: v
+                            for k, v in self.config.items()
+                            if k not in ["ner", "relation"]
+                        },
+                    )
+                entities = self._ner_extractor.extract_entities(text, **options)
 
             # Extract relations if not provided
             if relations is None:
@@ -320,15 +402,16 @@ class SemanticNetworkExtractor:
                 # Pass method if specified
                 if "relation_method" in self.config:
                     rel_config["method"] = self.config["relation_method"]
-                rel_extractor = RelationExtractor(
-                    **rel_config,
-                    **{
-                        k: v
-                        for k, v in self.config.items()
-                        if k not in ["ner", "relation"]
-                    },
-                )
-                relations = rel_extractor.extract_relations(text, entities, **options)
+                if self._relation_extractor is None:
+                    self._relation_extractor = RelationExtractor(
+                        **rel_config,
+                        **{
+                            k: v
+                            for k, v in self.config.items()
+                            if k not in ["ner", "relation"]
+                        },
+                    )
+                relations = self._relation_extractor.extract_relations(text, entities, **options)
 
             # Build network
             total_steps = 2  # Create nodes, create edges
@@ -355,6 +438,12 @@ class SemanticNetworkExtractor:
             self.progress_tracker.stop_tracking(
                 tracking_id, status="failed", message=str(e)
             )
+            verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+            if verbose_mode:
+                import sys
+                print(f"    [SemanticNetworkExtractor] ERROR: Extraction failed: {e}", flush=True, file=sys.stderr)
+                import traceback
+                traceback.print_exc(file=sys.stderr)
             raise
 
     def _build_network(

semantica/semantic_extract/triplet_extractor.py

@@ -143,6 +143,13 @@ class TripletExtractor:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        if method is not None:
+            self.config["ner_method"] = method
+            self.config["relation_method"] = method
+
+        self._ner_extractor = None
+        self._relation_extractor = None
+
         # Store parameters
         self.triplet_types = triplet_types
         self.include_temporal = include_temporal
@@ -191,9 +198,10 @@ class TripletExtractor:
             )
 
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
                 total_triplets_count = 0
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -206,50 +214,91 @@ class TripletExtractor:
                     tracking_id,
                     processed=0,
                     total=total_items,
-                    message=f"Starting batch extraction... 0/{total_items} (remaining: {total_items})"
+                    message=f"Starting batch extraction... 0/{total_items}"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
-                    
-                    doc_entities = None
-                    if entities and isinstance(entities, list) and idx < len(entities):
-                        doc_entities = entities[idx]
-                    
-                    doc_relations = None
-                    if relations and isinstance(relations, list) and idx < len(relations):
-                        doc_relations = relations[idx]
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=self.method,
+                )
 
-                    # Extract
-                    current_triplets = self.extract_triplets(
-                        doc_text, 
-                        entities=doc_entities, 
-                        relations=doc_relations, 
-                        **kwargs
-                    )
+                def process_item(idx, item):
+                    try:
+                        # Prepare arguments for single item
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                        
+                        doc_entities = None
+                        if entities and isinstance(entities, list) and idx < len(entities):
+                            doc_entities = entities[idx]
+                        
+                        doc_relations = None
+                        if relations and isinstance(relations, list) and idx < len(relations):
+                            doc_relations = relations[idx]
 
-                    # Add provenance metadata
-                    for triplet in current_triplets:
-                        if triplet.metadata is None:
-                            triplet.metadata = {}
-                        triplet.metadata["batch_index"] = idx
-                        if isinstance(item, dict) and "id" in item:
-                            triplet.metadata["document_id"] = item["id"]
-
-                    results.append(current_triplets)
-                    total_triplets_count += len(current_triplets)
-
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining}) - Extracted {total_triplets_count} triplets"
+                        # Extract
+                        current_triplets = self.extract_triplets(
+                            doc_text, 
+                            entities=doc_entities, 
+                            relations=doc_relations, 
+                            **kwargs
                         )
 
+                        # Add provenance metadata
+                        for triplet in current_triplets:
+                            if triplet.metadata is None:
+                                triplet.metadata = {}
+                            triplet.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                triplet.metadata["document_id"] = item["id"]
+                        
+                        return idx, current_triplets
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {idx}: {e}")
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
+                    
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {
+                            executor.submit(process_item, idx, item): idx 
+                            for idx, item in enumerate(text)
+                        }
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, triplets = future.result()
+                            results[idx] = triplets
+                            total_triplets_count += len(triplets)
+                            processed_count += 1
+                            
+                            if processed_count % update_interval == 0 or processed_count == total_items:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_triplets_count} triplets"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, triplets = process_item(idx, item)
+                        results[idx] = triplets
+                        total_triplets_count += len(triplets)
+                        processed_count += 1
+                        
+                        if processed_count % update_interval == 0 or processed_count == total_items:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_triplets_count} triplets"
+                            )
+
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
@@ -269,11 +318,12 @@ class TripletExtractor:
 
     def extract_triplets(
         self,
-        text: str,
-        entities: Optional[List[Entity]] = None,
-        relations: Optional[List[Relation]] = None,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        entities: Optional[Union[List[Entity], List[List[Entity]]]] = None,
+        relations: Optional[Union[List[Relation], List[List[Relation]]]] = None,
+        pipeline_id: Optional[str] = None,
         **options,
-    ) -> List[Triplet]:
+    ) -> Union[List[Triplet], List[List[Triplet]]]:
         """
         Extract RDF triplets from text.
 
@@ -281,11 +331,29 @@ class TripletExtractor:
             text: Input text
             entities: Pre-extracted entities (optional)
             relations: Pre-extracted relations (optional)
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Extraction options
 
         Returns:
             list: List of extracted triplets
         """
+        if isinstance(text, list):
+            entities_batch = entities
+            if entities is not None and isinstance(entities, list) and (not entities or all(isinstance(e, Entity) for e in entities)):
+                entities_batch = [entities for _ in range(len(text))] if entities else [[] for _ in range(len(text))]
+
+            relations_batch = relations
+            if relations is not None and isinstance(relations, list) and (not relations or all(isinstance(r, Relation) for r in relations)):
+                relations_batch = [relations for _ in range(len(text))] if relations else [[] for _ in range(len(text))]
+
+            return self.extract(
+                text,
+                entities=entities_batch,
+                relations=relations_batch,
+                pipeline_id=pipeline_id,
+                **options,
+            )
+
         from .methods import get_triplet_method
 
         tracking_id = self.progress_tracker.start_tracking(
@@ -298,27 +366,62 @@ class TripletExtractor:
             from .ner_extractor import NERExtractor
             from .relation_extractor import RelationExtractor
 
-            # Extract entities if not provided
-            if entities is None:
-                self.progress_tracker.update_tracking(
-                    tracking_id, message="Extracting entities..."
-                )
-                ner = NERExtractor(**self.config.get("ner", {}))
-                entities = ner.extract_entities(text)
-
-            # Extract relations if not provided
-            if relations is None:
-                self.progress_tracker.update_tracking(
-                    tracking_id, message="Extracting relations..."
-                )
-                rel_extractor = RelationExtractor(**self.config.get("relation", {}))
-                relations = rel_extractor.extract_relations(text, entities)
-
             # Use method-based extraction
             methods = options.get("method", self.method)
             if isinstance(methods, str):
                 methods = [methods]
 
+            # Determine if we need to extract entities/relations based on method
+            # HuggingFace (Seq2Seq) does not need pre-extracted entities/relations
+            needs_entities_relations = any(m not in ["huggingface"] for m in methods)
+
+            # Extract entities if not provided
+            if entities is None and needs_entities_relations:
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Extracting entities..."
+                )
+                if self._ner_extractor is None:
+                    ner_config = self.config.get("ner", {})
+                    if "ner_method" in self.config:
+                        ner_config = {**ner_config, "method": self.config["ner_method"]}
+                    
+                    # Filter out 'model' and 'huggingface_model' from shared config
+                    # to prevent passing triplet model to NER extractor
+                    shared_config = {
+                        k: v
+                        for k, v in self.config.items()
+                        if k not in ["ner", "relation", "validator", "serializer", "quality", "model", "huggingface_model"]
+                    }
+
+                    self._ner_extractor = NERExtractor(
+                        **ner_config,
+                        **shared_config,
+                    )
+                entities = self._ner_extractor.extract_entities(text)
+
+            # Extract relations if not provided
+            if relations is None and needs_entities_relations:
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Extracting relations..."
+                )
+                if self._relation_extractor is None:
+                    rel_config = self.config.get("relation", {})
+                    if "relation_method" in self.config:
+                        rel_config = {**rel_config, "method": self.config["relation_method"]}
+                    
+                    # Filter out 'model' and 'huggingface_model' from shared config
+                    shared_config = {
+                        k: v
+                        for k, v in self.config.items()
+                        if k not in ["ner", "relation", "validator", "serializer", "quality", "model", "huggingface_model"]
+                    }
+
+                    self._relation_extractor = RelationExtractor(
+                        **rel_config,
+                        **shared_config,
+                    )
+                relations = self._relation_extractor.extract_relations(text, entities)
+
             triplet_types = options.get("triplet_types", self.triplet_types)
             
             # Merge config with options
@@ -360,8 +463,12 @@ class TripletExtractor:
                         method_options["triplet_types"] = triplet_types
 
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", all_options.get("model")
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model")
+                            or self.config.get("huggingface_model") 
+                            or self.config.get("model")
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -371,18 +478,28 @@ class TripletExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key
 
+                    # Print progress if verbose mode is enabled (only for LLM method to avoid spam)
+                    verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+                    if verbose_mode and method_name == "llm":
+                        import sys
+                        print(f"    [TripletExtractor] Processing with {method_name}...", flush=True, file=sys.stdout)
+                        if "api_key" in method_options:
+                             masked = method_options["api_key"][:4] + "..." if method_options["api_key"] else "None"
+                             print(f"    [TripletExtractor Debug] api_key present: {masked}", flush=True, file=sys.stdout)
+                        else:
+                             print(f"    [TripletExtractor Debug] api_key NOT present", flush=True, file=sys.stdout)
+
                     triplets = method_func(
                         text,
                         entities=entities,
@@ -390,6 +507,11 @@ class TripletExtractor:
                         **method_options,
                     )
 
+                    # Print result count if verbose (only for LLM method)
+                    if verbose_mode and method_name == "llm" and len(triplets) > 0:
+                        import sys
+                        print(f"    [TripletExtractor] Extracted {len(triplets)} triplets", flush=True, file=sys.stdout)
+
                     # Apply weighted scoring if triplet_types are provided
                     if triplet_types:
                         try:
@@ -425,6 +547,12 @@ class TripletExtractor:
 
                 except Exception as e:
                     self.logger.warning(f"Method {method_name} failed: {e}")
+                    verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [TripletExtractor] ERROR: Method {method_name} failed: {e}", flush=True, file=sys.stderr)
+                        import traceback
+                        traceback.print_exc(file=sys.stderr)
                     continue
 
             # Use first successful method or fallback to relation conversion

semantica/utils/progress_tracker.py

@@ -34,6 +34,7 @@ License: MIT
 """
 
 import inspect
+import os
 import sys
 import threading
 import time
@@ -46,6 +47,13 @@ from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
 from .logging import get_logger
 
+DISABLE_JUPYTER_PROGRESS = os.getenv("SEMANTICA_DISABLE_JUPYTER_PROGRESS", "").strip().lower() in (
+    "1",
+    "true",
+    "yes",
+    "on",
+)
+
 # Try to import IPython for Jupyter support
 try:
     from IPython import get_ipython
@@ -1017,6 +1025,7 @@ class ProgressTracker:
 
         # Detect environment - will be checked dynamically
         self.is_jupyter = self._detect_jupyter()
+        self.disable_jupyter_progress = DISABLE_JUPYTER_PROGRESS
 
         # Create displays
         self.displays: List[ProgressDisplay] = []
@@ -1024,7 +1033,7 @@ class ProgressTracker:
         # Always try Jupyter first if available, fallback to console
         if IPYTHON_AVAILABLE:
             # Try to detect Jupyter - if available, use it
-            if self.is_jupyter:
+            if self.is_jupyter and not self.disable_jupyter_progress:
                 self.displays.append(JupyterProgressDisplay(use_emoji=use_emoji))
             # Also add console as fallback for immediate feedback
             self.displays.append(
@@ -1213,7 +1222,11 @@ class ProgressTracker:
         if IPYTHON_AVAILABLE and not self.is_jupyter:
             self.is_jupyter = self._detect_jupyter()
             # If Jupyter is now detected and we don't have a Jupyter display, add it
-            if self.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays):
+            if (
+                self.is_jupyter
+                and not self.disable_jupyter_progress
+                and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays)
+            ):
                 # Insert Jupyter display at the beginning for priority
                 self.displays.insert(0, JupyterProgressDisplay(use_emoji=self.use_emoji))
 
@@ -1341,7 +1354,11 @@ class ProgressTracker:
         if IPYTHON_AVAILABLE and not self.is_jupyter:
             self.is_jupyter = self._detect_jupyter()
             # If Jupyter is now detected and we don't have a Jupyter display, add it
-            if self.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays):
+            if (
+                self.is_jupyter
+                and not self.disable_jupyter_progress
+                and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays)
+            ):
                 # Insert Jupyter display at the beginning for priority
                 self.displays.insert(0, JupyterProgressDisplay(use_emoji=self.use_emoji))
 
@@ -1526,7 +1543,11 @@ def get_progress_tracker() -> ProgressTracker:
     if IPYTHON_AVAILABLE and not _global_tracker.is_jupyter:
         _global_tracker.is_jupyter = _global_tracker._detect_jupyter()
         # If Jupyter is now detected and we don't have a Jupyter display, add it
-        if _global_tracker.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in _global_tracker.displays):
+        if (
+            _global_tracker.is_jupyter
+            and not _global_tracker.disable_jupyter_progress
+            and not any(isinstance(d, JupyterProgressDisplay) for d in _global_tracker.displays)
+        ):
             # Insert Jupyter display at the beginning for priority
             _global_tracker.displays.insert(0, JupyterProgressDisplay(use_emoji=_global_tracker.use_emoji))
     

semantica/vector_store/__init__.py

@@ -3,7 +3,7 @@ Vector Store Management Module
 
 This module provides comprehensive vector storage and retrieval capabilities for the
 Semantica framework, including support for multiple vector store backends (FAISS,
-Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and
+Weaviate, Qdrant, Pinecone, Milvus), hybrid search combining vector similarity and
 metadata filtering, metadata management, and namespace isolation.
 
 Algorithms Used:
@@ -58,6 +58,7 @@ Supported Backends:
     - FAISS: In-memory/local disk (Facebook AI Similarity Search)
     - Weaviate: Cloud/Self-hosted (Schema-aware vector database)
     - Qdrant: Cloud/Self-hosted (Vector database for the next generation of AI)
+    - Pinecone: Cloud-managed (Managed vector database service)
     - Milvus: Cloud/Self-hosted (Highly scalable vector database)
     - InMemory: Simple list-based storage for testing/small datasets
 
@@ -73,7 +74,7 @@ Dependencies:
     - pymilvus
 
 Key Features:
-    - Multi-backend vector store support (FAISS, Weaviate, Qdrant, Milvus)
+    - Multi-backend vector store support (FAISS, Weaviate, Qdrant, Pinecone, Milvus)
     - Vector indexing and similarity search
     - Metadata indexing and filtering
     - Hybrid search combining vector and metadata queries
@@ -91,6 +92,7 @@ Main Classes:
     - FAISSStore: FAISS integration for local vector storage
     - WeaviateStore: Weaviate vector database integration
     - QdrantStore: Qdrant vector database integration
+    - PineconeStore: Pinecone vector database integration
     - MilvusStore: Milvus vector database integration
     - HybridSearch: Hybrid vector and metadata search
     - MetadataStore: Metadata indexing and management
@@ -145,6 +147,7 @@ from .methods import (
 )
 from .milvus_store import MilvusStore, MilvusClient, MilvusCollection, MilvusSearch
 from .namespace_manager import Namespace, NamespaceManager
+from .pinecone_store import PineconeStore, PineconeClient, PineconeIndex, PineconeSearch
 from .qdrant_store import QdrantStore, QdrantClient, QdrantCollection, QdrantSearch
 from .registry import MethodRegistry, method_registry
 from .vector_store import VectorIndexer, VectorManager, VectorRetriever, VectorStore
@@ -181,6 +184,11 @@ __all__ = [
     "MilvusClient",
     "MilvusCollection",
     "MilvusSearch",
+    # Pinecone
+    "PineconeStore",
+    "PineconeClient",
+    "PineconeIndex",
+    "PineconeSearch",
     # Hybrid search
     "HybridSearch",
     "MetadataFilter",

semantica/vector_store/pinecone_store.py

@@ -0,0 +1,639 @@
+"""
+Pinecone Store Module
+
+This module provides Pinecone vector database integration for vector storage and
+similarity search in the Semantica framework, supporting managed vector database
+service with serverless and pod-based indexes, namespace isolation, and efficient
+vector operations with metadata filtering.
+
+Key Features:
+ - Serverless and Pod-based index management
+ - Namespace isolation for multi-tenant support
+ - Metadata filtering during search
+ - Batch operations for efficient data loading
+ - Index creation, deletion, and listing
+ - Optional dependency handling
+
+Main Classes:
+ - PineconeStore: Main Pinecone store for vector operations
+ - PineconeClient: Pinecone client wrapper
+ - PineconeIndex: Index wrapper with operations
+ - PineconeSearch: Search operations and filtering
+
+Example Usage:
+ >>> from semantica.vector_store import PineconeStore
+ >>> store = PineconeStore(api_key="your-api-key")
+ >>> store.connect()
+ >>> store.create_index("my-index", dimension=768)
+ >>> store.upsert_vectors(vectors, ids, metadata=metadata)
+ >>> results = store.search_vectors(query_vector, k=10, filter={"category": "science"})
+ >>> stats = store.get_stats()
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any, Dict, List, Optional, Union
+
+import numpy as np
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from ..utils.progress_tracker import get_progress_tracker
+
+# Optional Pinecone import
+try:
+    from pinecone import Pinecone as PineconeClientLib, ServerlessSpec, PodSpec
+
+    PINECONE_AVAILABLE = True
+except (ImportError, OSError):
+    PINECONE_AVAILABLE = False
+    PineconeClientLib = None
+    ServerlessSpec = None
+    PodSpec = None
+
+
+class PineconeClient:
+    """Pinecone client wrapper."""
+
+    def __init__(self, client: Any):
+        """Initialize Pinecone client wrapper."""
+        self.client = client
+        self.logger = get_logger("pinecone_client")
+
+    def create_index(
+        self,
+        index_name: str,
+        dimension: int,
+        metric: str = "cosine",
+        spec: Optional[Dict[str, Any]] = None,
+        **options,
+    ) -> bool:
+        """Create an index in Pinecone."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            # Default to serverless spec if not provided
+            if spec is None:
+                spec = ServerlessSpec(cloud="aws", region="us-east-1")
+
+            # Map metric names
+            metric_map = {
+                "cosine": "cosine",
+                "euclidean": "euclidean_distance",
+                "dot": "dotproduct",
+            }
+            pinecone_metric = metric_map.get(metric.lower(), "cosine")
+
+            self.client.create_index(
+                name=index_name,
+                dimension=dimension,
+                metric=pinecone_metric,
+                spec=spec,
+                **options,
+            )
+            return True
+        except Exception as e:
+            raise ProcessingError(f"Failed to create index: {str(e)}")
+
+    def delete_index(self, index_name: str) -> bool:
+        """Delete an index from Pinecone."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            self.client.delete_index(index_name)
+            return True
+        except Exception as e:
+            raise ProcessingError(f"Failed to delete index: {str(e)}")
+
+    def list_indexes(self) -> List[str]:
+        """List available indexes."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            indexes = self.client.list_indexes()
+            return [index.name for index in indexes]
+        except Exception as e:
+            raise ProcessingError(f"Failed to list indexes: {str(e)}")
+
+    def get_index(self, index_name: str) -> Any:
+        """Get index object."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            return self.client.Index(index_name)
+        except Exception as e:
+            raise ProcessingError(f"Failed to get index: {str(e)}")
+
+
+class PineconeIndex:
+    """Pinecone index wrapper."""
+
+    def __init__(self, index: Any):
+        """Initialize Pinecone index wrapper."""
+        self.index = index
+        self.logger = get_logger("pinecone_index")
+
+    def upsert_vectors(
+        self,
+        vectors: List[List[float]],
+        ids: List[str],
+        metadata: Optional[List[Dict[str, Any]]] = None,
+        namespace: str = "",
+        **options,
+    ) -> Dict[str, Any]:
+        """Upsert vectors to index."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            # Prepare vectors for upsert
+            upsert_data = []
+            for i, (vector, vector_id) in enumerate(zip(vectors, ids)):
+                vector_dict = {"id": vector_id, "values": vector}
+                if metadata and i < len(metadata):
+                    vector_dict["metadata"] = metadata[i]
+                upsert_data.append(vector_dict)
+
+            response = self.index.upsert(
+                vectors=upsert_data, namespace=namespace, **options
+            )
+            return {"upserted_count": response.upserted_count}
+        except Exception as e:
+            raise ProcessingError(f"Failed to upsert vectors: {str(e)}")
+
+    def search_vectors(
+        self,
+        query_vector: List[float],
+        k: int = 10,
+        filter: Optional[Dict[str, Any]] = None,
+        namespace: str = "",
+        **options,
+    ) -> List[Dict[str, Any]]:
+        """Search for similar vectors."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            response = self.index.query(
+                vector=query_vector,
+                top_k=k,
+                filter=filter,
+                namespace=namespace,
+                include_metadata=True,
+                include_values=False,
+                **options,
+            )
+
+            results = []
+            for match in response.matches:
+                results.append(
+                    {
+                        "id": match.id,
+                        "score": match.score,
+                        "metadata": match.metadata or {},
+                    }
+                )
+
+            return results
+        except Exception as e:
+            raise ProcessingError(f"Failed to search vectors: {str(e)}")
+
+    def delete_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """Delete vectors from index."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            response = self.index.delete(ids=vector_ids, namespace=namespace, **options)
+            return {"deleted": True}
+        except Exception as e:
+            raise ProcessingError(f"Failed to delete vectors: {str(e)}")
+
+    def fetch_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """Fetch vectors by ID."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            response = self.index.fetch(ids=vector_ids, namespace=namespace, **options)
+            return {
+                "vectors": {
+                    vector_id: {
+                        "values": vector.values,
+                        "metadata": vector.metadata or {},
+                    }
+                    for vector_id, vector in response.vectors.items()
+                }
+            }
+        except Exception as e:
+            raise ProcessingError(f"Failed to fetch vectors: {str(e)}")
+
+    def describe_index_stats(self, **options) -> Dict[str, Any]:
+        """Get index statistics."""
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            stats = self.index.describe_index_stats(**options)
+            return {
+                "dimension": stats.dimension,
+                "index_fullness": stats.index_fullness,
+                "total_vector_count": stats.total_vector_count,
+                "namespaces": stats.namespaces,
+            }
+        except Exception as e:
+            raise ProcessingError(f"Failed to get index stats: {str(e)}")
+
+
+class PineconeSearch:
+    """Pinecone search operations."""
+
+    def __init__(self, index: PineconeIndex):
+        """Initialize Pinecone search."""
+        self.index = index
+        self.logger = get_logger("pinecone_search")
+
+    def similarity_search(
+        self,
+        query_vector: np.ndarray,
+        limit: int = 10,
+        filter: Optional[Dict[str, Any]] = None,
+        namespace: str = "",
+        **options,
+    ) -> List[Dict[str, Any]]:
+        """
+        Perform similarity search.
+
+        Args:
+            query_vector: Query vector
+            limit: Number of results
+            filter: Metadata filter
+            namespace: Namespace to search in
+            **options: Additional options
+
+        Returns:
+            List of search results
+        """
+        return self.index.search_vectors(
+            query_vector.tolist(), limit, filter, namespace, **options
+        )
+
+
+class PineconeStore:
+    """
+    Pinecone store for vector storage and similarity search.
+
+    • Pinecone connection and authentication
+    • Index and namespace management
+    • Vector storage and retrieval
+    • Similarity search and filtering
+    • Performance optimization
+    • Error handling and recovery
+    """
+
+    def __init__(
+        self,
+        api_key: Optional[str] = None,
+        environment: Optional[str] = None,
+        **config,
+    ):
+        """Initialize Pinecone store."""
+        self.logger = get_logger("pinecone_store")
+        self.config = config
+        self.progress_tracker = get_progress_tracker()
+        # Ensure progress tracker is enabled
+        if not self.progress_tracker.enabled:
+            self.progress_tracker.enabled = True
+
+        self.api_key = api_key or config.get("api_key")
+        self.environment = environment or config.get("environment")
+
+        self.client: Optional[PineconeClient] = None
+        self.index: Optional[PineconeIndex] = None
+        self.search_engine: Optional[PineconeSearch] = None
+
+        # Check Pinecone availability
+        if not PINECONE_AVAILABLE:
+            self.logger.warning(
+                "Pinecone not available. Install with: pip install pinecone-client"
+            )
+
+    def connect(self, **kwargs) -> bool:
+        """
+        Connect to Pinecone service.
+
+        Args:
+            **kwargs: Connection options
+
+        Returns:
+            True if connected successfully
+        """
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError(
+                "Pinecone is not available. Install it with: pip install pinecone-client"
+            )
+
+        api_key = kwargs.get("api_key") or self.api_key
+        if not api_key:
+            raise ValidationError("Pinecone API key is required")
+
+        try:
+            pinecone_client = PineconeClientLib(api_key=api_key, **kwargs)
+            self.client = PineconeClient(pinecone_client)
+
+            self.logger.info("Connected to Pinecone")
+            return True
+        except Exception as e:
+            raise ProcessingError(f"Failed to connect to Pinecone: {str(e)}")
+
+    def create_index(
+        self,
+        index_name: str,
+        dimension: int,
+        metric: str = "cosine",
+        spec: Optional[Dict[str, Any]] = None,
+        **kwargs,
+    ):
+        """
+        Create a Pinecone index.
+
+        Args:
+            index_name: Name of the index
+            dimension: Vector dimension
+            metric: Distance metric ("cosine", "euclidean", "dot")
+            spec: Index specification (ServerlessSpec or PodSpec)
+            **kwargs: Additional options
+
+        Returns:
+            PineconeIndex instance
+        """
+        if self.client is None:
+            self.connect()
+
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            # Create index spec if not provided
+            if spec is None:
+                spec = ServerlessSpec(cloud="aws", region="us-east-1")
+
+            self.client.create_index(index_name, dimension, metric, spec, **kwargs)
+
+            # Get the index
+            pinecone_index = self.client.get_index(index_name)
+            self.index = PineconeIndex(pinecone_index)
+            self.search_engine = PineconeSearch(self.index)
+
+            self.logger.info(f"Created Pinecone index: {index_name}")
+            return self.index
+
+        except Exception as e:
+            raise ProcessingError(f"Failed to create index: {str(e)}")
+
+    def get_index(self, index_name: str) -> PineconeIndex:
+        """
+        Get existing index.
+
+        Args:
+            index_name: Name of the index
+
+        Returns:
+            PineconeIndex instance
+        """
+        if self.client is None:
+            self.connect()
+
+        if not PINECONE_AVAILABLE:
+            raise ProcessingError("Pinecone not available")
+
+        try:
+            pinecone_index = self.client.get_index(index_name)
+            self.index = PineconeIndex(pinecone_index)
+            self.search_engine = PineconeSearch(self.index)
+            return self.index
+        except Exception as e:
+            raise ProcessingError(f"Failed to get index: {str(e)}")
+
+    def delete_index(self, index_name: str) -> bool:
+        """
+        Delete an index.
+
+        Args:
+            index_name: Name of the index to delete
+
+        Returns:
+            True if deleted successfully
+        """
+        if self.client is None:
+            self.connect()
+
+        return self.client.delete_index(index_name)
+
+    def list_indexes(self) -> List[str]:
+        """
+        List available indexes.
+
+        Returns:
+            List of index names
+        """
+        if self.client is None:
+            self.connect()
+
+        return self.client.list_indexes()
+
+    def upsert_vectors(
+        self,
+        vectors: List[Any],
+        ids: List[str],
+        metadata: Optional[List[Dict[str, Any]]] = None,
+        namespace: str = "",
+        **options,
+    ) -> Dict[str, Any]:
+        """
+        Upsert vectors to index.
+
+        Args:
+            vectors: List of vectors
+            ids: Vector IDs
+            metadata: Optional metadata for each vector
+            namespace: Namespace to upsert into
+            **options: Additional options
+
+        Returns:
+            Upsert response
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="vector_store",
+            submodule="PineconeStore",
+            message=f"Upserting {len(vectors)} vectors to Pinecone index",
+        )
+
+        try:
+            if self.index is None:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message="Index not initialized"
+                )
+                raise ProcessingError(
+                    "Index not initialized. Call create_index() or get_index() first."
+                )
+
+            if not PINECONE_AVAILABLE:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message="Pinecone not available"
+                )
+                raise ProcessingError("Pinecone not available")
+
+            self.progress_tracker.update_tracking(
+                tracking_id, message="Preparing vectors..."
+            )
+
+            # Convert vectors to list format
+            vector_list = []
+            for vector in vectors:
+                if isinstance(vector, np.ndarray):
+                    vector_list.append(vector.tolist())
+                else:
+                    vector_list.append(list(vector))
+
+            self.progress_tracker.update_tracking(
+                tracking_id, message="Upserting vectors to index..."
+            )
+            result = self.index.upsert_vectors(
+                vector_list, ids, metadata, namespace, **options
+            )
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Upserted {len(vectors)} vectors",
+            )
+            return result
+
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise ProcessingError(f"Failed to upsert vectors: {str(e)}")
+
+    def search_vectors(
+        self,
+        query_vector: Any,
+        k: int = 10,
+        filter: Optional[Dict[str, Any]] = None,
+        namespace: str = "",
+        **options,
+    ) -> List[Dict[str, Any]]:
+        """
+        Search vectors in index.
+
+        Args:
+            query_vector: Query vector
+            k: Number of results
+            filter: Metadata filter
+            namespace: Namespace to search in
+            **options: Additional options
+
+        Returns:
+            List of search results
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="vector_store",
+            submodule="PineconeStore",
+            message=f"Searching for {k} similar vectors in Pinecone",
+        )
+
+        try:
+            if self.search_engine is None:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message="Index not initialized"
+                )
+                raise ProcessingError(
+                    "Index not initialized. Call create_index() or get_index() first."
+                )
+
+            self.progress_tracker.update_tracking(
+                tracking_id, message="Performing similarity search..."
+            )
+
+            # Convert query vector to list
+            if isinstance(query_vector, np.ndarray):
+                query_vector = query_vector.tolist()
+            else:
+                query_vector = list(query_vector)
+
+            results = self.search_engine.similarity_search(
+                np.array(query_vector), k, filter, namespace, **options
+            )
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Found {len(results)} similar vectors",
+            )
+            return results
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
+    def delete_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """
+        Delete vectors from index.
+
+        Args:
+            vector_ids: Vector IDs to delete
+            namespace: Namespace to delete from
+            **options: Additional options
+
+        Returns:
+            Delete response
+        """
+        if self.index is None:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() or get_index() first."
+            )
+
+        return self.index.delete_vectors(vector_ids, namespace, **options)
+
+    def fetch_vectors(
+        self, vector_ids: List[str], namespace: str = "", **options
+    ) -> Dict[str, Any]:
+        """
+        Fetch vectors by ID.
+
+        Args:
+            vector_ids: Vector IDs to fetch
+            namespace: Namespace to fetch from
+            **options: Additional options
+
+        Returns:
+            Fetch response
+        """
+        if self.index is None:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() or get_index() first."
+            )
+
+        return self.index.fetch_vectors(vector_ids, namespace, **options)
+
+    def get_stats(self, **options) -> Dict[str, Any]:
+        """Get index statistics."""
+        if self.index is None:
+            raise ProcessingError(
+                "Index not initialized. Call create_index() or get_index() first."
+            )
+
+        return self.index.describe_index_stats(**options)

semantica/vector_store/vector_store.py

@@ -38,6 +38,7 @@ License: MIT
 """
 
 from typing import Any, Dict, List, Optional, Tuple, Union
+import concurrent.futures
 
 import numpy as np
 
@@ -59,9 +60,9 @@ class VectorStore:
     • Provides vector store operations
     """
 
-    SUPPORTED_BACKENDS = {"faiss", "weaviate", "qdrant", "milvus", "inmemory"}
+    SUPPORTED_BACKENDS = {"faiss", "weaviate", "qdrant", "milvus", "pinecone", "inmemory"}
 
-    def __init__(self, backend="faiss", config=None, **kwargs):
+    def __init__(self, backend="faiss", config=None, max_workers: int = 6, **kwargs):
         """Initialize vector store."""
         if backend.lower() not in self.SUPPORTED_BACKENDS:
             raise ValueError(
@@ -72,6 +73,7 @@ class VectorStore:
         self.logger = get_logger("vector_store")
         self.config = config or {}
         self.config.update(kwargs)
+        self.max_workers = max_workers
         self.progress_tracker = get_progress_tracker()
         # Ensure progress tracker is enabled
         if not self.progress_tracker.enabled:
@@ -127,6 +129,141 @@ class VectorStore:
         self.logger.warning("Using random fallback embedding")
         return np.random.rand(self.dimension).astype(np.float32)
 
+    def embed_batch(self, texts: List[str]) -> List[np.ndarray]:
+        """
+        Generate embeddings for a list of texts using the internal embedder.
+        
+        Args:
+            texts: List of texts to embed
+            
+        Returns:
+            List of numpy arrays
+        """
+        if self.embedder:
+            try:
+                # generate_embeddings handles list input
+                embeddings = self.embedder.generate_embeddings(texts)
+                # Ensure it returns a list of arrays (it returns 2D array or list)
+                if isinstance(embeddings, np.ndarray):
+                    return list(embeddings)
+                return embeddings
+            except Exception as e:
+                self.logger.warning(f"Batch embedding generation failed: {e}")
+        
+        # Fallback
+        self.logger.warning("Using random fallback embeddings for batch")
+        return [np.random.rand(self.dimension).astype(np.float32) for _ in texts]
+
+    def add_documents(
+        self,
+        documents: List[str],
+        metadata: Optional[List[Dict[str, Any]]] = None,
+        batch_size: int = 32,
+        parallel: bool = True,
+        **options,
+    ) -> List[str]:
+        """
+        Add multiple documents to the store with parallel embedding generation.
+        
+        Args:
+            documents: List of document texts
+            metadata: List of metadata dictionaries
+            batch_size: Number of documents to process in one batch
+            parallel: Whether to use parallel processing for embeddings
+            **options: Additional options
+            
+        Returns:
+            List[str]: Vector IDs
+        """
+        if not documents:
+            return []
+            
+        num_docs = len(documents)
+        metadata = metadata or [{} for _ in range(num_docs)]
+        
+        if len(metadata) != num_docs:
+            raise ValueError("Metadata list length must match documents length")
+            
+        all_vectors = [None] * num_docs
+        
+        # Helper for processing a batch
+        def process_batch(start_idx: int, end_idx: int):
+            batch_texts = documents[start_idx:end_idx]
+            batch_embeddings = self.embed_batch(batch_texts)
+            return start_idx, batch_embeddings
+
+        # Calculate batches
+        batches = []
+        for i in range(0, num_docs, batch_size):
+            batches.append((i, min(i + batch_size, num_docs)))
+            
+        tracking_id = self.progress_tracker.start_tracking(
+            module="vector_store",
+            submodule="VectorStore",
+            message=f"Processing {num_docs} documents (parallel={parallel})",
+        )
+
+        try:
+            if parallel and self.max_workers > 1:
+                self.progress_tracker.update_tracking(
+                    tracking_id, message=f"Embedding with {self.max_workers} workers..."
+                )
+                with concurrent.futures.ThreadPoolExecutor(max_workers=self.max_workers) as executor:
+                    futures = [
+                        executor.submit(process_batch, start, end)
+                        for start, end in batches
+                    ]
+                    
+                    completed = 0
+                    for future in concurrent.futures.as_completed(futures):
+                        start_idx, embeddings = future.result()
+                        # Place results in correct order
+                        for i, emb in enumerate(embeddings):
+                            all_vectors[start_idx + i] = emb
+                        
+                        completed += 1
+                        if completed % 5 == 0:  # Update progress periodically
+                            self.progress_tracker.update_tracking(
+                                tracking_id, 
+                                message=f"Embedded batch {completed}/{len(batches)}"
+                            )
+            else:
+                # Sequential processing
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Embedding sequentially..."
+                )
+                for i, (start, end) in enumerate(batches):
+                    _, embeddings = process_batch(start, end)
+                    for j, emb in enumerate(embeddings):
+                        all_vectors[start + j] = emb
+                    
+                    if i % 5 == 0:
+                        self.progress_tracker.update_tracking(
+                            tracking_id, 
+                            message=f"Embedded batch {i+1}/{len(batches)}"
+                        )
+                        
+            # Verify all embeddings generated
+            if any(v is None for v in all_vectors):
+                raise ProcessingError("Failed to generate all embeddings")
+                
+            # Store all vectors in one go
+            self.progress_tracker.update_tracking(tracking_id, message="Storing vectors...")
+            vector_ids = self.store_vectors(all_vectors, metadata=metadata, **options)
+            
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Added {len(vector_ids)} documents",
+            )
+            return vector_ids
+            
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
     def store(
         self,
         vectors: List[np.ndarray],

semantica/vector_store/vector_store_usage.md

@@ -1,6 +1,6 @@
 # Vector Store Module Usage Guide
 
-This comprehensive guide demonstrates how to use the vector store module for vector storage and retrieval, supporting multiple vector store backends (FAISS, Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and metadata filtering, metadata management, and namespace isolation.
+This comprehensive guide demonstrates how to use the vector store module for vector storage and retrieval, supporting multiple vector store backends (FAISS, Weaviate, Qdrant, Pinecone, Milvus), hybrid search combining vector similarity and metadata filtering, metadata management, and namespace isolation.
 
 ## Table of Contents
 
@@ -754,6 +754,39 @@ results = store.search(
 print(f"Found {len(results)} results")
 ```
 
+### Pinecone Store
+
+```python
+from semantica.vector_store import PineconeStore
+import numpy as np
+
+# Create Pinecone store
+store = PineconeStore(api_key="your-api-key")
+
+# Connect
+store.connect()
+
+# Create index
+store.create_index("my-index", dimension=768, metric="cosine")
+
+# Upsert vectors
+vectors = [np.random.rand(768).tolist() for _ in range(100)]
+ids = [f"vec_{i}" for i in range(100)]
+metadata = [{"category": "science"} for _ in range(100)]
+store.upsert_vectors(vectors, ids, metadata=metadata, namespace="my-namespace")
+
+# Search
+query_vector = np.random.rand(768).tolist()
+results = store.search_vectors(
+    query_vector,
+    k=10,
+    filter={"category": {"$eq": "science"}},
+    namespace="my-namespace"
+)
+
+print(f"Found {len(results)} results")
+```
+
 ### Milvus Store
 
 ```python

tests/ingest/test_ontology_ingestor.py

@@ -0,0 +1,214 @@
+import os
+import shutil
+import tempfile
+import pytest
+from pathlib import Path
+from semantica.ingest import OntologyIngestor, ingest, ingest_ontology, OntologyData
+
+class TestOntologyIngestor:
+    @pytest.fixture
+    def sample_ttl_content(self):
+        return """
+        @prefix : <http://example.org/ontology/> .
+        @prefix owl: <http://www.w3.org/2002/07/owl#> .
+        @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+        @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+        @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
+
+        <http://example.org/ontology/> rdf:type owl:Ontology ;
+            rdfs:label "Test Ontology" .
+
+        :Person rdf:type owl:Class ;
+            rdfs:label "Person" .
+
+        :hasName rdf:type owl:DatatypeProperty ;
+            rdfs:domain :Person ;
+            rdfs:range xsd:string .
+        """
+
+    def test_ingest_single_file(self, sample_ttl_content):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".ttl", mode="w") as tmp:
+            tmp.write(sample_ttl_content)
+            tmp_path = tmp.name
+
+        try:
+            ingestor = OntologyIngestor()
+            result = ingestor.ingest_ontology(tmp_path)
+            
+            assert isinstance(result, OntologyData)
+            assert result.data["name"] == "Test Ontology" or result.data["name"] == os.path.basename(tmp_path)
+            assert any(cls["name"] == "Person" for cls in result.data["classes"])
+            assert any(prop["name"] == "hasName" for prop in result.data["properties"])
+            assert result.metadata["format"] == "ttl" or result.metadata["format"] == "turtle"
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_ingest_directory(self, sample_ttl_content):
+        with tempfile.TemporaryDirectory() as tmp_dir:
+            # Create two ontology files
+            file1 = os.path.join(tmp_dir, "ont1.ttl")
+            file2 = os.path.join(tmp_dir, "ont2.rdf")
+            
+            with open(file1, "w") as f:
+                f.write(sample_ttl_content)
+            
+            # Simple RDF/XML content for the second file
+            rdf_content = """
+            <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+                     xmlns:owl="http://www.w3.org/2002/07/owl#">
+              <owl:Ontology rdf:about="http://example.org/ont2"/>
+              <owl:Class rdf:about="http://example.org/ont2/Animal"/>
+            </rdf:RDF>
+            """
+            with open(file2, "w") as f:
+                f.write(rdf_content)
+            
+            ingestor = OntologyIngestor()
+            results = ingestor.ingest_directory(tmp_dir)
+            
+            assert len(results) == 2
+            assert all(isinstance(r, OntologyData) for r in results)
+            
+            # Verify results contain expected classes
+            classes = [cls["name"] for res in results for cls in res.data["classes"]]
+            assert "Person" in classes
+            assert "Animal" in classes
+
+    def test_unified_ingest_function(self, sample_ttl_content):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".ttl", mode="w") as tmp:
+            tmp.write(sample_ttl_content)
+            tmp_path = tmp.name
+            
+        try:
+            # Test auto-detection via unified ingest
+            result = ingest(tmp_path)
+            assert "ontology" in result
+            assert isinstance(result["ontology"], OntologyData)
+            assert len(result["ontology"].data["classes"]) > 0
+            
+            # Test explicit source type
+            result_explicit = ingest(tmp_path, source_type="ontology")
+            assert "ontology" in result_explicit
+            assert result_explicit["ontology"].metadata["source_path"] == tmp_path
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_convenience_function(self, sample_ttl_content):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".n3", mode="w") as tmp:
+            tmp.write(sample_ttl_content)
+            tmp_path = tmp.name
+            
+        try:
+            result = ingest_ontology(tmp_path)
+            assert isinstance(result, OntologyData)
+            assert len(result.data["classes"]) > 0
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_ingest_formats(self):
+        """Test ingestion of all supported formats."""
+        ingestor = OntologyIngestor()
+        
+        # 1. JSON-LD
+        jsonld_content = """
+        {
+          "@context": {
+            "owl": "http://www.w3.org/2002/07/owl#",
+            "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
+            "rdfs": "http://www.w3.org/2000/01/rdf-schema#"
+          },
+          "@id": "http://example.org/jsonld",
+          "@type": "owl:Ontology",
+          "rdfs:label": "JSON-LD Ontology",
+          "owl:versionInfo": "1.0"
+        }
+        """
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".jsonld", mode="w") as tmp:
+            tmp.write(jsonld_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            assert result.data["name"] == "JSON-LD Ontology"
+            assert result.metadata["format"] == "json-ld"
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+        # 2. N-Triples
+        nt_content = '<http://example.org/nt/Class> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2002/07/owl#Class> .\n'
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".nt", mode="w") as tmp:
+            tmp.write(nt_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            # N-Triples often doesn't have ontology metadata, so name might default to basename
+            assert result.data["name"] == os.path.basename(tmp_path)
+            assert any(cls["uri"] == "http://example.org/nt/Class" for cls in result.data["classes"])
+            assert result.metadata["format"] == "nt"
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+        # 3. Notation3
+        n3_content = """
+        @prefix : <http://example.org/n3/> .
+        @prefix owl: <http://www.w3.org/2002/07/owl#> .
+        :N3Class a owl:Class .
+        """
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".n3", mode="w") as tmp:
+            tmp.write(n3_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            assert any(cls["uri"] == "http://example.org/n3/N3Class" for cls in result.data["classes"])
+            # format might be 'n3' or 'turtle' depending on rdflib detection as they are similar
+            assert result.metadata["format"] in ["n3", "turtle"] 
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+        # 4. RDF/XML (.owl)
+        owl_content = """
+        <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+                 xmlns:owl="http://www.w3.org/2002/07/owl#"
+                 xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">
+          <owl:Ontology rdf:about="http://example.org/owl"/>
+          <owl:Class rdf:about="http://example.org/owl/OwlClass">
+            <rdfs:label>OwlClass</rdfs:label>
+          </owl:Class>
+        </rdf:RDF>
+        """
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".owl", mode="w") as tmp:
+            tmp.write(owl_content)
+            tmp_path = tmp.name
+        try:
+            result = ingestor.ingest_ontology(tmp_path)
+            assert any(cls["name"] == "OwlClass" for cls in result.data["classes"])
+            assert result.metadata["format"] in ["xml", "rdf", "owl"]
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)
+
+    def test_error_handling(self):
+        ingestor = OntologyIngestor()
+        with pytest.raises(Exception): # Specific exception type depends on implementation, likely ValidationError or FileNotFoundError
+            ingestor.ingest_ontology("non_existent_file.ttl")
+
+    def test_invalid_content(self):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".ttl", mode="w") as tmp:
+            tmp.write("This is not valid turtle content")
+            tmp_path = tmp.name
+            
+        try:
+            ingestor = OntologyIngestor()
+            # Depending on implementation, this might raise an exception or return partial/empty result with error in metadata
+            # Given current implementation uses g.parse(), it likely raises an exception which is caught or propagated
+            # If propagated:
+            with pytest.raises(Exception):
+                ingestor.ingest_ontology(tmp_path)
+        finally:
+            if os.path.exists(tmp_path):
+                os.remove(tmp_path)

tests/integration/test_relations_groq.py

@@ -0,0 +1,98 @@
+import os
+import unittest
+import time
+
+from semantica.semantic_extract import NERExtractor, RelationExtractor
+
+# Use environment variable for API key
+_GROQ_KEY = os.getenv("GROQ_API_KEY") or os.getenv("GROQ_TEST_API_KEY")
+@unittest.skipUnless(_GROQ_KEY, "Groq key not set; skipping live integration test")
+class TestGroqRelationsIntegration(unittest.TestCase):
+    def setUp(self):
+        self.api_key = _GROQ_KEY
+        self.model = "llama-3.1-8b-instant"
+        # Short, unambiguous finance snippet
+        self.text_short = (
+            "Apple reported revenue of $4.4 billion in Q1 2024 and provided guidance for FY 2025."
+        )
+        # Longer text to exercise chunking and ensure no hang
+        self.text_long = (
+            "Apple reported revenue of $4.4 billion in Q1 2024. "
+            "The company also reported growth of 12% year-over-year and provided guidance for FY 2025. "
+            "Microsoft reported revenue of $6.1 billion in Q2 2024 and expects sequential growth. "
+            "NVIDIA reported record revenue in 2024 Q1 and guided for higher revenue in Q2 2024. "
+        ) * 20  # expand length
+
+    def _extract_entities(self, text):
+        ner = NERExtractor(
+            method="llm",
+            provider="groq",
+            llm_model=self.model,
+            api_key=self.api_key,
+            temperature=0.0,
+        )
+        entities = ner.extract_entities(text, entity_types=["ORGANIZATION", "MONEY", "DATE", "EVENT", "PERCENT"])
+        self.assertIsInstance(entities, list)
+        return entities
+
+    def test_relations_short_text(self):
+        entities = self._extract_entities(self.text_short)
+        self.assertGreater(len(entities), 0, "NER should extract entities for short text")
+
+        relation_extractor = RelationExtractor(
+            method="llm",
+            relation_types=[
+                "HAS_REVENUE",
+                "HAS_GROWTH",
+                "PROVIDES_GUIDANCE",
+                "IN_QUARTER",
+                "FOR_PERIOD",
+                "RELATED_TO",
+            ],
+            provider="groq",
+            llm_model=self.model,
+            api_key=self.api_key,
+            temperature=0.0,
+            verbose=True,
+        )
+
+        start = time.time()
+        relations = relation_extractor.extract_relations(text=self.text_short, entities=entities)
+        elapsed = time.time() - start
+
+        self.assertIsInstance(relations, list)
+        # Ensure call completes reasonably fast (network dependent; allow generous bound)
+        self.assertLess(elapsed, 60, f"Extraction took too long: {elapsed:.2f}s")
+        # Do not strictly assert >0 as model output may vary, but log for diagnostics
+        if relations:
+            sample = relations[0]
+            self.assertTrue(hasattr(sample, "subject") and hasattr(sample, "predicate") and hasattr(sample, "object"))
+
+    def test_relations_long_text_chunking(self):
+        entities = self._extract_entities(self.text_long)
+        self.assertGreater(len(entities), 0, "NER should extract entities for long text")
+
+        relation_extractor = RelationExtractor(
+            method="llm",
+            relation_types=["RELATED_TO", "HAS_REVENUE", "IN_QUARTER"],
+            provider="groq",
+            llm_model=self.model,
+            api_key=self.api_key,
+            temperature=0.0,
+            verbose=True,
+        )
+
+        start = time.time()
+        relations = relation_extractor.extract_relations(text=self.text_long, entities=entities)
+        elapsed = time.time() - start
+
+        self.assertIsInstance(relations, list)
+        # Ensure completion (chunked path) and no hang
+        self.assertLess(elapsed, 120, f"Chunked extraction took too long: {elapsed:.2f}s")
+        if relations:
+            for r in relations[:3]:
+                self.assertTrue(hasattr(r, "subject") and hasattr(r, "predicate") and hasattr(r, "object"))
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/kg/test_graph_builder_external.py

@@ -0,0 +1,232 @@
+import unittest
+from unittest.mock import MagicMock, patch
+
+from semantica.kg.graph_builder import GraphBuilder
+
+
+class TestGraphBuilderExternal(unittest.TestCase):
+    def setUp(self):
+        self.mock_tracker_patcher = patch("semantica.utils.progress_tracker.get_progress_tracker")
+        self.mock_get_tracker = self.mock_tracker_patcher.start()
+        self.mock_tracker = MagicMock()
+        self.mock_get_tracker.return_value = self.mock_tracker
+
+        self.mock_resolver_patcher = patch("semantica.kg.entity_resolver.EntityResolver")
+        self.mock_resolver_cls = self.mock_resolver_patcher.start()
+
+        self.mock_conflict_patcher = patch("semantica.conflicts.conflict_detector.ConflictDetector")
+        self.mock_conflict_cls = self.mock_conflict_patcher.start()
+
+    def tearDown(self):
+        self.mock_tracker_patcher.stop()
+        self.mock_resolver_patcher.stop()
+        self.mock_conflict_patcher.stop()
+
+    def test_single_source_dict_with_source_id_target_id(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "drug:1", "name": "Aspirin", "type": "Drug"},
+            {"id": "disease:1", "name": "Myocardial infarction", "type": "Disease"},
+        ]
+        relationships = [
+            {"source_id": "drug:1", "target_id": "disease:1", "type": "TREATS"},
+        ]
+
+        source = {"entities": entities, "relationships": relationships}
+
+        kg = builder.build(source)
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "drug:1")
+        self.assertEqual(rel.get("target"), "disease:1")
+        self.assertEqual(kg["metadata"]["num_relationships"], 1)
+
+    def test_sources_list_merge_with_external_relationships(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source1 = {
+            "entities": [{"id": "1", "name": "A"}],
+            "relationships": [{"source_id": "1", "target_id": "2", "type": "REL_1"}],
+        }
+        source2 = {
+            "entities": [{"id": "2", "name": "B"}],
+            "relationships": [{"source_id": "2", "target_id": "1", "type": "REL_2"}],
+        }
+
+        kg = builder.build([source1, source2])
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 2)
+        sources = {r["source"] for r in kg["relationships"]}
+        targets = {r["target"] for r in kg["relationships"]}
+        self.assertEqual(sources, {"1", "2"})
+        self.assertEqual(targets, {"1", "2"})
+
+    def test_build_with_explicit_relationships_argument_external_ids(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "REL"},
+        ]
+
+        kg = builder.build(entities, relationships=relationships)
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "2")
+
+    def test_build_single_source_external_graph(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source = {
+            "entities": [{"id": "1", "name": "A"}],
+            "relationships": [{"source_id": "1", "target_id": "1", "type": "SELF"}],
+        }
+
+        kg = builder.build_single_source(source)
+
+        self.assertEqual(len(kg["entities"]), 1)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "1")
+
+    def test_relationship_key_variants_normalized(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+            {"id": "3", "name": "C"},
+            {"id": "4", "name": "D"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "R1"},
+            {"source": "2", "target": "3", "type": "R2"},
+            {"subject": "3", "object": "4", "type": "R3"},
+        ]
+
+        kg = builder.build({"entities": entities, "relationships": relationships})
+
+        self.assertEqual(len(kg["relationships"]), 3)
+        ids = {(r["source"], r["target"]) for r in kg["relationships"]}
+        self.assertIn(("1", "2"), ids)
+        self.assertIn(("2", "3"), ids)
+        self.assertIn(("3", "4"), ids)
+
+    def test_warning_when_all_relationships_dropped(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source = {
+            "entities": [],
+            "relationships": [{"foo": "x"}, {"bar": "y"}],
+        }
+
+        with patch.object(builder.logger, "warning") as mock_warning:
+            kg = builder.build(source)
+
+        self.assertEqual(len(kg["relationships"]), 0)
+        mock_warning.assert_called()
+        args, _ = mock_warning.call_args
+        self.assertIn("All relationships were dropped", args[0])
+
+    def test_no_warning_when_some_relationships_kept(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        source = {
+            "entities": [{"id": "1"}, {"id": "2"}],
+            "relationships": [
+                {"source_id": "1", "target_id": "2", "type": "REL"},
+                {"foo": "x"},
+            ],
+        }
+
+        with patch.object(builder.logger, "warning") as mock_warning:
+            kg = builder.build(source)
+
+        self.assertEqual(len(kg["relationships"]), 2)
+        mock_warning.assert_not_called()
+
+    def test_issue_208_minimal_reproduction_shape(self):
+        builder = GraphBuilder(
+            merge_entities=False,
+            entity_resolution_strategy="none",
+            resolve_conflicts=False,
+        )
+
+        entities = [
+            {"id": "e1", "name": "Entity 1"},
+            {"id": "e2", "name": "Entity 2"},
+            {"id": "e3", "name": "Entity 3"},
+        ]
+        relationships = [
+            {"source_id": "e1", "target_id": "e2", "type": "REL_1"},
+            {"source_id": "e2", "target_id": "e3", "type": "REL_2"},
+        ]
+
+        entity_ids = {e["id"] for e in entities}
+        for r in relationships:
+            self.assertIn(r["source_id"], entity_ids)
+            self.assertIn(r["target_id"], entity_ids)
+
+        kg = builder.build(
+            sources=[{"entities": entities, "relationships": relationships}],
+            merge_entities=False,
+        )
+
+        self.assertEqual(len(kg["entities"]), 3)
+        self.assertEqual(len(kg["relationships"]), 2)
+        pairs = {(r["source"], r["target"]) for r in kg["relationships"]}
+        self.assertIn(("e1", "e2"), pairs)
+        self.assertIn(("e2", "e3"), pairs)
+
+    def test_issue_206_earnings_call_shape(self):
+        builder = GraphBuilder(
+            merge_entities=False,
+            entity_resolution_strategy="none",
+            resolve_conflicts=False,
+        )
+
+        entities = [
+            {
+                "id": "entity_446_MDA Space Ltd.",
+                "name": "MDA Space Ltd.",
+                "type": "ORGANIZATION",
+            },
+            {
+                "id": "entity_500_$409.8 million",
+                "name": "$409.8 million",
+                "type": "MONEY",
+            },
+        ]
+
+        relationships = [
+            {
+                "id": None,
+                "source_id": "MDA Space Ltd.",
+                "target_id": "$409.8 million",
+                "type": "HAS_REVENUE",
+                "confidence": 0.975,
+                "metadata": {},
+            }
+        ]
+
+        kg = builder.build(
+            sources=[{"entities": entities, "relationships": relationships}],
+            merge_entities=False,
+        )
+
+        self.assertEqual(len(kg["entities"]), 2)
+        self.assertEqual(len(kg["relationships"]), 1)
+        rel = kg["relationships"][0]
+        self.assertEqual(rel.get("source"), "MDA Space Ltd.")
+        self.assertEqual(rel.get("target"), "$409.8 million")

tests/kg/test_kg.py

@@ -91,6 +91,30 @@ class TestGraphBuilder(unittest.TestCase):
         graph2 = builder.build(source_list)
         self.assertEqual(len(graph2["entities"]), 2)
 
+    def test_build_with_external_relationship_ids(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "rel"},
+        ]
+
+        source = {
+            "entities": entities,
+            "relationships": relationships,
+        }
+
+        graph = builder.build(source)
+
+        self.assertEqual(len(graph["entities"]), 2)
+        self.assertEqual(len(graph["relationships"]), 1)
+        rel = graph["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "2")
+
     def test_build_with_conflict_resolution(self):
         """Test building with conflict resolution enabled"""
         builder = GraphBuilder(resolve_conflicts=True)
@@ -106,6 +130,50 @@ class TestGraphBuilder(unittest.TestCase):
         self.mock_conflict_cls.return_value.detect_conflicts.assert_called_once()
         self.mock_conflict_cls.return_value.resolve_conflicts.assert_called_once()
 
+    def test_build_single_source(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+        source = {
+            "entities": [{"id": "1", "name": "A"}],
+            "relationships": [{"source_id": "1", "target_id": "1", "type": "self"}],
+        }
+        graph = builder.build_single_source(source)
+        self.assertEqual(len(graph["entities"]), 1)
+        self.assertEqual(len(graph["relationships"]), 1)
+
+    def test_build_with_explicit_relationships_argument(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+
+        entities = [
+            {"id": "1", "name": "A"},
+            {"id": "2", "name": "B"},
+        ]
+        relationships = [
+            {"source_id": "1", "target_id": "2", "type": "rel"},
+        ]
+
+        graph = builder.build(entities, relationships=relationships)
+
+        self.assertEqual(len(graph["entities"]), 2)
+        self.assertEqual(len(graph["relationships"]), 1)
+        rel = graph["relationships"][0]
+        self.assertEqual(rel.get("source"), "1")
+        self.assertEqual(rel.get("target"), "2")
+
+    def test_build_warns_when_all_relationships_dropped(self):
+        builder = GraphBuilder(merge_entities=False, resolve_conflicts=False)
+        source = {
+            "entities": [],
+            "relationships": [{"foo": "x"}, {"bar": "y"}],
+        }
+
+        with patch.object(builder.logger, "warning") as mock_warning:
+            graph = builder.build(source)
+
+        self.assertEqual(len(graph["relationships"]), 0)
+        mock_warning.assert_called()
+        args, _ = mock_warning.call_args
+        self.assertIn("All relationships were dropped", args[0])
+
 class TestGraphAnalyzer(unittest.TestCase):
     def setUp(self):
         self.mock_tracker_patcher = patch("semantica.kg.graph_analyzer.get_progress_tracker")

tests/pipeline/test_pipeline.py

@@ -107,5 +107,25 @@ class TestPipelineModule(unittest.TestCase):
         
         self.assertEqual(execution_order, ["A", "B", "C"])
 
+    def test_imports_no_circular_dependencies(self):
+        import semantica
+
+        _ = semantica.pipeline
+
+        from semantica.pipeline import PipelineBuilder, PipelineValidator
+        from semantica.deduplication import DuplicateDetector
+
+        builder = PipelineBuilder()
+        builder.add_step("step1", "dummy")
+        pipeline = builder.build("import_test_pipeline")
+
+        validator = PipelineValidator()
+        result = validator.validate_pipeline(pipeline)
+
+        self.assertTrue(result.valid)
+
+        detector = DuplicateDetector()
+        self.assertIsNotNone(detector)
+
 if __name__ == "__main__":
     unittest.main()

tests/reproduce_issue_176.py

@@ -0,0 +1,100 @@
+
+import unittest
+from unittest.mock import MagicMock, patch
+from semantica.semantic_extract.methods import extract_relations_llm, extract_entities_llm, extract_triplets_llm
+from semantica.semantic_extract.ner_extractor import Entity
+
+class TestMaxTokensPropagation(unittest.TestCase):
+    @patch("semantica.semantic_extract.methods.create_provider")
+    def test_max_tokens_propagation_relations(self, mock_create_provider):
+        """Test that max_tokens is passed to generate_typed in extract_relations_llm."""
+        # Setup mock
+        mock_llm = MagicMock()
+        mock_create_provider.return_value = mock_llm
+        mock_llm.is_available.return_value = True
+        
+        # Setup return value to avoid pydantic validation errors
+        mock_response = MagicMock()
+        mock_response.relations = []
+        mock_llm.generate_typed.return_value = mock_response
+
+        # Create dummy entities
+        entities = [Entity(text="Foo", label="ORG", start_char=0, end_char=3)]
+
+        # Call the function with max_tokens
+        extract_relations_llm(
+            text="some text",
+            entities=entities,
+            provider="openai",
+            model="gpt-4",
+            max_tokens=128000
+        )
+
+        # Check if generate_typed was called with max_tokens
+        args, kwargs = mock_llm.generate_typed.call_args
+        
+        print(f"Relations Call kwargs: {kwargs}")
+        
+        self.assertIn("max_tokens", kwargs)
+        self.assertEqual(kwargs["max_tokens"], 128000)
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    def test_max_tokens_propagation_entities(self, mock_create_provider):
+        """Test that max_tokens is passed to generate_typed in extract_entities_llm."""
+        # Setup mock
+        mock_llm = MagicMock()
+        mock_create_provider.return_value = mock_llm
+        mock_llm.is_available.return_value = True
+        
+        # Setup return value to avoid pydantic validation errors
+        mock_response = MagicMock()
+        mock_response.entities = []
+        mock_llm.generate_typed.return_value = mock_response
+
+        # Call the function with max_tokens
+        extract_entities_llm(
+            text="some text",
+            provider="openai",
+            model="gpt-4",
+            max_tokens=128000
+        )
+
+        # Check if generate_typed was called with max_tokens
+        args, kwargs = mock_llm.generate_typed.call_args
+        
+        print(f"Entities Call kwargs: {kwargs}")
+        
+        self.assertIn("max_tokens", kwargs)
+        self.assertEqual(kwargs["max_tokens"], 128000)
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    def test_max_tokens_propagation_triplets(self, mock_create_provider):
+        """Test that max_tokens is passed to generate_typed in extract_triplets_llm."""
+        # Setup mock
+        mock_llm = MagicMock()
+        mock_create_provider.return_value = mock_llm
+        mock_llm.is_available.return_value = True
+        
+        # Setup return value to avoid pydantic validation errors
+        mock_response = MagicMock()
+        mock_response.triplets = []
+        mock_llm.generate_typed.return_value = mock_response
+
+        # Call the function with max_tokens
+        extract_triplets_llm(
+            text="some text",
+            provider="openai",
+            model="gpt-4",
+            max_tokens=128000
+        )
+
+        # Check if generate_typed was called with max_tokens
+        args, kwargs = mock_llm.generate_typed.call_args
+        
+        print(f"Triplets Call kwargs: {kwargs}")
+        
+        self.assertIn("max_tokens", kwargs)
+        self.assertEqual(kwargs["max_tokens"], 128000)
+
+if __name__ == "__main__":
+    unittest.main()

tests/semantic_extract/perf_real_use_cases.py

@@ -0,0 +1,125 @@
+import statistics
+import time
+import os
+import sys
+from typing import Dict, List
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
+
+from semantica.semantic_extract.event_detector import EventDetector
+from semantica.semantic_extract.ner_extractor import NERExtractor
+from semantica.semantic_extract.relation_extractor import RelationExtractor
+from semantica.semantic_extract.semantic_analyzer import SemanticAnalyzer
+from semantica.semantic_extract.semantic_network_extractor import SemanticNetworkExtractor
+from semantica.semantic_extract.triplet_extractor import TripletExtractor
+from semantica.utils.progress_tracker import get_progress_tracker
+
+
+def _make_documents(n: int) -> List[Dict[str, str]]:
+    base = (
+        "Apple Inc. was founded by Steve Jobs in 1976 and is headquartered in Cupertino, California. "
+        "Microsoft Corporation was founded by Bill Gates and Paul Allen in 1975. "
+        "In 2014, Apple acquired Beats Electronics for $3 billion. "
+        "In 2023, Google announced a partnership with OpenAI to improve search experiences."
+    )
+    return [{"id": f"doc_{i}", "content": f"{base} Document number {i}."} for i in range(n)]
+
+
+def _median_seconds(fn, repeats: int = 3) -> float:
+    times = []
+    for _ in range(repeats):
+        start = time.perf_counter()
+        fn()
+        times.append(time.perf_counter() - start)
+    return statistics.median(times)
+
+
+def _bench(label: str, fn, repeats: int = 3) -> dict:
+    fn()
+    seconds = _median_seconds(fn, repeats=repeats)
+    return {"label": label, "seconds": seconds}
+
+
+def main():
+    progress = get_progress_tracker()
+    progress.displays = []
+
+    docs = _make_documents(80)
+    texts = [d["content"] for d in docs]
+
+    ner = NERExtractor(method="pattern")
+    rel = RelationExtractor(method="pattern")
+    trip = TripletExtractor(method="pattern")
+    events = EventDetector(method="pattern")
+    analyzer = SemanticAnalyzer()
+    net = SemanticNetworkExtractor(ner_method="pattern", relation_method="pattern")
+
+    results = []
+
+    def ner_parallel():
+        ner.extract(texts)
+
+    def ner_seq():
+        ner.extract(texts, max_workers=1)
+
+    results.append(_bench("NER batch (default workers)", ner_parallel))
+    results.append(_bench("NER batch (max_workers=1)", ner_seq))
+
+    entities_batch = ner.extract(texts)
+
+    def rel_parallel():
+        rel.extract(texts, entities_batch)
+
+    def rel_seq():
+        rel.extract(texts, entities_batch, max_workers=1)
+
+    results.append(_bench("Relation batch (default workers)", rel_parallel))
+    results.append(_bench("Relation batch (max_workers=1)", rel_seq))
+
+    def trip_parallel():
+        trip.extract(texts)
+
+    def trip_seq():
+        trip.extract(texts, max_workers=1)
+
+    results.append(_bench("Triplet pipeline (default workers)", trip_parallel))
+    results.append(_bench("Triplet pipeline (max_workers=1)", trip_seq))
+
+    def ev_parallel():
+        events.detect_events(texts)
+
+    def ev_seq():
+        events.detect_events(texts, max_workers=1)
+
+    results.append(_bench("Event detection (default workers)", ev_parallel))
+    results.append(_bench("Event detection (max_workers=1)", ev_seq))
+
+    def analyzer_parallel():
+        analyzer.analyze(texts)
+
+    def analyzer_seq():
+        analyzer.analyze(texts, max_workers=1)
+
+    results.append(_bench("Semantic analysis (default workers)", analyzer_parallel))
+    results.append(_bench("Semantic analysis (max_workers=1)", analyzer_seq))
+
+    def net_parallel():
+        net.extract_network(texts)
+
+    def net_seq():
+        net.extract_network(texts, max_workers=1)
+
+    results.append(_bench("Semantic network (default workers)", net_parallel))
+    results.append(_bench("Semantic network (max_workers=1)", net_seq))
+
+    per_doc = []
+    for row in results:
+        per_doc.append({**row, "ms_per_doc": (row["seconds"] / len(texts)) * 1000.0})
+
+    print(f"Documents: {len(texts)}")
+    for row in per_doc:
+        print(f"{row['label']}: {row['seconds']:.3f}s  ({row['ms_per_doc']:.2f} ms/doc)")
+
+
+if __name__ == "__main__":
+    main()

tests/semantic_extract/test_extractors.py

@@ -7,8 +7,12 @@ import os
 sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
 
 from semantica.semantic_extract.ner_extractor import NERExtractor
+from semantica.semantic_extract.ner_extractor import Entity as NEREntity
 from semantica.semantic_extract.relation_extractor import RelationExtractor
 from semantica.semantic_extract.triplet_extractor import TripletExtractor
+from semantica.semantic_extract.event_detector import EventDetector
+from semantica.semantic_extract.semantic_analyzer import SemanticAnalyzer
+from semantica.semantic_extract.semantic_network_extractor import SemanticNetworkExtractor
 from semantica.semantic_extract.named_entity_recognizer import Entity
 from semantica.semantic_extract.relation_extractor import Relation
 
@@ -51,6 +55,21 @@ class TestExtractors(unittest.TestCase):
         self.assertIsInstance(entities, list)
         mock_get_method.assert_called()
 
+    @patch("semantica.semantic_extract.methods.get_entity_method")
+    def test_ner_extraction_batch_via_extract_entities(self, mock_get_method):
+        mock_method = MagicMock()
+        mock_method.extract_entities.return_value = []
+        mock_get_method.return_value = mock_method
+
+        extractor = NERExtractor(method="pattern")
+        results = extractor.extract_entities(
+            ["Test text 1", "Test text 2"],
+        )
+
+        self.assertIsInstance(results, list)
+        self.assertEqual(len(results), 2)
+        self.assertTrue(all(isinstance(r, list) for r in results))
+
     @patch("semantica.semantic_extract.methods.get_relation_method")
     def test_relation_extraction(self, mock_get_method):
         """Test relation extraction call"""
@@ -65,6 +84,21 @@ class TestExtractors(unittest.TestCase):
         self.assertIsInstance(relations, list)
         mock_get_method.assert_called()
 
+    @patch("semantica.semantic_extract.methods.get_relation_method")
+    def test_relation_extraction_batch_via_extract_relations(self, mock_get_method):
+        mock_method = MagicMock()
+        mock_method.extract_relations.return_value = []
+        mock_get_method.return_value = mock_method
+
+        extractor = RelationExtractor(method="pattern")
+        texts = ["A knows B", "A knows B"]
+        entities = [NEREntity(text="A", label="PERSON", start_char=0, end_char=1, confidence=1.0)]
+
+        results = extractor.extract_relations(texts, entities)
+        self.assertIsInstance(results, list)
+        self.assertEqual(len(results), 2)
+        self.assertTrue(all(isinstance(r, list) for r in results))
+
     @patch("semantica.semantic_extract.methods.get_triplet_method")
     def test_triplet_extraction(self, mock_get_method):
         """Test triplet extraction call"""
@@ -81,5 +115,55 @@ class TestExtractors(unittest.TestCase):
         self.assertIsInstance(triplets, list)
         mock_get_method.assert_called()
 
+    @patch("semantica.semantic_extract.methods.get_triplet_method")
+    def test_triplet_extraction_batch_via_extract_triplets(self, mock_get_method):
+        mock_method = MagicMock()
+        mock_method.extract_triplets.return_value = []
+        mock_get_method.return_value = mock_method
+
+        extractor = TripletExtractor(method="pattern")
+        texts = ["A knows A", "A knows A"]
+        entities_batch = [[NEREntity(text="A", label="PERSON", start_char=0, end_char=1, confidence=1.0)] for _ in texts]
+        relations_batch = [[] for _ in texts]
+
+        results = extractor.extract_triplets(
+            texts,
+            entities=entities_batch,
+            relations=relations_batch,
+        )
+
+        self.assertIsInstance(results, list)
+        self.assertEqual(len(results), 2)
+        self.assertTrue(all(isinstance(r, list) for r in results))
+
+    def test_event_detector_batch_via_detect_events(self):
+        detector = EventDetector()
+        texts = ["Apple acquired Beats in 2014.", "Google announced a partnership in 2023."]
+        results = detector.detect_events(texts)
+        self.assertIsInstance(results, list)
+        self.assertEqual(len(results), 2)
+        self.assertTrue(all(isinstance(r, list) for r in results))
+
+    def test_semantic_analyzer_batch_parallel(self):
+        analyzer = SemanticAnalyzer()
+        texts = ["A short sentence.", "Another short sentence."]
+        results = analyzer.analyze(texts)
+        self.assertIsInstance(results, list)
+        self.assertEqual(len(results), 2)
+        self.assertTrue(all(isinstance(r, dict) for r in results))
+
+    def test_semantic_network_batch_via_extract_network(self):
+        extractor = SemanticNetworkExtractor()
+        texts = ["A knows B.", "C knows D."]
+        entities_batch = [[] for _ in texts]
+        relations_batch = [[] for _ in texts]
+        results = extractor.extract_network(
+            texts,
+            entities=entities_batch,
+            relations=relations_batch,
+        )
+        self.assertIsInstance(results, list)
+        self.assertEqual(len(results), 2)
+
 if __name__ == "__main__":
     unittest.main()

tests/semantic_extract/test_groq_real_world.py

@@ -0,0 +1,378 @@
+
+import unittest
+import time
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+from semantica.semantic_extract.ner_extractor import NERExtractor
+from semantica.semantic_extract.relation_extractor import RelationExtractor
+from semantica.semantic_extract.triplet_extractor import TripletExtractor
+from semantica.semantic_extract.event_detector import EventDetector
+from semantica.semantic_extract.semantic_network_extractor import SemanticNetworkExtractor
+from semantica.semantic_extract.methods import _result_cache
+
+class TestGroqRealWorldPerformance(unittest.TestCase):
+    """
+    Real-world performance test suite using Groq LLM.
+    Tests parallel processing, caching, and correctness.
+    """
+    
+    @classmethod
+    def setUpClass(cls):
+        cls.api_key = os.getenv("GROQ_API_KEY")
+        if not cls.api_key:
+            raise unittest.SkipTest("GROQ_API_KEY is not set")
+        cls.metrics_file = os.path.join(os.getcwd(), "groq_metrics.txt")
+        
+        # Real-world sample texts (mix of tech, business, and general)
+        cls.sample_texts = [
+            """Apple Inc. is planning to launch a new AI-powered iPhone in late 2024. 
+            CEO Tim Cook announced that the device will feature a neural engine capable of 
+            processing 50 trillion operations per second. The company's stock rose 5% following the news.""",
+            
+            """Microsoft Corporation has acquired Activision Blizzard for $68.7 billion. 
+            Satya Nadella, Microsoft's Chairman and CEO, stated that this acquisition will 
+            accelerate growth in Microsoft's gaming business across mobile, PC, console, and cloud.""",
+            
+            """Elon Musk's SpaceX successfully launched the Starship rocket from Boca Chica, Texas. 
+            The mission aims to test new heat shield technology essential for future Mars missions. 
+            NASA Administrator Bill Nelson congratulated the team on the achievement.""",
+            
+            """Google DeepMind introduced Gemini, a new multimodal AI model. 
+            Sundar Pichai emphasized that Gemini represents a significant leap forward in 
+            AI capabilities, outperforming GPT-4 on several benchmarks including MMLU.""",
+            
+            """Amazon Web Services (AWS) announced a partnership with Anthropic to develop 
+            reliable and high-performance foundation models. Amazon is investing up to $4 billion 
+            in the AI safety startup founded by Dario Amodei."""
+        ]
+        
+        # Warm up: Ensure modules are loaded
+        print("\n[Setup] Initializing extractors...")
+        cls.extractor = NERExtractor(
+            method="llm",
+            provider="groq",
+            llm_model="llama-3.3-70b-versatile",
+            api_key=cls.api_key,
+        )
+        cls.relation_extractor = RelationExtractor(
+            method="llm",
+            provider="groq",
+            llm_model="llama-3.3-70b-versatile",
+            api_key=cls.api_key,
+        )
+        cls.triplet_extractor = TripletExtractor(
+            method="llm",
+            provider="groq",
+            llm_model="llama-3.3-70b-versatile",
+            api_key=cls.api_key,
+        )
+        cls.event_detector = EventDetector(
+            method="llm",
+            provider="groq",
+            llm_model="llama-3.3-70b-versatile",
+            api_key=cls.api_key,
+        )
+        cls.network_extractor = SemanticNetworkExtractor(
+            method="llm",
+            provider="groq",
+            llm_model="llama-3.3-70b-versatile",
+            api_key=cls.api_key,
+        )
+        
+    def setUp(self):
+        # Clear cache before specific performance tests to ensure fair comparison
+        # (Unless testing cache specifically)
+        if _result_cache:
+            _result_cache._caches["entities"].clear()
+            _result_cache._caches["relations"].clear()
+            
+    def log_metrics(self, message):
+        print(message)
+        with open(self.metrics_file, "a") as f:
+            f.write(message + "\n")
+            f.flush()
+
+    def test_01_parallel_vs_sequential_performance(self):
+        """Compare sequential vs parallel extraction speed."""
+        try:
+            self.log_metrics("\n" + "="*60)
+            self.log_metrics("TEST 1: Sequential vs Parallel Processing Performance")
+            self.log_metrics("="*60)
+            
+            extractor = NERExtractor(method="llm", provider="groq", api_key=self.api_key, model="llama-3.3-70b-versatile")
+            
+            # 1. Sequential Run (Max workers = 1)
+            self.log_metrics("\nStarting Sequential Extraction (5 documents)...")
+            start_time = time.time()
+            seq_results = extractor.extract(self.sample_texts, max_workers=1)
+            seq_time = time.time() - start_time
+            self.log_metrics(f"Sequential Time: {seq_time:.4f}s")
+            self.log_metrics(f"Average Latency: {seq_time/len(self.sample_texts):.4f}s per doc")
+            
+            # Clear cache to force re-extraction for parallel test
+            _result_cache._caches["entities"].clear()
+            
+            # 2. Parallel Run (Max workers = 5)
+            self.log_metrics("\nStarting Parallel Extraction (5 documents, 5 workers)...")
+            start_time = time.time()
+            par_results = extractor.extract(self.sample_texts, max_workers=5)
+            par_time = time.time() - start_time
+            self.log_metrics(f"Parallel Time:   {par_time:.4f}s")
+            self.log_metrics(f"Average Latency: {par_time/len(self.sample_texts):.4f}s per doc")
+            
+            # Analysis
+            speedup = seq_time / par_time if par_time > 0 else 0
+            self.log_metrics(f"\n>>> Performance Gain: {speedup:.2f}x speedup")
+            self.log_metrics(f">>> Latency Reduction: {(seq_time - par_time):.4f}s total time saved")
+            
+            self.assertLess(par_time, seq_time * 1.35, "Parallel processing should not be significantly slower")
+            self.assertEqual(len(seq_results), len(self.sample_texts))
+            self.assertEqual(len(par_results), len(self.sample_texts))
+        except Exception as e:
+            self.log_metrics(f"ERROR in Test 1: {e}")
+            raise
+
+    def test_02_caching_latency_reduction(self):
+        """Measure latency reduction from caching."""
+        try:
+            self.log_metrics("\n" + "="*60)
+            self.log_metrics("TEST 2: Caching Performance & Latency Reduction")
+            self.log_metrics("="*60)
+            
+            extractor = NERExtractor(method="llm", provider="groq", api_key=self.api_key, model="llama-3.3-70b-versatile")
+            text = [self.sample_texts[0]]
+            
+            # 1. Cold Cache
+            _result_cache._caches["entities"].clear()
+            self.log_metrics("\nCold Cache Request...")
+            start_time = time.time()
+            extractor.extract(text)
+            cold_time = time.time() - start_time
+            self.log_metrics(f"Cold Cache Time: {cold_time:.4f}s")
+            cache_size_after_cold = _result_cache.get_stats()["entities"]["size"]
+            
+            # 2. Warm Cache
+            self.log_metrics("\nWarm Cache Request (Identical Query)...")
+            start_time = time.time()
+            extractor.extract(text)
+            warm_time = time.time() - start_time
+            self.log_metrics(f"Warm Cache Time: {warm_time:.6f}s")
+            cache_size_after_warm = _result_cache.get_stats()["entities"]["size"]
+            
+            # Analysis
+            reduction = (cold_time - warm_time) / cold_time * 100
+            self.log_metrics(f"\n>>> Latency Reduction: {reduction:.2f}%")
+            
+            self.assertLess(warm_time, 1.0, "Warm cache response should be fast (<1.0s)")
+            # self.assertGreater(reduction, 50, "Caching should reduce latency by >50%")
+            if reduction < 30:
+                self.log_metrics(f"WARNING: Caching reduction is low ({reduction:.2f}%)")
+            self.assertGreater(reduction, 20, "Caching should reduce latency by >20%")
+            self.assertGreater(cache_size_after_cold, 0, "Cache should store entity results")
+            self.assertEqual(cache_size_after_warm, cache_size_after_cold, "Warm request should hit the cache")
+        except Exception as e:
+            self.log_metrics(f"ERROR in Test 2: {e}")
+            raise
+
+    def test_03_correctness_and_entity_matching(self):
+        """Verify extraction correctness and data quality."""
+        try:
+            self.log_metrics("\n" + "="*60)
+            self.log_metrics("TEST 3: Extraction Correctness & Data Quality")
+            self.log_metrics("="*60)
+            
+            # Use a specific text with clear entities
+            text = "Satya Nadella is the CEO of Microsoft."
+            
+            extractor = NERExtractor(method="llm", provider="groq", api_key=self.api_key, model="llama-3.3-70b-versatile")
+            entities = extractor.extract([text])[0] # List of lists
+            
+            self.log_metrics(f"\nInput: {text}")
+            self.log_metrics(f"Extracted Entities: {[e.text + '(' + e.label + ')' for e in entities]}")
+            
+            # Validation
+            found_person = any(e.label == "PERSON" and "Satya" in e.text for e in entities)
+            found_org = any(e.label == "ORG" and "Microsoft" in e.text for e in entities)
+            
+            if not found_org:
+                self.log_metrics("FAILURE: Did not find Microsoft as ORG. Found entities:")
+                for e in entities:
+                    self.log_metrics(f" - {e.text}: {e.label}")
+
+            self.assertTrue(found_person, "Failed to extract Satya Nadella as PERSON")
+            self.assertTrue(found_org, "Failed to extract Microsoft as ORG")
+            
+            self.log_metrics("\n>>> Correctness Verification: PASS")
+            self.log_metrics("    - Identified PERSON entity")
+            self.log_metrics("    - Identified ORG entity")
+            self.log_metrics("    - Pydantic models validated successfully")
+        except Exception as e:
+            self.log_metrics(f"ERROR in Test 3: {e}")
+            raise
+
+    def test_4_relation_extraction(self):
+        """Test Relation Extraction capabilities"""
+        print("\n" + "="*60)
+        print("TEST 4: Relation Extraction")
+        print("="*60)
+        
+        text = self.sample_texts[1]  # Microsoft acquisition text
+        print(f"\nInput: {text[:100]}...")
+        
+        # First extract entities
+        entities = self.__class__.extractor.extract_entities(text)
+        self.assertTrue(len(entities) > 0, "Should extract entities first")
+        
+        # Extract relations
+        start_time = time.time()
+        relations = self.__class__.relation_extractor.extract_relations(text, entities)
+        duration = time.time() - start_time
+        
+        print(f"Extracted {len(relations)} relations in {duration:.4f}s")
+        for r in relations:
+            print(f"  - {r.subject.text} -> {r.predicate} -> {r.object.text}")
+            
+        self.assertTrue(len(relations) > 0, "Should extract relations")
+        
+        # Verify specific relation (Microsoft -> acquired -> Activision Blizzard)
+        found_acquisition = False
+        for r in relations:
+            if "Microsoft" in r.subject.text and "Activision" in r.object.text:
+                found_acquisition = True
+                break
+        
+        if not found_acquisition:
+            # Fallback check - sometimes subject/object might be swapped or different wording
+            for r in relations:
+                if "Activision" in r.subject.text and "Microsoft" in r.object.text:
+                    found_acquisition = True
+                    break
+                    
+        self.assertTrue(found_acquisition, "Should find acquisition relation between Microsoft and Activision")
+
+    def test_5_triplet_extraction(self):
+        """Test RDF Triplet Extraction capabilities"""
+        print("\n" + "="*60)
+        print("TEST 5: Triplet Extraction")
+        print("="*60)
+        
+        text = self.sample_texts[0]  # Apple text
+        print(f"\nInput: {text[:100]}...")
+        
+        # Pipeline: Entities -> Relations -> Triplets
+        entities = self.__class__.extractor.extract_entities(text)
+        relations = self.__class__.relation_extractor.extract_relations(text, entities)
+        
+        start_time = time.time()
+        triplets = self.__class__.triplet_extractor.extract_triplets(text, entities, relations)
+        duration = time.time() - start_time
+        
+        print(f"Extracted {len(triplets)} triplets in {duration:.4f}s")
+        for t in triplets:
+            print(f"  - <{t.subject}> <{t.predicate}> <{t.object}>")
+            
+        self.assertTrue(len(triplets) > 0, "Should extract triplets")
+        
+        # Check for Apple related triplet
+        found_apple = False
+        for t in triplets:
+            if "Apple" in t.subject or "Apple" in t.object:
+                found_apple = True
+                break
+        self.assertTrue(found_apple, "Should find Apple-related triplet")
+
+    def test_6_event_detection(self):
+        """Test Event Detection capabilities"""
+        print("\n" + "="*60)
+        print("TEST 6: Event Detection")
+        print("="*60)
+        
+        text = self.sample_texts[2]  # SpaceX launch text
+        print(f"\nInput: {text[:100]}...")
+        
+        start_time = time.time()
+        events = self.__class__.event_detector.detect_events(text)
+        duration = time.time() - start_time
+        
+        print(f"Detected {len(events)} events in {duration:.4f}s")
+        for e in events:
+            print(f"  - [{e.event_type}] {e.text} (Participants: {e.participants})")
+            
+        self.assertTrue(len(events) > 0, "Should detect events")
+        
+        # Verify launch event
+        found_launch = False
+        for e in events:
+            if "launch" in e.event_type.lower() or "launch" in e.text.lower():
+                found_launch = True
+                break
+        self.assertTrue(found_launch, "Should detect launch event")
+
+    def test_7_semantic_network(self):
+        """Test Semantic Network Extraction capabilities"""
+        print("\n" + "="*60)
+        print("TEST 7: Semantic Network Extraction")
+        print("="*60)
+        
+        text = self.sample_texts[3]  # Google DeepMind text
+        print(f"\nInput: {text[:100]}...")
+        
+        # Extract base components first
+        entities = self.__class__.extractor.extract_entities(text)
+        relations = self.__class__.relation_extractor.extract_relations(text, entities)
+        
+        start_time = time.time()
+        network = self.__class__.network_extractor.extract_network(text, entities=entities, relations=relations)
+        duration = time.time() - start_time
+        
+        print(f"Extracted Network in {duration:.4f}s")
+        print(f"  - Nodes: {len(network.nodes)}")
+        print(f"  - Edges: {len(network.edges)}")
+        
+        self.assertTrue(len(network.nodes) > 0, "Should have nodes")
+        self.assertTrue(len(network.edges) > 0, "Should have edges")
+        
+        # Verify Google/DeepMind/Gemini nodes exist
+        node_labels = [n.label for n in network.nodes]
+        print(f"  - Node Labels: {node_labels}")
+        self.assertTrue(any("Gemini" in l for l in node_labels), "Should contain Gemini node")
+
+    def test_8_parallel_event_detection(self):
+        """Test Parallel Event Detection capabilities"""
+        print("\n" + "="*60)
+        print("TEST 8: Parallel Event Detection")
+        print("="*60)
+        
+        # Create a larger batch by duplicating sample texts
+        batch_texts = self.sample_texts * 2  # 10 documents
+        
+        # 1. Sequential Run
+        print("\nStarting Sequential Event Detection (10 documents)...")
+        start_time = time.time()
+        seq_results = self.__class__.event_detector.extract(batch_texts, max_workers=1)
+        seq_time = time.time() - start_time
+        print(f"Sequential Time: {seq_time:.4f}s")
+        
+        # 2. Parallel Run
+        print("\nStarting Parallel Event Detection (10 documents, 5 workers)...")
+        start_time = time.time()
+        par_results = self.__class__.event_detector.extract(batch_texts, max_workers=5)
+        par_time = time.time() - start_time
+        print(f"Parallel Time:   {par_time:.4f}s")
+        
+        # Analysis
+        speedup = seq_time / par_time if par_time > 0 else 0
+        print(f"\n>>> Performance Gain: {speedup:.2f}x speedup")
+        
+        self.assertEqual(len(seq_results), len(batch_texts))
+        self.assertEqual(len(par_results), len(batch_texts))
+        
+        # Verify results match (order should be preserved)
+        for i in range(len(batch_texts)):
+            self.assertEqual(len(seq_results[i]), len(par_results[i]), f"Result count mismatch at index {i}")
+
+if __name__ == "__main__":
+    unittest.main()

tests/semantic_extract/test_llm_smoke_groq.py

@@ -0,0 +1,63 @@
+import os
+
+import pytest
+
+try:
+    from dotenv import load_dotenv
+
+    load_dotenv()
+except Exception:
+    pass
+
+pytest.importorskip("groq")
+
+from semantica.semantic_extract.ner_extractor import NERExtractor
+from semantica.semantic_extract.relation_extractor import RelationExtractor
+from semantica.semantic_extract.triplet_extractor import TripletExtractor
+
+
+def test_groq_llm_smoke_entities_relations_triplets():
+    if not os.getenv("GROQ_API_KEY"):
+        pytest.skip("GROQ_API_KEY is not set")
+
+    text = (
+        "Apple acquired Beats in 2014 for $3 billion. "
+        "Steve Jobs founded Apple. "
+        "Beats is based in California."
+    )
+    model = "llama-3.3-70b-versatile"
+
+    entities = NERExtractor(method="llm").extract(
+        text,
+        provider="groq",
+        model=model,
+        temperature=0.0,
+        max_tokens=250,
+    )
+    assert isinstance(entities, list)
+    assert len(entities) > 0
+    assert len(entities) <= 30
+
+    relations = RelationExtractor(method="llm").extract(
+        text,
+        entities=entities,
+        provider="groq",
+        model=model,
+        temperature=0.0,
+        max_tokens=350,
+        max_entities_prompt=12,
+    )
+    assert isinstance(relations, list)
+    assert len(relations) <= 30
+
+    triplets = TripletExtractor(method="llm").extract(
+        text,
+        entities=entities,
+        relations=relations,
+        provider="groq",
+        model=model,
+        temperature=0.0,
+        max_tokens=350,
+    )
+    assert isinstance(triplets, list)
+    assert len(triplets) <= 40

tests/semantic_extract/test_performance.py

@@ -0,0 +1,286 @@
+import time
+import unittest
+print("Starting tests module...")
+from unittest.mock import MagicMock, patch
+from semantica.semantic_extract.providers import create_provider, ProviderPool, _provider_pool
+from semantica.semantic_extract.ner_extractor import NERExtractor
+from semantica.semantic_extract.relation_extractor import RelationExtractor
+from semantica.semantic_extract.triplet_extractor import TripletExtractor, Triplet
+from semantica.semantic_extract.methods import _result_cache, extract_entities_llm, extract_relations_llm, extract_triplets_llm, match_entity
+from semantica.semantic_extract.ner_extractor import Entity
+
+class TestSemanticExtractImprovements(unittest.TestCase):
+    def setUp(self):
+        _provider_pool.clear()
+        # Clear cache before each test
+        if _result_cache:
+            _result_cache._caches["entities"].clear()
+            _result_cache._caches["relations"].clear()
+            _result_cache._caches["triplets"].clear()
+
+    def test_entity_matching(self):
+        print("\nTesting Entity Matching...")
+        entities = [
+            Entity(text="Apple Inc.", label="ORG", start_char=0, end_char=10, confidence=1.0),
+            Entity(text="Steve Jobs", label="PERSON", start_char=0, end_char=10, confidence=1.0)
+        ]
+        
+        # Exact match
+        m1 = match_entity("Apple Inc.", entities)
+        self.assertIsNotNone(m1)
+        self.assertEqual(m1.text, "Apple Inc.")
+        
+        # Case insensitive
+        m2 = match_entity("apple inc.", entities)
+        self.assertIsNotNone(m2)
+        self.assertEqual(m2.text, "Apple Inc.")
+        
+        # Substring/Partial match (should work via calculate_similarity)
+        # "Apple" is contained in "Apple Inc."
+        # calculate_similarity gives a boost for containment
+        m3 = match_entity("Apple", entities)
+        if m3:
+            self.assertEqual(m3.text, "Apple Inc.")
+            print("  Partial match 'Apple' -> 'Apple Inc.' successful.")
+        else:
+            print("  Partial match 'Apple' -> 'Apple Inc.' failed (score too low).")
+            
+        # No match
+        m4 = match_entity("Microsoft", entities)
+        self.assertIsNone(m4)
+        print("  No match verified.")
+
+        # Synonym match
+        # We need entities that match the synonym keys in methods.py (e.g. "acquired" -> "bought")
+        # Let's create an entity "bought"
+        rel_entities = [Entity(text="bought", label="RELATION", start_char=0, end_char=6, confidence=1.0)]
+        m5 = match_entity("acquired", rel_entities)
+        self.assertIsNotNone(m5)
+        self.assertEqual(m5.text, "bought")
+        print("  Synonym match 'acquired' -> 'bought' verified.")
+        
+        # Empty input
+        m6 = match_entity("", entities)
+        self.assertIsNone(m6)
+        print("  Empty input handled.")
+
+    def test_caching(self):
+        print("\nTesting Caching...")
+        
+        text = "Apple Inc. was founded in 1976."
+        
+        # Mock provider
+        mock_provider = MagicMock()
+        mock_provider.is_available.return_value = True
+        # Setup mock response for entities
+        mock_entities_response = MagicMock()
+        mock_entities_response.entities = [
+            MagicMock(text="Apple Inc.", label="ORG", confidence=0.9),
+            MagicMock(text="1976", label="DATE", confidence=0.9)
+        ]
+        mock_provider.generate_typed.return_value = mock_entities_response
+        
+        with patch('semantica.semantic_extract.methods.create_provider', return_value=mock_provider) as mock_create:
+            # First call - should hit provider
+            print("  First call (cache miss)...")
+            results1 = extract_entities_llm(text, provider="openai", model="gpt-4", api_key="test")
+            self.assertEqual(len(results1), 2)
+            self.assertEqual(mock_provider.generate_typed.call_count, 1)
+            
+            # Check cache state
+            print(f"  Cache size: {len(_result_cache._caches['entities'])}")
+            
+            # Second call - should hit cache
+            print("  Second call (cache hit)...")
+            results2 = extract_entities_llm(text, provider="openai", model="gpt-4", api_key="test")
+            self.assertEqual(len(results2), 2)
+            # Provider should NOT be called again
+            self.assertEqual(mock_provider.generate_typed.call_count, 1)
+            
+            print("  Cache hit verified for entities.")
+
+    def test_secure_caching(self):
+        """Test that sensitive parameters are excluded from cache keys."""
+        print("\nTesting Secure Caching...")
+        
+        text = "Security test."
+        
+        # Mock provider
+        mock_provider = MagicMock()
+        mock_provider.is_available.return_value = True
+        mock_entities_response = MagicMock()
+        mock_entities_response.entities = [MagicMock(text="Test", label="TEST", confidence=1.0)]
+        mock_provider.generate_typed.return_value = mock_entities_response
+        
+        with patch('semantica.semantic_extract.methods.create_provider', return_value=mock_provider):
+            # First call with one API key
+            extract_entities_llm(text, provider="openai", model="gpt-4", api_key="secret_key_1")
+            
+            # Second call with DIFFERENT API key
+            # If secure caching is working, this should be a CACHE HIT because api_key is ignored
+            extract_entities_llm(text, provider="openai", model="gpt-4", api_key="secret_key_2")
+            
+            # Provider should have been called ONLY ONCE
+            self.assertEqual(mock_provider.generate_typed.call_count, 1)
+            print("  Secure caching verified: Changing API key did not trigger new extraction.")
+            
+            # Verify cache content
+            self.assertIn("entities", _result_cache._caches)
+            self.assertTrue(len(_result_cache._caches["entities"]) > 0)
+
+    def test_provider_pool(self):
+        print("\nTesting Provider Pool...")
+        # Create provider twice with same args
+        # We need to mock the actual provider init to avoid API keys requirement if not present
+        with patch('semantica.semantic_extract.providers.OpenAIProvider') as MockProvider:
+            MockProvider.side_effect = lambda *args, **kwargs: MagicMock()
+            
+            p1 = create_provider("openai", api_key="test", model_name="gpt-4")
+            p2 = create_provider("openai", api_key="test", model_name="gpt-4")
+            
+            # Should be same instance
+            self.assertIs(p1, p2)
+            print("  Provider reuse verified.")
+            
+            # Different args
+            p3 = create_provider("openai", api_key="test", model_name="gpt-3.5")
+            self.assertIsNot(p1, p3)
+            print("  Different args create new instance verified.")
+            
+            # Explicitly not using pool
+            p4 = create_provider("openai", use_pool=False, api_key="test", model_name="gpt-4")
+            self.assertIsNot(p1, p4)
+            print("  Opt-out of pool verified.")
+
+    def test_ner_parallel_processing(self):
+        print("\nTesting NER Parallel Processing...")
+        extractor = NERExtractor(method="pattern") # Use pattern which is fast/local
+        
+        # Mock extract_entities to simulate work and track thread execution
+        original_extract = extractor.extract_entities
+        
+        def mock_extract(text, **kwargs):
+            time.sleep(0.1) # Simulate delay
+            return original_extract(text, **kwargs)
+            
+        extractor.extract_entities = mock_extract
+        
+        texts = ["Text 1", "Text 2", "Text 3", "Text 4"]
+        
+        start_time = time.time()
+        results = extractor.extract(texts)
+        end_time = time.time()
+        
+        duration = end_time - start_time
+        print(f"  Parallel NER (default workers) took {duration:.4f}s")
+        
+        self.assertEqual(len(results), 4)
+        
+        # Verify sequential fallback
+        start_time_seq = time.time()
+        extractor.extract(texts, max_workers=1)
+        end_time_seq = time.time()
+        duration_seq = end_time_seq - start_time_seq
+        print(f"  Sequential NER took {duration_seq:.4f}s")
+
+        # Check if parallel was indeed parallel (faster)
+        # With 0.1s sleep * 4 items:
+        # Sequential ~ 0.4s
+        # Parallel (2 workers) ~ 0.2s + overhead
+        self.assertLess(duration, duration_seq * 0.8) 
+        print("  Parallel execution speedup verified.")
+
+    def test_relation_parallel_processing(self):
+        print("\nTesting Relation Parallel Processing...")
+        extractor = RelationExtractor(method="pattern")
+        
+        # Mock extract_relations
+        original_extract = extractor.extract_relations
+        def mock_extract(text, entities, **kwargs):
+            time.sleep(0.1)
+            return original_extract(text, entities, **kwargs)
+        extractor.extract_relations = mock_extract
+        
+        texts = ["Text 1", "Text 2", "Text 3", "Text 4"]
+        entities = [[], [], [], []]
+        
+        start_time = time.time()
+        results = extractor.extract(texts, entities)
+        end_time = time.time()
+        
+        duration = end_time - start_time
+        print(f"  Parallel RE (default workers) took {duration:.4f}s")
+        
+        self.assertEqual(len(results), 4)
+        
+        # Sequential
+        start_time_seq = time.time()
+        extractor.extract(texts, entities, max_workers=1)
+        end_time_seq = time.time()
+        duration_seq = end_time_seq - start_time_seq
+        print(f"  Sequential RE took {duration_seq:.4f}s")
+        
+        self.assertLess(duration, duration_seq * 0.8)
+        print("  Parallel execution speedup verified.")
+
+    def test_relation_extraction_fuzzy_matching(self):
+        print("\nTesting Relation Extraction Fuzzy Matching...")
+        extractor = RelationExtractor(method="pattern")
+        
+        # Entities have formal names
+        entities = [
+            Entity(text="Apple Inc.", label="ORG", start_char=0, end_char=10, confidence=1.0),
+            Entity(text="Steve Jobs", label="PERSON", start_char=21, end_char=31, confidence=1.0)
+        ]
+        
+        # Text uses informal name "Apple"
+        text = "Apple was founded by Steve Jobs."
+        
+        relations = extractor.extract(text, entities)
+        
+        found = False
+        for rel in relations:
+            # Check if subject matches "Apple Inc." even though text said "Apple"
+            if rel.subject.text == "Apple Inc." and rel.object.text == "Steve Jobs" and rel.predicate == "founded_by":
+                found = True
+                print("  Successfully matched 'Apple' -> 'Apple Inc.' in relation extraction.")
+                break
+                
+        self.assertTrue(found, "Failed to extract relation with fuzzy entity matching")
+
+    def test_triplet_parallel_processing(self):
+        print("\nTesting Triplet Parallel Processing...")
+        extractor = TripletExtractor(method="pattern")
+        
+        # Mock extract_triplets
+        original_extract = extractor.extract_triplets
+        def mock_extract(text, **kwargs):
+            time.sleep(0.1)
+            # Return dummy triplets to avoid actual extraction overhead
+            return [Triplet(subject="s", predicate="p", object="o")]
+        extractor.extract_triplets = mock_extract
+        
+        texts = ["Text 1", "Text 2", "Text 3", "Text 4"]
+        
+        start_time = time.time()
+        results = extractor.extract(texts)
+        end_time = time.time()
+        
+        duration = end_time - start_time
+        print(f"  Parallel TE (default workers) took {duration:.4f}s")
+        
+        self.assertEqual(len(results), 4)
+        
+        # Sequential
+        start_time_seq = time.time()
+        extractor.extract(texts, max_workers=1)
+        end_time_seq = time.time()
+        duration_seq = end_time_seq - start_time_seq
+        print(f"  Sequential TE took {duration_seq:.4f}s")
+        
+        self.assertLess(duration, duration_seq * 0.8)
+        print("  Parallel execution speedup verified.")
+
+if __name__ == "__main__":
+    suite = unittest.TestLoader().loadTestsFromTestCase(TestSemanticExtractImprovements)
+    unittest.TextTestRunner(verbosity=2).run(suite)

tests/semantic_extract/test_provider_limits.py

@@ -0,0 +1,157 @@
+
+import unittest
+from unittest.mock import patch, MagicMock
+import sys
+import os
+
+# Ensure we test the local code, not the installed package
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
+
+from semantica.semantic_extract.ner_extractor import Entity
+from semantica.semantic_extract.relation_extractor import Relation
+from semantica.semantic_extract.triplet_extractor import Triplet
+from semantica.semantic_extract.methods import extract_entities_llm
+# We import providers later inside tests to allow patching
+
+class TestSemanticClasses:
+    """Test that core semantic classes do not have hardcoded max lengths."""
+
+    def test_entity_no_max_length(self):
+        long_text = "a" * 10000
+        entity = Entity(text=long_text, label="TEST", start_char=0, end_char=10000)
+        assert entity.text == long_text
+        assert len(entity.text) == 10000
+
+    def test_relation_no_max_length(self):
+        long_text = "a" * 10000
+        e1 = Entity(text="s", label="S", start_char=0, end_char=1)
+        e2 = Entity(text="o", label="O", start_char=0, end_char=1)
+        relation = Relation(subject=e1, predicate=long_text, object=e2)
+        assert relation.predicate == long_text
+
+    def test_triplet_no_max_length(self):
+        long_text = "a" * 10000
+        triplet = Triplet(subject=long_text, predicate="r", object="t")
+        assert triplet.subject == long_text
+
+
+class TestProviderLimits:
+    """Test that providers pass through correct length parameters."""
+
+    def test_openai_max_completion_tokens(self):
+        from semantica.semantic_extract.providers import OpenAIProvider
+        
+        # Patch _init_client to avoid real client creation and import issues
+        with patch.object(OpenAIProvider, '_init_client', return_value=None):
+            provider = OpenAIProvider(api_key="fake")
+            
+            # Manually mock client
+            mock_client = MagicMock()
+            mock_response = MagicMock()
+            mock_response.choices[0].message.content = "result"
+            mock_client.chat.completions.create.return_value = mock_response
+            provider.client = mock_client
+            
+            provider.generate("prompt", max_completion_tokens=12345, top_p=0.9)
+
+            call_kwargs = mock_client.chat.completions.create.call_args[1]
+            assert call_kwargs["max_completion_tokens"] == 12345
+            assert call_kwargs["top_p"] == 0.9
+            assert "max_tokens" not in call_kwargs
+
+    def test_anthropic_max_tokens_defaults(self):
+        from semantica.semantic_extract.providers import AnthropicProvider
+        
+        with patch.object(AnthropicProvider, '_init_client', return_value=None):
+            provider = AnthropicProvider(api_key="fake")
+            
+            mock_client = MagicMock()
+            mock_response = MagicMock()
+            mock_response.content = [MagicMock(text="result")]
+            mock_client.messages.create.return_value = mock_response
+            provider.client = mock_client
+            
+            provider.generate("prompt")
+            
+            # Verify default is 8192 (new limit)
+            call_kwargs = mock_client.messages.create.call_args[1]
+            assert call_kwargs["max_tokens"] == 8192
+
+            # Test override
+            provider.generate("prompt", max_tokens=9999)
+            call_kwargs = mock_client.messages.create.call_args[1]
+            assert call_kwargs["max_tokens"] == 9999
+
+    def test_groq_max_completion_tokens(self):
+        from semantica.semantic_extract.providers import GroqProvider
+        
+        with patch.object(GroqProvider, '_init_client', return_value=None):
+            provider = GroqProvider(api_key="fake")
+            
+            mock_client = MagicMock()
+            mock_response = MagicMock()
+            mock_response.choices[0].message.content = "result"
+            mock_client.chat.completions.create.return_value = mock_response
+            provider.client = mock_client
+            
+            provider.generate("prompt", max_completion_tokens=5000)
+
+            # Verify
+            call_kwargs = mock_client.chat.completions.create.call_args[1]
+            assert call_kwargs["max_completion_tokens"] == 5000
+
+    def test_gemini_params(self):
+        from semantica.semantic_extract.providers import GeminiProvider
+        
+        with patch.object(GeminiProvider, '_init_client', return_value=None):
+            provider = GeminiProvider(api_key="fake")
+            
+            mock_model = MagicMock()
+            mock_response = MagicMock()
+            mock_response.text = "result"
+            mock_model.generate_content.return_value = mock_response
+            provider.client = mock_model
+            
+            provider.generate("prompt", top_k=10, candidate_count=2)
+
+            # Verify
+            call_kwargs = mock_model.generate_content.call_args[1]
+            gen_config = call_kwargs["generation_config"]
+            assert gen_config["top_k"] == 10
+            assert gen_config["candidate_count"] == 2
+
+class TestChunkingDefaults:
+    """Test that chunking defaults have been increased."""
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    @patch("semantica.semantic_extract.methods._extract_entities_chunked")
+    def test_openai_chunking_limit(self, mock_chunked, mock_create_provider):
+        # Setup
+        mock_llm = MagicMock()
+        mock_llm.is_available.return_value = True
+        mock_create_provider.return_value = mock_llm
+        
+        # Text length = 10000 (Greater than old 4000, less than new 64000)
+        long_text = "a" * 10000
+        
+        # Call without explicit max_text_length
+        extract_entities_llm(long_text, provider="openai", api_key="fake")
+        
+        # Should NOT call chunked extraction because default is now 64000
+        mock_chunked.assert_not_called()
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    @patch("semantica.semantic_extract.methods._extract_entities_chunked")
+    def test_groq_chunking_limit(self, mock_chunked, mock_create_provider):
+        # Setup
+        mock_llm = MagicMock()
+        mock_llm.is_available.return_value = True
+        mock_create_provider.return_value = mock_llm
+        
+        # Text length = 10000 (Greater than old 8000, less than new 64000)
+        long_text = "a" * 10000
+        
+        extract_entities_llm(long_text, provider="groq", api_key="fake")
+        
+        # Should NOT call chunked extraction because default is now 64000
+        mock_chunked.assert_not_called()

tests/semantic_extract/test_regression_performance_edge_cases.py

@@ -0,0 +1,123 @@
+import multiprocessing
+import time
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from semantica.semantic_extract.config import resolve_max_workers
+from semantica.semantic_extract.ner_extractor import Entity, NERExtractor
+from semantica.semantic_extract.relation_extractor import RelationExtractor
+from semantica.semantic_extract.triplet_extractor import TripletExtractor
+from semantica.semantic_extract.semantic_network_extractor import SemanticNetworkExtractor
+from semantica.semantic_extract.methods import filter_entities_for_text
+from semantica.semantic_extract.schemas import RelationsResponse, RelationOut
+
+
+def test_resolve_max_workers_defaults_and_clamps():
+    cpu_count = multiprocessing.cpu_count() or 1
+
+    assert resolve_max_workers(explicit=0) == 1
+    assert resolve_max_workers(explicit=-10) == 1
+    assert resolve_max_workers(explicit=1) == 1
+    assert resolve_max_workers(explicit=10**9) == min(cpu_count, 32)
+
+    assert resolve_max_workers(explicit=None, methods=["ml"]) == 1
+
+
+def test_filter_entities_for_text_keeps_short_tokens():
+    text = "US AI lab in NY"
+    entities = [
+        Entity(text="US", label="GPE", start_char=0, end_char=2, confidence=1.0),
+        Entity(text="AI", label="TECH", start_char=3, end_char=5, confidence=1.0),
+        Entity(text="NY", label="GPE", start_char=13, end_char=15, confidence=1.0),
+    ]
+    kept = filter_entities_for_text(text, entities, max_keep=2)
+    kept_texts = {e.text for e in kept}
+    assert "US" in kept_texts or "AI" in kept_texts or "NY" in kept_texts
+
+
+def test_pattern_batch_defaults_to_single_worker_low_latency():
+    extractor = NERExtractor(method="pattern")
+    texts = [f"Text {i}" for i in range(8)]
+    extractor.extract(texts)
+
+
+def test_relation_llm_prompt_filter_does_not_break_mapping():
+    entities = [Entity(text=f"VeryLongEntityName{i}", label="ORG", start_char=0, end_char=1, confidence=1.0) for i in range(120)]
+    ghost = Entity(text="Ghost", label="ORG", start_char=0, end_char=1, confidence=1.0)
+    entities.append(ghost)
+
+    captured = {}
+
+    class FakeLLM:
+        def is_available(self):
+            return True
+
+        def generate_typed(self, prompt, schema, **kwargs):
+            captured["prompt"] = prompt
+            return RelationsResponse(
+                relations=[
+                    RelationOut(subject="Ghost", predicate="related_to", object="VeryLongEntityName0", confidence=0.9)
+                ]
+            )
+
+    with patch("semantica.semantic_extract.methods.create_provider", return_value=FakeLLM()):
+        from semantica.semantic_extract.methods import extract_relations_llm
+
+        relations = extract_relations_llm(
+            "Short text mentioning VeryLongEntityName0 only.",
+            entities=entities,
+            provider="openai",
+            model="gpt-4",
+            max_entities_prompt=20,
+        )
+
+    assert "Ghost" not in captured["prompt"]
+    assert len(relations) == 1
+    assert relations[0].subject.text == "Ghost"
+
+
+def test_triplet_extractor_reuses_sub_extractors():
+    ner_instance = MagicMock()
+    ner_instance.extract_entities.return_value = [
+        Entity(text="A", label="PERSON", start_char=0, end_char=1, confidence=1.0)
+    ]
+
+    rel_instance = MagicMock()
+    rel_instance.extract_relations.return_value = []
+
+    ner_ctor = MagicMock(return_value=ner_instance)
+    rel_ctor = MagicMock(return_value=rel_instance)
+
+    with patch("semantica.semantic_extract.ner_extractor.NERExtractor", ner_ctor), patch(
+        "semantica.semantic_extract.relation_extractor.RelationExtractor", rel_ctor
+    ), patch("semantica.semantic_extract.methods.get_triplet_method", return_value=lambda *args, **kwargs: []):
+        extractor = TripletExtractor(method="pattern")
+        extractor.extract_triplets("A text.")
+        extractor.extract_triplets("A text again.")
+
+    assert ner_ctor.call_count == 1
+    assert rel_ctor.call_count == 1
+
+
+def test_semantic_network_extractor_reuses_sub_extractors():
+    ner_instance = MagicMock()
+    ner_instance.extract_entities.return_value = [
+        Entity(text="A", label="PERSON", start_char=0, end_char=1, confidence=1.0)
+    ]
+
+    rel_instance = MagicMock()
+    rel_instance.extract_relations.return_value = []
+
+    ner_ctor = MagicMock(return_value=ner_instance)
+    rel_ctor = MagicMock(return_value=rel_instance)
+
+    with patch("semantica.semantic_extract.ner_extractor.NERExtractor", ner_ctor), patch(
+        "semantica.semantic_extract.relation_extractor.RelationExtractor", rel_ctor
+    ):
+        extractor = SemanticNetworkExtractor(method="pattern")
+        extractor.extract_network("A text.")
+        extractor.extract_network("A text again.")
+
+    assert ner_ctor.call_count == 1
+    assert rel_ctor.call_count == 1

tests/semantic_extract/test_retry_logic.py

@@ -0,0 +1,264 @@
+
+import unittest
+from unittest.mock import MagicMock, patch, ANY
+import sys
+import os
+from typing import List, Optional
+from pydantic import BaseModel
+import importlib.util
+
+# Add project root to path
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
+
+# Mock external dependencies with __spec__ for importlib checks
+mock_spacy = MagicMock()
+mock_spacy.__spec__ = MagicMock()
+sys.modules["spacy"] = mock_spacy
+
+sys.modules["instructor"] = MagicMock()
+sys.modules["groq"] = MagicMock()
+
+# Better mock for openai
+mock_openai = MagicMock()
+mock_openai.__spec__ = MagicMock()
+sys.modules["openai"] = mock_openai
+
+# Mock sentence_transformers and transformers to avoid heavy imports and dependency checks
+sys.modules["sentence_transformers"] = MagicMock()
+mock_transformers = MagicMock()
+mock_transformers.__spec__ = MagicMock()
+sys.modules["transformers"] = mock_transformers
+
+from semantica.semantic_extract import NERExtractor
+from semantica.semantic_extract.methods import extract_entities_llm, _extract_entities_chunked, extract_relations_llm, extract_triplets_llm
+from semantica.semantic_extract.providers import BaseProvider
+
+class EntitiesResponse(BaseModel):
+    entities: List[dict]
+
+class TestRetryLogic(unittest.TestCase):
+    
+    def setUp(self):
+        self.mock_provider = MagicMock()
+        self.mock_provider.is_available.return_value = True
+        self.mock_provider.generate_typed.return_value = MagicMock(entities=[])
+        
+    def test_ner_extractor_init_default(self):
+        """Test default max_retries in NERExtractor"""
+        ner = NERExtractor(method="llm", provider="test")
+        # Check internal config, max_retries not in config means default behavior downstream
+        self.assertIsNone(ner.config.get("max_retries"))
+
+    def test_ner_extractor_init_custom(self):
+        """Test custom max_retries in NERExtractor init"""
+        ner = NERExtractor(method="llm", provider="test", max_retries=5)
+        self.assertEqual(ner.config.get("max_retries"), 5)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_extract_entities_uses_init_value(self, mock_create_provider):
+        """Test extract_entities uses initialized max_retries"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        ner = NERExtractor(method="llm", provider="test", max_retries=5)
+        ner.extract_entities("test text")
+        
+        # Verify generate_typed called with max_retries=5
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 5)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_extract_entities_override(self, mock_create_provider):
+        """Test extract_entities override max_retries"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        ner = NERExtractor(method="llm", provider="test", max_retries=5)
+        # Override with 1
+        ner.extract_entities("test text", max_retries=1)
+        
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 1)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_chunked_extraction_propagation(self, mock_create_provider):
+        """Test max_retries propagation in chunked extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        # Patch TextSplitter where it lives
+        with patch('semantica.split.TextSplitter') as MockSplitter:
+            mock_splitter_instance = MockSplitter.return_value
+            # Mock split to return 2 chunks
+            mock_chunk1 = MagicMock()
+            mock_chunk1.text = "chunk1"
+            mock_chunk2 = MagicMock()
+            mock_chunk2.text = "chunk2"
+            mock_splitter_instance.split.return_value = [mock_chunk1, mock_chunk2]
+            
+            # Force chunking by setting max_text_length small
+            extract_entities_llm(
+                "very long text",
+                provider="test",
+                model="test-model",
+                max_text_length=10, # Force chunking
+                max_retries=7,
+                structured_output_mode="typed"
+            )
+            
+            # Check if generate_typed was called with max_retries=7 for chunks
+            # It should be called twice (once for each chunk)
+            self.assertEqual(self.mock_provider.generate_typed.call_count, 2)
+            
+            # Check arguments of the calls
+            call_args_list = self.mock_provider.generate_typed.call_args_list
+            for args, kwargs in call_args_list:
+                self.assertEqual(kwargs.get("max_retries"), 7)
+
+    def test_provider_base_logic(self):
+        """Test BaseProvider logic for max_retries with manual loop"""
+        provider = BaseProvider()
+        provider.client = MagicMock()
+        provider.logger = MagicMock()
+        provider.generate_structured = MagicMock(side_effect=Exception("Fail"))
+        
+        # Mock instructor failing
+        with patch('semantica.semantic_extract.providers.instructor') as mock_instructor:
+             # Make instructor client fail
+            mock_client = MagicMock()
+            mock_client.chat.completions.create.side_effect = Exception("Instructor Fail")
+            mock_instructor.from_provider.return_value = mock_client
+            mock_instructor.from_openai.return_value = mock_client
+            
+            # Run with max_retries=2
+            try:
+                provider.generate_typed("prompt", EntitiesResponse, max_retries=2)
+            except Exception:
+                pass
+            
+            # Should try manual generation exactly 2 times
+            self.assertEqual(provider.generate_structured.call_count, 2)
+
+    def test_provider_zero_retries(self):
+        """Test BaseProvider with max_retries=0"""
+        provider = BaseProvider()
+        provider.client = MagicMock()
+        provider.logger = MagicMock()
+        provider.generate_structured = MagicMock(side_effect=Exception("Fail"))
+        
+        # Mock instructor failing
+        with patch('semantica.semantic_extract.providers.instructor') as mock_instructor:
+            mock_client = MagicMock()
+            mock_client.chat.completions.create.side_effect = Exception("Instructor Fail")
+            mock_instructor.from_provider.return_value = mock_client
+            mock_instructor.from_openai.return_value = mock_client
+            
+            try:
+                provider.generate_typed("prompt", EntitiesResponse, max_retries=0)
+            except Exception:
+                pass
+            
+            # Should NOT try manual generation loop (range(0) is empty)
+            self.assertEqual(provider.generate_structured.call_count, 0)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_relations_retry_propagation(self, mock_create_provider):
+        """Test max_retries propagation in relation extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        # Create a mock entity
+        mock_entity = MagicMock()
+        mock_entity.text = "entity"
+        mock_entity.start_char = 0
+        mock_entity.end_char = 5
+        
+        extract_relations_llm(
+            "test text",
+            entities=[mock_entity],
+            provider="test",
+            max_retries=4
+        )
+        
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 4)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_relations_chunked_propagation(self, mock_create_provider):
+        """Test max_retries propagation in chunked relation extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        with patch('semantica.split.TextSplitter') as MockSplitter:
+            mock_splitter_instance = MockSplitter.return_value
+            mock_chunk1 = MagicMock()
+            mock_chunk1.text = "chunk1"
+            mock_chunk1.start_index = 0
+            mock_chunk1.end_index = 6
+            mock_splitter_instance.split.return_value = [mock_chunk1]
+            
+            # Create a mock entity
+            mock_entity = MagicMock()
+            mock_entity.text = "entity"
+            mock_entity.start_char = 0
+            mock_entity.end_char = 5
+            
+            extract_relations_llm(
+                "very long text",
+                entities=[mock_entity],
+                provider="test",
+                max_text_length=10,
+                max_retries=6
+            )
+            
+            # Check call count - should be called for the chunk
+            # Note: _extract_relations_chunked creates a new future for each chunk
+            # which calls extract_relations_llm, which calls generate_typed
+            self.assertEqual(self.mock_provider.generate_typed.call_count, 1)
+            
+            args, kwargs = self.mock_provider.generate_typed.call_args
+            self.assertEqual(kwargs.get("max_retries"), 6)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_triplets_retry_propagation(self, mock_create_provider):
+        """Test max_retries propagation in triplet extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        extract_triplets_llm(
+            "test text",
+            entities=[],
+            relations=[],
+            provider="test",
+            max_retries=7
+        )
+        
+        args, kwargs = self.mock_provider.generate_typed.call_args
+        self.assertEqual(kwargs.get("max_retries"), 7)
+
+    @patch('semantica.semantic_extract.methods.create_provider')
+    def test_triplets_chunked_propagation(self, mock_create_provider):
+        """Test max_retries propagation in chunked triplet extraction"""
+        mock_create_provider.return_value = self.mock_provider
+        
+        with patch('semantica.split.TextSplitter') as MockSplitter:
+            mock_splitter_instance = MockSplitter.return_value
+            mock_chunk1 = MagicMock()
+            mock_chunk1.text = "chunk1"
+            mock_chunk1.start_index = 0
+            mock_chunk1.end_index = 6
+            mock_splitter_instance.split.return_value = [mock_chunk1]
+            
+            # Use max_text_length > 100 to pass the minimum viable chunk size check
+            # and make text longer than that
+            extract_triplets_llm(
+                "very long text " * 20, # length > 101
+                entities=[],
+                relations=[],
+                provider="test",
+                max_text_length=101,
+                max_retries=8
+            )
+            
+            # Check call count
+            self.assertEqual(self.mock_provider.generate_typed.call_count, 1)
+            
+            args, kwargs = self.mock_provider.generate_typed.call_args
+            self.assertEqual(kwargs.get("max_retries"), 8)
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_extractors_dispatch.py

@@ -0,0 +1,168 @@
+import unittest
+from unittest.mock import MagicMock, patch
+import sys
+import os
+
+# Add project root to path
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
+
+# Mock dependencies to avoid import hangs and external calls
+sys.modules['spacy'] = MagicMock()
+sys.modules['semantica.semantic_extract.methods'] = MagicMock()
+sys.modules['semantica.utils.logging'] = MagicMock()
+sys.modules['semantica.utils.progress_tracker'] = MagicMock()
+sys.modules['semantica.semantic_extract.providers'] = MagicMock()
+
+# Mock get_logger and get_progress_tracker
+mock_logger = MagicMock()
+sys.modules['semantica.utils.logging'].get_logger.return_value = mock_logger
+
+mock_tracker = MagicMock()
+sys.modules['semantica.utils.progress_tracker'].get_progress_tracker.return_value = mock_tracker
+
+# Mock the methods module functions specifically
+mock_methods = sys.modules['semantica.semantic_extract.methods']
+mock_methods.get_entity_method = MagicMock()
+mock_methods.get_relation_method = MagicMock()
+mock_methods.get_triplet_method = MagicMock()
+
+# Mock specific extraction functions
+mock_extract_entities_hf = MagicMock()
+mock_extract_relations_hf = MagicMock()
+mock_extract_triplets_hf = MagicMock()
+
+# Setup the registry mocks to return our mock functions
+mock_methods.get_entity_method.return_value = mock_extract_entities_hf
+mock_methods.get_relation_method.return_value = mock_extract_relations_hf
+mock_methods.get_triplet_method.return_value = mock_extract_triplets_hf
+
+# Now import the classes under test
+# We need to patch where they import 'methods' locally if they do
+with patch.dict(sys.modules):
+    from semantica.semantic_extract.ner_extractor import NERExtractor
+    from semantica.semantic_extract.relation_extractor import RelationExtractor
+    from semantica.semantic_extract.triplet_extractor import TripletExtractor
+    from semantica.semantic_extract.ner_extractor import Entity
+    from semantica.semantic_extract.relation_extractor import Relation
+
+class TestExtractorsDispatch(unittest.TestCase):
+    def setUp(self):
+        self.mock_extract_entities_hf = mock_extract_entities_hf
+        self.mock_extract_relations_hf = mock_extract_relations_hf
+        self.mock_extract_triplets_hf = mock_extract_triplets_hf
+        
+        self.mock_extract_entities_hf.reset_mock()
+        self.mock_extract_relations_hf.reset_mock()
+        self.mock_extract_triplets_hf.reset_mock()
+        
+        # Configure mocks to return something iterable/valid
+        self.mock_extract_entities_hf.return_value = [MagicMock(spec=Entity, confidence=0.9, text="Test Entity")]
+        self.mock_extract_relations_hf.return_value = [MagicMock(spec=Relation, confidence=0.9)]
+        self.mock_extract_triplets_hf.return_value = [MagicMock(confidence=0.9)]
+
+    def test_ner_extractor_huggingface_dispatch(self):
+        print("\nTesting NERExtractor dispatch to HuggingFace...")
+        # Initialize with HuggingFace method
+        extractor = NERExtractor(method="huggingface")
+        
+        # Call extract_entities
+        text = "Steve Jobs founded Apple."
+        # Use a specific model via kwargs
+        extractor.extract_entities(text, model="my-custom-ner-model")
+        
+        # Verify get_entity_method was called with "huggingface"
+        mock_methods.get_entity_method.assert_called_with("huggingface")
+        
+        # Verify the extraction function was called with correct model
+        # We need to check the call args to see if 'model' was passed correctly
+        # The logic we implemented: method_options["model"] = all_options.get("huggingface_model") or all_options.get("model") or self.huggingface_model
+        
+        call_args = self.mock_extract_entities_hf.call_args
+        self.assertIsNotNone(call_args, "extract_entities_huggingface should have been called")
+        
+        _, kwargs = call_args
+        self.assertEqual(kwargs.get("model"), "my-custom-ner-model", "Should use model passed in kwargs")
+        
+        print("NERExtractor dispatch verified.")
+
+    def test_relation_extractor_huggingface_dispatch(self):
+        print("\nTesting RelationExtractor dispatch to HuggingFace...")
+        extractor = RelationExtractor(method="huggingface")
+        
+        text = "Steve Jobs founded Apple."
+        entities = [MagicMock(spec=Entity)]
+        
+        # Call extract_relations with explicit model
+        extractor.extract_relations(text, entities, model="my-relation-model")
+        
+        # Verify dispatch
+        mock_methods.get_relation_method.assert_called_with("huggingface")
+        
+        call_args = self.mock_extract_relations_hf.call_args
+        self.assertIsNotNone(call_args, "extract_relations_huggingface should have been called")
+        
+        _, kwargs = call_args
+        self.assertEqual(kwargs.get("model"), "my-relation-model", "Should use model passed in kwargs")
+        
+        print("RelationExtractor dispatch verified.")
+
+    def test_triplet_extractor_huggingface_dispatch(self):
+        print("\nTesting TripletExtractor dispatch to HuggingFace...")
+        extractor = TripletExtractor(method="huggingface")
+        
+        text = "Steve Jobs founded Apple."
+        
+        # Call extract_triplets with explicit model
+        extractor.extract_triplets(text, model="my-triplet-model")
+        
+        # Verify dispatch
+        mock_methods.get_triplet_method.assert_called_with("huggingface")
+        
+        call_args = self.mock_extract_triplets_hf.call_args
+        self.assertIsNotNone(call_args, "extract_triplets_huggingface should have been called")
+        
+        _, kwargs = call_args
+        self.assertEqual(kwargs.get("model"), "my-triplet-model", "Should use model passed in kwargs")
+        
+        print("TripletExtractor dispatch verified.")
+
+    def test_ner_extractor_huggingface_fallback(self):
+        print("\nTesting NERExtractor fallback logic...")
+        # Init with huggingface_model in config
+        extractor = NERExtractor(method="huggingface", huggingface_model="config-model")
+        
+        extractor.extract_entities("text")
+        
+        _, kwargs = self.mock_extract_entities_hf.call_args
+        self.assertEqual(kwargs.get("model"), "config-model", "Should prioritize huggingface_model from config")
+        
+        # Now override with kwargs model
+        extractor.extract_entities("text", model="kwargs-model")
+        _, kwargs = self.mock_extract_entities_hf.call_args
+        self.assertEqual(kwargs.get("model"), "kwargs-model", "Should allow overriding config huggingface_model via model kwarg")
+        
+        # Let's test passing 'huggingface_model' in kwargs
+        extractor.extract_entities("text", huggingface_model="override-model")
+        _, kwargs = self.mock_extract_entities_hf.call_args
+        self.assertEqual(kwargs.get("model"), "override-model", "Should allow overriding huggingface_model via kwargs")
+
+    def test_triplet_extractor_lazy_loading(self):
+        print("\nTesting TripletExtractor lazy loading for HuggingFace...")
+        # Initialize with HuggingFace method
+        extractor = TripletExtractor(method="huggingface")
+        
+        # Check initial state
+        self.assertIsNone(extractor._ner_extractor)
+        self.assertIsNone(extractor._relation_extractor)
+        
+        # Run extraction
+        extractor.extract_triplets("Steve Jobs founded Apple.")
+        
+        # Check state AFTER extraction - should STILL be None because huggingface (REBEL) doesn't need them
+        self.assertIsNone(extractor._ner_extractor, "NERExtractor should not be initialized for HuggingFace method")
+        self.assertIsNone(extractor._relation_extractor, "RelationExtractor should not be initialized for HuggingFace method")
+        
+        print("TripletExtractor lazy loading verified.")
+
+if __name__ == '__main__':
+    unittest.main()

tests/test_hf_deep_verify.py

@@ -0,0 +1,167 @@
+import unittest
+from unittest.mock import MagicMock, patch
+import sys
+import os
+
+# Ensure project root is in path
+sys.path.append(os.getcwd())
+
+from semantica.semantic_extract.ner_extractor import NERExtractor, Entity
+from semantica.semantic_extract.relation_extractor import RelationExtractor
+from semantica.semantic_extract.triplet_extractor import TripletExtractor
+from semantica.utils.exceptions import ProcessingError
+
+class TestHuggingFaceDeepIntegration(unittest.TestCase):
+    """
+    Comprehensive test suite for Hugging Face models integration
+    in NER, Relation, and Triplet extraction modules.
+    """
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_ner_extraction_flow(self, MockLoaderClass):
+        """Test NER extraction with detailed IOB parsing and aggregation."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Simulate IOB output (Raw token classification)
+        mock_loader.extract_entities.return_value = [
+            {"entity": "B-PER", "score": 0.99, "index": 1, "word": "John", "start": 0, "end": 4, "label": "B-PER"},
+            {"entity": "I-PER", "score": 0.98, "index": 2, "word": "Doe", "start": 5, "end": 8, "label": "I-PER"},
+            {"entity": "O", "score": 0.99, "index": 3, "word": "lives", "start": 9, "end": 14, "label": "O"},
+            {"entity": "B-LOC", "score": 0.95, "index": 4, "word": "New", "start": 18, "end": 21, "label": "B-LOC"},
+            {"entity": "I-LOC", "score": 0.96, "index": 5, "word": "York", "start": 22, "end": 26, "label": "I-LOC"},
+        ]
+        
+        extractor = NERExtractor(method="huggingface", huggingface_model="dslim/bert-base-NER")
+        entities = extractor.extract_entities("John Doe lives in New York")
+        
+        # Verify aggregation worked (John Doe should be one entity)
+        # Note: The logic in extract_entities_huggingface handles manual aggregation
+        # if "entity_group" is missing and labels start with B-/I-
+        
+        # Let's debug what we expect. 
+        # "John" (B-PER) -> current_entity="John"
+        # "Doe" (I-PER) -> match! -> current_entity="John Doe"
+        # "lives" (O) -> append John Doe, current=None
+        # "New" (B-LOC) -> current="New"
+        # "York" (I-LOC) -> match! -> current="New York"
+        # End -> append New York
+        
+        self.assertEqual(len(entities), 2)
+        
+        person = next((e for e in entities if e.label == "PER"), None)
+        self.assertIsNotNone(person)
+        self.assertEqual(person.text, "John Doe")
+        
+        loc = next((e for e in entities if e.label == "LOC"), None)
+        self.assertIsNotNone(loc)
+        self.assertEqual(loc.text, "New York")
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_ner_aggregation_strategy_simple(self, MockLoaderClass):
+        """Test NER extraction when the pipeline handles aggregation (strategy='simple')."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Simulate Aggregated output
+        mock_loader.extract_entities.return_value = [
+            {"entity_group": "PER", "score": 0.99, "word": "John Doe", "start": 0, "end": 8},
+            {"entity_group": "LOC", "score": 0.95, "word": "New York", "start": 18, "end": 26},
+        ]
+        
+        extractor = NERExtractor(
+            method="huggingface", 
+            huggingface_model="dslim/bert-base-NER",
+            aggregation_strategy="simple" # Explicitly requesting simple
+        )
+        entities = extractor.extract_entities("John Doe lives in New York")
+        
+        self.assertEqual(len(entities), 2)
+        self.assertEqual(entities[0].text, "John Doe")
+        self.assertEqual(entities[0].label, "PER")
+        
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_relation_extraction_flow(self, MockLoaderClass):
+        """Test Relation extraction with Hugging Face model."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Mock extract_relations output
+        mock_loader.extract_relations.return_value = [{
+            "subject": Entity(text="Apple", label="ORG", start_char=0, end_char=5),
+            "object": Entity(text="Steve Jobs", label="PERSON", start_char=21, end_char=31),
+            "relation": "founded_by",
+            "score": 0.9
+        }]
+        
+        # We need to provide entities for relation extraction usually
+        entities = [
+            Entity(text="Apple", label="ORG", start_char=0, end_char=5),
+            Entity(text="Steve Jobs", label="PERSON", start_char=21, end_char=31)
+        ]
+        
+        extractor = RelationExtractor(method="huggingface", huggingface_model="facebook/bart-large-mnli")
+        relations = extractor.extract_relations("Apple was founded by Steve Jobs", entities=entities)
+        
+        # Check if relation is found
+        self.assertEqual(len(relations), 1)
+        self.assertEqual(relations[0].predicate, "founded_by")
+        self.assertEqual(relations[0].subject.text, "Apple")
+        self.assertEqual(relations[0].object.text, "Steve Jobs")
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_triplet_extraction_rebel(self, MockLoaderClass):
+        """Test Triplet extraction using REBEL parsing logic."""
+        mock_loader = MockLoaderClass.return_value
+        
+        # Mock extract_triplets output
+        # The extract_triplets method in Loader returns [{"triplet": decoded_text}]
+        # But wait, methods.py extract_triplets_huggingface handles parsing?
+        # No, let's check methods.py again.
+        
+        # Actually, methods.py for triplets calls loader.extract_triplets and then parses the result?
+        # Or does loader.extract_triplets return the raw generation?
+        # Let's check the code I read earlier.
+        # loader.extract_triplets returns [{"triplet": decoded}]
+        
+        # But methods.py `extract_triplets_huggingface` logic needs to be verified.
+        # I didn't read extract_triplets_huggingface in methods.py yet (I read entities).
+        # Assuming standard behavior, let's return what loader returns.
+        
+        mock_loader.extract_triplets.return_value = [{"triplet": "<triplet> Apple <subj> founded by <obj> Steve Jobs"}]
+        
+        # Wait, if methods.py expects raw text and parses it, then I need to know IF methods.py does the parsing or if it expects pre-parsed.
+        # Usually, if it's REBEL, the parsing happens after generation.
+        # Let's assume methods.py parses the REBEL format.
+        
+        extractor = TripletExtractor(method="huggingface", huggingface_model="Babelscape/rebel-large")
+        
+        # If the extractor relies on methods.py to parse, and methods.py relies on REBEL format:
+        triplets = extractor.extract_triplets("Apple was founded by Steve Jobs")
+        
+        # Note: If this fails, it might be because I need to check how extract_triplets_huggingface is implemented.
+        # But let's try.
+        if not triplets:
+             # Fallback: maybe methods.py expects the model to return parsed triplets?
+             pass
+             
+        self.assertTrue(len(triplets) > 0)
+        self.assertEqual(triplets[0].subject, "Apple")
+        self.assertEqual(triplets[0].object, "Steve Jobs")
+        self.assertEqual(triplets[0].predicate, "founded by")
+
+    @patch('semantica.semantic_extract.methods.HuggingFaceModelLoader')
+    def test_byom_override(self, MockLoaderClass):
+        """Verify Bring Your Own Model (runtime override) works for all extractors."""
+        mock_loader = MockLoaderClass.return_value
+        mock_loader.extract_entities.return_value = []
+        
+        # NER
+        ner = NERExtractor(method="huggingface", huggingface_model="default-ner")
+        ner.extract_entities("test", huggingface_model="runtime-ner")
+        
+        # Check if load_ner_model was called with runtime model
+        # mock_loader.load_ner_model.assert_called_with("runtime-ner", ...)
+        # args[0] should be "runtime-ner"
+        call_args = mock_loader.load_ner_model.call_args
+        self.assertEqual(call_args[0][0], "runtime-ner")
+            
+if __name__ == "__main__":
+    unittest.main()

tests/test_huggingface_impl.py

@@ -0,0 +1,178 @@
+
+import sys
+import os
+import traceback
+from unittest.mock import MagicMock
+
+print("Starting test script...", flush=True)
+
+# Mock transformers and torch BEFORE any project imports
+try:
+    mock_transformers = MagicMock()
+    mock_pipeline = MagicMock()
+    mock_transformers.pipeline = mock_pipeline
+    sys.modules["transformers"] = mock_transformers
+    sys.modules["torch"] = MagicMock()
+    sys.modules["torch"].cuda.is_available.return_value = False
+    
+    # Mock spacy
+    mock_spacy = MagicMock()
+    sys.modules["spacy"] = mock_spacy
+    
+    # Mock instructor
+    sys.modules["instructor"] = MagicMock()
+    
+    # Also mock semantica.semantic_extract.config to avoid initialization issues
+    mock_config_module = MagicMock()
+    mock_config_instance = MagicMock()
+    # Setup default return values for config
+    mock_config_instance.get.return_value = {}
+    mock_config_instance.get_optimization_config.return_value = {"enable_cache": False}
+    
+    mock_config_module.config = mock_config_instance
+    mock_config_module.Config = MagicMock(return_value=mock_config_instance)
+    sys.modules["semantica.semantic_extract.config"] = mock_config_module
+    
+    print("Mocks setup complete.", flush=True)
+except Exception as e:
+    print(f"Error setting up mocks: {e}", flush=True)
+    sys.exit(1)
+
+# Add project root
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
+print(f"Added to path: {sys.path[0]}", flush=True)
+
+try:
+    print("Importing methods...", flush=True)
+    from semantica.semantic_extract.methods import extract_entities_huggingface, extract_relations_huggingface, extract_triplets_huggingface
+    print("Importing Entity class...", flush=True)
+    from semantica.semantic_extract.ner_extractor import Entity
+    print("Imports successful.", flush=True)
+except Exception as e:
+    print(f"Import failed: {e}", flush=True)
+    traceback.print_exc()
+    sys.exit(1)
+
+def test_enhanced_impl():
+    print("Testing enhanced implementation...", flush=True)
+
+    try:
+        # 1. Test NER with aggregation strategy
+        print("\n--- Testing NER ---", flush=True)
+        
+        # Setup mock pipeline return value
+        mock_ner_pipeline = MagicMock()
+        mock_ner_pipeline.return_value = [
+            {"entity_group": "PERSON", "score": 0.99, "word": "Elon Musk", "start": 0, "end": 9},
+        ]
+        
+        # Configure pipeline side effect
+        def pipeline_side_effect(task, **kwargs):
+            if task == "ner": return mock_ner_pipeline
+            return MagicMock()
+            
+        mock_pipeline.side_effect = pipeline_side_effect
+        
+        # Test calling with aggregation_strategy
+        entities = extract_entities_huggingface(
+            "Elon Musk founded SpaceX.", 
+            model="dslim/bert-base-NER", 
+            aggregation_strategy="max"
+        )
+        print(f"Entities: {entities}", flush=True)
+        
+        # Verify aggregation_strategy was passed
+        mock_pipeline.assert_any_call(
+            "ner", 
+            model="dslim/bert-base-NER", 
+            device=-1, 
+            aggregation_strategy="max",
+            tokenizer=None
+        )
+
+        # 2. Test Relations with Input Formatting
+        print("\n--- Testing Relations ---", flush=True)
+        e1 = Entity(text="Elon Musk", label="PERSON", start_char=0, end_char=9)
+        e2 = Entity(text="SpaceX", label="ORG", start_char=18, end_char=24)
+        
+        mock_rel_pipeline = MagicMock()
+        mock_rel_pipeline.return_value = [{"label": "founded", "score": 0.9}]
+        
+        # Update pipeline mock to return rel pipeline
+        def pipeline_side_effect_rel(task, **kwargs):
+            if task == "ner": return mock_ner_pipeline
+            if task == "text-classification": return mock_rel_pipeline
+            return MagicMock()
+            
+        mock_pipeline.side_effect = pipeline_side_effect_rel
+        
+        relations = extract_relations_huggingface(
+            "Elon Musk founded SpaceX.", 
+            entities=[e1, e2], 
+            model="some-relation-model"
+        )
+        print(f"Relations: {relations}", flush=True)
+        
+        # Verify input formatting
+        # Check if ANY call contained the correct formatting
+        found_match = False
+        for call in mock_rel_pipeline.call_args_list:
+            args, _ = call
+            if "<subj> Elon Musk </subj>" in args[0] and "<obj> SpaceX </obj>" in args[0]:
+                found_match = True
+                break
+        
+        if not found_match:
+            print("Failed to find expected call args in:", flush=True)
+            for call in mock_rel_pipeline.call_args_list:
+                print(f"  {call[0]}", flush=True)
+                
+        assert found_match, "Did not find relation call with Elon Musk as subject"
+
+        # 3. Test Triplets with REBEL parsing
+        print("\n--- Testing Triplets ---", flush=True)
+        
+        # Mock Tokenizer and Model
+        mock_tokenizer_instance = MagicMock()
+        mock_transformers.AutoTokenizer.from_pretrained.return_value = mock_tokenizer_instance
+        mock_tokenizer_instance.encode.return_value = MagicMock()
+        # Mock decode to return REBEL format
+        mock_tokenizer_instance.decode.return_value = "<s><triplet> Elon Musk <subj> founded <obj> SpaceX <triplet> SpaceX <subj> created <obj> Starship</s>"
+        
+        mock_model_instance = MagicMock()
+        mock_transformers.AutoModelForSeq2SeqLM.from_pretrained.return_value = mock_model_instance
+        mock_model_instance.generate.return_value = [MagicMock()]
+        
+        triplets = extract_triplets_huggingface(
+            "Elon Musk founded SpaceX and created Starship.",
+            model="Babelscape/rebel-large"
+        )
+        print(f"Triplets: {triplets}", flush=True)
+        
+        # Verify parsing
+        assert len(triplets) == 2
+        assert triplets[0].subject == "Elon Musk"
+        assert triplets[0].predicate == "founded"
+        assert triplets[0].object == "SpaceX"
+        assert triplets[1].subject == "SpaceX"
+        assert triplets[1].predicate == "created"
+        assert triplets[1].object == "Starship"
+        
+        # Verify skip_special_tokens=False was passed
+        mock_tokenizer_instance.decode.assert_called_with(
+            mock_model_instance.generate.return_value[0], 
+            skip_special_tokens=False
+        )
+
+    except Exception as e:
+        print(f"Error during test execution: {e}", flush=True)
+        traceback.print_exc()
+        sys.exit(1)
+
+if __name__ == "__main__":
+    try:
+        test_enhanced_impl()
+        print("\nAll tests passed!", flush=True)
+    except Exception as e:
+        print(f"\nTest failed: {e}", flush=True)
+        traceback.print_exc()

tests/test_relations_llm.py

@@ -0,0 +1,118 @@
+import unittest
+
+from semantica.semantic_extract.methods import extract_relations_llm
+from semantica.semantic_extract.ner_extractor import Entity
+
+
+class FakeProvider:
+    def __init__(self, typed_payload=None, structured_payload=None):
+        self._typed_payload = typed_payload
+        self._structured_payload = structured_payload
+
+    def is_available(self):
+        return True
+
+    # Simulate typed output return: can be dict or an object with relations
+    def generate_typed(self, prompt, schema, **kwargs):
+        return self._typed_payload if self._typed_payload is not None else {"relations": []}
+
+    def generate_structured(self, prompt, **kwargs):
+        return self._structured_payload if self._structured_payload is not None else {"relations": []}
+
+
+class TestLLMRelationExtraction(unittest.TestCase):
+    def setUp(self):
+        # Minimal realistic text and entities
+        self.text = "Apple reported revenue of $4.4 billion in Q1 2024."
+        self.entities = [
+            Entity(text="Apple", label="ORGANIZATION", start_char=0, end_char=5, confidence=0.99),
+            Entity(text="$4.4 billion", label="MONEY", start_char=26, end_char=39, confidence=0.99),
+            Entity(text="Q1 2024", label="DATE", start_char=43, end_char=51, confidence=0.99),
+        ]
+
+    def _monkeypatch_provider(self, provider_instance):
+        # Monkeypatch create_provider used by extract_relations_llm
+        import semantica.semantic_extract.methods as methods
+        self._orig_create_provider = methods.create_provider
+
+        def _fake_create_provider(provider, model=None, **kwargs):
+            return provider_instance
+
+        methods.create_provider = _fake_create_provider
+
+    def tearDown(self):
+        # Restore original create_provider if patched
+        try:
+            import semantica.semantic_extract.methods as methods
+            if hasattr(self, "_orig_create_provider"):
+                methods.create_provider = self._orig_create_provider
+        except Exception:
+            pass
+
+    def test_typed_relations_parsed(self):
+        # Typed returns a dict compatible with parser
+        typed_payload = {
+            "relations": [
+                {
+                    "subject": "Apple",
+                    "predicate": "HAS_REVENUE",
+                    "object": "$4.4 billion",
+                    "confidence": 0.92,
+                },
+                {
+                    "subject": "Apple",
+                    "predicate": "IN_QUARTER",
+                    "object": "Q1 2024",
+                    "confidence": 0.9,
+                },
+            ]
+        }
+        fake = FakeProvider(typed_payload=typed_payload)
+        self._monkeypatch_provider(fake)
+
+        rels = extract_relations_llm(
+            text=self.text,
+            entities=self.entities,
+            provider="groq",
+            model="llama-3.1-8b-instant",
+            relation_types=["HAS_REVENUE", "IN_QUARTER"],
+            verbose=True,
+        )
+        self.assertGreaterEqual(len(rels), 2, "Expected at least two relations from typed payload")
+        preds = {(r.subject.text, r.predicate, r.object.text) for r in rels}
+        self.assertIn(("Apple", "HAS_REVENUE", "$4.4 billion"), preds)
+        self.assertIn(("Apple", "IN_QUARTER", "Q1 2024"), preds)
+
+    def test_structured_fallback_used(self):
+        # Typed returns zero, structured has content
+        typed_payload = {"relations": []}
+        structured_payload = {
+            "relations": [
+                {
+                    "subject": "Apple",
+                    "predicate": "HAS_REVENUE",
+                    "object": "$4.4 billion",
+                    "confidence": 0.88,
+                }
+            ]
+        }
+        fake = FakeProvider(typed_payload=typed_payload, structured_payload=structured_payload)
+        self._monkeypatch_provider(fake)
+
+        rels = extract_relations_llm(
+            text=self.text,
+            entities=self.entities,
+            provider="groq",
+            model="llama-3.1-8b-instant",
+            relation_types=["HAS_REVENUE"],
+            verbose=True,
+        )
+        self.assertEqual(len(rels), 1, "Expected fallback to structured JSON to yield one relation")
+        r = rels[0]
+        self.assertEqual(r.subject.text, "Apple")
+        self.assertEqual(r.object.text, "$4.4 billion")
+        self.assertEqual(r.predicate, "HAS_REVENUE")
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/vector_store/test_pinecone_removal.py

@@ -1,62 +1,3 @@
-import unittest
-from unittest.mock import MagicMock, patch
-import os
-import sys
-
-# Ensure semantica is in path
-sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
-
-from semantica.vector_store.vector_store import VectorStore
-from semantica.vector_store.registry import method_registry
-from semantica.vector_store.config import vector_store_config
-
-class TestPineconeRemoval(unittest.TestCase):
-    """Verify that Pinecone has been completely removed from the system."""
-
-    def test_pinecone_backend_rejected(self):
-        """Test that initializing VectorStore with backend='pinecone' raises an error."""
-        with self.assertRaises(ValueError) as context:
-            VectorStore(backend="pinecone")
-        
-        # The error message might be generic "Unknown backend" or specific.
-        # We just want to ensure it fails.
-        self.assertTrue("pinecone" in str(context.exception).lower() or "unknown" in str(context.exception).lower())
-
-    def test_registry_clean(self):
-        """Test that no Pinecone methods are registered."""
-        # Check all task types
-        task_types = ["store", "search", "index", "hybrid_search", "metadata", "namespace"]
-        
-        for task in task_types:
-            methods = method_registry.list_all(task)
-            # Flatten if it's a dict
-            if isinstance(methods, dict):
-                method_names = methods.get(task, [])
-            else:
-                method_names = methods
-                
-            for name in method_names:
-                self.assertNotIn("pinecone", name.lower(), f"Found pinecone reference in registry task {task}: {name}")
-
-    def test_config_clean(self):
-        """Test that configuration does not contain Pinecone keys."""
-        config = vector_store_config.get_all()
-        
-        for key in config.keys():
-            self.assertNotIn("pinecone", key.lower(), f"Found pinecone key in config: {key}")
-
-    def test_stores_existence(self):
-        """Verify that other stores exist but PineconeStore does not."""
-        try:
-            from semantica.vector_store import faiss_store
-            from semantica.vector_store import weaviate_store
-            from semantica.vector_store import qdrant_store
-            from semantica.vector_store import milvus_store
-        except ImportError as e:
-            self.fail(f"Failed to import a required store: {e}")
-
-        with self.assertRaises(ImportError):
-            from semantica.vector_store import pinecone_store
-
-if __name__ == '__main__':
-    unittest.main()
+# This test file has been updated as Pinecone support has been re-added to Semantica.
+# Pinecone is now a supported vector store backend (PR #220).
+# See test_pinecone_store.py for Pinecone-specific tests.

tests/vector_store/test_pinecone_store.py

@@ -0,0 +1,250 @@
+import unittest
+from unittest.mock import MagicMock, patch
+import numpy as np
+import sys
+import os
+
+# Ensure semantica is in path if running directly
+if __name__ == "__main__":
+    sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
+
+try:
+    from semantica.vector_store.pinecone_store import (
+        PineconeStore,
+        PineconeClient,
+        PineconeIndex,
+        PineconeSearch,
+        PINECONE_AVAILABLE
+    )
+    from semantica.utils.exceptions import ProcessingError
+except ImportError:
+    # If we can't import, we can't run these tests
+    # But we should not crash silently.
+    # We will define dummy classes if needed or fail loudly.
+    raise
+
+class TestPineconeStore(unittest.TestCase):
+    """Test Pinecone store functionality."""
+
+    def setUp(self):
+        self.mock_logger = MagicMock()
+        self.mock_tracker = MagicMock()
+
+        self.logger_patcher = patch('semantica.vector_store.pinecone_store.get_logger', return_value=self.mock_logger)
+        self.tracker_patcher = patch('semantica.vector_store.pinecone_store.get_progress_tracker', return_value=self.mock_tracker)
+        self.mock_logger_instance = self.logger_patcher.start()
+        self.mock_tracker_instance = self.tracker_patcher.start()
+
+    def tearDown(self):
+        self.logger_patcher.stop()
+        self.tracker_patcher.stop()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_connect(self, mock_pinecone_client):
+        """Test connecting to Pinecone."""
+        mock_client_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        self.assertIsNotNone(store.client)
+        mock_pinecone_client.assert_called_once_with(api_key="test-key")
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', False)
+    def test_connect_unavailable(self):
+        """Test connecting when Pinecone is not available."""
+        store = PineconeStore(api_key="test-key")
+        with self.assertRaises(ProcessingError):
+            store.connect()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_create_index(self, mock_pinecone_client):
+        """Test creating an index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+        mock_client_instance.Index.return_value = mock_index_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Mock the client's create_index method
+        store.client.create_index = MagicMock()
+        store.client.get_index = MagicMock(return_value=mock_index_instance)
+
+        result = store.create_index("test-index", dimension=768, metric="cosine")
+
+        self.assertIsInstance(result, PineconeIndex)
+        self.assertIsInstance(store.index, PineconeIndex)
+        self.assertIsInstance(store.search_engine, PineconeSearch)
+        store.client.create_index.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_upsert_vectors(self, mock_pinecone_client):
+        """Test upserting vectors to Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up index
+        store.index = PineconeIndex(mock_index_instance)
+        store.index.upsert_vectors = MagicMock(return_value={"upserted_count": 2})
+
+        vectors = [np.array([0.1, 0.2, 0.3]), np.array([0.4, 0.5, 0.6])]
+        ids = ["id1", "id2"]
+        metadata = [{"key": "value1"}, {"key": "value2"}]
+
+        result = store.upsert_vectors(vectors, ids, metadata)
+
+        self.assertEqual(result["upserted_count"], 2)
+        store.index.upsert_vectors.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_search_vectors(self, mock_pinecone_client):
+        """Test searching vectors in Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up search engine
+        store.search_engine = PineconeSearch(PineconeIndex(mock_index_instance))
+        store.search_engine.similarity_search = MagicMock(return_value=[
+            {"id": "id1", "score": 0.9, "metadata": {"key": "value1"}}
+        ])
+
+        query_vector = np.array([0.1, 0.2, 0.3])
+        results = store.search_vectors(query_vector, k=5)
+
+        self.assertEqual(len(results), 1)
+        self.assertEqual(results[0]["id"], "id1")
+        store.search_engine.similarity_search.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_delete_vectors(self, mock_pinecone_client):
+        """Test deleting vectors from Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up index
+        store.index = PineconeIndex(mock_index_instance)
+        store.index.delete_vectors = MagicMock(return_value={"deleted": True})
+
+        result = store.delete_vectors(["id1", "id2"])
+
+        self.assertEqual(result["deleted"], True)
+        # Fix: assert called without the empty dict
+        store.index.delete_vectors.assert_called_once_with(["id1", "id2"], "")
+
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_fetch_vectors(self, mock_pinecone_client):
+        """Test fetching vectors from Pinecone index."""
+        mock_client_instance = MagicMock()
+        mock_index_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        store = PineconeStore(api_key="test-key")
+        store.connect()
+
+        # Set up index
+        store.index = PineconeIndex(mock_index_instance)
+        store.index.fetch_vectors = MagicMock(return_value={
+            "vectors": {
+                "id1": {"values": [0.1, 0.2], "metadata": {"key": "value1"}}
+            }
+        })
+
+        result = store.fetch_vectors(["id1"])
+
+        self.assertIn("vectors", result)
+        # Fix: assert called without the empty dict
+        store.index.fetch_vectors.assert_called_once_with(["id1"], "")
+
+
+class TestPineconeClient(unittest.TestCase):
+    """Test PineconeClient wrapper."""
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_create_index(self, mock_pinecone_client):
+        """Test creating an index via PineconeClient."""
+        mock_client_instance = MagicMock()
+        mock_pinecone_client.return_value = mock_client_instance
+
+        client = PineconeClient(mock_client_instance)
+        client.create_index("test-index", 768, "cosine")
+
+        mock_client_instance.create_index.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    @patch('semantica.vector_store.pinecone_store.PineconeClientLib')
+    def test_list_indexes(self, mock_pinecone_client):
+        """Test listing indexes via PineconeClient."""
+        mock_client_instance = MagicMock()
+        mock_index_obj = MagicMock()
+        mock_index_obj.name = "test-index"
+        mock_client_instance.list_indexes.return_value = [mock_index_obj]
+        mock_pinecone_client.return_value = mock_client_instance
+
+        client = PineconeClient(mock_client_instance)
+        result = client.list_indexes()
+
+        self.assertEqual(result, ["test-index"])
+
+
+class TestPineconeIndex(unittest.TestCase):
+    """Test PineconeIndex wrapper."""
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_upsert_vectors(self):
+        """Test upserting vectors via PineconeIndex."""
+        mock_index = MagicMock()
+        mock_response = MagicMock()
+        mock_response.upserted_count = 2
+        mock_index.upsert.return_value = mock_response
+
+        index = PineconeIndex(mock_index)
+        result = index.upsert_vectors(
+            [[0.1, 0.2], [0.3, 0.4]],
+            ["id1", "id2"],
+            [{"key": "value1"}]
+        )
+
+        self.assertEqual(result["upserted_count"], 2)
+        mock_index.upsert.assert_called_once()
+
+    @patch('semantica.vector_store.pinecone_store.PINECONE_AVAILABLE', True)
+    def test_search_vectors(self):
+        """Test searching vectors via PineconeIndex."""
+        mock_index = MagicMock()
+        mock_match = MagicMock()
+        mock_match.id = "id1"
+        mock_match.score = 0.9
+        mock_match.metadata = {"key": "value1"}
+        mock_response = MagicMock()
+        mock_response.matches = [mock_match]
+        mock_index.query.return_value = mock_response
+
+        index = PineconeIndex(mock_index)
+        result = index.search_vectors([0.1, 0.2], k=5)
+
+        self.assertEqual(len(result), 1)
+        self.assertEqual(result[0]["id"], "id1")
+        mock_index.query.assert_called_once()
+
+
+if __name__ == '__main__':
+    print("DEBUG: Starting unittest.main()")
+    unittest.main()

tests/vector_store/test_vector_store_parallel.py

@@ -0,0 +1,149 @@
+
+import unittest
+import numpy as np
+import time
+import sys
+import os
+import logging
+from unittest.mock import MagicMock, patch
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../..")))
+
+from semantica.vector_store import VectorStore
+from semantica.utils.exceptions import ProcessingError
+
+
+class TestVectorStoreParallel(unittest.TestCase):
+    def setUp(self):
+        logging.getLogger("vector_store").setLevel(logging.ERROR)
+
+        self.dimension = 4
+        self.store = VectorStore(
+            backend="inmemory",
+            dimension=self.dimension,
+        )
+
+        self.store.embedder = MagicMock()
+
+    def test_embed_batch_success(self):
+        texts = ["a", "b", "c"]
+        expected_embeddings = [
+            np.array([0.1] * 4, dtype=np.float32),
+            np.array([0.2] * 4, dtype=np.float32),
+            np.array([0.3] * 4, dtype=np.float32),
+        ]
+
+        self.store.embedder.generate_embeddings.return_value = expected_embeddings
+
+        results = self.store.embed_batch(texts)
+
+        self.assertEqual(len(results), 3)
+        self.assertTrue(np.allclose(results[0], expected_embeddings[0]))
+        self.store.embedder.generate_embeddings.assert_called_once_with(texts)
+
+    def test_embed_batch_fallback(self):
+        texts = ["a", "b"]
+
+        self.store.embedder.generate_embeddings.side_effect = Exception("Model error")
+
+        results = self.store.embed_batch(texts)
+
+        self.assertEqual(len(results), 2)
+        self.assertEqual(results[0].shape, (self.dimension,))
+        self.assertTrue(isinstance(results[0], np.ndarray))
+
+    def test_add_documents_empty(self):
+        ids = self.store.add_documents([])
+        self.assertEqual(ids, [])
+
+    def test_add_documents_metadata_mismatch(self):
+        with self.assertRaises(ValueError):
+            self.store.add_documents(["doc1"], metadata=[{}, {}])
+
+    def test_add_documents_parallel_success(self):
+        num_docs = 10
+        documents = [f"doc_{i}" for i in range(num_docs)]
+        metadata = [{"id": i} for i in range(num_docs)]
+
+        def mock_embed_batch(texts):
+            return [np.full(self.dimension, float(i)) for i, _ in enumerate(texts)]
+
+        with patch.object(self.store, "embed_batch", side_effect=mock_embed_batch):
+            ids = self.store.add_documents(
+                documents,
+                metadata,
+                batch_size=2,
+                parallel=True,
+            )
+
+        self.assertEqual(len(ids), num_docs)
+        self.assertEqual(len(self.store.vectors), num_docs)
+
+        for i, vec_id in enumerate(ids):
+            stored_meta = self.store.get_metadata(vec_id)
+            self.assertEqual(stored_meta["id"], i)
+
+    def test_add_documents_sequential_success(self):
+        num_docs = 5
+        documents = [f"doc_{i}" for i in range(num_docs)]
+
+        with patch.object(self.store, "embed_batch") as mock_batch:
+            mock_batch.return_value = [np.zeros(self.dimension) for _ in range(num_docs)]
+
+            ids = self.store.add_documents(documents, parallel=False)
+
+            self.assertEqual(len(ids), num_docs)
+            self.assertEqual(mock_batch.call_count, 1)
+
+    def test_add_documents_error_propagation(self):
+        documents = ["doc1", "doc2"]
+
+        with patch.object(self.store, "embed_batch", side_effect=ValueError("Embedding Error")):
+            with self.assertRaises(Exception):
+                self.store.add_documents(documents, parallel=True)
+
+    def test_performance_simulation(self):
+        num_batches = 4
+        batch_delay = 0.1
+        batch_size = 1
+        documents = [f"doc_{i}" for i in range(num_batches)]
+
+        def slow_embed(texts):
+            time.sleep(batch_delay)
+            return [np.zeros(self.dimension) for _ in texts]
+
+        with patch.object(self.store, "embed_batch", side_effect=slow_embed):
+            start_seq = time.time()
+            self.store.add_documents(documents, batch_size=batch_size, parallel=False)
+            dur_seq = time.time() - start_seq
+
+            self.store.vectors = {}
+
+            start_par = time.time()
+            self.store.add_documents(documents, batch_size=batch_size, parallel=True)
+            dur_par = time.time() - start_par
+
+            print(f"\nPerformance Test:")
+            print(f"Sequential Duration: {dur_seq:.4f}s")
+            print(f"Parallel Duration:   {dur_par:.4f}s")
+            print(f"Speedup:             {dur_seq / dur_par:.2f}x")
+
+            self.assertLess(dur_par, dur_seq * 0.7)
+
+    def test_add_documents_batch_size_edge_cases(self):
+        documents = ["a", "b", "c"]
+
+        with patch.object(self.store, "embed_batch") as mock_batch:
+            mock_batch.side_effect = lambda texts: [np.zeros(4) for _ in texts]
+
+            self.store.add_documents(documents, batch_size=100)
+            self.assertEqual(mock_batch.call_count, 1)
+
+            mock_batch.reset_mock()
+
+            self.store.add_documents(documents, batch_size=1)
+            self.assertEqual(mock_batch.call_count, 3)
+
+
+if __name__ == "__main__":
+    unittest.main()