Release v0.2.6

Fix Critical Test Issues and Add JenaStore Empty Graph Tests

.github/FUNDING.yml

@@ -1,8 +1,3 @@
 # Funding options for Semantica
-# Uncomment and add your usernames/links below
-
-# github: [username]
-# patreon: username
-# ko_fi: username
-# custom: ["https://your-funding-page.com"]
+github: Hawksight-AI
 

.github/SUPPORT.md

@@ -32,6 +32,8 @@ For enterprise support, custom development, or consulting services:
 
 ## Sponsorship
 
+### Sponsor this project
+
 Support Semantica development:
 - [GitHub Sponsors](https://github.com/sponsors/Hawksight-AI)
 

.pre-commit-config.yaml

@@ -5,6 +5,7 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-yaml
+        exclude: 'neptune-setup\.yaml$'
       - id: check-json
       - id: check-toml
       - id: check-added-large-files
@@ -49,9 +50,15 @@ repos:
     hooks:
       - id: yamllint
         args: ['-d', '{extends: default, rules: {line-length: {max: 120}}}']
+        exclude: 'neptune-setup\.yaml$'
+
+  - repo: https://github.com/aws-cloudformation/cfn-lint
+    rev: v1.43.3
+    hooks:
+      - id: cfn-lint
+        files: 'neptune-setup\.yaml$'
 
   # Removed slow hooks for faster development:
   # - mypy: Type checking (can be run manually or in CI)
   # - bandit: Security scanning (can be run separately)
   # - pytest: Testing (should be run manually, not on every commit)
-

CHANGELOG.md

@@ -7,6 +7,244 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.6] - 2026-02-03
+
+### Added / Changed
+
+- **W3C PROV-O Compliant Provenance Tracking** (#254, #246):
+  - Comprehensive provenance tracking system with W3C PROV-O compliance across all 17 Semantica modules
+  - **Core Module**: `ProvenanceManager`, W3C PROV-O schemas, storage backends (InMemory, SQLite), SHA-256 integrity verification
+  - **Module Integrations**: Semantic Extract, LLMs (Groq, OpenAI, HuggingFace, LiteLLM), Pipeline, Context, Ingest, Embeddings, Graph/Vector/Triplet stores, Reasoning, Conflicts, Deduplication, Export, Parse, Normalize, Ontology, Visualization
+  - **Features**: Complete lineage tracking (Document → Chunk → Entity → Relationship → Graph), LLM tracking (tokens, costs, latency), source tracking, bridge axioms for domain transformations
+  - **Compliance Infrastructure**: W3C PROV-O, FDA 21 CFR Part 11, SOX, HIPAA, TNFD
+  - **Testing**: 237 tests covering core functionality, all 17 module integrations, edge cases, backward compatibility
+  - **Design**: Opt-in with `provenance=False` by default, zero breaking changes, no new dependencies
+  - Contributed by @KaifAhmad1
+
+- **Enhanced Change Management Module** (#248, #243):
+  - Enterprise-grade version control for knowledge graphs and ontologies with persistent storage and audit trails
+  - **Core Classes**: `TemporalVersionManager` (KG versioning), `OntologyVersionManager` (ontology versioning), `ChangeLogEntry` (metadata)
+  - **Storage**: SQLite (persistent) and in-memory backends with thread-safe operations
+  - **Features**: SHA-256 checksums, detailed entity/relationship diffs, structural ontology comparison, email validation
+  - **Compliance Infrastructure**: HIPAA, SOX, FDA 21 CFR Part 11 with immutable audit trails
+  - **Testing**: 104 tests (100% pass) - unit, integration, compliance, performance, edge cases
+  - **Performance**: 17.6ms for 10k entities, 510+ ops/sec concurrent, handles 5k+ entity graphs
+  - **Migration**: Backward compatible, simplified class names, zero external dependencies
+  - Contributed by @KaifAhmad1
+
+- CSV Ingestion Enhancements (PR #244 by @saloni0318)
+  - Auto-detect CSV encoding (chardet) and delimiter (csv.Sniffer)
+  - Tolerant decoding and malformed-row handling (`on_bad_lines='warn'`)
+  - Optional chunked reading for large files; metadata tracks detected values
+  - Expanded unit tests covering delimiters, quoted/multiline fields, header overrides, chunks, and NaN preservation
+
+- Tests: Comprehensive units for TextNormalizer (PR #242 by @ZohaibHassan16)
+  - Added focused test coverage for TextNormalizer behavior across inputs
+
+- Tests: Register integration mark and tidy ingest test warnings (PR #241 by @KaifAhmad1)
+  - Introduced integration test marker and reduced noisy warnings in ingest tests
+
+- **Ingest Unit Tests** (#239, #232):
+  - Comprehensive unit tests for ingestion modules (file, web, and feed ingestors)
+  - **Coverage**: File scanning (local/cloud S3/GCS/Azure), web ingestion (URL/sitemap/robots.txt), RSS/Atom feed parsing
+  - **Testing**: 998 lines of test code with mocked external dependencies for fast, isolated execution
+  - **Results**: file_ingestor (86%), web_ingestor (86%), feed_ingestor (80%) coverage
+  - Covers happy paths, edge cases, and error handling
+  - Contributed by @Mohammed2372
+
+### Fixed
+
+- **Temperature Compatibility Fix** (#256, #252):
+  - Fixed hardcoded `temperature=0.3` that broke compatibility with models requiring specific temperature values (e.g., gpt-5-mini)
+  - Added `_add_if_set` helper method to `BaseProvider` that only passes parameters when explicitly set
+  - When `temperature=None`, parameter is omitted allowing APIs to use model defaults
+  - Updated all 5 providers: OpenAI, Groq, Gemini, Ollama, DeepSeek
+  - Reduced code by ~85 lines with cleaner parameter handling
+  - Comprehensive test coverage added (10 temperature tests, all passing)
+  - Backward compatible - no breaking changes
+  - Contributed by @F0rt1s and @IGES-Institut
+
+- **JenaStore Empty Graph Bug** (#257, #258):
+  - Fixed `ProcessingError: Graph not initialized` when operating on empty (but initialized) graphs
+  - Replaced implicit `if not self.graph:` checks with explicit `if self.graph is None:` validation in 5 methods (`add_triplets`, `get_triplets`, `delete_triplet`, `execute_sparql`, `serialize`)
+  - Properly distinguishes `None` (uninitialized) from empty graphs (initialized with 0 triplets)
+  - Unblocks benchmarking suite, fresh deployments, and testing workflows
+  - Contributed by @ZohaibHassan16
+
+## [0.2.5] - 2026-01-27
+
+### Added
+- **Pinecone Vector Store Support**:
+    - Implemented native Pinecone support (`PineconeStore`) with full CRUD capabilities.
+    - Added support for serverless and pod-based indexes, namespaces, and metadata filtering.
+    - Integrated with `VectorStore` unified interface and registry.
+    - (Closes #219, Resolves #220)
+- **Configurable LLM Retry Logic**:
+    - Exposed `max_retries` parameter in `NERExtractor`, `RelationExtractor`, `TripletExtractor` and low-level extraction methods (`extract_entities_llm`, `extract_relations_llm`, `extract_triplets_llm`).
+    - Defaults to 3 retries to prevent infinite loops during JSON validation failures or API timeouts.
+    - Propagated retry configuration through chunked processing helpers to ensure consistent behavior for long documents.
+    - Updated `03_Earnings_Call_Analysis.ipynb` to use `max_retries=3` by default.
+
+### Added
+- **Bring Your Own Model (BYOM) Support**:
+    - Enabled full support for custom Hugging Face models in `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Added support for custom tokenizers in `HuggingFaceModelLoader` to handle models with non-standard tokenization requirements.
+    - Implemented robust fallback logic for model selection: runtime options (`extract(model=...)`) now correctly override configuration defaults.
+- **Enhanced NER Implementation**:
+    - Added configurable aggregation strategies (`simple`, `first`, `average`, `max`) to `extract_entities_huggingface` for better sub-word token handling.
+    - Implemented robust IOB/BILOU parsing to reconstruct entities from raw model outputs when structured output is unavailable.
+    - Added confidence scoring for aggregated entities.
+- **Relation Extraction Improvements**:
+    - Implemented standard entity marker technique (wrapping subject/object with `<subj>`, `<obj>` tags) in `extract_relations_huggingface` for compatibility with sequence classification models.
+    - Added structured output parsing to convert raw model predictions into validated `Relation` objects.
+- **Triplet Extraction Completion**:
+    - Added specialized parsing for Seq2Seq models (e.g., REBEL) in `extract_triplets_huggingface` to generate structured triplets directly from text.
+    - Implemented post-processing logic to clean and validate generated triplets.
+
+### Fixed
+- **LLM Extraction Stability**:
+    - Fixed infinite retry loops in `BaseProvider` by strictly enforcing `max_retries` limit during structured output generation.
+    - Resolved stuck execution in earnings call analysis notebooks when using smaller models (e.g., Llama 3 8B) that frequently produce invalid JSON.
+- **Model Parameter Precedence**:
+    - Fixed issue where configuration defaults took precedence over runtime arguments in Hugging Face extractors. Runtime options now correctly override config values.
+- **Import Handling**:
+    - Fixed circular import issues in test suites by implementing robust mocking strategies.
+
+## [0.2.4] - 2026-01-22
+
+### Added
+- **Ontology Ingestion Module**:
+    - Implemented `OntologyIngestor` in `semantica.ingest` for parsing RDF/OWL files (Turtle, RDF/XML, JSON-LD, N3) into standardized `OntologyData` objects.
+    - Added `ingest_ontology` convenience function and integrated it into the unified `ingest(source_type="ontology")` interface.
+    - Added recursive directory scanning support for batch ontology ingestion.
+    - Exposed ingestion tools in `semantica.ontology` for better discoverability.
+    - Added `OntologyData` dataclass for consistent metadata handling (source path, format, timestamps).
+- **Documentation**:
+    - **Ontology Usage Guide**: Updated `ontology_usage.md` with comprehensive examples for single-file and directory ingestion.
+    - **API Reference**: Updated `ontology.md` with `OntologyIngestor` class documentation and method details.
+- **Tests**:
+    - **Comprehensive Test Suite**: Added `tests/ingest/test_ontology_ingestor.py` covering all supported formats, error handling, and unified interface integration.
+    - **Demo Script**: Added `examples/demo_ontology_ingest.py` for end-to-end usage demonstration.
+
+## [0.2.3] - 2026-01-20
+
+### Fixed
+- **LLM Relation Extraction Parsing**:
+    - Fixed relation extraction returning zero relations despite successful API calls to Groq and other providers
+    - Normalized typed responses from instructor/OpenAI/Groq to consistent dict format before parsing
+    - Added structured JSON fallback when typed generation yields zero relations to avoid silent empty outputs
+    - Removed acceptance of extra kwargs (`max_tokens`, `max_entities_prompt`) from relation extraction internals
+    - Filtered kwargs passed to provider LLM calls to only `temperature` and `verbose`
+- **API Parameter Handling**:
+    - Limited kwargs forwarded in chunked extraction helper to prevent parameter leakage
+    - Ensured minimal, safe parameters are passed to provider calls
+- **Pipeline Circular Import (Issues #192, #193)**:
+    - Fixed circular import between `pipeline_builder` and `pipeline_validator` triggered during `semantica.pipeline` import
+    - Lazy-loaded `PipelineValidator` inside `PipelineBuilder.__init__` and guarded type hints with `TYPE_CHECKING`
+    - Ensured `from semantica.deduplication import DuplicateDetector` no longer fails even when pipeline module is imported
+- **JupyterLab Progress Output (Issue #181)**:
+    - Added `SEMANTICA_DISABLE_JUPYTER_PROGRESS` environment variable to disable rich Jupyter/Colab progress tables
+    - When enabled, progress falls back to console-style output, preventing infinite scrolling and JupyterLab out-of-memory errors
+
+### Added
+- **Comprehensive Test Suite**:
+-    - Added unit tests (`tests/test_relations_llm.py`) with mocked LLM provider covering both typed and structured response paths
+-    - Added integration tests (`tests/integration/test_relations_groq.py`) for real Groq API calls with environment variable API key
+-    - Tests validate relation extraction completion and result parsing across different response formats
+- **Amazon Neptune Dev Environment**:
+-    - Added CloudFormation template (`cookbook/introduction/neptune-setup.yaml`) to provision a dev Neptune cluster with public endpoint and IAM auth enabled
+-    - Documented deployment, cost estimates, and IAM User vs IAM Role best practices in `cookbook/introduction/21_Amazon_Neptune_Store.ipynb`
+-    - Added `cfn-lint` to `.pre-commit-config.yaml` for validating CloudFormation templates while excluding `neptune-setup.yaml` from generic YAML linters
+- **Vector Store High-Performance Ingestion**:
+-    - Added `VectorStore.add_documents` for high-throughput ingestion with automatic embedding generation, batching, and parallel processing
+-    - Added `VectorStore.embed_batch` helper for generating embeddings for lists of texts without immediately storing them
+-    - Enabled default parallel ingestion in `VectorStore` with `max_workers=6` for common workloads
+-    - Added dedicated documentation page `docs/vector_store_usage.md` describing high-performance vector store usage and configuration
+-    - Added `tests/vector_store/test_vector_store_parallel.py` covering parallel vs sequential performance, error handling, and edge cases for `add_documents` and `embed_batch`
+
+### Changed
+- **Relation Extraction API**:
+-    - Simplified parameter interface by removing unused kwargs that were previously ignored
+-    - Improved error handling and verbose logging for debugging relation extraction issues
+-    - Enhanced robustness of post-response parsing across different LLM providers
+- **Vector Store Defaults and Examples**:
+-    - Standardized `VectorStore` default concurrency to `max_workers=6` for parallel ingestion
+-    - Updated vector store reference documentation and usage guides to rely on implicit defaults instead of requiring manual `max_workers` configuration in examples
+
+
+## [0.2.2] - 2026-01-15
+
+### Added
+- **Parallel Extraction Engine**:
+    - Implemented high-throughput parallel batch processing across all core extractors (`NERExtractor`, `RelationExtractor`, `TripletExtractor`, `EventDetector`, `SemanticNetworkExtractor`) using `concurrent.futures.ThreadPoolExecutor`.
+    - Added `max_workers` configuration parameter (default: 1) to all extractor `extract()` methods, allowing users to tune concurrency based on available CPU cores or API rate limits.
+    - **Parallel Chunking**: Implemented parallel processing for large document chunking in `_extract_entities_chunked` and `_extract_relations_chunked`, significantly reducing latency for long-form text analysis.
+    - **Thread-Safe Progress Tracking**: Enhanced `ProgressTracker` to handle concurrent updates from multiple threads without race conditions during batch processing.
+- **Semantic Extract Performance & Regression**:
+    - Added edge-case regression suite covering max worker defaults, LLM prompt entity filtering, and extractor reuse.
+    - Added a runnable real-use-case benchmark script for batch latency across `NERExtractor`, `RelationExtractor`, `TripletExtractor`, `EventDetector`, `SemanticAnalyzer`, and `SemanticNetworkExtractor`.
+    - Added Groq LLM smoke tests that exercise LLM-based entities/relations/triplets when `GROQ_API_KEY` is available via environment configuration.
+
+### Security
+- **Credential Sanitization**:
+    - Removed hardcoded API keys from 8 cookbook notebooks to prevent secret leakage.
+    - Enforced environment variable usage for `GROQ_API_KEY` across all examples.
+- **Secure Caching**:
+    - Updated `ExtractionCache` to exclude sensitive parameters (e.g., `api_key`, `token`, `password`) from cache key generation, preventing secret leakage and enabling safe cache sharing.
+    - Upgraded cache key hashing algorithm from MD5 to **SHA-256** for enhanced collision resistance and security.
+
+### Changed
+- **Gemini SDK Migration**:
+    - Migrated `GeminiProvider` to use the new `google-genai` SDK (v0.1.0+) to address deprecation warnings.
+    - Implemented graceful fallback to `google.generativeai` for backward compatibility.
+- **Dependency Resolution**:
+    - Pinned `opentelemetry-api` and `opentelemetry-sdk` to `1.37.0` to resolve pip conflicts.
+    - Updated `protobuf` and `grpcio` constraints for better stability.
+- **Entity Filtering Scope**:
+    - Removed entity filtering from non-LLM extraction flows to avoid accuracy regressions.
+    - Applied entity downselection only to LLM relation prompt construction, while matching returned entities against the full original entity list.
+- **Batch Concurrency Defaults**:
+    - Standardized `max_workers` defaulting across `semantic_extract` and tuned for low-latency: ML-backed methods default to single-worker, while pattern/regex/rules/LLM/huggingface methods use a higher parallelism default capped by CPU.
+    - Raised the global `optimization.max_workers` default to 8 for better throughput on batch workloads.
+
+### Performance
+- **Bottleneck Optimization (GitHub Issue #186)**:
+    - **Resolved Bottleneck #1 (Sequential Processing)**: Replaced sequential `for` loops with parallel execution for both document-level batches and intra-document chunks.
+    - **Performance Gains**: Achieved **~1.89x speedup** in real-world extraction scenarios (tested with Groq `llama-3.3-70b-versatile` on standard datasets).
+    - **Initialization Optimization**: Refactored test suite to use class-level `setUpClass` for LLM provider initialization, eliminating redundant API client creation overhead.
+- **Low-Latency Entity Matching**:
+    - Avoided heavyweight embedding stack imports on common matches by improving fast matching heuristics and short-circuiting before embedding similarity.
+    - Optimized entity matching to prioritize exact/substring/word-boundary matches and only fall back to embedding similarity when needed, reducing CPU overhead in LLM relation/triplet mapping.
+
+
+## [0.2.1] - 2026-01-12
+
+### Fixed
+- **LLM Output Stability (Bug #176)**:
+    - Fixed incomplete JSON output issues by correctly propagating `max_tokens` parameter in `extract_relations_llm`.
+    - Implemented automatic error handling that halves chunk sizes and retries when LLM context or output limits are exceeded.
+    - Fixed `AttributeError` in provider integration by ensuring consistent parameter passing via `**kwargs`.
+- **Constraint Relaxations**:
+    - Removed hardcoded `max_length` constraints from `Entity`, `Relation`, and `Triplet` classes to support long-form semantic extraction (e.g., long descriptions or names).
+- Fixed orchestrator lazy property initialization and configuration normalization logic in `Orchestrator`.
+- Resolved `AssertionError` in orchestrator tests by aligning test mocks with production component usage.
+- Fixed dependency compatibility issues by pinning `protobuf>=5.29.1,<7.0` and `grpcio>=1.71.2`.
+- Added missing dependencies `GitPython` and `chardet` to `pyproject.toml`.
+- Verified and aligned `FileObject.text` property usage in GraphRAG notebooks for consistent content decoding.
+
+### Changed
+- **Chunking Defaults**:
+    - Increased default `max_text_length` for auto-chunking to **64,000 characters** (from 32k/16k) for OpenAI, Anthropic, Gemini, Groq, and DeepSeek providers.
+    - Unified chunking logic across `extract_entities_llm`, `extract_relations_llm`, and `extract_triplets_llm`.
+- **Groq Support**:
+    - Standardized Groq provider defaults to use `llama-3.3-70b-versatile` with a 64k context window.
+    - Added native support for `max_tokens` and `max_completion_tokens` to prevent output truncation.
+
+### Added
+- **Testing**:
+    - Added `tests/reproduce_issue_176.py` to validate `max_tokens` propagation and chunking behavior across all extractors.
+
+
 ## [0.2.0] - 2026-01-10
 
 ### Added

CONTRIBUTING.md

@@ -1,306 +1,266 @@
 # Contributing to Semantica
 
-Thank you for your interest in contributing to Semantica! This document provides guidelines and instructions for contributing to the project.
+Thank you for your interest in contributing! Every contribution, no matter how small, is valuable. 🎉
 
-## Table of Contents
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-- [Code of Conduct](#code-of-conduct)
-- [Getting Started](#getting-started)
-- [Development Setup](#development-setup)
-- [Code Style Guidelines](#code-style-guidelines)
-- [Testing Requirements](#testing-requirements)
-- [Commit Message Conventions](#commit-message-conventions)
-- [Pull Request Process](#pull-request-process)
-- [Documentation Standards](#documentation-standards)
-- [Types of Contributions](#types-of-contributions)
-- [Getting Help](#getting-help)
+> **New to contributing?** Start with a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/vqRt2qbx) community.
 
-## Code of Conduct
+---
 
-This project adheres to a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the maintainers.
+## 🚀 Quick Start
 
-## Getting Started
+1. Find a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue)
+2. [Fork Semantica](https://github.com/Hawksight-AI/semantica/fork) & clone the repository
+3. Make your changes
+4. Submit a pull request!
 
-1. **Fork the repository** on GitHub
-2. **Clone your fork** locally:
-   ```bash
-   git clone https://github.com/your-username/semantica.git
-   cd semantica
-   ```
-3. **Add the upstream remote**:
-   ```bash
-   git remote add upstream https://github.com/Hawksight-AI/semantica.git
-   ```
+**Need help?** Join [Discord](https://discord.gg/vqRt2qbx) or [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
 
-## Development Setup
+---
 
-### Prerequisites
+## 🎯 Ways to Contribute
 
-- Python 3.8 or higher (3.9+ recommended)
-- pip package manager
-- Git
+### 💻 Code
 
-### Installation
+**What you can do:**
+- Fix bugs
+- Add new features
+- Improve code quality (add type hints, docstrings, improve error messages)
+- Optimize performance
 
-1. **Create a virtual environment** (recommended):
-   ```bash
-   python -m venv venv
-   source venv/bin/activate  # On Windows: venv\Scripts\activate
-   ```
+**Where:** `semantica/` directory
 
-2. **Install the project in editable mode with dev dependencies**:
-   ```bash
-   pip install -e ".[dev]"
-   ```
+**Good first issues:** Add docstrings, type hints, or improve error messages
 
-3. **Install pre-commit hooks**:
-   ```bash
-   pre-commit install
-   ```
+---
 
-### Verify Installation
+### 📝 Documentation
+
+**What you can do:**
+- Fix typos and grammar errors
+- Improve clarity and readability
+- Add code examples and tutorials
+- Create new cookbook notebooks
+- Improve API documentation (docstrings)
+- Create troubleshooting guides
+- Update installation instructions
+- Add missing documentation
+
+**Where:** `README.md`, `docs/`, `cookbook/`, docstrings in code
+
+**Good first issues:** Fix typos, add examples, create cookbook tutorials, improve docstrings
+
+**Documentation formatting:**
+- Use clear, concise language
+- Include code examples where helpful
+- Follow markdown best practices
+- Use proper headings hierarchy
+- Add links to related sections
+- Include screenshots for UI-related docs
+
+---
+
+### 🧪 Testing
+
+**What you can do:**
+- Add unit tests
+- Improve test coverage
+- Add integration tests
+
+**Where:** `tests/` directory
+
+**Good first issues:** Add tests for specific functions or classes
+
+---
+
+### 🐛 Bug Reports
+
+**What:** Report bugs you find
+
+**How:** Use the [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
+
+**Include:** Description, steps to reproduce, expected vs actual behavior, environment details
+
+---
+
+### 💡 Feature Requests
+
+**What:** Suggest new features or improvements
+
+**How:** Use the [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
+
+**Include:** Problem statement, proposed solution, use cases
+
+---
+
+### 🎨 Cookbook & Examples
+
+**What:** Create tutorials and examples
+
+**Where:** `cookbook/` directory
+
+**Examples:** Create new notebooks, add examples, improve existing tutorials
+
+---
+
+### 💬 Community Support
+
+**What:** Help others in the community
+
+**Where:** [Discord](https://discord.gg/vqRt2qbx), [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
+
+**Examples:** Answer questions, review PRs, share your projects
+
+---
+
+### 🎓 Educational Content
+
+**What:** Create educational materials
+
+**Examples:** Blog posts, video tutorials, talks, workshops, case studies
+
+---
+
+### 🔧 Other Contributions
+
+- **Design & Graphics:** Logos, diagrams, visualizations
+- **Tools & Integrations:** CLI tools, integrations with other frameworks
+- **Infrastructure:** CI/CD improvements, Docker optimization
+- **Security:** Report security vulnerabilities (privately)
+
+---
+
+## 📋 Getting Started
+
+### 1. Fork & Clone
+
+First, [fork Semantica](https://github.com/Hawksight-AI/semantica/fork) on GitHub, then:
 
 ```bash
-python -c "import semantica; print(semantica.__version__)"
-pytest --version
-black --version
+git clone https://github.com/your-username/semantica.git
+cd semantica
+git remote add upstream https://github.com/Hawksight-AI/semantica.git
 ```
 
-## Code Style Guidelines
-
-We use several tools to maintain code quality and consistency:
-
-### Formatting
-
-- **Black**: Code formatting (line length: 88)
-  ```bash
-  black semantica/
-  ```
-
-- **isort**: Import sorting
-  ```bash
-  isort semantica/
-  ```
-
-### Linting
-
-- **flake8**: Style guide enforcement
-  ```bash
-  flake8 semantica/
-  ```
-
-- **mypy**: Static type checking
-  ```bash
-  mypy semantica/
-  ```
-
-### Running All Checks
+### 2. Set Up Environment
 
 ```bash
-# Format code
-black semantica/ tests/
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # Windows: venv\Scripts\activate
 
-# Sort imports
-isort semantica/ tests/
+# Install dev dependencies
+pip install -e ".[dev]"
 
-# Lint
-flake8 semantica/ tests/
-
-# Type check
-mypy semantica/
+# Install pre-commit hooks (optional)
+pre-commit install
 ```
 
-Or use pre-commit hooks (automatically runs on commit):
-```bash
-pre-commit run --all-files
-```
-
-## Testing Requirements
-
-### Running Tests
+### 3. Create Branch
 
 ```bash
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=semantica --cov-report=html
-
-# Run specific test file
-pytest tests/test_specific.py
-
-# Run with verbose output
-pytest -v
+git checkout -b feature/your-feature-name
+# or
+git checkout -b fix/bug-description
 ```
 
-### Test Coverage
+### 4. Make Changes
 
-- Minimum coverage: **80%**
-- Critical modules: **90%+**
-- Coverage reports are generated in `htmlcov/`
+- Follow code style (see below)
+- Add tests for new features
+- Update documentation
 
-### Writing Tests
+### 5. Run Checks
 
-- Follow pytest conventions
-- Use descriptive test names
-- Include docstrings for complex tests
-- Test both success and failure cases
-- Use fixtures for common setup
-
-Example:
-```python
-def test_entity_extraction():
-    """Test basic entity extraction functionality."""
-    from semantica.semantic_extract import NamedEntityRecognizer
-    
-    ner = NamedEntityRecognizer()
-    entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
-    
-    assert len(entities) > 0
-    assert any(e.text == "Apple Inc." for e in entities)
+```bash
+pytest                          # Run tests
+black semantica/ tests/        # Format code
+isort semantica/ tests/         # Sort imports
+flake8 semantica/ tests/        # Lint
 ```
 
-## Commit Message Conventions
+Or use pre-commit hooks: `pre-commit run --all-files`
 
-We follow [Conventional Commits](https://www.conventionalcommits.org/) specification:
+### 6. Commit & Push
 
-### Format
-
-```
-<type>(<scope>): <subject>
-
-<body>
-
-<footer>
+```bash
+git commit -m "feat(module): add new feature"
+git push origin feature/your-feature-name
 ```
 
-### Types
+Then create a pull request on GitHub!
 
-- `feat`: New feature
-- `fix`: Bug fix
-- `docs`: Documentation changes
-- `style`: Code style changes (formatting, etc.)
-- `refactor`: Code refactoring
-- `test`: Adding or updating tests
-- `chore`: Maintenance tasks
-- `perf`: Performance improvements
-- `ci`: CI/CD changes
+---
 
-### Examples
+## 📐 Code Style
+
+We use automated tools:
+
+| Tool     | Purpose                    | Command                    |
+|----------|----------------------------|----------------------------|
+| **Black** | Code formatting            | `black semantica/ tests/` |
+| **isort** | Import sorting             | `isort semantica/ tests/` |
+| **flake8** | Style enforcement          | `flake8 semantica/ tests/` |
+| **mypy** | Type checking              | `mypy semantica/`          |
+
+**Run all:** `black semantica/ tests/ && isort semantica/ tests/ && flake8 semantica/ tests/ && mypy semantica/`
+
+---
+
+## 🧪 Testing
+
+```bash
+pytest                          # Run all tests
+pytest --cov=semantica         # With coverage
+pytest tests/test_file.py      # Specific file
+```
+
+**Coverage goal:** 80% minimum, 90%+ for critical modules
+
+---
+
+## 📝 Commit Messages
+
+Use [Conventional Commits](https://www.conventionalcommits.org/):
 
 ```
 feat(kg): add temporal graph support
-
-Add support for temporal knowledge graphs with version tracking
-and time-based queries.
-
-Closes #123
+fix(parse): handle empty PDF files
+docs(readme): add installation guide
+test(extract): add unit tests
 ```
 
-```
-fix(parse): handle empty PDF files gracefully
+**Types:** `feat`, `fix`, `docs`, `test`, `refactor`, `perf`, `style`, `chore`
 
-Previously, empty PDF files would cause a crash. Now they return
-an empty document with appropriate warnings.
+---
 
-Fixes #456
-```
+## ✅ PR Checklist
 
-## Pull Request Process
-
-### Before Submitting
-
-1. **Update your fork**:
-   ```bash
-   git fetch upstream
-   git checkout main
-   git merge upstream/main
-   ```
-
-2. **Create a feature branch**:
-   ```bash
-   git checkout -b feature/your-feature-name
-   # or
-   git checkout -b fix/bug-description
-   ```
-
-3. **Make your changes** and commit following our conventions
-
-4. **Run all checks**:
-   ```bash
-   pytest
-   black semantica/ tests/
-   isort semantica/ tests/
-   flake8 semantica/ tests/
-   mypy semantica/
-   ```
-
-5. **Push to your fork**:
-   ```bash
-   git push origin feature/your-feature-name
-   ```
-
-### PR Checklist
+Before submitting:
 
 - [ ] Code follows style guidelines
 - [ ] Tests pass locally
-- [ ] New tests added for new features
+- [ ] New tests added (if applicable)
 - [ ] Documentation updated
 - [ ] Commit messages follow conventions
 - [ ] No merge conflicts
-- [ ] PR description is clear and complete
 
-### PR Description Template
+---
 
-```markdown
-## Description
-Brief description of changes
+## 📖 Documentation Standards
 
-## Type of Change
-- [ ] Bug fix
-- [ ] New feature
-- [ ] Breaking change
-- [ ] Documentation update
+### Code Documentation (Docstrings)
 
-## Related Issues
-Closes #123
-Related to #456
+**Format:** Use Google-style docstrings
 
-## Testing
-- [ ] Tests pass locally
-- [ ] Added new tests
-- [ ] Updated existing tests
-
-## Checklist
-- [ ] Code follows style guidelines
-- [ ] Self-review completed
-- [ ] Comments added for complex code
-- [ ] Documentation updated
-- [ ] No new warnings generated
-```
-
-## Documentation Standards
-
-### Code Documentation
-
-- Use Google-style docstrings
-- Include type hints
-- Document all public functions and classes
-- Include examples for complex functions
-
-Example:
 ```python
-def extract_entities(
-    text: str,
-    model: str = "transformer",
-    confidence_threshold: float = 0.7
-) -> List[Entity]:
+def extract_entities(text: str, model: str = "transformer") -> List[Entity]:
     """Extract named entities from text.
     
     Args:
         text: Input text to process
         model: NER model to use (default: "transformer")
-        confidence_threshold: Minimum confidence score (default: 0.7)
     
     Returns:
         List of extracted Entity objects
@@ -309,92 +269,98 @@ def extract_entities(
         ValueError: If text is empty or model is invalid
     
     Example:
-        >>> ner = NamedEntityRecognizer()
+        >>> from semantica.semantic_extract import NERExtractor
+        >>> ner = NERExtractor(method="ml", model="en_core_web_sm")
         >>> entities = ner.extract("Apple Inc. was founded in 1976.")
         >>> len(entities)
         2
     """
-    ...
 ```
 
-### Documentation Files
+### Markdown Documentation Formatting
 
-- Update relevant documentation in `docs/`
-- Add examples to cookbook if applicable
-- Update API reference if adding new public APIs
-- Keep README.md up to date
+**General Guidelines:**
+- Use clear headings (H1 for title, H2 for main sections, H3 for subsections)
+- Keep paragraphs short and focused
+- Use bullet points for lists
+- Add code blocks with syntax highlighting
+- Include links to related documentation
 
-## Types of Contributions
+**Code Blocks:**
+- Use triple backticks with language identifier: ` ```python `, ` ```bash `
+- Include comments in code examples
+- Show expected output when helpful
 
-### 💻 Code Contributions
+**Examples:**
 
-- **Bug Fixes**: Resolving issues reported in the issue tracker.
-- **New Features**: Implementing new capabilities (please discuss via an issue first!).
-- **Refactoring**: Improving code structure and maintainability without changing behavior.
-- **Algorithm Optimization**: Improving the efficiency of graph algorithms and vector search.
+```markdown
+## Section Title
 
-#### ⚡ Performance and Latency
-We deeply value efficiency. Contributions that make Semantica faster and lighter are highly appreciated!
+Brief introduction paragraph.
 
-- **Latency Reduction**: Optimize critical paths and RAG pipeline response times.
-- **Memory Optimization**: Reduce graph/vector processing memory footprint.
-- **Throughput**: Improve operations per second (bulk ingestion, parallel queries).
-- **Benchmarks**: Add performance benchmarks to track regressions.
-- **Async/Concurrency**: Enhance asynchronous execution and concurrency.
+### Subsection
 
-### 📚 Documentation Contributions
+- Bullet point 1
+- Bullet point 2
 
-- Fix typos and grammar
-- Improve clarity
-- Add examples
-- Create tutorials
-- Translate documentation
+**Code example:**
 
-### Testing Contributions
+```python
+from semantica import SomeClass
 
-- Add test coverage
-- Improve test quality
-- Add integration tests
-- Performance benchmarks
+instance = SomeClass()
+result = instance.method()
+```
 
-### Other Contributions
+**Note:** Additional context or warnings.
+```
 
-- Answer questions in discussions
-- Help with issues
-- Review pull requests
-- Share use cases
-- Report bugs
-- Suggest features
+**Best Practices:**
+- Start with an overview/introduction
+- Use consistent terminology
+- Include "See also" links
+- Add examples for complex concepts
+- Keep formatting consistent across docs
 
-## Getting Help
+---
 
-### Communication Channels
+## 🆘 Getting Help
 
-- **GitHub Discussions**: General questions and discussions
-- **GitHub Issues**: Bug reports and feature requests
-- **Discord**: Real-time chat and community support
+- 💬 [Discord](https://discord.gg/vqRt2qbx) - Real-time chat
+- 💭 [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) - Q&A
+- 🐛 [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) - Bug reports
 
-### Before Asking for Help
+**Before asking:** Check existing documentation, search issues/discussions, review cookbook examples
 
-1. Check existing documentation
-2. Search GitHub issues and discussions
-3. Review code examples in cookbook
-4. Check FAQ in documentation
+---
 
-### Asking Good Questions
+## 🏆 Recognition
 
-- Provide context and environment details
-- Include code examples
-- Show what you've tried
-- Include error messages and logs
-- Be specific about what you need
-
-## Recognition
-
-Contributors are recognized in:
+All contributors are recognized in:
 - [CONTRIBUTORS.md](CONTRIBUTORS.md)
 - GitHub contributors page
-- Release notes for significant contributions
+- Release notes
 
-Thank you for contributing to Semantica! 🎉
+We follow the [all-contributors](https://allcontributors.org) specification!
 
+---
+
+## 📜 Code of Conduct
+
+This project follows a [Code of Conduct](CODE_OF_CONDUCT.md). Be respectful and inclusive.
+
+---
+
+## 📚 Resources
+
+- [README.md](README.md) - Project overview
+- [Cookbook](cookbook/) - Tutorials and examples
+- [Documentation](docs/) - Comprehensive guides
+
+---
+
+**Thank you for contributing!** 🚀
+
+Every contribution matters - whether it's a single line of code, a typo fix, a helpful answer, or a bug report. We appreciate you! 🙏
+
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**

CONTRIBUTORS.md

@@ -4,44 +4,31 @@ Thank you to all the people who have contributed to Semantica! 🎉
 
 This project follows the [all-contributors](https://allcontributors.org) specification. Contributions of any kind are welcome!
 
-## How to Contribute
+⭐ **Give us a Star** • 🍴 **Fork us** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-We welcome contributions of all kinds! Whether you're:
-- Writing code
-- Improving documentation
-- Reporting bugs
-- Suggesting features
-- Answering questions
-- Reviewing pull requests
-- Sharing use cases
-- Creating examples
-
-All contributions are valuable and appreciated!
+---
 
 ## Contribution Types
 
+We recognize all types of contributions:
+
 - 💻 **Code**: Writing code, fixing bugs, implementing features
 - 📝 **Documentation**: Writing docs, tutorials, examples
+- 🧪 **Testing**: Writing tests, improving test coverage
 - 🐛 **Bug Reports**: Finding and reporting bugs
 - 💡 **Ideas**: Suggesting new features or improvements
 - 🎨 **Design**: UI/UX improvements, graphics, branding
-- 📖 **Examples**: Creating code examples and tutorials
-- 🔍 **Testing**: Writing tests, improving test coverage
-- 💬 **Answering Questions**: Helping others in discussions
-- 📢 **Talks**: Giving talks, presentations, workshops
-- 🌍 **Translation**: Translating documentation
+- 🎨 **Cookbook**: Creating tutorials and examples
+- 💬 **Community**: Answering questions, reviewing PRs
+- 🎓 **Education**: Blog posts, video tutorials, talks, workshops
 - 🔧 **Tools**: Creating tools, scripts, integrations
 - 📦 **Packaging**: Improving build, release, distribution
 - ⚠️ **Security**: Reporting security vulnerabilities
-- 🎓 **Education**: Teaching, mentoring, tutorials
-- 📹 **Video**: Creating video content, tutorials
-- 🎵 **Audio**: Podcasts, audio content
-- 📸 **Photography**: Screenshots, images
-- 🔬 **Research**: Research, analysis, studies
-- 💰 **Financial**: Sponsoring, funding
 - 🏗️ **Infrastructure**: CI/CD, hosting, infrastructure
 - 🚇 **Maintenance**: Maintenance, triage, project management
 
+---
+
 ## Contributors
 
 <!-- ALL-CONTRIBUTORS-LIST:START -->
@@ -50,48 +37,78 @@ All contributions are valuable and appreciated!
 
 <!-- ALL-CONTRIBUTORS-LIST:END -->
 
+---
+
 ## Recognition
 
-### Top Contributors
+All contributors are recognized in:
 
-Contributors are recognized based on their contributions to the project. Recognition includes:
+- This contributors list
+- [GitHub contributors page](https://github.com/Hawksight-AI/semantica/graphs/contributors)
+- Release notes for significant contributions
+- Community appreciation
 
-- Listing in this file
-- GitHub contributor statistics
-- Special mentions in release notes
-- Featured showcases for significant contributions
-
-### Hall of Fame
-
-Special recognition for exceptional contributions:
-
-- **Coming soon** - We'll feature outstanding contributors here!
+---
 
 ## How to Add Yourself
 
-If you've contributed to Semantica and want to be added to this list:
+### Automatic Recognition
 
-1. **Automatic**: If you've made a commit, you'll appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors)
-2. **Manual**: Open a PR adding yourself to this file, or use the [@all-contributors bot](https://allcontributors.org/docs/en/bot/usage)
+If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors).
 
-Example:
-```markdown
-- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
-```
+### Using All-Contributors Bot
 
-## All Contributors Bot
-
-We use the [all-contributors](https://allcontributors.org) bot to automatically recognize contributors. To add a contributor, comment on an issue or PR:
+Comment on any issue or PR with:
 
 ```
 @all-contributors please add @username for code, docs, bug
 ```
 
-## Thank You!
+**Examples:**
 
-Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community!
+```
+@all-contributors please add @johndoe for code
+@all-contributors please add @janedoe for docs, bug
+@all-contributors please add @devuser for code, test, maintenance
+```
+
+### Manual Addition
+
+Open a PR adding yourself to this file:
+
+```markdown
+- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
+```
 
 ---
 
-**Want to contribute?** Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
+## Contribution Type Codes
 
+When using the all-contributors bot, use these codes:
+
+- `code` - Code contributions
+- `doc` - Documentation
+- `test` - Testing
+- `bug` - Bug reports
+- `ideas` - Feature requests/ideas
+- `design` - Design work
+- `example` - Cookbook/examples
+- `question` - Answering questions
+- `talk` - Talks/presentations
+- `tool` - Tools/integrations
+- `packaging` - Packaging/distribution
+- `security` - Security reports
+- `infra` - Infrastructure
+- `maintenance` - Maintenance
+
+See [all-contributors specification](https://allcontributors.org/docs/en/emoji-key) for complete list.
+
+---
+
+## Thank You!
+
+Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community! 🙏
+
+**Want to contribute?** 
+
+⭐ Give us a Star • 🍴 [Fork us](https://github.com/Hawksight-AI/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!

README.md

@@ -1,204 +1,237 @@
 <div align="center">
 
-<img src="semantica_logo.png" alt="Semantica Logo" width="450" height="auto">
+<img src="Semantica Updated Logo.png" alt="Semantica Logo" width="460"/>
 
 # 🧠 Semantica
+### Open-Source Semantic Layer & Knowledge Engineering Framework
 
-[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/)
-[![Monthly Downloads](https://img.shields.io/pypi/dm/semantica)](https://pypi.org/project/semantica/)
+[![PyPI](https://img.shields.io/pypi/v/semantica.svg)](https://pypi.org/project/semantica/)
 [![Total Downloads](https://static.pepy.tech/badge/semantica)](https://pepy.tech/project/semantica)
-[![Discord](https://img.shields.io/badge/Discord-Join%20Us-7289da?style=flat&logo=discord&logoColor=white)](https://discord.gg/pMHguUzG)
 [![CI](https://github.com/Hawksight-AI/semantica/workflows/CI/badge.svg)](https://github.com/Hawksight-AI/semantica/actions)
+[![Discord](https://img.shields.io/badge/Discord-Join-7289da?logo=discord&logoColor=white)](https://discord.gg/RgaGTj9J)
 
-<p align="center">
-    <a href="https://github.com/Hawksight-AI/semantica/stargazers">
-        <img src="https://img.shields.io/badge/Give%20a%20Star-%E2%AD%90-yellow?style=for-the-badge&labelColor=555555" alt="Give a Star">
-    </a>
-    &nbsp;&nbsp;
-    <a href="https://github.com/Hawksight-AI/semantica/fork">
-        <img src="https://img.shields.io/badge/Support%20Project-Fork%20Us-blue?style=for-the-badge&labelColor=555555" alt="Support Project">
-    </a>
-</p>
+### ⭐ Give us a Star • 🍴 Fork us • 💬 Join our Discord
 
-**Open Source Framework for Semantic Layer & Knowledge Engineering**
-
-> **Transform chaotic data into intelligent knowledge.**
-
-*The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.*
-
-**100% Open Source** • **MIT Licensed** • **Latest Version: 0.2.0** • **Production Ready** • **Community Driven**
-
-[**Discord**](https://discord.gg/pMHguUzG)
+> **Transform Chaos into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.**
 
 </div>
 
-## What is Semantica?
-
-Semantica bridges the gap between raw data chaos and AI-ready knowledge. It's a **semantic intelligence platform** that transforms unstructured data into structured, queryable knowledge graphs powering GraphRAG, AI agents, and multi-agent systems.
-
-### What Makes Semantica Different?
-
-Unlike traditional approaches that process isolated documents and extract text into vectors, Semantica understands **semantic relationships across all content**, provides **automated ontology generation**, and builds a **unified semantic layer** with **production-grade QA**.
-
-| **Traditional Approaches** | **Semantica's Approach** |
-|:---------------------------|:-------------------------|
-| Process data as isolated documents | Understands semantic relationships across all content |
-| Extract text and store vectors | Builds knowledge graphs with meaningful connections |
-| Generic entity recognition | General-purpose ontology generation and validation |
-| Manual schema definition | Automatic semantic modeling from content patterns |
-| Disconnected data silos | Unified semantic layer across all data sources |
-| Basic quality checks | Production-grade QA with conflict detection & resolution |
 
 ---
 
-## 🎯 The Problem We Solve
+## 🚀 Why Semantica?
 
-### The Semantic Gap
+**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy.
 
-Organizations today face a **fundamental mismatch** between how data exists and how AI systems need it.
-
-#### The Semantic Gap: Problem vs. Solution
-
-Organizations have **unstructured data** (PDFs, emails, logs), **messy data** (inconsistent formats, duplicates, conflicts), and **disconnected silos** (no shared context, missing relationships). AI systems need **clear rules** (formal ontologies), **structured entities** (validated, consistent), and **relationships** (semantic connections, context-aware reasoning).
-
-| **What Organizations Have** | **What AI Systems Require** |
-|:------------------------------|:------------------------------|
-| **Unstructured Data** | **Clear Rules** |
-| PDFs, emails, logs | Formal ontologies |
-| Mixed schemas | Graphs & Networks |
-| Conflicting facts | |
-| **Messy, Noisy Data** | **Structured Entities** |
-| Inconsistent formats | Validated entities |
-| Duplicate records | Domain Knowledge |
-| Missing relationships | |
-| **Disconnected, Siloed Data** | **Relationships** |
-| Data in separate systems | Semantic connections |
-| No shared context | Context-Aware Reasoning |
-| Isolated knowledge | |
-
-### **SEMANTICA FRAMEWORK**
-
-Semantica operates through three integrated layers that transform raw data into AI-ready knowledge:
-
-**Input Layer** — Universal ingestion from multiple data formats (PDFs, DOCX, HTML, JSON, CSV, databases, live feeds, APIs, streams, archives, multi-modal content) into a unified pipeline.
-
-**Semantic Layer** — Core intelligence engine performing entity extraction, relationship mapping, ontology generation, context engineering, and quality assurance. Includes **advanced entity deduplication** (Jaro-Winkler, disjoint property handling) to ensure a clean single source of truth.
-
-**Output Layer** — Production-ready knowledge graphs, vector embeddings, and validated ontologies that power GraphRAG systems, AI agents, and multi-agent systems.
-
-**Powers: GraphRAG, AI Agents, Multi-Agent Systems**
-
-
-
-
-
-### What Happens Without Semantics?
-
-**They Break** — Systems crash due to inconsistent formats and missing structure.
-
-**They Hallucinate** — AI models generate false information without semantic context to validate outputs.
-
-**They Fail Silently** — Systems return wrong answers without warnings, leading to bad decisions.
-
-**Why?** Systems have data — not semantics. They can't connect concepts, understand relationships, validate against domain rules, or detect conflicts.
+Perfect for **high-stakes domains** where mistakes have real consequences.
 
 ---
 
-## 💡 The Semantica Solution
+### ⚡ Get Started in 30 Seconds
 
-**Semantica** is an **open-source framework** that closes the semantic gap between real-world messy data and the structured semantic layers required by advanced AI systems — GraphRAG, agents, multi-agent systems, reasoning models, and more.
+```bash
+pip install semantica
+```
 
-### How Semantica Solves These Problems
+```python
+from semantica.semantic_extract import NERExtractor
+from semantica.kg import GraphBuilder
 
-**Efficient Embeddings** — Uses **FastEmbed** by default for high-performance, lightweight local embedding generation (faster than sentence-transformers).
+# Extract entities and build knowledge graph
+ner = NERExtractor(method="ml", model="en_core_web_sm")
+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
+kg = GraphBuilder().build({"entities": entities, "relationships": []})
 
-**Universal Data Ingestion** — Handles multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams) with unified pipeline, no custom parsers needed.
+print(f"Built KG with {len(kg.get('entities', []))} entities")
+```
 
-**Automated Semantic Extraction** — NER, relationship extraction, and triplet generation with LLM enhancement. Includes **auto-chunking** for long documents and **robust error handling** with automatic retry logic.
-
-**Knowledge Graph Construction** — Production-ready graphs with entity resolution, temporal support, and graph analytics. Queryable knowledge ready for AI applications.
-
-**GraphRAG Engine** — Hybrid vector + graph retrieval achieves 91% accuracy (30% improvement) via semantic search + graph traversal for multi-hop reasoning. Features LLM-generated responses grounded in knowledge graph context with reasoning traces. [See Comparison Benchmark](cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
-
-**AI Agent Context Engineering** — Persistent memory with RAG + knowledge graphs enables context maintenance, action validation, and structured knowledge access.
-
-**Automated Ontology Generation** — 6-stage LLM pipeline generates validated OWL ontologies with HermiT/Pellet validation, eliminating manual engineering.
-
-**Production-Grade QA** — Conflict detection, deduplication, quality scoring, and provenance tracking ensure trusted, production-ready knowledge graphs.
-
-**Pipeline Orchestration** — Flexible pipeline builder with parallel execution enables scalable processing via orchestrator-worker pattern.
-
-### Core Features at a Glance
-
-| **Feature Category** | **Capabilities** | **Key Benefits** |
-|:---------------------|:-----------------|:------------------|
-| **Data Ingestion** | Multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams, archives) | Universal ingestion, no custom parsers needed |
-| **Semantic Extraction** | NER, relations, triplets, LLM enhancement, **auto-chunking** | Automated discovery with robust error handling |
-| **Knowledge Graphs** | Entity resolution, temporal support, graph analytics, query interface | Production-ready, queryable knowledge structures |
-| **Ontology Generation** | 6-stage LLM pipeline, OWL generation, HermiT/Pellet validation | Automated ontology creation from documents |
-| **GraphRAG** | Hybrid vector + graph retrieval, multi-hop reasoning, LLM-generated responses | 91% accuracy, 30% improvement over vector-only, reasoning traces |
-| **LLM Providers** | Unified interface to 100+ LLMs (Groq, OpenAI, HuggingFace, LiteLLM) | Clean imports, multiple providers, structured output |
-| **Agent Memory** | Persistent memory (Save/Load), Hybrid Retrieval (Vector+Graph), FastEmbed support | Context-aware agents with semantic understanding |
-| **Pipeline Orchestration** | Parallel execution, custom steps, orchestrator-worker pattern | Scalable, flexible data processing |
-| **Quality Assurance** | Conflict detection, deduplication, quality scoring, provenance | Trusted knowledge graphs ready for production |
+**[📖 Full Quick Start](#-quick-start)** • **[🍳 Cookbook Examples](#-semantica-cookbook)** • **[💬 Join Discord](https://discord.gg/RgaGTj9J)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)**
 
 ---
 
-## 👥 Who Is This For?
+## Core Value Proposition
 
-Semantica is designed for **developers, data engineers, and organizations** building the next generation of AI applications that require semantic understanding and knowledge graphs.
+| **Trustworthy** | **Explainable** | **Auditable** |
+|:------------------:|:------------------:|:-----------------:|
+| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking |
+| Rule-based governance | Entity relationships & ontologies | W3C PROV-O compliant lineage |
+| Production-grade QA | Multi-hop graph reasoning | Source tracking & integrity verification |
 
-### Who Uses Semantica
+---
 
-**AI/ML Engineers & Data Scientists** — Build GraphRAG systems, AI agents, and multi-agent systems.
+## Key Features & Benefits
 
-**Data Engineers** — Build scalable pipelines with semantic enrichment.
+### Not Just Another Agentic Framework
 
-**Knowledge Engineers & Ontologists** — Create knowledge graphs and ontologies with automated pipelines.
+**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with:
 
-**Enterprise Data Teams** — Unify semantic layers, improve data quality, resolve conflicts.
+| Feature | Benefit |
+|:--------|:--------|
+| **Auditable** | Complete provenance tracking with W3C PROV-O compliance |
+| **Explainable** | Transparent reasoning paths with entity relationships |
+| **Provenance-Aware** | End-to-end lineage from documents to responses |
+| **Validated** | Built-in conflict detection, deduplication, QA |
+| **Governed** | Rule-based validation and semantic consistency |
+| **Version Control** | Enterprise-grade change management with integrity verification |
 
-**Software & DevOps Engineers** — Build semantic APIs and infrastructure with production-ready SDK.
+### Perfect For High-Stakes Use Cases
 
-**Analysts & Researchers** — Transform data into queryable knowledge graphs for insights.
+| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** |
+|:-----------------:|:--------------:|:------------:|
+| Clinical decisions | Fraud detection | Evidence-backed research |
+| Drug interactions | Regulatory support | Contract analysis |
+| Patient safety | Risk assessment | Case law reasoning |
 
-**Security & Compliance Teams** — Threat intelligence, regulatory reporting, audit trails.
+| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** |
+|:-------------------:|:----------------:|:-------------------:|:-----------------:|
+| Threat attribution | Policy decisions | Power grids | Decision logs |
+| Incident response | Classified info | Transportation | Safety validation |
 
-**Product Teams & Startups** — Rapid prototyping of AI products and semantic features.
+### Powers Your AI Stack
+
+- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search
+- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory
+- **Reasoning Models** — Explainable AI decisions with reasoning paths
+- **Enterprise AI** — Governed, auditable platforms that support compliance
+
+### Integrations
+
+- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)
+- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication
+- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)
+
+> **Built for environments where every answer must be explainable and governed.**
+
+
+---
+
+## 🚨 The Problem: The Semantic Gap
+
+### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.
+
+### Understanding the Semantic Gap
+
+The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).
+
+**Traditional AI approaches:**
+- Rely on statistical patterns and text similarity
+- Cannot understand relationships between entities
+- Cannot reason about domain-specific rules
+- Cannot explain why decisions were made
+- Cannot trace back to original sources with confidence
+
+**High-stakes AI requires:**
+- Semantic understanding of entities and their relationships
+- Domain knowledge encoded as formal rules (ontologies)
+- Explainable reasoning paths
+- Source-level provenance
+- Conflict detection and resolution
+
+**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.
+
+### What Organizations Have vs What They Need
+
+| **Current State** | **Required for High-Stakes AI** |
+|:---------------------|:-----------------------------------|
+| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |
+| APIs, databases, streams | Structured and validated entities |
+| Conflicting facts and duplicates | Explicit semantic relationships |
+| Siloed systems with no lineage | **Explainable reasoning paths** |
+| | **Source-level provenance** |
+| | **Audit-ready compliance** |
+
+### The Cost of Missing Semantics
+
+- **Decisions cannot be explained** — No transparency in AI reasoning
+- **Errors cannot be traced** — No way to debug or improve
+- **Conflicts go undetected** — Contradictory information causes failures
+- **Compliance becomes impossible** — No audit trails for regulations
+
+**Trustworthy AI requires semantic accountability.**
+
+---
+
+## 🆚 Semantica vs Traditional RAG
+
+| Feature | Traditional RAG | Semantica |
+|:--------|:----------------|:----------|
+| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths |
+| **Provenance** | ❌ No provenance | ✅ W3C PROV-O compliant lineage tracking |
+| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning |
+| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection |
+| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments |
+| **Compliance** | ❌ No audit trails | ✅ Complete audit trails with integrity verification |
+
+---
+
+## 🧩 Semantica Architecture
+
+### 1️⃣ Input Layer — Governed Ingestion
+- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX
+- 🔧 **Docling Support** — Docling parser for table extraction
+- 💾 **Data Sources** — Databases, APIs, streams, archives, web content
+- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction
+- 📊 **Single Pipeline** — Unified ingestion with metadata and source tracking
+
+### 2️⃣ Semantic Layer — Trust & Reasoning Engine
+- 🔍 **Entity Extraction** — NER, normalization, classification
+- 🔗 **Relationship Discovery** — Triplet generation, semantic links
+- 📐 **Ontology Induction** — Automated domain rule generation
+- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution
+- ✅ **Quality Assurance** — Conflict detection, validation
+- 📊 **Provenance Tracking** — W3C PROV-O compliant lineage tracking across all modules
+- 🧠 **Reasoning Traces** — Explainable inference paths
+- 🔐 **Change Management** — Version control with audit trails, checksums, compliance support
+
+### 3️⃣ Output Layer — Auditable Knowledge Assets
+- 📊 **Knowledge Graphs** — Queryable, temporal, explainable
+- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support
+- 🔢 **Vector Embeddings** — FastEmbed by default
+- ☁️ **AWS Neptune** — Amazon Neptune graph database support
+- 🔍 **Provenance** — Every AI response links back to:
+  - 📄 Source documents
+  - 🏷️ Extracted entities & relations
+  - 📐 Ontology rules applied
+  - 🧠 Reasoning steps used
+
+---
+
+## 🏥 Built for High-Stakes Domains
+
+Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:
+
+- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking
+- **💰 Finance & Risk** — Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation
+- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning
+- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis
+- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence
+- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response
+- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation  
+
+---
+
+## 👥 Who Uses Semantica?
+
+- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents
+- **⚙️ Data Engineers** — Creating governed semantic pipelines
+- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale
+- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure
+- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems  
 
 ---
 
 ## 📦 Installation
 
-> **✅ Available on PyPI!** Semantica is now published on PyPI. Install it with a single command: `pip install semantica`
-
-**Prerequisites:** Python 3.8+ (3.9+ recommended) • pip (latest version)
-
 ### Install from PyPI (Recommended)
 
 ```bash
-# Install latest version from PyPI
 pip install semantica
-
-# Or install with optional dependencies
+# or
 pip install semantica[all]
-
-# GitHub Workaround (if PyPI version has issues)
-pip install git+https://github.com/Hawksight-AI/semantica.git@main
-
-# Verify installation
-python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')"
 ```
 
-**Current Version:** [![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/) • [View on PyPI](https://pypi.org/project/semantica/)
-
-!!! info "Windows PyTorch Note"
-    If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica.
-
-
-
 ### Install from Source (Development)
 
 ```bash
@@ -258,7 +291,7 @@ print(f" Ingested {len(sources)} sources")
 
 ### Document Parsing & Processing
 
-> **Multi-format parsing** • **Text normalization** • **Intelligent chunking**
+> **Multi-format parsing** • **Docling Support** • **Text normalization** • **Intelligent chunking**
 
 ```python
 from semantica.parse import DocumentParser, DoclingParser
@@ -269,7 +302,7 @@ from semantica.split import TextSplitter
 parser = DocumentParser()
 parsed = parser.parse("document.pdf", format="auto")
 
-# Enhanced parsing with Docling (recommended for complex layouts/tables)
+# Parsing with Docling (for complex layouts/tables)
 # Requires: pip install docling
 docling_parser = DoclingParser(enable_ocr=True)
 result = docling_parser.parse("complex_table.pdf")
@@ -360,7 +393,7 @@ results = vector_store.search(query="supply chain", top_k=5)
 
 ### Graph Store & Triplet Store
 
-> **Neo4j, FalkorDB, Amazon Neptune support** • **SPARQL queries** • **RDF triplets**
+> **Neo4j, FalkorDB, Amazon Neptune** • **SPARQL queries** • **RDF triplets**
 
 ```python
 from semantica.graph_store import GraphStore
@@ -398,19 +431,137 @@ results = triplet_store.execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT
 
 ### Ontology Generation & Management
 
-> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation
+> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation • **Custom Ontology Import** (OWL, RDF, Turtle, JSON-LD)
 
 ```python
 from semantica.ontology import OntologyGenerator
+from semantica.ingest import ingest_ontology
 
+# Generate ontology automatically
 generator = OntologyGenerator(llm_provider="openai", model="gpt-4")
 ontology = generator.generate_from_documents(sources=["domain_docs/"])
 
-print(f"Classes: {len(ontology.classes)}")
+# Or import your existing ontology
+custom_ontology = ingest_ontology("my_ontology.ttl")  # Supports OWL, RDF, Turtle, JSON-LD
+print(f"Classes: {len(custom_ontology.classes)}")
 ```
 
 [**Cookbook: Ontology**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/14_Ontology.ipynb)
 
+### Change Management & Version Control
+
+> **Version Control for Knowledge Graphs & Ontologies** • **SQLite & In-Memory Storage** • **SHA-256 Integrity Verification**
+
+```python
+from semantica.change_management import TemporalVersionManager, OntologyVersionManager
+
+# Knowledge Graph versioning with audit trails
+kg_manager = TemporalVersionManager(storage_path="kg_versions.db")
+
+# Create versioned snapshot
+snapshot = kg_manager.create_snapshot(
+    knowledge_graph,
+    version_label="v1.0",
+    author="user@company.com",
+    description="Initial patient record"
+)
+
+# Compare versions with detailed diffs
+diff = kg_manager.compare_versions("v1.0", "v2.0")
+print(f"Entities added: {diff['summary']['entities_added']}")
+print(f"Entities modified: {diff['summary']['entities_modified']}")
+
+# Verify data integrity
+is_valid = kg_manager.verify_checksum(snapshot)
+```
+
+**What We Provide:**
+- 🔐 **Persistent Storage** — SQLite and in-memory backends implemented
+- 📊 **Detailed Diffs** — Entity-level and relationship-level change tracking
+- ✅ **Data Integrity** — SHA-256 checksums with tamper detection
+- 📝 **Standardized Metadata** — ChangeLogEntry with author, timestamp, description
+- ⚡ **Performance Tested** — Benchmarked with 10k entities
+- 🧪 **Test Coverage** — 104 tests covering core functionality
+
+**Compliance Note:** Provides technical infrastructure (audit trails, checksums, temporal tracking) that supports compliance efforts for HIPAA, SOX, FDA 21 CFR Part 11. Organizations must implement additional policies and procedures for full regulatory compliance.
+
+[**Documentation: Change Management**](docs/reference/change_management.md) • [**Usage Guide**](semantica/change_management/change_management_usage.md)
+
+### Provenance Tracking — W3C PROV-O Compliant Lineage
+
+> **W3C PROV-O Implementation** • **17 Module Integrations** • **Opt-In Design** • **Zero Breaking Changes**
+
+**⚠️ Compliance Note:** Provides technical infrastructure for provenance tracking that supports compliance efforts. Organizations must implement additional policies, procedures, and controls for full regulatory compliance.
+
+```python
+from semantica.semantic_extract.semantic_extract_provenance import NERExtractorWithProvenance
+from semantica.llms.llms_provenance import GroqLLMWithProvenance
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+
+# Enable provenance tracking - just add provenance=True
+ner = NERExtractorWithProvenance(provenance=True)
+entities = ner.extract(
+    text="Apple Inc. was founded by Steve Jobs.",
+    source="biography.pdf"
+)
+
+# Track LLM calls with costs and latency
+llm = GroqLLMWithProvenance(provenance=True, model="llama-3.1-70b")
+response = llm.generate("Summarize the document")
+
+# Store in graph with complete lineage
+graph = GraphStoreWithProvenance(provenance=True)
+graph.add_node(entity, source="biography.pdf")
+
+# Retrieve complete provenance
+lineage = ner._prov_manager.get_lineage("entity_id")
+print(f"Source: {lineage['source']}")
+print(f"Lineage chain: {lineage['lineage_chain']}")
+```
+
+**What We Provide:**
+- ✅ **W3C PROV-O Implementation** — Data schemas implementing prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom
+- ✅ **17 Module Integrations** — Provenance-enabled versions of semantic extract, LLMs, pipeline, context, ingest, embeddings, reasoning, conflicts, deduplication, export, parse, normalize, ontology, visualization, graph/vector/triplet stores
+- ✅ **Opt-In Design** — Zero breaking changes, `provenance=False` by default
+- ✅ **Lineage Tracking** — Document → Chunk → Entity → Relationship → Graph lineage chains
+- ✅ **LLM Tracking** — Token counts, costs, and latency tracking for LLM calls
+- ✅ **Source Tracking Fields** — Document identifiers, page numbers, sections, and quote fields in schemas
+- ✅ **Storage Backends** — InMemoryStorage (fast) and SQLiteStorage (persistent) implemented
+- ✅ **Bridge Axioms** — BridgeAxiom and TranslationChain classes for domain transformations (L1 → L2 → L3)
+- ✅ **Integrity Verification** — SHA-256 checksum computation and verification functions
+- ✅ **No New Dependencies** — Uses Python stdlib only (sqlite3, json, dataclasses)
+
+**Supported Modules:**
+```python
+# Semantic Extract
+from semantica.semantic_extract.semantic_extract_provenance import (
+    NERExtractorWithProvenance, RelationExtractorWithProvenance, EventDetectorWithProvenance
+)
+
+# LLM Providers
+from semantica.llms.llms_provenance import (
+    GroqLLMWithProvenance, OpenAILLMWithProvenance, HuggingFaceLLMWithProvenance
+)
+
+# Storage & Processing
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+from semantica.vector_store.vector_store_provenance import VectorStoreWithProvenance
+from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
+
+# ... and 12 more modules
+```
+
+**High-Stakes Use Cases:**
+- 🏥 **Healthcare** — Clinical decision audit trails with source tracking
+- 💰 **Finance** — Fraud detection provenance with complete lineage
+- ⚖️ **Legal** — Evidence chain of custody with temporal tracking
+- 🔒 **Cybersecurity** — Threat attribution with relationship tracking
+- 🏛️ **Government** — Policy decision audit trails with integrity verification
+
+**Note:** Provenance tracking provides the *technical infrastructure* for compliance. Organizations must implement additional policies and procedures to meet specific regulatory requirements (HIPAA, SOX, FDA 21 CFR Part 11, etc.).
+
+[**Documentation: Provenance Tracking**](semantica/provenance/provenance_usage.md)
+
 ### Context Engineering & Memory Systems
 
 > **Persistent Memory** • **Context Graph** • **Context Retriever** • **Hybrid Retrieval (Vector + Graph)** • **Production Graph Store (Neo4j)** • **Entity Linking** • **Multi-Hop Reasoning**
@@ -476,7 +627,7 @@ reasoned_result = context.query_with_reasoning(
 
 ### Knowledge Graph-Powered RAG (GraphRAG)
 
-> **30% Accuracy Improvement** • Vector + Graph Hybrid Search • 91% Accuracy • **Multi-Hop Reasoning** • **LLM-Generated Responses**
+> **Vector + Graph Hybrid Search** • **Multi-Hop Reasoning** • **LLM-Generated Responses** • **Semantic Re-ranking**
 
 ```python
 from semantica.context import AgentContext
@@ -525,7 +676,7 @@ print(f"Confidence: {result['confidence']:.3f}")
 from semantica.llms import Groq, OpenAI, HuggingFaceLLM, LiteLLM
 import os
 
-# Groq - Fast inference
+# Groq
 groq = Groq(
     model="llama-3.1-8b-instant",
     api_key=os.getenv("GROQ_API_KEY")
@@ -555,7 +706,7 @@ structured = groq.generate_structured("Extract entities from: Apple Inc. was fou
 ```
 
 **Supported Providers:**
-- **Groq**: Fast inference with Llama models
+- **Groq**: Inference with Llama models
 - **OpenAI**: GPT-3.5, GPT-4, and other OpenAI models
 - **HuggingFace**: Local LLM inference with Transformers
 - **LiteLLM**: Unified interface to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, and more)
@@ -755,7 +906,7 @@ print(f"Found {len(results)} results")
 
 #### Cybersecurity
 - [**Real-Time Anomaly Detection**](cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) - CVE RSS, Kafka streams, temporal KGs, sentence chunking
-- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, enhanced GraphRAG, deduplication
+- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, GraphRAG, deduplication
 
 #### Intelligence & Law Enforcement
 - [**Criminal Network Analysis**](cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) - OSINT RSS, deduplication, network centrality, graph analytics
@@ -772,12 +923,16 @@ print(f"Found {len(results)} results")
 
 ## 🔬 Advanced Features
 
+**Docling Integration** — Document parsing with table extraction for PDFs, DOCX, PPTX, and XLSX files. Supports OCR and multiple export formats.
+
+**AWS Neptune Support** — Amazon Neptune graph database integration with IAM authentication and OpenCypher queries.
+
+**Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD, N3) and extend Schema.org, FOAF, Dublin Core, or custom ontologies.
+
 **Incremental Updates** — Real-time stream processing with Kafka, RabbitMQ, Kinesis for live updates.
 
 **Multi-Language Support** — Process multiple languages with automatic detection.
 
-**Custom Ontology Import** — Import and extend Schema.org and custom ontologies.
-
 **Advanced Reasoning** — Forward/backward chaining, Rete-based pattern matching, and automated explanation generation.
 
 **Graph Analytics** — Centrality, community detection, path finding, temporal analysis.
@@ -788,20 +943,6 @@ print(f"Found {len(results)} results")
 
 [**See Advanced Examples**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced) — Advanced extraction, graph analytics, reasoning, and more.
 
-## 🗺️ Roadmap
-
-### Q1 2026
-- [x] Core framework (v1.0)
-- [x] GraphRAG engine
-- [x] 6-stage ontology pipeline
-- [x] Advanced reasoning v2 (Rete, Forward/Backward Chaining)
-- [ ] Quality assurance features and Quality Assurance module
-- [ ] Enhanced multi-language support
-- [ ] Evals
-- [ ] Real-time streaming improvements
-
-### Q2 2026
-- [ ] Multi-modal processing
 
 ---
 
@@ -867,20 +1008,11 @@ git push origin feature/your-feature
 4. **Feature Requests** - [Request feature](https://github.com/Hawksight-AI/semantica/issues/new)
 
 
-###  Contributors
-
-<a href="https://github.com/Hawksight-AI/semantica/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=Hawksight-AI/semantica" alt="Contributors" />
-</a>
-
 ## 📜 License
 
 Semantica is licensed under the **MIT License** - see the [LICENSE](https://github.com/Hawksight-AI/semantica/blob/main/LICENSE) file for details.
 
-<div align="center">
-
 **Built by the Semantica Community**
 
-[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/pMHguUzG)
 
-</div>
+[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/RgaGTj9J)

RELEASE.md

@@ -26,10 +26,10 @@ Before releasing, ensure:
 
 The project uses GitHub Actions for automated releases to PyPI.
 
-1. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.0`).
+1.29. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.3`).
    ```bash
-   git tag -a v0.2.0 -m "Release v0.2.0"
-   git push origin v0.2.0
+   git tag -a v0.2.3 -m "Release v0.2.3"
+   git push origin v0.2.3
    ```
 2. **GitHub Action**: The `Release` workflow will automatically trigger, build the package, create a GitHub Release, and publish to PyPI using Trusted Publishing.
 

SECURITY.md

@@ -6,6 +6,9 @@ We actively support the following versions of Semantica with security updates:
 
 | Version | Supported          |
 | ------- | ------------------ |
+| 0.2.3   | :white_check_mark: |
+| 0.2.2   | :white_check_mark: |
+| 0.2.1   | :white_check_mark: |
 | 0.2.0   | :white_check_mark: |
 | 0.1.1   | :white_check_mark: |
 | 0.1.0   | :white_check_mark: |

cookbook/introduction/21_Amazon_Neptune_Store.ipynb

@@ -25,6 +25,57 @@
     "- AWS credentials configured (boto3, environment variables, or IAM role)\n",
     "- Network access to your Neptune cluster (VPC, security groups)\n",
     "\n",
+    "#### Quick Setup with CloudFormation\n",
+    "\n",
+    "If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n",
+    "\n",
+    "```bash\n",
+    "# Deploy the Neptune stack (takes ~15-20 minutes)\n",
+    "aws cloudformation create-stack \\\n",
+    "  --stack-name semantica-neptune \\\n",
+    "  --template-body file://neptune-setup.yaml \\\n",
+    "  --capabilities CAPABILITY_NAMED_IAM\n",
+    "\n",
+    "# Wait for stack creation to complete\n",
+    "aws cloudformation wait stack-create-complete --stack-name semantica-neptune\n",
+    "\n",
+    "# Get the outputs (endpoint, port, credentials)\n",
+    "aws cloudformation describe-stacks --stack-name semantica-neptune \\\n",
+    "  --query 'Stacks[0].Outputs' --output table\n",
+    "```\n",
+    "\n",
+    "The template creates:\n",
+    "- VPC with public subnets and Internet Gateway\n",
+    "- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
+    "- IAM user with least-privilege access for OpenCypher queries\n",
+    "- Security group allowing Bolt protocol (port 8182) access\n",
+    "\n",
+    "> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
+    "\n",
+    "**Outputs:**\n",
+    "- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
+    "- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
+    "- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
+    "- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
+    "- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
+    "\n",
+    "**Cleanup:**\n",
+    "```bash\n",
+    "aws cloudformation delete-stack --stack-name semantica-neptune\n",
+    "```\n",
+    "\n",
+    "**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n",
+    "\n",
+    "| Resource | Cost (USD) |\n",
+    "| --- | --- |\n",
+    "| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n",
+    "| Storage (10 GB) | ~1/month |\n",
+    "| I/O requests | ~1-5/month |\n",
+    "| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
+    "| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
+    "\n",
+    "> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
+    "\n",
     "---"
    ]
   },
@@ -70,14 +121,21 @@
     "import os\n",
     "\n",
     "# Neptune cluster configuration - REPLACE WITH YOUR VALUES\n",
+    "# (Get these from CloudFormation stack outputs)\n",
     "os.environ[\"NEPTUNE_ENDPOINT\"] = \"your-cluster.us-east-1.neptune.amazonaws.com\"\n",
     "os.environ[\"NEPTUNE_PORT\"] = \"8182\"\n",
     "os.environ[\"AWS_REGION\"] = \"us-east-1\"\n",
     "\n",
-    "# AWS credentials (if using IAM Auth and not relying on IAM role or ~/.aws/credentials)\n",
-    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n",
-    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n",
-    "# os.environ[\"AWS_SESSION_TOKEN\"] = \"your-session-token\"\n",
+    "# AWS credentials for IAM Authentication\n",
+    "# Option 1: IAM User (static credentials from CloudFormation template)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"AKIA...\"  # From AwsAccessKeyId output\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # From AwsSecretAccessKey output\n",
+    "# Note: No AWS_SESSION_TOKEN needed for IAM users\n",
+    "\n",
+    "# Option 2: IAM Role / Temporary credentials (e.g., STS AssumeRole, EC2 instance role)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"ASIA...\"  # Temporary access key\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # Temporary secret key\n",
+    "# os.environ[\"AWS_SESSION_TOKEN\"] = \"...\"      # REQUIRED for temporary credentials\n",
     "\n",
     "print(f\"Neptune Endpoint: {os.environ.get('NEPTUNE_ENDPOINT')}\")\n",
     "print(f\"AWS Region: {os.environ.get('AWS_REGION')}\")"

cookbook/introduction/neptune-setup.yaml

@@ -0,0 +1,228 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: >
+  Amazon Neptune cluster with public endpoint, IAM authentication, and least-privilege
+  IAM user for Semantica cookbook. Uses db.t3.medium (most cost-effective Neptune instance type).
+
+Parameters:
+  EnvironmentName:
+    Type: String
+    Default: semantica-neptune
+    Description: Environment name prefix for resource naming
+
+Resources:
+  # =============================================================================
+  # VPC & NETWORKING
+  # =============================================================================
+
+  VPC:
+    Type: AWS::EC2::VPC
+    Properties:
+      CidrBlock: 10.0.0.0/16
+      EnableDnsHostnames: true
+      EnableDnsSupport: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-vpc
+
+  InternetGateway:
+    Type: AWS::EC2::InternetGateway
+    Properties:
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-igw
+
+  InternetGatewayAttachment:
+    Type: AWS::EC2::VPCGatewayAttachment
+    Properties:
+      InternetGatewayId: !Ref InternetGateway
+      VpcId: !Ref VPC
+
+  PublicSubnet1:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [0, !GetAZs '']
+      CidrBlock: 10.0.1.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-1
+
+  PublicSubnet2:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [1, !GetAZs '']
+      CidrBlock: 10.0.2.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-2
+
+  PublicRouteTable:
+    Type: AWS::EC2::RouteTable
+    Properties:
+      VpcId: !Ref VPC
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-rt
+
+  DefaultPublicRoute:
+    Type: AWS::EC2::Route
+    DependsOn: InternetGatewayAttachment
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      DestinationCidrBlock: 0.0.0.0/0
+      GatewayId: !Ref InternetGateway
+
+  PublicSubnet1RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet1
+
+  PublicSubnet2RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet2
+
+  # =============================================================================
+  # SECURITY GROUP
+  # =============================================================================
+
+  NeptuneSecurityGroup:
+    Type: AWS::EC2::SecurityGroup
+    Properties:
+      GroupName: !Sub ${EnvironmentName}-neptune-sg
+      GroupDescription: Security group for Neptune cluster - allows Bolt protocol access
+      VpcId: !Ref VPC
+      SecurityGroupIngress:
+        - IpProtocol: tcp
+          FromPort: 8182
+          ToPort: 8182
+          CidrIp: 0.0.0.0/0
+          Description: Allow Bolt protocol access from anywhere
+      SecurityGroupEgress:
+        - IpProtocol: -1
+          CidrIp: 0.0.0.0/0
+          Description: Allow all outbound traffic
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-neptune-sg
+
+  # =============================================================================
+  # NEPTUNE CLUSTER
+  # =============================================================================
+
+  NeptuneSubnetGroup:
+    Type: AWS::Neptune::DBSubnetGroup
+    Properties:
+      DBSubnetGroupDescription: Subnet group for Neptune cluster
+      DBSubnetGroupName: !Sub ${EnvironmentName}-subnet-group
+      SubnetIds:
+        - !Ref PublicSubnet1
+        - !Ref PublicSubnet2
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-subnet-group
+
+  NeptuneCluster:
+    Type: AWS::Neptune::DBCluster
+    Properties:
+      DBClusterIdentifier: !Sub ${EnvironmentName}-cluster
+      DBSubnetGroupName: !Ref NeptuneSubnetGroup
+      VpcSecurityGroupIds:
+        - !Ref NeptuneSecurityGroup
+      EngineVersion: '1.4.6.3'
+      IamAuthEnabled: true
+      StorageEncrypted: true
+      DeletionProtection: false
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-cluster
+
+  NeptuneInstance:
+    Type: AWS::Neptune::DBInstance
+    Properties:
+      DBInstanceIdentifier: !Sub ${EnvironmentName}-instance
+      DBInstanceClass: db.t3.medium
+      DBClusterIdentifier: !Ref NeptuneCluster
+      PubliclyAccessible: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-instance
+
+  # =============================================================================
+  # IAM USER WITH LEAST PRIVILEGES
+  # =============================================================================
+
+  NeptuneUser:
+    Type: AWS::IAM::User
+    Properties:
+      UserName: !Sub ${EnvironmentName}-user
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-user
+
+  NeptuneUserPolicy:
+    Type: AWS::IAM::Policy
+    Properties:
+      PolicyName: !Sub ${EnvironmentName}-neptune-access
+      Users:
+        - !Ref NeptuneUser
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Sid: NeptuneDataAccess
+            Effect: Allow
+            Action:
+              - neptune-db:connect
+              - neptune-db:ReadDataViaQuery
+              - neptune-db:WriteDataViaQuery
+              - neptune-db:DeleteDataViaQuery
+            Resource: !Sub
+              - arn:aws:neptune-db:${AWS::Region}:${AWS::AccountId}:${ClusterResourceId}/*
+              - ClusterResourceId: !GetAtt NeptuneCluster.ClusterResourceId
+
+  NeptuneUserAccessKey:
+    Type: AWS::IAM::AccessKey
+    Properties:
+      UserName: !Ref NeptuneUser
+
+# =============================================================================
+# OUTPUTS
+# =============================================================================
+
+Outputs:
+  NeptuneEndpoint:
+    Description: Neptune cluster endpoint (hostname only) - use as NEPTUNE_ENDPOINT
+    Value: !GetAtt NeptuneCluster.Endpoint
+
+  NeptunePort:
+    Description: Neptune cluster port - use as NEPTUNE_PORT
+    Value: !GetAtt NeptuneCluster.Port
+
+  AwsAccessKeyId:
+    Description: Access key ID for the Neptune IAM user - use as AWS_ACCESS_KEY_ID
+    Value: !Ref NeptuneUserAccessKey
+
+  AwsSecretAccessKey:
+    Description: Secret access key for the Neptune IAM user - use as AWS_SECRET_ACCESS_KEY
+    Value: !GetAtt NeptuneUserAccessKey.SecretAccessKey
+
+  AwsRegion:
+    Description: AWS region where Neptune is deployed - use as AWS_REGION
+    Value: !Ref AWS::Region
+
+  NeptuneClusterResourceId:
+    Description: Neptune cluster resource ID (for IAM policy reference)
+    Value: !GetAtt NeptuneCluster.ClusterResourceId
+
+  VpcId:
+    Description: VPC ID
+    Value: !Ref VPC
+
+  SecurityGroupId:
+    Description: Neptune security group ID
+    Value: !Ref NeptuneSecurityGroup

cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb

@@ -110,7 +110,7 @@
    "source": [
     "# Set up API keys\n",
     "# Note: In production, use environment variables: export GROQ_API_KEY=\"your-key\"\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"Your Groq API\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb

@@ -30,7 +30,7 @@
     "# Environment Setup\n",
     "import os\n",
     "\n",
-    "os.environ['GROQ_API_KEY'] = os.getenv('GROQ_API_KEY', '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')\n",
+    "os.environ['GROQ_API_KEY'] = os.getenv('GROQ_API_KEY', '')\n",
     "\n",
     "# Install Semantica and all required dependencies\n",
     "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers\n"
@@ -84,7 +84,7 @@
    "source": [
     "# Set up API keys\n",
     "# Note: In production, use environment variables: export GROQ_API_KEY=\"your-key\"\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-groq-api-key-here\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "print(\"API keys configured.\")\n"
    ]

cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb

@@ -109,7 +109,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb

@@ -85,7 +85,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb

@@ -81,7 +81,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/finance/02_Fraud_Detection.ipynb

@@ -98,7 +98,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb

@@ -83,7 +83,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb

@@ -80,7 +80,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

docs/DOCS_README.md

@@ -46,7 +46,7 @@ semantica/
 │   │   └── custom.css   # Custom styling
 │   └── assets/
 │       └── img/
-│           └── semantica_logo.png
+│           └── Semantica Updated Logo.png
 └── site/                # Generated site (created by mkdocs build)
 ```
 

docs/citation.md

@@ -12,22 +12,22 @@ How to cite Semantica in academic papers and research.
   author = {Hawksight AI},
   year = {2026},
   url = {https://github.com/Hawksight-AI/semantica},
-  version = {0.2.0},
+  version = {0.2.6},
   doi = {10.5281/zenodo.XXXXXXX}
 }
 ```
 
 ### APA
-Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.0) [Computer software]. https://github.com/Hawksight-AI/semantica
+Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.6) [Computer software]. https://github.com/Hawksight-AI/semantica
 
 ### MLA
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.0, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.6, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
 
 ### Chicago
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.0. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.6. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
 
 ### IEEE
-Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.0, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
+Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.6, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
 
 ---
 

docs/index.md

@@ -1,5 +1,5 @@
 <div align="center">
-  <img src="assets/img/semantica_logo.png" alt="Semantica Logo" width="450" height="auto">
+  <img src="assets/img/Semantica Updated Logo.png" alt="Semantica Logo" width="450" height="auto">
   
   <h1>🧠 Semantica</h1>
   
@@ -17,7 +17,7 @@
   
   <p><em>The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.</em></p>
   
-  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.1.1</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
+  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.2.3</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
   
   <p>
     <a href="getting-started/" class="md-button md-button--primary">Get Started</a>

docs/js/version-selector.js

@@ -9,7 +9,12 @@ document.addEventListener("DOMContentLoaded", function () {
 
         // Define versions
         var versions = [
-            { name: "0.1.1", url: "#", current: true },
+            { name: "0.2.4", url: "#", current: true },
+            { name: "0.2.3", url: "#", current: false },
+            { name: "0.2.2", url: "#", current: false },
+            { name: "0.2.1", url: "#", current: false },
+            { name: "0.2.0", url: "#", current: false },
+            { name: "0.1.1", url: "#", current: false },
             { name: "0.1.0", url: "#", current: false }
         ];
 

docs/reference/change_management.md

@@ -0,0 +1,938 @@
+# Change Management API Reference
+
+Comprehensive API documentation for the Enhanced Change Management module in Semantica.
+
+## Overview
+
+The `semantica.change_management` module provides enterprise-grade version control, audit trails, and compliance tracking for knowledge graphs and ontologies. It includes persistent storage backends, detailed change tracking, data integrity verification, and standardized metadata structures.
+
+## Module Structure
+
+```
+semantica.change_management/
+├── change_log.py              # Standardized metadata structures
+├── version_storage.py         # Storage abstraction and implementations
+├── managers.py                # Enhanced version managers
+├── ontology_version_manager.py # Ontology version management
+└── change_management_usage.md # Usage guide
+```
+
+## Quick Import
+
+```python
+from semantica.change_management import (
+    # Metadata
+    ChangeLogEntry,
+    
+    # Storage
+    VersionStorage,
+    InMemoryVersionStorage,
+    SQLiteVersionStorage,
+    
+    # Utilities
+    compute_checksum,
+    verify_checksum,
+    
+    # Version Managers
+    BaseVersionManager,
+    TemporalVersionManager,
+    OntologyVersionManager,
+    VersionManager,
+    OntologyVersion
+)
+```
+
+---
+
+## Core Classes
+
+### ChangeLogEntry
+
+Standardized metadata structure for version changes with validation.
+
+#### Class Definition
+
+```python
+@dataclass
+class ChangeLogEntry:
+    """
+    Standardized change log entry with validation.
+    
+    Attributes:
+        timestamp: ISO 8601 formatted timestamp
+        author: Email address of the change author
+        description: Change description (max 500 characters)
+        change_id: Optional ID linking to external systems
+    """
+    timestamp: str
+    author: str
+    description: str
+    change_id: Optional[str] = None
+```
+
+#### Methods
+
+##### `__post_init__()`
+
+Validates all fields after initialization.
+
+**Raises:**
+- `ValidationError`: If any field validation fails
+
+**Example:**
+```python
+entry = ChangeLogEntry(
+    timestamp="2024-01-30T12:00:00Z",
+    author="user@example.com",
+    description="Updated entity relationships",
+    change_id="TICKET-123"
+)
+```
+
+##### `create_now(author, description, change_id=None)` (classmethod)
+
+Creates a change log entry with the current timestamp.
+
+**Parameters:**
+- `author` (str): Email address of the change author
+- `description` (str): Change description (max 500 characters)
+- `change_id` (str, optional): ID linking to external systems
+
+**Returns:**
+- `ChangeLogEntry`: New instance with current timestamp
+
+**Example:**
+```python
+entry = ChangeLogEntry.create_now(
+    author="developer@company.com",
+    description="Fixed entity resolution bug",
+    change_id="JIRA-1234"
+)
+```
+
+#### Validation Rules
+
+- **Timestamp**: Must be valid ISO 8601 format with 'T' separator
+- **Author**: Must be valid email format (RFC 5322)
+- **Description**: Maximum 500 characters
+- **Change ID**: Optional, no validation
+
+---
+
+### VersionStorage
+
+Abstract base class for storage implementations.
+
+#### Class Definition
+
+```python
+class VersionStorage(ABC):
+    """
+    Abstract base class for version storage backends.
+    
+    Provides interface for saving, retrieving, and managing version snapshots.
+    """
+```
+
+#### Abstract Methods
+
+##### `save(snapshot)`
+
+Save a version snapshot.
+
+**Parameters:**
+- `snapshot` (Dict[str, Any]): Version snapshot dictionary with metadata
+
+**Raises:**
+- `ValidationError`: If snapshot data is invalid
+- `ProcessingError`: If save operation fails
+
+**Example:**
+```python
+snapshot = {
+    "label": "v1.0",
+    "timestamp": "2024-01-30T12:00:00Z",
+    "author": "user@example.com",
+    "description": "Initial version",
+    "data": {...}
+}
+storage.save(snapshot)
+```
+
+##### `get(label)`
+
+Retrieve a version snapshot by label.
+
+**Parameters:**
+- `label` (str): Version label to retrieve
+
+**Returns:**
+- `Optional[Dict[str, Any]]`: Snapshot dictionary or None if not found
+
+**Example:**
+```python
+snapshot = storage.get("v1.0")
+if snapshot:
+    print(f"Retrieved: {snapshot['label']}")
+```
+
+##### `list_all()`
+
+List all version snapshots.
+
+**Returns:**
+- `List[Dict[str, Any]]`: List of snapshot metadata dictionaries
+
+**Example:**
+```python
+versions = storage.list_all()
+for v in versions:
+    print(f"{v['label']}: {v['description']}")
+```
+
+##### `exists(label)`
+
+Check if a version exists.
+
+**Parameters:**
+- `label` (str): Version label to check
+
+**Returns:**
+- `bool`: True if version exists, False otherwise
+
+**Example:**
+```python
+if storage.exists("v1.0"):
+    print("Version exists")
+```
+
+##### `delete(label)`
+
+Delete a version snapshot.
+
+**Parameters:**
+- `label` (str): Version label to delete
+
+**Returns:**
+- `bool`: True if deleted, False if not found
+
+**Example:**
+```python
+if storage.delete("v1.0"):
+    print("Version deleted")
+```
+
+---
+
+### InMemoryVersionStorage
+
+In-memory version storage implementation.
+
+#### Class Definition
+
+```python
+class InMemoryVersionStorage(VersionStorage):
+    """
+    In-memory version storage implementation.
+    
+    Fast, volatile storage for development and testing.
+    Data is lost when the process ends.
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self):
+    """Initialize in-memory storage."""
+```
+
+**Example:**
+```python
+storage = InMemoryVersionStorage()
+```
+
+#### Performance Characteristics
+
+- **Save**: 0.37-16ms (10-1000 entities)
+- **Get**: 0.20-16ms (10-1000 entities)
+- **List**: <0.03ms
+- **Thread-safe**: Yes (uses RLock)
+
+#### Use Cases
+
+- Development and testing
+- Temporary version tracking
+- High-performance scenarios where persistence is not required
+
+---
+
+### SQLiteVersionStorage
+
+SQLite-based persistent version storage implementation.
+
+#### Class Definition
+
+```python
+class SQLiteVersionStorage(VersionStorage):
+    """
+    SQLite-based persistent version storage implementation.
+    
+    Provides persistence across process restarts with ACID guarantees.
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: str):
+    """
+    Initialize SQLite storage.
+    
+    Args:
+        storage_path: Path to SQLite database file
+    """
+```
+
+**Parameters:**
+- `storage_path` (str): Path to SQLite database file (created if doesn't exist)
+
+**Example:**
+```python
+storage = SQLiteVersionStorage("versions.db")
+```
+
+#### Database Schema
+
+```sql
+CREATE TABLE versions (
+    label TEXT PRIMARY KEY,
+    timestamp TEXT NOT NULL,
+    author TEXT NOT NULL,
+    description TEXT,
+    checksum TEXT,
+    snapshot_data TEXT NOT NULL,
+    created_at TEXT NOT NULL
+)
+```
+
+#### Performance Characteristics
+
+- **Save**: 7-25ms (10-1000 entities)
+- **Get**: 2-8ms (10-1000 entities)
+- **List**: 0.6-13ms
+- **Thread-safe**: Yes (uses RLock)
+- **ACID**: Full transaction support
+
+#### Use Cases
+
+- Production deployments
+- Long-term version storage
+- Compliance and audit requirements
+- Multi-process environments
+
+---
+
+### BaseVersionManager
+
+Abstract base class for version managers.
+
+#### Class Definition
+
+```python
+class BaseVersionManager(ABC):
+    """
+    Abstract base class for version managers.
+    
+    Provides common functionality for version management across
+    different data types (knowledge graphs, ontologies, etc.).
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: Optional[str] = None):
+    """
+    Initialize base version manager.
+    
+    Args:
+        storage_path: Path to SQLite database file for persistent storage.
+                     If None, uses in-memory storage.
+    """
+```
+
+**Parameters:**
+- `storage_path` (str, optional): Path to SQLite database file
+
+**Example:**
+```python
+# In-memory storage
+manager = BaseVersionManager()
+
+# Persistent storage
+manager = BaseVersionManager(storage_path="versions.db")
+```
+
+#### Abstract Methods
+
+##### `create_snapshot(data, version_label, author, description, **options)`
+
+Create a versioned snapshot of the data.
+
+**Parameters:**
+- `data` (Any): Data to snapshot
+- `version_label` (str): Version label
+- `author` (str): Email address of the author
+- `description` (str): Change description
+- `**options`: Additional options
+
+**Returns:**
+- `Dict[str, Any]`: Snapshot with metadata and checksum
+
+##### `compare_versions(version1, version2, **options)`
+
+Compare two versions and return detailed differences.
+
+**Parameters:**
+- `version1` (Any): First version (label or snapshot)
+- `version2` (Any): Second version (label or snapshot)
+- `**options`: Comparison options
+
+**Returns:**
+- `Dict[str, Any]`: Detailed differences
+
+#### Concrete Methods
+
+##### `list_versions()`
+
+List all version snapshots.
+
+**Returns:**
+- `List[Dict[str, Any]]`: List of version metadata
+
+**Example:**
+```python
+versions = manager.list_versions()
+for v in versions:
+    print(f"{v['label']}: {v['description']}")
+```
+
+##### `get_version(label)`
+
+Retrieve specific version by label.
+
+**Parameters:**
+- `label` (str): Version label
+
+**Returns:**
+- `Optional[Dict[str, Any]]`: Version snapshot or None
+
+**Example:**
+```python
+version = manager.get_version("v1.0")
+```
+
+##### `verify_checksum(snapshot)`
+
+Verify data integrity using checksum.
+
+**Parameters:**
+- `snapshot` (Dict[str, Any]): Snapshot to verify
+
+**Returns:**
+- `bool`: True if checksum is valid
+
+**Example:**
+```python
+is_valid = manager.verify_checksum(snapshot)
+```
+
+---
+
+### TemporalVersionManager
+
+Enhanced temporal version management engine for knowledge graphs.
+
+#### Class Definition
+
+```python
+class TemporalVersionManager(BaseVersionManager):
+    """
+    Enhanced temporal version management engine for knowledge graphs.
+    
+    Features:
+        - Persistent snapshot storage (SQLite or in-memory)
+        - Detailed change tracking with entity-level diffs
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+        - Version comparison with backward compatibility
+        - Input validation and security features
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: Optional[str] = None, **config):
+    """
+    Initialize enhanced temporal version manager.
+    
+    Args:
+        storage_path: Path to SQLite database file for persistent storage.
+                     If None, uses in-memory storage
+        **config: Additional configuration options
+    """
+```
+
+**Parameters:**
+- `storage_path` (str, optional): Path to SQLite database file
+- `**config`: Additional configuration options
+
+**Example:**
+```python
+# In-memory storage
+manager = TemporalVersionManager()
+
+# Persistent storage
+manager = TemporalVersionManager(storage_path="kg_versions.db")
+```
+
+#### Methods
+
+##### `create_snapshot(graph, version_label, author, description, **options)`
+
+Create and store snapshot with checksum and metadata.
+
+**Parameters:**
+- `graph` (Dict[str, Any]): Knowledge graph dict with "entities" and "relationships"
+- `version_label` (str): Version string (e.g., "v1.0")
+- `author` (str): Email address of the change author
+- `description` (str): Change description (max 500 chars)
+- `**options`: Additional options
+
+**Returns:**
+- `Dict[str, Any]`: Snapshot with metadata and checksum
+
+**Raises:**
+- `ValidationError`: If input validation fails
+- `ProcessingError`: If snapshot creation fails
+
+**Example:**
+```python
+graph = {
+    "entities": [
+        {"id": "e1", "name": "Entity 1", "type": "Person"},
+        {"id": "e2", "name": "Entity 2", "type": "Organization"}
+    ],
+    "relationships": [
+        {"source": "e1", "target": "e2", "type": "works_for"}
+    ]
+}
+
+snapshot = manager.create_snapshot(
+    graph,
+    version_label="v1.0",
+    author="user@example.com",
+    description="Initial knowledge graph"
+)
+
+print(f"Created: {snapshot['label']}")
+print(f"Checksum: {snapshot['checksum']}")
+```
+
+##### `compare_versions(version1, version2, **options)`
+
+Compare two versions with detailed entity and relationship diffs.
+
+**Parameters:**
+- `version1` (Union[str, Dict]): First version (label or snapshot dict)
+- `version2` (Union[str, Dict]): Second version (label or snapshot dict)
+- `**options`: Comparison options
+
+**Returns:**
+- `Dict[str, Any]`: Detailed differences including:
+  - `summary`: Aggregate statistics
+  - `entity_changes`: Entity-level changes
+  - `relationship_changes`: Relationship-level changes
+
+**Example:**
+```python
+diff = manager.compare_versions("v1.0", "v2.0")
+
+print(f"Entities added: {diff['summary']['entities_added']}")
+print(f"Entities modified: {diff['summary']['entities_modified']}")
+print(f"Relationships added: {diff['summary']['relationships_added']}")
+
+# Detailed entity changes
+for entity_id, changes in diff['entity_changes'].items():
+    print(f"Entity {entity_id}: {changes['status']}")
+    if changes['status'] == 'modified':
+        print(f"  Before: {changes['before']}")
+        print(f"  After: {changes['after']}")
+```
+
+#### Performance
+
+- **Snapshot Creation**: 1.40-54ms (50-2000 entities)
+- **Version Retrieval**: 0.65-26ms (50-2000 entities)
+- **Version Comparison**: 3.46-33ms (100-1000 entities)
+- **Concurrent Throughput**: 500+ operations/second
+
+---
+
+### OntologyVersionManager
+
+Enhanced version management for ontologies.
+
+#### Class Definition
+
+```python
+class OntologyVersionManager(BaseVersionManager):
+    """
+    Enhanced version management for ontologies.
+    
+    Features:
+        - Persistent ontology snapshot storage
+        - Structural comparison (classes, properties, axioms)
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: Optional[str] = None, **config):
+    """
+    Initialize enhanced version manager for ontologies.
+    
+    Args:
+        storage_path: Path to SQLite database file for persistent storage.
+                     If None, uses in-memory storage
+        **config: Additional configuration options
+    """
+```
+
+**Example:**
+```python
+manager = OntologyVersionManager(storage_path="ontology_versions.db")
+```
+
+#### Methods
+
+##### `create_snapshot(ontology, version_label, author, description, **options)`
+
+Create ontology snapshot with metadata.
+
+**Parameters:**
+- `ontology` (Dict[str, Any]): Ontology dict with structure information
+- `version_label` (str): Version label
+- `author` (str): Email address of the author
+- `description` (str): Change description
+- `**options`: Additional options
+
+**Returns:**
+- `Dict[str, Any]`: Ontology snapshot with metadata
+
+**Example:**
+```python
+ontology = {
+    "uri": "https://example.com/ontology",
+    "version_info": {"version": "1.0", "date": "2024-01-30"},
+    "structure": {
+        "classes": ["Person", "Organization", "Location"],
+        "properties": ["name", "address", "email"],
+        "individuals": ["JohnDoe", "ACME_Corp"],
+        "axioms": ["Person hasAddress exactly 1 Location"]
+    }
+}
+
+snapshot = manager.create_snapshot(
+    ontology,
+    version_label="ont_v1.0",
+    author="architect@example.com",
+    description="Initial ontology design"
+)
+```
+
+##### `compare_versions(version1, version2, **options)`
+
+Compare ontology versions with structural analysis.
+
+**Parameters:**
+- `version1` (Union[str, Dict]): First version
+- `version2` (Union[str, Dict]): Second version
+- `**options`: Comparison options
+
+**Returns:**
+- `Dict[str, Any]`: Structural differences including:
+  - `classes_added`, `classes_removed`
+  - `properties_added`, `properties_removed`
+  - `individuals_added`, `individuals_removed`
+  - `axioms_added`, `axioms_removed`, `axioms_modified`
+
+**Example:**
+```python
+diff = manager.compare_versions("ont_v1.0", "ont_v2.0")
+
+print(f"Classes added: {diff['classes_added']}")
+print(f"Properties added: {diff['properties_added']}")
+print(f"Axioms modified: {diff['axioms_modified']}")
+```
+
+---
+
+## Utility Functions
+
+### compute_checksum
+
+Compute SHA-256 checksum for data integrity.
+
+#### Function Signature
+
+```python
+def compute_checksum(data: Dict[str, Any]) -> str:
+    """
+    Compute SHA-256 checksum for data.
+    
+    Args:
+        data: Dictionary to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+    """
+```
+
+**Parameters:**
+- `data` (Dict[str, Any]): Dictionary to compute checksum for
+
+**Returns:**
+- `str`: SHA-256 checksum as hexadecimal string
+
+**Example:**
+```python
+from semantica.change_management import compute_checksum
+
+data = {"entities": [...], "relationships": [...]}
+checksum = compute_checksum(data)
+print(f"Checksum: {checksum}")
+```
+
+**Performance:** 1.29-110ms (100-10,000 entities)
+
+---
+
+### verify_checksum
+
+Verify data integrity using stored checksum.
+
+#### Function Signature
+
+```python
+def verify_checksum(snapshot: Dict[str, Any]) -> bool:
+    """
+    Verify data integrity using checksum.
+    
+    Args:
+        snapshot: Snapshot dictionary with 'checksum' field
+        
+    Returns:
+        True if checksum is valid, False otherwise
+    """
+```
+
+**Parameters:**
+- `snapshot` (Dict[str, Any]): Snapshot dictionary with 'checksum' field
+
+**Returns:**
+- `bool`: True if checksum is valid, False otherwise
+
+**Example:**
+```python
+from semantica.change_management import verify_checksum
+
+snapshot = manager.get_version("v1.0")
+is_valid = verify_checksum(snapshot)
+
+if not is_valid:
+    print("WARNING: Data integrity compromised!")
+```
+
+**Performance:** 0.82-96ms (100-10,000 entities)
+
+---
+
+## Legacy Classes
+
+### VersionManager
+
+Original ontology version manager (moved from `semantica.ontology`).
+
+#### Import
+
+```python
+from semantica.change_management import VersionManager, OntologyVersion
+```
+
+**Note:** This class is maintained for backward compatibility. New projects should use `OntologyVersionManager`.
+
+---
+
+## Error Handling
+
+### ValidationError
+
+Raised when input validation fails.
+
+**Common Causes:**
+- Invalid email format
+- Description exceeds 500 characters
+- Invalid ISO 8601 timestamp
+- Missing required fields
+
+**Example:**
+```python
+from semantica.utils.exceptions import ValidationError
+
+try:
+    entry = ChangeLogEntry(
+        timestamp="invalid",
+        author="not-an-email",
+        description="x" * 501
+    )
+except ValidationError as e:
+    print(f"Validation failed: {e}")
+```
+
+### ProcessingError
+
+Raised when operations fail.
+
+**Common Causes:**
+- Database connection issues
+- File system errors
+- Concurrent modification conflicts
+
+**Example:**
+```python
+from semantica.utils.exceptions import ProcessingError
+
+try:
+    storage.save(snapshot)
+except ProcessingError as e:
+    print(f"Save failed: {e}")
+```
+
+---
+
+## Performance Considerations
+
+### Benchmarks
+
+Based on comprehensive performance testing:
+
+| Component | Small (100) | Medium (500) | Large (2000) |
+|-----------|-------------|--------------|--------------|
+| Snapshot Creation | 2.33ms | 10.70ms | 54.23ms |
+| Version Retrieval | 1.88ms | 7.33ms | 26.04ms |
+| Version Comparison | 3.46ms | 17.39ms | 32.83ms |
+| Checksum Compute | 1.29ms | 5.48ms | 22.15ms |
+| SQLite Save | 8.69ms | 13.37ms | 25.33ms |
+| InMemory Save | 1.18ms | 10.60ms | 14.11ms |
+
+### Optimization Tips
+
+1. **Use appropriate storage backend:**
+   - Development: `InMemoryVersionStorage`
+   - Production: `SQLiteVersionStorage`
+
+2. **Batch operations when possible:**
+   ```python
+   for data in batch:
+       manager.create_snapshot(data, ...)
+   ```
+
+3. **Implement retention policies:**
+   ```python
+   # Delete old versions periodically
+   for version in old_versions:
+       storage.delete(version['label'])
+   ```
+
+4. **Use concurrent operations:**
+   - Thread-safe: 500+ operations/second
+   - No performance degradation under load
+
+---
+
+## Compliance Features
+
+### HIPAA Compliance
+
+- Complete audit trails with author attribution
+- Timestamp tracking for all changes
+- Data integrity verification with checksums
+- Secure storage with access controls
+
+### SOX Compliance
+
+- Immutable change records
+- Detailed change descriptions
+- External system linking (change IDs)
+- Comprehensive audit reports
+
+### FDA 21 CFR Part 11
+
+- Electronic signatures (author email)
+- Data integrity verification
+- Audit trail generation
+- Tamper detection
+
+---
+
+## Examples
+
+### Complete Healthcare Example
+
+```python
+from semantica.change_management import TemporalVersionManager
+
+# Initialize with HIPAA-compliant storage
+manager = TemporalVersionManager(storage_path="hipaa_records.db")
+
+# Patient knowledge graph
+patient_kg = {
+    "entities": [
+        {"id": "patient_001", "type": "Patient", "name": "Jane Smith"},
+        {"id": "diagnosis_001", "type": "Diagnosis", "code": "I10"}
+    ],
+    "relationships": [
+        {"source": "patient_001", "target": "diagnosis_001", "type": "has_diagnosis"}
+    ]
+}
+
+# Create versioned record
+snapshot = manager.create_snapshot(
+    patient_kg,
+    "patient_001_v1.0",
+    "dr.williams@hospital.com",
+    "Initial diagnosis - Essential hypertension"
+)
+
+# Verify integrity
+assert manager.verify_checksum(snapshot), "Data integrity check failed"
+
+# Generate audit report
+for version in manager.list_versions():
+    print(f"{version['timestamp']}: {version['label']} by {version['author']}")
+```
+
+---
+
+## See Also
+
+- **Usage Guide**: `semantica/change_management/change_management_usage.md`
+- **Performance Tests**: `tests/change_management/test_performance.py`
+- **CHANGELOG**: `CHANGELOG.md`
+- **GitHub**: https://github.com/Hawksight-AI/semantica

docs/reference/ontology.md

@@ -63,6 +63,29 @@ The module uses several inference algorithms:
 
 ---
 
+## Ontology Ingestion
+
+Ingest existing ontology files directly into usable data structures using `OntologyIngestor`.
+
+**Function:** `ingest_ontology(source, method="file")`
+
+| Argument | Description |
+|----------|-------------|
+| `source` | File path, directory path, or list of paths |
+| `method` | Ingestion method (default: "file") |
+
+**Example:**
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest file
+data = ingest_ontology("ontology.ttl")
+
+# Ingest directory
+dataset = ingest_ontology("ontologies/")
+```
+
 ## Main Classes
 
 ### OntologyEngine
@@ -170,6 +193,17 @@ Manages external dependencies.
 | `import_external_ontology(uri, ontology)` | Load and merge external ontology |
 | `evaluate_alignment(uri, ontology)` | Assess alignment and compatibility |
 
+### OntologyIngestor
+
+Handles ingestion of existing ontologies from files and directories.
+
+**Methods:**
+
+| Method | Description |
+|--------|-------------|
+| `ingest_ontology(file_path)` | Ingest a single ontology file |
+| `ingest_directory(directory_path)` | Recursively ingest ontology files from a directory |
+
 ---
 
 ## Unified Engine Examples

docs/reference/provenance.md

@@ -0,0 +1,667 @@
+# Provenance Tracking Module
+
+**W3C PROV-O compliant provenance tracking for high-stakes domains requiring complete traceability**
+
+## Overview
+
+The Semantica provenance module provides W3C PROV-O compliant tracking for knowledge graphs, enabling complete end-to-end lineage from source documents to query responses. Designed for high-stakes domains where every decision must be explainable and auditable.
+
+### Key Features
+
+- ✅ **W3C PROV-O Compliant** — Implements PROV-O ontology (prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom)
+- ✅ **All 17 Modules Integrated** — Complete coverage across Semantica
+- ✅ **Source Tracking** — Document identifiers, page numbers, sections, and direct quotes supported
+- ✅ **Zero Breaking Changes** — 100% backward compatible, opt-in only
+- ✅ **Multiple Storage Backends** — InMemory (fast) and SQLite (persistent)
+- ✅ **Bridge Axiom Support** — Translation chain tracking for domain transformations (L1 → L2 → L3)
+- ✅ **Integrity Verification** — SHA-256 checksums for tamper detection
+- ✅ **Complete Lineage Tracing** — End-to-end from document to response
+
+---
+
+## Installation
+
+The provenance module is included with Semantica. No additional installation required.
+
+```python
+from semantica.provenance import ProvenanceManager
+```
+
+---
+
+## Core Components
+
+### ProvenanceManager
+
+Central manager for all provenance tracking operations.
+
+```python
+from semantica.provenance import ProvenanceManager
+
+# Initialize with in-memory storage (default)
+manager = ProvenanceManager()
+
+# Initialize with persistent SQLite storage
+manager = ProvenanceManager(storage_path="provenance.db")
+```
+
+**Methods:**
+- `track_entity(entity_id, source, entity_type, **metadata)` — Track entity provenance
+- `track_relationship(relationship_id, source, subject, predicate, obj, **metadata)` — Track relationship provenance
+- `track_chunk(chunk_id, source_document, chunk_text, start_char, end_char, **metadata)` — Track document chunk provenance
+- `track_property_source(entity_id, property_name, value, source, **metadata)` — Track property-level provenance
+- `get_lineage(entity_id)` — Retrieve complete lineage for an entity
+- `get_statistics()` — Get provenance statistics
+- `get_all_entries()` — Retrieve all provenance entries
+
+### Storage Backends
+
+#### InMemoryStorage
+
+Fast, non-persistent storage for development and testing.
+
+```python
+from semantica.provenance import ProvenanceManager, InMemoryStorage
+
+manager = ProvenanceManager(storage=InMemoryStorage())
+```
+
+#### SQLiteStorage
+
+Persistent storage for production use.
+
+```python
+from semantica.provenance import ProvenanceManager, SQLiteStorage
+
+storage = SQLiteStorage("provenance.db")
+manager = ProvenanceManager(storage=storage)
+```
+
+### Data Schemas
+
+#### ProvenanceEntry
+
+Core data structure for provenance tracking.
+
+```python
+from semantica.provenance import ProvenanceEntry
+from datetime import datetime
+
+entry = ProvenanceEntry(
+    entity_id="entity_1",
+    source="document.pdf",
+    timestamp=datetime.now(),
+    entity_type="named_entity",
+    metadata={"text": "Apple Inc.", "confidence": 0.95}
+)
+```
+
+#### SourceReference
+
+Structured source information with page and section details.
+
+```python
+from semantica.provenance import SourceReference
+
+source = SourceReference(
+    document="research_paper.pdf",
+    page=5,
+    section="Results",
+    confidence=0.98
+)
+```
+
+---
+
+## Module Integrations
+
+All Semantica modules have provenance-enabled versions. Enable tracking by setting `provenance=True`.
+
+### Semantic Extract
+
+```python
+from semantica.semantic_extract.semantic_extract_provenance import (
+    NERExtractorWithProvenance,
+    RelationExtractorWithProvenance,
+    EventDetectorWithProvenance,
+    CoreferenceResolverWithProvenance,
+    TripletExtractorWithProvenance
+)
+
+# Named Entity Recognition with provenance
+ner = NERExtractorWithProvenance(provenance=True)
+entities = ner.extract(
+    text="Apple Inc. was founded by Steve Jobs in Cupertino.",
+    source="company_history.pdf"
+)
+
+# Access provenance manager
+prov_manager = ner._prov_manager
+lineage = prov_manager.get_lineage("entity_id")
+```
+
+**Tracks:** Entity text, labels, confidence scores, source documents, character positions, extraction timestamps
+
+### LLM Providers
+
+```python
+from semantica.llms.llms_provenance import (
+    GroqLLMWithProvenance,
+    OpenAILLMWithProvenance,
+    HuggingFaceLLMWithProvenance,
+    LiteLLMWithProvenance
+)
+
+# Groq LLM with provenance
+llm = GroqLLMWithProvenance(
+    provenance=True,
+    model="llama-3.1-70b"
+)
+
+response = llm.generate("What is artificial intelligence?")
+
+# Access cost and performance data
+stats = llm._prov_manager.get_statistics()
+```
+
+**Tracks:** Model name, prompt/completion tokens, API costs, latency, generation parameters, prompts and responses
+
+### Pipeline Execution
+
+```python
+from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
+
+pipeline = PipelineWithProvenance(provenance=True)
+result = pipeline.run(data=input_data, source="input_file.json")
+```
+
+**Tracks:** Pipeline steps executed, duration, input/output data, execution status
+
+### Context Management
+
+```python
+from semantica.context.context_provenance import ContextManagerWithProvenance
+
+ctx = ContextManagerWithProvenance(provenance=True)
+ctx.add_context("Relevant background information", source="knowledge_base.txt")
+```
+
+**Tracks:** Context additions, sources, timestamps
+
+### Document Ingestion
+
+```python
+from semantica.ingest.ingest_provenance import PDFIngestorWithProvenance
+
+ingestor = PDFIngestorWithProvenance(provenance=True)
+documents = ingestor.ingest("research_paper.pdf")
+```
+
+**Tracks:** File paths, page counts, file metadata, ingestion timestamps
+
+### Embeddings Generation
+
+```python
+from semantica.embeddings.embeddings_provenance import EmbeddingGeneratorWithProvenance
+
+embedder = EmbeddingGeneratorWithProvenance(
+    provenance=True,
+    model="sentence-transformers/all-mpnet-base-v2"
+)
+embeddings = embedder.embed(["Text 1", "Text 2"], source="corpus.txt")
+```
+
+**Tracks:** Model name, embedding dimensions, generation timestamps
+
+### Graph Store
+
+```python
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+
+store = GraphStoreWithProvenance(provenance=True)
+store.add_node(entity_node, source="knowledge_graph.json")
+```
+
+**Tracks:** Nodes added, node properties, graph structure changes
+
+### Vector Store
+
+```python
+from semantica.vector_store.vector_store_provenance import VectorStoreWithProvenance
+
+store = VectorStoreWithProvenance(provenance=True)
+store.add_vectors(embedding_vectors, source="embeddings.npy")
+```
+
+**Tracks:** Vectors stored, dimensions, storage timestamps
+
+### Triplet Store
+
+```python
+from semantica.triplet_store.triplet_store_provenance import TripletStoreWithProvenance
+
+store = TripletStoreWithProvenance(provenance=True)
+store.add_triplet("Steve_Jobs", "founded", "Apple_Inc", source="knowledge_base.ttl")
+```
+
+**Tracks:** Subject, predicate, object, confidence scores, timestamps
+
+### Other Modules
+
+All remaining modules follow the same pattern:
+
+- **Reasoning** — `ReasoningEngineWithProvenance`
+- **Conflicts** — `SourceTrackerWithUnifiedBackend`
+- **Deduplication** — `DeduplicatorWithProvenance`
+- **Export** — `ExporterWithProvenance`
+- **Parse** — `ParserWithProvenance`
+- **Normalize** — `NormalizerWithProvenance`
+- **Ontology** — `OntologyManagerWithProvenance`
+- **Visualization** — `VisualizerWithProvenance`
+
+---
+
+## Usage Examples
+
+### Basic Entity Tracking
+
+```python
+from semantica.provenance import ProvenanceManager
+
+manager = ProvenanceManager()
+
+# Track entity
+manager.track_entity(
+    entity_id="entity_1",
+    source="document.pdf",
+    entity_type="organization",
+    metadata={
+        "name": "Apple Inc.",
+        "confidence": 0.95,
+        "extraction_method": "NER"
+    }
+)
+
+# Retrieve lineage
+lineage = manager.get_lineage("entity_1")
+print(f"Source: {lineage['source']}")
+print(f"Timestamp: {lineage['timestamp']}")
+print(f"Metadata: {lineage['metadata']}")
+```
+
+### Relationship Tracking
+
+```python
+# Track entities
+manager.track_entity("steve_jobs", "biography.pdf", "person")
+manager.track_entity("apple_inc", "biography.pdf", "organization")
+
+# Track relationship
+manager.track_relationship(
+    relationship_id="rel_1",
+    source="biography.pdf",
+    subject="steve_jobs",
+    predicate="founded",
+    obj="apple_inc",
+    metadata={"confidence": 0.92}
+)
+```
+
+### Lineage Chain Tracking
+
+```python
+# Create lineage chain: document → chunk → entity
+manager.track_entity("doc_1", "research_paper.pdf", "document")
+
+manager.track_chunk(
+    chunk_id="chunk_1",
+    source_document="doc_1",
+    chunk_text="Sample text content",
+    start_char=0,
+    end_char=100
+)
+
+manager.track_entity(
+    entity_id="entity_1",
+    source="chunk_1",
+    entity_type="named_entity",
+    metadata={"text": "Apple"}
+)
+
+# Retrieve complete lineage
+lineage = manager.get_lineage("entity_1")
+print(f"Lineage chain: {lineage['lineage_chain']}")
+```
+
+### Property-Level Provenance
+
+```python
+from semantica.provenance import SourceReference
+
+# Track entity
+manager.track_entity("company_1", "doc.pdf", "organization")
+
+# Track property sources
+manager.track_property_source(
+    entity_id="company_1",
+    property_name="revenue",
+    value="$394.3B",
+    source=SourceReference(
+        document="annual_report_2023.pdf",
+        page=5,
+        section="Financial Summary",
+        confidence=0.98
+    )
+)
+
+manager.track_property_source(
+    entity_id="company_1",
+    property_name="employees",
+    value="500",
+    source=SourceReference(
+        document="company_profile.pdf",
+        page=2,
+        confidence=0.90
+    )
+)
+```
+
+### End-to-End Workflow
+
+```python
+from semantica.provenance import ProvenanceManager
+from semantica.ingest.ingest_provenance import PDFIngestorWithProvenance
+from semantica.semantic_extract.semantic_extract_provenance import NERExtractorWithProvenance
+from semantica.llms.llms_provenance import GroqLLMWithProvenance
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+
+# Initialize
+manager = ProvenanceManager()
+
+# Step 1: Ingest
+ingestor = PDFIngestorWithProvenance(provenance=True)
+documents = ingestor.ingest("research_paper.pdf")
+
+# Step 2: Extract
+ner = NERExtractorWithProvenance(provenance=True)
+entities = ner.extract(documents[0].text, source="research_paper.pdf")
+
+# Step 3: LLM Analysis
+llm = GroqLLMWithProvenance(provenance=True)
+summary = llm.generate(f"Summarize: {documents[0].text[:500]}")
+
+# Step 4: Store
+graph = GraphStoreWithProvenance(provenance=True)
+for entity in entities:
+    graph.add_node(entity, source="research_paper.pdf")
+
+# Step 5: Retrieve provenance
+lineage = ner._prov_manager.get_lineage("entity_id")
+stats = ner._prov_manager.get_statistics()
+print(f"Total operations: {stats['total_entries']}")
+```
+
+---
+
+## Bridge Axioms
+
+Bridge axioms enable translation chain tracking across multiple abstraction layers.
+
+```python
+from semantica.provenance.bridge_axiom import BridgeAxiom, TranslationChain
+
+# Create bridge axiom
+axiom = BridgeAxiom(
+    source_layer="L1_ecological",
+    target_layer="L2_financial",
+    translation_rule="fish_biomass_to_revenue",
+    confidence=0.89
+)
+
+# Add provenance
+axiom.add_source_provenance(
+    document="DOI:10.1371/journal.pone.0023601",
+    location="Figure 2",
+    quote="Total fish biomass increased by 463%"
+)
+
+# Create translation chain
+chain = TranslationChain()
+chain.add_axiom(axiom)
+
+# Track complete chain
+provenance_data = chain.get_complete_provenance()
+```
+
+**Use Cases:**
+- Blue Finance: Ecological data → Financial metrics
+- Healthcare: Clinical data → Treatment recommendations
+- Legal: Evidence → Legal conclusions
+- Pharmaceutical: Research data → Drug efficacy claims
+
+---
+
+## Best Practices
+
+### 1. Always Provide Source Information
+
+```python
+# ✅ GOOD - Provides source
+entities = ner.extract(text, source="document.pdf")
+
+# ❌ BAD - No source information
+entities = ner.extract(text)
+```
+
+### 2. Use Descriptive Entity IDs
+
+```python
+# ✅ GOOD - Descriptive IDs
+manager.track_entity("company_apple_inc", source, "organization")
+
+# ❌ BAD - Generic IDs
+manager.track_entity("entity_1", source, "organization")
+```
+
+### 3. Include Rich Metadata
+
+```python
+# ✅ GOOD - Rich metadata
+manager.track_entity(
+    entity_id="person_steve_jobs",
+    source="biography.pdf",
+    entity_type="person",
+    metadata={
+        "full_name": "Steve Jobs",
+        "birth_year": 1955,
+        "confidence": 0.95,
+        "extraction_method": "NER_spacy"
+    }
+)
+```
+
+### 4. Enable Provenance for High-Stakes Operations
+
+```python
+# For high-stakes requirements
+llm = GroqLLMWithProvenance(provenance=True)  # Track all LLM calls
+ner = NERExtractorWithProvenance(provenance=True)  # Track all extractions
+```
+
+### 5. Use Persistent Storage for Production
+
+```python
+from semantica.provenance import ProvenanceManager, SQLiteStorage
+
+# Use SQLite for persistence
+storage = SQLiteStorage("provenance.db")
+manager = ProvenanceManager(storage=storage)
+```
+
+---
+
+## Performance
+
+### Benchmarks
+
+- **Entity tracking:** <5ms per operation
+- **Lineage retrieval:** <10ms for chains up to 100 levels
+- **Batch operations:** 1000+ entities/second
+- **Storage:** InMemory (fastest), SQLite (persistent)
+
+### Optimization Tips
+
+1. **Batch Operations:** Use batch methods for multiple entities
+2. **Selective Tracking:** Only track provenance for critical entities
+3. **Storage Choice:** Use InMemory for development, SQLite for production
+4. **Index Optimization:** SQLite automatically indexes entity_id and source_document
+
+---
+
+## Compliance Standards Support
+
+The provenance module provides **technical infrastructure** that supports compliance efforts:
+
+- **W3C PROV-O** — Implements PROV-O ontology data structures and relationships
+- **FDA 21 CFR Part 11** — Provides audit trails, checksums, and temporal tracking for electronic records
+- **SOX** — Enables financial data lineage tracking and integrity verification
+- **HIPAA** — Supports healthcare data integrity through checksums and source tracking
+- **TNFD** — Enables bridge axiom tracking for nature-to-financial translations
+
+**Important:** This module provides the *technical capabilities* for compliance. Organizations must implement additional policies, procedures, validation, and controls to meet specific regulatory requirements. Semantica does not provide regulatory certification or legal compliance guarantees.
+
+---
+
+## API Reference
+
+### ProvenanceManager
+
+#### `__init__(storage=None, storage_path=None)`
+
+Initialize provenance manager.
+
+**Parameters:**
+- `storage` (ProvenanceStorage, optional): Storage backend instance
+- `storage_path` (str, optional): Path for SQLite storage
+
+#### `track_entity(entity_id, source, entity_type, **metadata)`
+
+Track entity provenance.
+
+**Parameters:**
+- `entity_id` (str): Unique identifier for entity
+- `source` (str): Source document or identifier
+- `entity_type` (str): Type of entity
+- `**metadata`: Additional metadata
+
+**Returns:** ProvenanceEntry
+
+#### `track_relationship(relationship_id, source, subject, predicate, obj, **metadata)`
+
+Track relationship provenance.
+
+**Parameters:**
+- `relationship_id` (str): Unique identifier for relationship
+- `source` (str): Source document
+- `subject` (str): Subject entity ID
+- `predicate` (str): Relationship type
+- `obj` (str): Object entity ID
+- `**metadata`: Additional metadata
+
+**Returns:** ProvenanceEntry
+
+#### `track_chunk(chunk_id, source_document, chunk_text, start_char, end_char, **metadata)`
+
+Track document chunk provenance.
+
+**Parameters:**
+- `chunk_id` (str): Unique identifier for chunk
+- `source_document` (str): Source document ID
+- `chunk_text` (str): Text content of chunk
+- `start_char` (int): Start character position
+- `end_char` (int): End character position
+- `**metadata`: Additional metadata
+
+**Returns:** ProvenanceEntry
+
+#### `get_lineage(entity_id)`
+
+Retrieve complete lineage for an entity.
+
+**Parameters:**
+- `entity_id` (str): Entity identifier
+
+**Returns:** dict with lineage information
+
+#### `get_statistics()`
+
+Get provenance statistics.
+
+**Returns:** dict with statistics (total_entries, entities, relationships, chunks)
+
+---
+
+## Testing
+
+Run the provenance test suite:
+
+```bash
+# All provenance tests
+pytest tests/provenance/ -v
+
+# Specific test categories
+pytest tests/provenance/test_manager.py -v
+pytest tests/provenance/test_storage.py -v
+pytest tests/provenance/test_bridge_axiom.py -v
+pytest tests/provenance/test_integration.py -v
+
+# Module integration tests
+pytest tests/provenance/test_semantic_extract_provenance.py -v
+pytest tests/provenance/test_llms_provenance.py -v
+```
+
+---
+
+## Troubleshooting
+
+### Provenance Not Being Tracked
+
+```python
+# Check if provenance is enabled
+print(f"Provenance enabled: {obj.provenance}")
+print(f"Manager available: {obj._prov_manager is not None}")
+```
+
+### Performance Issues
+
+```python
+# Use batch operations
+entities = [{"id": f"entity_{i}"} for i in range(1000)]
+manager.track_entities_batch(entities, source="doc_1")
+```
+
+### Storage Growing Too Large
+
+```python
+# Use separate databases for different time periods
+manager_2026 = ProvenanceManager(storage_path="provenance_2026.db")
+```
+
+---
+
+## See Also
+
+- [Provenance Usage Guide](https://github.com/Hawksight-AI/semantica/blob/main/semantica/provenance/provenance_usage.md) — Comprehensive usage documentation
+- [Change Management](change_management.md) — Version control and audit trails
+- [Conflicts Module](conflicts.md) — Source tracking and conflict resolution
+- [Knowledge Graph](kg.md) — Entity and relationship tracking
+
+---
+
+## License
+
+MIT License - See [LICENSE](../../LICENSE) for details.
+
+## Support
+
+For issues or questions, please open an issue on GitHub or join our [Discord](https://discord.gg/RgaGTj9J).

docs/reference/semantic_extract.md

@@ -23,7 +23,7 @@ The **Semantic Extract Module** extracts structured information from unstructure
 - **High Accuracy**: LLM-based extraction for complex schemas
 - **Flexible Configuration**: Customize extraction for your domain
 - **Confidence Scores**: Get confidence scores for all extractions
-- **Batch Processing**: Efficient batch processing for large datasets
+- **Batch Processing**: Efficient parallel batch processing for large datasets
 - **Coreference Resolution**: Resolve pronouns to their entity references
 
 ### How It Works
@@ -184,16 +184,15 @@ Core entity extraction implementation used by notebooks and lower-level integrat
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `method` | str or list | `"ml"` | Method(s): "ml", "llm", "pattern", "regex", "huggingface" |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
-| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `provider`) |
+| `entity_types` | list | `None` | Filter for specific entity types |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `aggregation_strategy`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text)` | Alias for `extract_entities`. Get list of entities. |
-| `extract_entities(text)` | Get list of entities |
+| `extract(text, pipeline_id=None, **kwargs)` | Alias for `extract_entities`. Supports `max_workers`. |
+| `extract_entities(text, pipeline_id=None, **kwargs)` | Get list of entities. Supports `max_workers`. |
 
 **Example:**
 
@@ -204,11 +203,12 @@ from semantica.semantic_extract import NERExtractor
 extractor = NERExtractor(method="ml", model="en_core_web_trf")
 entities = extractor.extract("Elon Musk leads SpaceX.")
 
-# 2. LLM (OpenAI/Gemini/etc)
+# 2. LLM (OpenAI/Gemini/Groq/etc)
 extractor = NERExtractor(
     method="llm", 
-    provider="openai", 
-    model="gpt-4",
+    provider="groq", 
+    model="llama-3.3-70b-versatile",
+    max_tokens=2048, # Increased output limit
     temperature=0.0
 )
 
@@ -228,17 +228,19 @@ Extracts relationships between entities.
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"dependency"` | Method: "dependency", "pattern", "cooccurrence", "huggingface", "llm" |
 | `relation_types` | list | `None` | Specific relation types to extract |
 | `bidirectional` | bool | `False` | Extract bidirectional relations |
 | `confidence_threshold` | float | `0.6` | Minimum confidence score |
 | `max_distance` | int | `50` | Max token distance between entities |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `device` for HuggingFace) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text, entities)` | Alias for `extract_relations`. Find links. |
-| `extract_relations(text, entities)` | Find links |
+| `extract(text, entities, pipeline_id=None, **kwargs)` | Alias for `extract_relations`. Supports `max_workers`. |
+| `extract_relations(text, entities, pipeline_id=None, **kwargs)` | Find links. Supports `max_workers`. |
 
 **Example:**
 
@@ -308,6 +310,7 @@ Identifies events with temporal information and participants.
 | `extract_participants` | bool | `True` | Extract event participants |
 | `extract_location` | bool | `True` | Extract event locations |
 | `extract_time` | bool | `True` | Extract temporal information |
+| `max_workers` | int | `1` | Threads for parallel batch processing |
 
 **Methods:**
 
@@ -336,17 +339,18 @@ Extracts RDF triplets (Subject-Predicate-Object).
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
+| `triplet_types` | list | `None` | Specific triplet types/predicates to extract |
 | `include_temporal` | bool | `False` | Include time information |
 | `include_provenance` | bool | `False` | Track source sentences |
-| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
+| `**kwargs` | dict | `{}` | Configuration options (e.g., `model`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract_triplets(text)` | Get (S, P, O) tuples |
+| `extract(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Alias for `extract_triplets`. Supports `max_workers`. |
+| `extract_triplets(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Get (S, P, O) tuples. Supports `max_workers`. |
 
 **Example:**
 
@@ -371,6 +375,7 @@ Extracts structured semantic networks with nodes and edges.
 |-----------|------|---------|-------------|
 | `ner_method` | str | `None` | Method for node extraction |
 | `relation_method` | str | `None` | Method for edge extraction |
+| `max_workers` | int | `1` | Threads for parallel batch processing |
 | `**config` | dict | `{}` | Configuration for underlying extractors |
 
 **Methods:**

docs/reference/vector_store.md

@@ -1,12 +1,12 @@
 # Vector Store
 
-> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, and Milvus with Hybrid Search.**
+> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, Pinecone, and Milvus with Hybrid Search.**
 
 ---
 
 ## 🎯 Overview
 
-The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Milvus) and enables semantic search, RAG, and similarity matching.
+The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Pinecone, Milvus) and enables semantic search, RAG, and similarity matching.
 
 ### What is a Vector Store?
 
@@ -18,7 +18,7 @@ A **vector store** is a database optimized for storing and searching high-dimens
 
 ### Why Use the Vector Store Module?
 
-- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, and Milvus
+- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, Pinecone, and Milvus
 - **Unified Interface**: Same API regardless of backend
 - **Hybrid Search**: Combine vector similarity with metadata filtering
 - **Performance**: Optimized for high-throughput search operations
@@ -38,8 +38,8 @@ A **vector store** is a database optimized for storing and searching high-dimens
 -   :material-database:{ .lg .middle } **Multi-Backend Support**
 
     ---
-
-    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, and Milvus
+    
+    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, Pinecone, and Milvus
 
 -   :material-magnify-plus:{ .lg .middle } **Hybrid Search**
 
@@ -112,6 +112,8 @@ The main facade for all vector operations.
 | Method | Description |
 |--------|-------------|
 | `store_vectors(vectors, metadata)` | Store embeddings |
+| `add_documents(documents, metadata, batch_size, parallel)` | **(New)** Store documents with automatic embedding generation and parallelization |
+| `embed_batch(texts)` | **(New)** Generate embeddings for a batch of texts |
 | `search(query, k)` | Semantic search |
 | `delete(ids)` | Remove vectors |
 
@@ -120,15 +122,24 @@ The main facade for all vector operations.
 ```python
 from semantica.vector_store import VectorStore
 
-# Initialize (defaults to FAISS)
+# Initialize (defaults to FAISS, parallel enabled by default with 6 workers)
 store = VectorStore(backend="faiss", dimension=1536)
 
-# Store
+# 1. Store pre-computed vectors
 ids = store.store_vectors(
     vectors=[[0.1, 0.2, ...], ...],
     metadata=[{"text": "Hello"}, ...]
 )
 
+# 2. Store raw documents (High Performance)
+# Automatically handles embedding generation in parallel batches (uses default 6 workers)
+ids = store.add_documents(
+    documents=["Doc 1", "Doc 2", ...],
+    metadata=[{"id": 1}, {"id": 2}, ...],
+    batch_size=32,
+    parallel=True
+)
+
 # Search
 results = store.search(query_vector=[0.1, 0.2, ...], k=5)
 ```
@@ -771,6 +782,7 @@ print(f"Context: {context}")
 ---
 
 ## See Also
+- [High-Performance Usage Guide](../vector_store_usage.md) - **(New)** Parallel ingestion and batching guide
 - [Embeddings Module](embeddings.md) - Generates the vectors
 - [Context Module](context.md) - Uses vector store for memory
 - [Ingest Module](ingest.md) - Source of data

docs/vector_store_usage.md

@@ -0,0 +1,101 @@
+# High-Performance Vector Store Usage
+
+This guide demonstrates how to leverage the new high-performance features of the Semantica Vector Store, specifically designed for efficient batch processing and parallel ingestion of large document sets.
+
+## 🚀 Key Features
+
+- **Parallel Ingestion**: Utilize multi-threading to embed and store documents concurrently.
+- **Batch Processing**: Automatically group documents into batches to minimize overhead.
+- **Unified API**: A single `add_documents` method handles embedding generation and storage.
+
+---
+
+## ⚡ Quick Start: Parallel Ingestion
+
+The fastest way to ingest documents is using the `add_documents` method. Parallelization is enabled by default with optimized settings (6 workers).
+
+```python
+from semantica.vector_store import VectorStore
+import time
+
+store = VectorStore(
+    backend="faiss",
+    dimension=768,
+)
+
+documents = [f"This is document number {i} with some content." for i in range(1000)]
+metadata = [{"source": "generated", "id": i} for i in range(1000)]
+
+start_time = time.time()
+ids = store.add_documents(
+    documents=documents,
+    metadata=metadata,
+    batch_size=64,
+    parallel=True,
+)
+print(f"Ingested {len(ids)} documents in {time.time() - start_time:.2f}s")
+```
+
+---
+
+## 📊 Performance Comparison
+
+### Old Method (Sequential Loop)
+*Slower due to sequential processing and overhead per single item.*
+
+```python
+for doc in documents:
+    emb = embedder.generate(doc)
+    store.store_vectors([emb], [{"text": doc}])
+```
+
+### New Method (Parallel Batching)
+*Significantly faster (3x-10x) by utilizing thread pools and batch operations.*
+
+```python
+store.add_documents(documents, parallel=True)
+```
+
+---
+
+## 🛠 Configuration & Tuning
+
+### `max_workers`
+Controls the number of concurrent threads used for embedding generation.
+- **Default**: 6 (Optimized for most systems)
+- **Recommendation**: You generally don't need to change this. If you have very high core counts or specific throughput needs, you can override it.
+
+```python
+store = VectorStore(max_workers=16)
+```
+
+### `batch_size`
+Controls how many documents are processed in a single chunk.
+- **Default**: 32
+- **Recommendation**: 
+    - **Local Models**: 32-64 usually works well.
+    - **API Models (OpenAI, etc.)**: Larger batches (e.g., 100-200) can reduce network latency overhead.
+
+```python
+store.add_documents(documents, batch_size=100)
+```
+
+---
+
+## 🧩 Advanced: Manual Batch Embedding
+
+If you need the embeddings without storing them immediately, use `embed_batch`.
+
+```python
+vectors = store.embed_batch(
+    texts=documents[:100],
+)
+
+print(f"Generated {len(vectors)} vectors")
+```
+
+## ⚠️ Best Practices
+
+1.  **Metadata Consistency**: Ensure your `metadata` list has the same length as your `documents` list.
+2.  **Error Handling**: The `add_documents` method will propagate exceptions if embedding fails. Ensure your data is clean.
+3.  **Memory Usage**: Very large `batch_size` combined with high `max_workers` can increase memory usage. Monitor your system resources.

integrations/__init__.py

@@ -0,0 +1,7 @@
+"""
+Semantica Framework Integrations
+
+Optional integration packages for agentic frameworks (Google ADK, Claude Agent SDK, Agno, etc.).
+Each integration is self-contained, independently installable via extras_require, and maintains
+zero impact on core Semantica - keeping the semantic layer lean while maximizing ecosystem reach.
+"""

mkdocs.yml

@@ -107,6 +107,7 @@ nav:
     - installation.md
     - quickstart.md
   - Docs:
+    - Change Management: reference/change_management.md
     - Conflicts: reference/conflicts.md
     - Context: reference/context.md
     - Core: reference/core.md
@@ -122,6 +123,7 @@ nav:
     - Ontology: reference/ontology.md
     - Parse: reference/parse.md
     - Pipeline: reference/pipeline.md
+    - Provenance: reference/provenance.md
     - Reasoning: reference/reasoning.md
     - Seed: reference/seed.md
     - Semantic Extract: reference/semantic_extract.md

pyproject.toml

@@ -4,317 +4,208 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantica"
-version = "0.2.0"
-description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering "
+version = "0.2.6"
+description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering"
 readme = "README.md"
-license = {text = "MIT"}
-authors = [
-    {name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com"}
-]
-maintainers = [
-    {name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com"}
-]
+license = { text = "MIT" }
+
+authors = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
+maintainers = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
+
 requires-python = ">=3.8"
+
 classifiers = [
-    "Development Status :: 3 - Alpha",
-    "Intended Audience :: Developers",
-    "Intended Audience :: Science/Research",
-    "License :: OSI Approved :: MIT License",
-    "Operating System :: OS Independent",
-    "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
-    "Programming Language :: Python :: 3.11",
-    "Programming Language :: Python :: 3.12",
-    "Topic :: Scientific/Engineering :: Artificial Intelligence",
-    "Topic :: Software Development :: Libraries :: Python Modules",
-    "Topic :: Text Processing :: Linguistic",
-    "Topic :: Database :: Database Engines/Servers",
-    "Topic :: Internet :: WWW/HTTP :: Indexing/Search"
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Software Development :: Libraries :: Python Modules"
 ]
+
 keywords = [
-    "semantic-layer", "knowledge-engineering", "nlp", "knowledge-graph",
-    "embeddings", "entity-extraction", "relationship-extraction", "rdf",
-    "ontology", "semantic-analysis", "ai", "machine-learning"
+  "semantic-layer", "knowledge-graph", "nlp", "embeddings",
+  "entity-extraction", "relationship-extraction", "rdf", "ontology"
 ]
+
+# ---------------- CORE DEPENDENCIES (SAFE DEFAULT) ----------------
 dependencies = [
-    "numpy>=1.21.0",
-    "pandas>=1.3.0",
-    "scikit-learn>=1.0.0",
-    "umap-learn>=0.5.0",
-    "spacy>=3.4.0",
-    "transformers>=4.20.0",
-    "torch>=1.12.0",
-    "sentence-transformers>=2.2.0",
-    "rdflib>=6.2.0",
-    "networkx>=2.8.0",
-    "matplotlib>=3.5.0",
-    "seaborn>=0.11.0",
-    "plotly>=5.10.0",
-    "ipywidgets>=8.0.0",
-    "requests>=2.28.0",
-    "GitPython>=3.1.30",
-    "chardet>=5.1.0",
-    "protobuf==4.25.8",
-    "grpcio==1.67.1",
-    "beautifulsoup4>=4.11.0",
-    "lxml>=4.9.0",
-    "pypdf2>=2.10.0",
-    "python-docx>=0.8.11",
-    "docling>=1.0.0",
-    "openpyxl>=3.0.10",
-    "pillow>=9.2.0",
-    "librosa>=0.9.0",
-    "opencv-python>=4.6.0",
-    "faiss-cpu>=1.7.0",
-    "fastembed>=0.2.0",
-    "onnxruntime>=1.17.0",
-    "tokenizers>=0.15.0",
-    "weaviate-client>=3.15.0",
-    "qdrant-client>=1.3.0",
-    "neo4j>=5.0.0",
-    "falkordb>=1.0.0",
-    "pymongo>=4.2.0",
-    "sqlalchemy>=1.4.0",
-    "psycopg2-binary>=2.9.0",
-    "pymysql>=1.0.0",
-    "redis>=4.3.0",
-    "celery>=5.2.0",
-    "kafka-python>=2.0.0",
-    "pulsar-client>=3.0.0",
-    "pika>=1.3.0",
-    "boto3>=1.24.0",
-    "azure-storage-blob>=12.12.0",
-    "google-cloud-storage>=2.5.0",
-    "pydantic>=2.0.0",
-    "fastmcp>=0.1.0",
-    "groq>=0.4.0",
-    "openai>=1.0.0",
-    "litellm>=1.0.0",
-    "instructor>=1.0.0",
-    "click>=8.1.0",
-    "rich>=12.5.0",
-    "tqdm>=4.64.0",
-    "pyyaml>=6.0",
-    "toml>=0.10.0",
-    "python-dotenv>=0.20.0",
-    "loguru>=0.6.0",
-    "structlog>=22.1.0",
-    "prometheus-client>=0.14.0",
-    "opentelemetry-api>=1.12.0",
-    "opentelemetry-sdk>=1.12.0",
-    "opentelemetry-instrumentation",
-    "fastapi>=0.78.0",
-    "uvicorn>=0.18.0",
-    "pytest>=7.1.0",
-    "pytest-cov>=3.0.0",
-    "pytest-asyncio>=0.19.0",
-    "black>=22.6.0",
-    "isort>=5.10.0",
-    "flake8>=4.0.0",
-    "mypy>=0.971",
-    "pre-commit>=2.19.0"
+  "numpy>=1.21.0",
+  "pandas>=1.3.0",
+  "scikit-learn>=1.0.0",
+  "umap-learn>=0.5.0",
+  "spacy>=3.4.0",
+  "transformers>=4.20.0",
+  "torch>=1.12.0",
+  "sentence-transformers>=2.2.0",
+  "rdflib>=6.2.0",
+  "networkx>=2.8.0",
+  "matplotlib>=3.5.0",
+  "seaborn>=0.11.0",
+  "plotly>=5.10.0",
+  "ipywidgets>=8.0.0",
+  "requests>=2.28.0",
+  "GitPython>=3.1.30",
+  "chardet>=5.1.0",
+  "protobuf>=5.29.1,<7.0",
+  "grpcio>=1.71.2",
+  "beautifulsoup4>=4.11.0",
+  "lxml>=4.9.0",
+  "pypdf2>=2.10.0",
+  "python-docx>=0.8.11",
+  "openpyxl>=3.0.10",
+  "pillow>=9.2.0",
+  "librosa>=0.9.0",
+  "opencv-python>=4.6.0",
+  "faiss-cpu>=1.7.0",
+  "fastembed>=0.2.0",
+  "onnxruntime>=1.17.0",
+  "tokenizers>=0.15.0",
+  "pydantic>=2.0.0",
+  "click>=8.1.0",
+  "rich>=12.5.0",
+  "tqdm>=4.64.0",
+  "pyyaml>=6.0",
+  "toml>=0.10.0",
+  "python-dotenv>=0.20.0",
+  "loguru>=0.6.0",
+  "structlog>=22.1.0"
 ]
-[project.urls]
-Homepage = "https://github.com/Hawksight-AI/semantica"
-Repository = "https://github.com/Hawksight-AI/semantica"
-"Bug Tracker" = "https://github.com/Hawksight-AI/semantica/issues"
-Discussions = "https://github.com/Hawksight-AI/semantica/discussions"
 
+# ---------------- OPTIONAL DEPENDENCIES ----------------
 [project.optional-dependencies]
-dev = [
-    "pytest>=7.1.0",
-    "pytest-cov>=3.0.0",
-    "pytest-asyncio>=0.19.0",
-    "black>=22.6.0",
-    "isort>=5.10.0",
-    "flake8>=4.0.0",
-    "mypy>=0.971",
-    "pre-commit>=2.19.0",
-    "jupyter>=1.0.0",
-    "ipykernel>=6.15.0",
-    "notebook>=6.4.0"
-]
-viz = [
-    "pyvis>=0.3.0",
-    "graphviz>=0.20.0",
-    "umap-learn>=0.5.0",
-    "d3blocks>=1.0.0"
-]
-gpu = [
-    "torch>=1.12.0",
-    "faiss-gpu>=1.7.0",
-    "cupy>=10.0.0"
-]
-cloud = [
-    "boto3>=1.24.0",
-    "azure-storage-blob>=12.12.0",
-    "google-cloud-storage>=2.5.0",
-    "kubernetes>=24.0.0",
-    "helm>=3.10.0"
-]
-monitoring = [
-    "prometheus-client>=0.14.0",
-    "opentelemetry-api>=1.12.0",
-    "opentelemetry-sdk>=1.12.0",
-    "opentelemetry-instrumentation>=0.32.0",
-    "grafana-api>=1.0.0",
-    "elasticsearch>=8.5.0"
-]
-llm-openai = [
-    "openai>=1.0.0"
-]
-llm-gemini = [
-    "google-generativeai>=0.3.0"
-]
-llm-groq = [
-    "groq>=0.4.0"
-]
-llm-anthropic = [
-    "anthropic>=0.18.0"
-]
-llm-ollama = [
-    "ollama>=0.1.0"
-]
-llm-deepseek = [
-    "deepseek>=0.1.0"
-]
-llm-litellm = [
-    "litellm>=1.0.0"
-]
-llm-instructor = [
-    "instructor>=1.0.0"
-]
+
+# ---- LLM Providers ----
+llm-openai = ["openai>=1.0.0"]
+llm-groq = ["groq>=0.4.0"]
+llm-gemini = ["google-genai>=0.1.0"]
+llm-anthropic = ["anthropic>=0.18.0"]
+llm-ollama = ["ollama>=0.1.0"]
+llm-deepseek = ["deepseek>=0.1.0"]
+llm-litellm = ["litellm>=1.0.0"]
+llm-instructor = ["instructor>=1.0.0"]
+
 llm-all = [
-    "semantica[llm-openai,llm-gemini,llm-groq,llm-anthropic,llm-ollama,llm-deepseek,llm-litellm,llm-instructor]"
-]
-models-huggingface = [
-    "transformers>=4.20.0",
-    "torch>=1.12.0"
-]
-split-tiktoken = [
-    "tiktoken>=0.5.0"
-]
-split-community = [
-    "python-louvain>=0.16"
-]
-split-topic = [
-    "bertopic>=0.15.0",
-    "gensim>=4.3.0"
-]
-split-all = [
-    "semantica[split-tiktoken,split-community,split-topic]"
-]
-graph-neo4j = [
-    "neo4j>=5.0.0"
-]
-graph-falkordb = [
-    "falkordb>=1.0.0",
-    "redis>=4.3.0"
-]
-graph-amazon-neptune = [
-    "boto3>=1.24.0",
-    "neo4j>=5.0.0"
-]
-graph-all = [
-    "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune]"
-]
-parse-docling = [
-    "docling>=1.0.0"
-]
-all = [
-    "semantica[dev,viz,gpu,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,parse-docling]"
+  "semantica[llm-openai,llm-groq,llm-gemini,llm-anthropic,llm-ollama,llm-deepseek,llm-litellm,llm-instructor]"
 ]
 
+# ---- Document Parsing ----
+parse-docling = ["docling>=1.0.0"]
+
+# ---- Embedding / Models ----
+models-huggingface = [
+  "transformers>=4.20.0",
+  "torch>=1.12.0"
+]
+
+# ---- Graph Backends ----
+graph-neo4j = ["neo4j>=5.0.0"]
+graph-falkordb = ["falkordb>=1.0.0", "redis>=4.3.0"]
+graph-amazon-neptune = ["boto3>=1.24.0", "neo4j>=5.0.0"]
+
+graph-all = [
+  "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune]"
+]
+
+# ---- Vector Store Backends ----
+vectorstore-qdrant = ["qdrant-client>=1.0.0"]
+vectorstore-weaviate = ["weaviate-client>=4.0.0"]
+vectorstore-pinecone = ["pinecone-client>=3.0.0"]
+vectorstore-milvus = ["pymilvus>=2.0.0"]
+
+vectorstore-all = [
+  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus]"
+]
+
+# ---- Infra / Queues / Workers ----
+infra = [
+  "redis>=4.3.0",
+  "celery>=5.2.0",
+  "kafka-python>=2.0.0",
+  "pulsar-client>=3.0.0",
+  "pika>=1.3.0"
+]
+
+# ---- Cloud Providers ----
+cloud = [
+  "boto3>=1.24.0",
+  "azure-storage-blob>=12.12.0",
+  "google-cloud-storage>=2.5.0"
+]
+
+# ---- Monitoring (FIXED) ----
+monitoring = [
+  "prometheus-client>=0.14.0",
+  "opentelemetry-api>=1.30.0,<2.0.0",
+  "opentelemetry-sdk>=1.30.0,<2.0.0",
+  "opentelemetry-semantic-conventions>=0.58b0,<0.61b0",
+  "opentelemetry-instrumentation>=0.58b0,<0.61b0"
+]
+
+# ---- Visualization ----
+viz = [
+  "pyvis>=0.3.0",
+  "graphviz>=0.20.0",
+  "d3blocks>=1.0.0"
+]
+
+# ---- GPU ----
+gpu = [
+  "faiss-gpu>=1.7.0",
+  "cupy>=10.0.0"
+]
+
+# ---- Splitting / Chunking ----
+split-tiktoken = ["tiktoken>=0.5.0"]
+split-community = ["python-louvain>=0.16"]
+split-topic = ["bertopic>=0.15.0", "gensim>=4.3.0"]
+
+split-all = [
+  "semantica[split-tiktoken,split-community,split-topic]"
+]
+
+# ---- Dev ----
+dev = [
+  "pytest>=7.1.0",
+  "pytest-cov>=3.0.0",
+  "pytest-asyncio>=0.19.0",
+  "black>=22.6.0",
+  "isort>=5.10.0",
+  "flake8>=4.0.0",
+  "mypy>=0.971",
+  "pre-commit>=2.19.0",
+  "jupyter>=1.0.0",
+  "ipykernel>=6.15.0"
+]
+
+# ---- Everything ----
+all = [
+  "semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling]"
+]
+
+# ---------------- ENTRYPOINTS ----------------
 [project.scripts]
 semantica = "semantica.cli:main"
 semantica-server = "semantica.server:main"
 semantica-worker = "semantica.worker:main"
 
+# ---------------- TOOLING ----------------
 [tool.setuptools.packages.find]
 where = ["."]
 include = ["semantica*"]
-exclude = ["tests*", "docs*", "examples*"]
-
-[tool.setuptools.package-data]
-semantica = ["*.yaml", "*.yml", "*.json", "*.toml", "*.txt", "*.md"]
 
 [tool.black]
 line-length = 88
-target-version = ['py38', 'py39', 'py310', 'py311', 'py312']
-include = '\.pyi?$'
-extend-exclude = '''
-/(
-  # directories
-  \.eggs
-  | \.git
-  | \.hg
-  | \.mypy_cache
-  | \.tox
-  | \.venv
-  | build
-  | dist
-)/
-'''
 
 [tool.isort]
 profile = "black"
-multi_line_output = 3
-line_length = 88
-known_first_party = ["semantica"]
-known_third_party = ["numpy", "pandas", "scikit-learn", "spacy", "transformers", "torch"]
-
-[tool.mypy]
-python_version = "3.9"
-warn_return_any = true
-warn_unused_configs = true
-disallow_untyped_defs = true
-disallow_incomplete_defs = true
-check_untyped_defs = true
-disallow_untyped_decorators = true
-no_implicit_optional = true
-warn_redundant_casts = true
-warn_unused_ignores = true
-warn_no_return = true
-warn_unreachable = true
-strict_equality = true
-show_error_codes = true
 
 [tool.pytest.ini_options]
-minversion = "7.0"
-addopts = "-ra -q --strict-markers --strict-config"
 testpaths = ["tests"]
-python_files = ["test_*.py", "*_test.py"]
-python_classes = ["Test*"]
-python_functions = ["test_*"]
-markers = [
-    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
-    "integration: marks tests as integration tests",
-    "unit: marks tests as unit tests",
-    "gpu: marks tests that require GPU",
-    "cloud: marks tests that require cloud services"
-]
-
-[tool.coverage.run]
-source = ["semantica"]
-omit = [
-    "*/tests/*",
-    "*/test_*",
-    "*/__pycache__/*",
-    "*/migrations/*"
-]
-
-[tool.coverage.report]
-exclude_lines = [
-    "pragma: no cover",
-    "def __repr__",
-    "if self.debug:",
-    "if settings.DEBUG",
-    "raise AssertionError",
-    "raise NotImplementedError",
-    "if 0:",
-    "if __name__ == .__main__.:",
-    "class .*\\bProtocol\\):",
-    "@(abc\\.)?abstractmethod"
-]

pytest.ini

@@ -0,0 +1,4 @@
+[pytest]
+markers =
+    integration: marks tests as integration (deselect with '-m "not integration"')
+addopts = -ra

semantica/__init__.py

@@ -10,7 +10,7 @@ Main exports:
     - Config: Configuration management
 """
 
-__version__ = "0.2.0"
+__version__ = "0.2.6"
 __author__ = "Semantica Contributors"
 __license__ = "MIT"
 

semantica/change_management/__init__.py

@@ -0,0 +1,63 @@
+"""
+Enhanced Change Management Module for Semantica
+
+This module provides comprehensive change management capabilities including:
+- Persistent version storage (SQLite and in-memory)
+- Detailed change tracking and diff algorithms
+- Standardized metadata and audit trails
+- Data integrity verification with checksums
+- Enhanced version managers for KG and ontologies
+- Enterprise compliance support (HIPAA, SOX, FDA)
+
+Public API:
+    ChangeLogEntry: Standardized metadata for version changes
+    VersionStorage: Abstract storage interface
+    InMemoryVersionStorage: Fast in-memory storage backend
+    SQLiteVersionStorage: Persistent SQLite storage backend
+    compute_checksum: SHA-256 checksum computation
+    verify_checksum: Data integrity verification
+    EnhancedTemporalVersionManager: Advanced KG version management
+    EnhancedVersionManager: Advanced ontology version management
+"""
+
+from .change_log import ChangeLogEntry
+from .version_storage import (
+    VersionStorage,
+    InMemoryVersionStorage,
+    SQLiteVersionStorage,
+    compute_checksum,
+    verify_checksum
+)
+from .managers import (
+    BaseVersionManager,
+    TemporalVersionManager,
+    OntologyVersionManager
+)
+from .ontology_version_manager import VersionManager, OntologyVersion
+
+__all__ = [
+    # Change metadata
+    "ChangeLogEntry",
+    
+    # Storage backends
+    "VersionStorage",
+    "InMemoryVersionStorage", 
+    "SQLiteVersionStorage",
+    
+    # Integrity utilities
+    "compute_checksum",
+    "verify_checksum",
+    
+    # Version managers
+    "BaseVersionManager",
+    "TemporalVersionManager",
+    "OntologyVersionManager",
+    
+    # Ontology version management
+    "VersionManager",
+    "OntologyVersion"
+]
+
+__version__ = "1.0.0"
+__author__ = "Semantica Team"
+__description__ = "Enhanced Change Management for Semantica"

semantica/change_management/change_log.py

@@ -0,0 +1,108 @@
+"""
+Change Log Module
+
+This module provides standardized metadata structures for version changes
+across both ontology and knowledge graph versioning systems.
+
+Key Features:
+    - Standardized ChangeLogEntry dataclass
+    - Email validation for authors
+    - Timestamp handling in ISO 8601 format
+    - Optional change linking and tracking
+
+Main Classes:
+    - ChangeLogEntry: Standard metadata for version changes
+
+Example Usage:
+    >>> from semantica.common.change_log import ChangeLogEntry
+    >>> entry = ChangeLogEntry(
+    ...     timestamp="2024-01-15T10:30:00Z",
+    ...     author="alice@company.com",
+    ...     description="Added Customer entity"
+    ... )
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import re
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import List, Optional
+
+from ..utils.exceptions import ValidationError
+
+
+@dataclass
+class ChangeLogEntry:
+    """
+    Standard metadata for version changes.
+    
+    This dataclass provides a consistent structure for tracking changes
+    across both ontology and knowledge graph versioning systems.
+    
+    Attributes:
+        timestamp: ISO 8601 timestamp of the change
+        author: Email address of the change author
+        description: Description of the change (max 500 characters)
+        change_id: Optional unique identifier for the change
+        related_changes: Optional list of related change IDs
+    """
+    
+    timestamp: str
+    author: str
+    description: str
+    change_id: Optional[str] = None
+    related_changes: List[str] = field(default_factory=list)
+    
+    def __post_init__(self):
+        """Validate fields after initialization."""
+        self._validate_timestamp()
+        self._validate_author()
+        self._validate_description()
+    
+    def _validate_timestamp(self):
+        """Validate timestamp is in ISO 8601 format."""
+        try:
+            # More strict validation for ISO 8601 format
+            if 'T' not in self.timestamp:
+                raise ValueError("Missing 'T' separator")
+            datetime.fromisoformat(self.timestamp.replace('Z', '+00:00'))
+        except ValueError:
+            raise ValidationError(f"Invalid timestamp format: {self.timestamp}. Expected ISO 8601 format.")
+    
+    def _validate_author(self):
+        """Validate author is a valid email address."""
+        email_pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
+        if not re.match(email_pattern, self.author):
+            raise ValidationError(f"Invalid email format: {self.author}")
+    
+    def _validate_description(self):
+        """Validate description length."""
+        if len(self.description) > 500:
+            raise ValidationError(f"Description too long: {len(self.description)} characters (max 500)")
+        if not self.description.strip():
+            raise ValidationError("Description cannot be empty")
+    
+    @classmethod
+    def create_now(cls, author: str, description: str, change_id: Optional[str] = None, 
+                   related_changes: Optional[List[str]] = None) -> 'ChangeLogEntry':
+        """
+        Create a ChangeLogEntry with current timestamp.
+        
+        Args:
+            author: Email address of the change author
+            description: Description of the change
+            change_id: Optional unique identifier for the change
+            related_changes: Optional list of related change IDs
+            
+        Returns:
+            ChangeLogEntry with current timestamp
+        """
+        return cls(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description,
+            change_id=change_id,
+            related_changes=related_changes or []
+        )

semantica/change_management/managers.py

@@ -0,0 +1,497 @@
+"""
+Enhanced Version Managers Module
+
+This module provides enhanced version management capabilities for both knowledge graphs
+and ontologies, with comprehensive change tracking, persistent storage, and audit trails.
+
+Key Features:
+    - Enhanced TemporalVersionManager for knowledge graphs
+    - Enhanced VersionManager for ontologies  
+    - Detailed diff algorithms for entities and relationships
+    - Structural comparison for ontology elements
+    - Integration with storage backends and metadata
+
+Main Classes:
+    - EnhancedTemporalVersionManager: Advanced KG version management
+    - EnhancedVersionManager: Advanced ontology version management
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from abc import ABC, abstractmethod
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from .change_log import ChangeLogEntry
+from .version_storage import VersionStorage, InMemoryVersionStorage, SQLiteVersionStorage, compute_checksum, verify_checksum
+from ..utils.exceptions import ValidationError, ProcessingError
+from ..utils.logging import get_logger
+
+
+class BaseVersionManager(ABC):
+    """
+    Abstract base class for enhanced version managers.
+    
+    Provides common functionality for version management across different data types.
+    """
+    
+    def __init__(self, storage_path: Optional[str] = None):
+        """
+        Initialize base version manager.
+        
+        Args:
+            storage_path: Path to SQLite database for persistent storage.
+                         If None, uses in-memory storage.
+        """
+        self.logger = get_logger(self.__class__.__name__.lower())
+        
+        # Initialize storage backend
+        if storage_path:
+            self.storage = SQLiteVersionStorage(storage_path)
+            self.logger.info(f"Initialized with SQLite storage: {storage_path}")
+        else:
+            self.storage = InMemoryVersionStorage()
+            self.logger.info("Initialized with in-memory storage")
+    
+    @abstractmethod
+    def create_snapshot(self, data: Any, version_label: str, author: str, description: str, **options) -> Dict[str, Any]:
+        """Create a versioned snapshot of the data."""
+        pass
+    
+    @abstractmethod
+    def compare_versions(self, version1: Any, version2: Any, **options) -> Dict[str, Any]:
+        """Compare two versions and return detailed differences."""
+        pass
+    
+    def list_versions(self) -> List[Dict[str, Any]]:
+        """List all version snapshots."""
+        return self.storage.list_all()
+    
+    def get_version(self, label: str) -> Optional[Dict[str, Any]]:
+        """Retrieve specific version by label."""
+        return self.storage.get(label)
+    
+    def verify_checksum(self, snapshot: Dict[str, Any]) -> bool:
+        """Verify the integrity of a snapshot using its checksum."""
+        return verify_checksum(snapshot)
+
+
+class TemporalVersionManager(BaseVersionManager):
+    """
+    Temporal version management engine for knowledge graphs.
+    
+    Provides comprehensive version/snapshot management capabilities including
+    persistent storage, detailed change tracking, and audit trails.
+    
+    Features:
+        - Persistent snapshot storage (SQLite or in-memory)
+        - Detailed change tracking with entity-level diffs
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+        - Version comparison with backward compatibility
+        - Input validation and security features
+    """
+    
+    def __init__(self, storage_path: Optional[str] = None, **config):
+        """
+        Initialize enhanced temporal version manager.
+        
+        Args:
+            storage_path: Path to SQLite database file for persistent storage.
+                         If None, uses in-memory storage
+            **config: Additional configuration options
+        """
+        super().__init__(storage_path)
+        self.config = config
+    
+    def create_snapshot(
+        self, 
+        graph: Dict[str, Any], 
+        version_label: str, 
+        author: str, 
+        description: str,
+        **options
+    ) -> Dict[str, Any]:
+        """
+        Create and store snapshot with checksum and metadata.
+        
+        Args:
+            graph: Knowledge graph dict with "entities" and "relationships"
+            version_label: Version string (e.g., "v1.0")
+            author: Email address of the change author
+            description: Change description (max 500 chars)
+            **options: Additional options
+            
+        Returns:
+            dict: Snapshot with metadata and checksum
+            
+        Raises:
+            ValidationError: If input validation fails
+            ProcessingError: If storage operation fails
+        """
+        # Validate inputs
+        change_entry = ChangeLogEntry(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description
+        )
+        
+        # Create snapshot
+        snapshot = {
+            "label": version_label,
+            "timestamp": change_entry.timestamp,
+            "author": change_entry.author,
+            "description": change_entry.description,
+            "entities": graph.get("entities", []).copy(),
+            "relationships": graph.get("relationships", []).copy(),
+            "metadata": options.get("metadata", {})
+        }
+        
+        # Compute and add checksum
+        snapshot["checksum"] = compute_checksum(snapshot)
+        
+        # Store snapshot
+        self.storage.save(snapshot)
+        
+        self.logger.info(f"Created snapshot '{version_label}' by {author}")
+        return snapshot
+    
+    def compare_versions(
+        self,
+        v1_label_or_dict,
+        v2_label_or_dict,
+        comparison_metrics: Optional[List[str]] = None,
+        **options,
+    ) -> Dict[str, Any]:
+        """
+        Compare two graph versions with detailed entity-level differences.
+        
+        Args:
+            v1_label_or_dict: First version (label string or snapshot dict)
+            v2_label_or_dict: Second version (label string or snapshot dict)
+            comparison_metrics: List of metrics to calculate (optional, unused)
+            **options: Additional comparison options (unused)
+            
+        Returns:
+            dict: Detailed version comparison results
+        """
+        # Handle both label strings and snapshot dictionaries
+        if isinstance(v1_label_or_dict, str):
+            version1 = self.storage.get(v1_label_or_dict)
+            if not version1:
+                raise ValidationError(f"Version not found: {v1_label_or_dict}")
+        else:
+            version1 = v1_label_or_dict
+            
+        if isinstance(v2_label_or_dict, str):
+            version2 = self.storage.get(v2_label_or_dict)
+            if not version2:
+                raise ValidationError(f"Version not found: {v2_label_or_dict}")
+        else:
+            version2 = v2_label_or_dict
+        
+        # Compute detailed diff
+        detailed_diff = self._compute_detailed_diff(version1, version2)
+        
+        # Maintain backward compatibility with summary
+        summary = {
+            "entities_added": len(detailed_diff["entities_added"]),
+            "entities_removed": len(detailed_diff["entities_removed"]),
+            "entities_modified": len(detailed_diff["entities_modified"]),
+            "relationships_added": len(detailed_diff["relationships_added"]),
+            "relationships_removed": len(detailed_diff["relationships_removed"]),
+            "relationships_modified": len(detailed_diff["relationships_modified"])
+        }
+        
+        return {
+            "version1": version1.get("label", "unknown"),
+            "version2": version2.get("label", "unknown"),
+            "summary": summary,
+            **detailed_diff
+        }
+    
+    def _compute_detailed_diff(self, version1: Dict[str, Any], version2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute detailed entity and relationship differences between versions.
+        
+        Args:
+            version1: First version snapshot
+            version2: Second version snapshot
+            
+        Returns:
+            Dict with detailed diff information
+        """
+        entities1 = {e.get("id", str(i)): e for i, e in enumerate(version1.get("entities", []))}
+        entities2 = {e.get("id", str(i)): e for i, e in enumerate(version2.get("entities", []))}
+        
+        relationships1 = {self._relationship_key(r): r for r in version1.get("relationships", [])}
+        relationships2 = {self._relationship_key(r): r for r in version2.get("relationships", [])}
+        
+        # Entity differences
+        entity_ids1 = set(entities1.keys())
+        entity_ids2 = set(entities2.keys())
+        
+        entities_added = [entities2[eid] for eid in entity_ids2 - entity_ids1]
+        entities_removed = [entities1[eid] for eid in entity_ids1 - entity_ids2]
+        
+        entities_modified = []
+        for eid in entity_ids1 & entity_ids2:
+            if entities1[eid] != entities2[eid]:
+                changes = self._compute_entity_changes(entities1[eid], entities2[eid])
+                entities_modified.append({
+                    "id": eid,
+                    "before": entities1[eid],
+                    "after": entities2[eid],
+                    "changes": changes
+                })
+        
+        # Relationship differences
+        rel_keys1 = set(relationships1.keys())
+        rel_keys2 = set(relationships2.keys())
+        
+        relationships_added = [relationships2[key] for key in rel_keys2 - rel_keys1]
+        relationships_removed = [relationships1[key] for key in rel_keys1 - rel_keys2]
+        
+        relationships_modified = []
+        for key in rel_keys1 & rel_keys2:
+            if relationships1[key] != relationships2[key]:
+                changes = self._compute_relationship_changes(relationships1[key], relationships2[key])
+                relationships_modified.append({
+                    "key": key,
+                    "before": relationships1[key],
+                    "after": relationships2[key],
+                    "changes": changes
+                })
+        
+        return {
+            "entities_added": entities_added,
+            "entities_removed": entities_removed,
+            "entities_modified": entities_modified,
+            "relationships_added": relationships_added,
+            "relationships_removed": relationships_removed,
+            "relationships_modified": relationships_modified
+        }
+    
+    def _relationship_key(self, relationship: Dict[str, Any]) -> str:
+        """Generate a unique key for a relationship."""
+        source = relationship.get("source", "")
+        target = relationship.get("target", "")
+        rel_type = relationship.get("type", relationship.get("relationship", ""))
+        return f"{source}|{rel_type}|{target}"
+    
+    def _compute_entity_changes(self, entity1: Dict[str, Any], entity2: Dict[str, Any]) -> Dict[str, Any]:
+        """Compute changes between two entity versions."""
+        changes = {}
+        all_keys = set(entity1.keys()) | set(entity2.keys())
+        
+        for key in all_keys:
+            val1 = entity1.get(key)
+            val2 = entity2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes
+    
+    def _compute_relationship_changes(self, rel1: Dict[str, Any], rel2: Dict[str, Any]) -> Dict[str, Any]:
+        """Compute changes between two relationship versions."""
+        changes = {}
+        all_keys = set(rel1.keys()) | set(rel2.keys())
+        
+        for key in all_keys:
+            val1 = rel1.get(key)
+            val2 = rel2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes
+
+
+class OntologyVersionManager(BaseVersionManager):
+    """
+    Version management for ontologies with structural comparison.
+    
+    Provides comprehensive version management for ontologies including
+    detailed structural analysis and change tracking.
+    
+    Features:
+        - Structural comparison of ontology elements
+        - Detailed diff for classes, properties, individuals, axioms
+        - Persistent storage with metadata
+        - Change tracking and audit trails
+    """
+    
+    def __init__(self, storage_path: Optional[str] = None, **config):
+        """
+        Initialize enhanced version manager.
+        
+        Args:
+            storage_path: Path to SQLite database file for persistent storage.
+                         If None, uses in-memory storage
+            **config: Additional configuration options
+        """
+        super().__init__(storage_path)
+        self.config = config
+        self.versions = {}  # In-memory version tracking for compatibility
+    
+    def create_snapshot(
+        self,
+        ontology_data: Dict[str, Any],
+        version_label: str,
+        author: str,
+        description: str,
+        **options
+    ) -> Dict[str, Any]:
+        """
+        Create ontology version snapshot.
+        
+        Args:
+            ontology_data: Ontology data dictionary
+            version_label: Version string (e.g., "v1.0")
+            author: Email address of the change author
+            description: Change description
+            **options: Additional options including metadata
+            
+        Returns:
+            dict: Ontology version snapshot
+        """
+        # Validate inputs
+        change_entry = ChangeLogEntry(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description
+        )
+        
+        # Create snapshot
+        snapshot = {
+            "label": version_label,
+            "timestamp": change_entry.timestamp,
+            "author": change_entry.author,
+            "description": change_entry.description,
+            "ontology_iri": ontology_data.get("uri", ""),
+            "version_info": ontology_data.get("version_info", {}),
+            "structure": ontology_data.get("structure", {}),
+            "metadata": options.get("metadata", {})
+        }
+        
+        # Compute and add checksum
+        snapshot["checksum"] = compute_checksum(snapshot)
+        
+        # Store snapshot
+        self.storage.save(snapshot)
+        
+        # Also store in memory for compatibility
+        self.versions[version_label] = snapshot
+        
+        self.logger.info(f"Created ontology snapshot '{version_label}' by {author}")
+        return snapshot
+    
+    def compare_versions(self, version1: str, version2: str, **options) -> Dict[str, Any]:
+        """
+        Compare two ontology versions with detailed structural analysis.
+        
+        Args:
+            version1: First version label
+            version2: Second version label
+            **options: Additional comparison options
+            
+        Returns:
+            Detailed comparison results including structural differences
+        """
+        # Get versions from storage
+        v1_snapshot = self.storage.get(version1)
+        v2_snapshot = self.storage.get(version2)
+        
+        if not v1_snapshot:
+            raise ValidationError(f"Version not found: {version1}")
+        if not v2_snapshot:
+            raise ValidationError(f"Version not found: {version2}")
+        
+        # Basic metadata comparison
+        metadata_changes = {}
+        if v1_snapshot.get("ontology_iri") != v2_snapshot.get("ontology_iri"):
+            metadata_changes["ontology_iri"] = {
+                "from": v1_snapshot.get("ontology_iri"),
+                "to": v2_snapshot.get("ontology_iri")
+            }
+        if v1_snapshot.get("version_info") != v2_snapshot.get("version_info"):
+            metadata_changes["version_info"] = {
+                "from": v1_snapshot.get("version_info"),
+                "to": v2_snapshot.get("version_info")
+            }
+        
+        # Structural comparison
+        structural_diff = self._compare_ontology_structures(v1_snapshot, v2_snapshot)
+        
+        return {
+            "version1": version1,
+            "version2": version2,
+            "metadata_changes": metadata_changes,
+            **structural_diff
+        }
+    
+    def _compare_ontology_structures(self, v1_snapshot: Dict[str, Any], v2_snapshot: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compare structural elements between two ontology versions.
+        
+        Args:
+            v1_snapshot: First ontology version snapshot
+            v2_snapshot: Second ontology version snapshot
+            
+        Returns:
+            Dictionary with structural differences
+        """
+        # Extract structural information
+        v1_structure = v1_snapshot.get("structure", {})
+        v2_structure = v2_snapshot.get("structure", {})
+        
+        # Compare classes
+        v1_classes = set(v1_structure.get("classes", []))
+        v2_classes = set(v2_structure.get("classes", []))
+        
+        classes_added = list(v2_classes - v1_classes)
+        classes_removed = list(v1_classes - v2_classes)
+        
+        # Compare properties
+        v1_properties = set(v1_structure.get("properties", []))
+        v2_properties = set(v2_structure.get("properties", []))
+        
+        properties_added = list(v2_properties - v1_properties)
+        properties_removed = list(v1_properties - v2_properties)
+        
+        # Compare individuals
+        v1_individuals = set(v1_structure.get("individuals", []))
+        v2_individuals = set(v2_structure.get("individuals", []))
+        
+        individuals_added = list(v2_individuals - v1_individuals)
+        individuals_removed = list(v1_individuals - v2_individuals)
+        
+        # Compare axioms/rules
+        v1_axioms = set(v1_structure.get("axioms", []))
+        v2_axioms = set(v2_structure.get("axioms", []))
+        
+        axioms_added = list(v2_axioms - v1_axioms)
+        axioms_removed = list(v1_axioms - v2_axioms)
+        
+        return {
+            "classes_added": classes_added,
+            "classes_removed": classes_removed,
+            "properties_added": properties_added,
+            "properties_removed": properties_removed,
+            "individuals_added": individuals_added,
+            "individuals_removed": individuals_removed,
+            "axioms_added": axioms_added,
+            "axioms_removed": axioms_removed,
+            "summary": {
+                "classes_added": len(classes_added),
+                "classes_removed": len(classes_removed),
+                "properties_added": len(properties_added),
+                "properties_removed": len(properties_removed),
+                "individuals_added": len(individuals_added),
+                "individuals_removed": len(individuals_removed),
+                "axioms_added": len(axioms_added),
+                "axioms_removed": len(axioms_removed)
+            }
+        }

semantica/change_management/ontology_version_manager.py

@@ -42,7 +42,7 @@ from typing import Any, Dict, List, Optional
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
-from .namespace_manager import NamespaceManager
+from ..ontology.namespace_manager import NamespaceManager
 
 
 @dataclass
@@ -198,14 +198,14 @@ class VersionManager:
 
     def compare_versions(self, version1: str, version2: str) -> Dict[str, Any]:
         """
-        Compare two ontology versions.
+        Compare two ontology versions with detailed structural analysis.
 
         Args:
             version1: First version
             version2: Second version
 
         Returns:
-            Comparison results
+            Detailed comparison results including structural differences
         """
         if version1 not in self.versions:
             raise ValidationError(f"Version not found: {version1}")
@@ -215,19 +215,85 @@ class VersionManager:
         v1 = self.versions[version1]
         v2 = self.versions[version2]
 
-        # Basic comparison
-        changes = []
+        # Basic metadata comparison
+        metadata_changes = {}
         if v1.ontology_iri != v2.ontology_iri:
-            changes.append("Ontology IRI changed")
+            metadata_changes["ontology_iri"] = {"from": v1.ontology_iri, "to": v2.ontology_iri}
         if v1.version_info != v2.version_info:
-            changes.append("Version info changed")
+            metadata_changes["version_info"] = {"from": v1.version_info, "to": v2.version_info}
+
+        # Structural comparison (if ontology data is available in metadata)
+        structural_diff = self._compare_ontology_structures(v1, v2)
 
         return {
             "version1": version1,
             "version2": version2,
-            "changes": changes,
-            "v1_iri": v1.ontology_iri,
-            "v2_iri": v2.ontology_iri,
+            "metadata_changes": metadata_changes,
+            **structural_diff
+        }
+
+    def _compare_ontology_structures(self, v1: OntologyVersion, v2: OntologyVersion) -> Dict[str, Any]:
+        """
+        Compare structural elements between two ontology versions.
+        
+        Args:
+            v1: First ontology version
+            v2: Second ontology version
+            
+        Returns:
+            Dictionary with structural differences
+        """
+        # Extract structural information from metadata if available
+        v1_structure = v1.metadata.get("structure", {})
+        v2_structure = v2.metadata.get("structure", {})
+        
+        # Compare classes
+        v1_classes = set(v1_structure.get("classes", []))
+        v2_classes = set(v2_structure.get("classes", []))
+        
+        classes_added = list(v2_classes - v1_classes)
+        classes_removed = list(v1_classes - v2_classes)
+        
+        # Compare properties
+        v1_properties = set(v1_structure.get("properties", []))
+        v2_properties = set(v2_structure.get("properties", []))
+        
+        properties_added = list(v2_properties - v1_properties)
+        properties_removed = list(v1_properties - v2_properties)
+        
+        # Compare individuals
+        v1_individuals = set(v1_structure.get("individuals", []))
+        v2_individuals = set(v2_structure.get("individuals", []))
+        
+        individuals_added = list(v2_individuals - v1_individuals)
+        individuals_removed = list(v1_individuals - v2_individuals)
+        
+        # Compare axioms/rules if available
+        v1_axioms = set(v1_structure.get("axioms", []))
+        v2_axioms = set(v2_structure.get("axioms", []))
+        
+        axioms_added = list(v2_axioms - v1_axioms)
+        axioms_removed = list(v1_axioms - v2_axioms)
+        
+        return {
+            "classes_added": classes_added,
+            "classes_removed": classes_removed,
+            "properties_added": properties_added,
+            "properties_removed": properties_removed,
+            "individuals_added": individuals_added,
+            "individuals_removed": individuals_removed,
+            "axioms_added": axioms_added,
+            "axioms_removed": axioms_removed,
+            "summary": {
+                "classes_added": len(classes_added),
+                "classes_removed": len(classes_removed),
+                "properties_added": len(properties_added),
+                "properties_removed": len(properties_removed),
+                "individuals_added": len(individuals_added),
+                "individuals_removed": len(individuals_removed),
+                "axioms_added": len(axioms_added),
+                "axioms_removed": len(axioms_removed)
+            }
         }
 
     def get_version(self, version: str) -> Optional[OntologyVersion]:

semantica/change_management/version_storage.py

@@ -0,0 +1,391 @@
+"""
+Version Storage Module
+
+This module provides abstract storage interfaces and concrete implementations
+for persistent version management in Semantica.
+
+Key Features:
+    - Abstract VersionStorage interface
+    - In-memory storage implementation
+    - SQLite-based persistent storage implementation
+    - Checksum computation and validation
+    - Thread-safe operations
+
+Main Classes:
+    - VersionStorage: Abstract base class for storage backends
+    - InMemoryVersionStorage: Dictionary-based in-memory storage
+    - SQLiteVersionStorage: SQLite-based persistent storage
+
+Example Usage:
+    >>> from semantica.common.version_storage import SQLiteVersionStorage
+    >>> storage = SQLiteVersionStorage("versions.db")
+    >>> storage.save(snapshot)
+    >>> versions = storage.list_all()
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import hashlib
+import json
+import sqlite3
+import threading
+from abc import ABC, abstractmethod
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+
+
+class VersionStorage(ABC):
+    """
+    Abstract base class for version storage backends.
+    
+    This interface defines the contract that all storage implementations
+    must follow for version management operations.
+    """
+    
+    @abstractmethod
+    def save(self, snapshot: Dict[str, Any]) -> None:
+        """
+        Save a version snapshot.
+        
+        Args:
+            snapshot: Version snapshot dictionary with metadata
+            
+        Raises:
+            ValidationError: If snapshot data is invalid
+            ProcessingError: If save operation fails
+        """
+        pass
+    
+    @abstractmethod
+    def get(self, label: str) -> Optional[Dict[str, Any]]:
+        """
+        Retrieve a version snapshot by label.
+        
+        Args:
+            label: Version label to retrieve
+            
+        Returns:
+            Snapshot dictionary or None if not found
+        """
+        pass
+    
+    @abstractmethod
+    def list_all(self) -> List[Dict[str, Any]]:
+        """
+        List all version snapshots.
+        
+        Returns:
+            List of snapshot metadata dictionaries
+        """
+        pass
+    
+    @abstractmethod
+    def exists(self, label: str) -> bool:
+        """
+        Check if a version exists.
+        
+        Args:
+            label: Version label to check
+            
+        Returns:
+            True if version exists, False otherwise
+        """
+        pass
+    
+    @abstractmethod
+    def delete(self, label: str) -> bool:
+        """
+        Delete a version snapshot.
+        
+        Args:
+            label: Version label to delete
+            
+        Returns:
+            True if deleted, False if not found
+        """
+        pass
+
+
+class InMemoryVersionStorage(VersionStorage):
+    """
+    In-memory version storage implementation.
+    
+    This implementation stores all version data in memory using a dictionary.
+    Data is lost when the process ends.
+    """
+    
+    def __init__(self):
+        """Initialize in-memory storage."""
+        self._storage: Dict[str, Dict[str, Any]] = {}
+        self._lock = threading.RLock()
+        self.logger = get_logger("in_memory_storage")
+    
+    def save(self, snapshot: Dict[str, Any]) -> None:
+        """Save snapshot to memory."""
+        label = snapshot.get("label")
+        if not label:
+            raise ValidationError("Snapshot must have a 'label' field")
+        
+        with self._lock:
+            if label in self._storage:
+                raise ValidationError(f"Version '{label}' already exists")
+            
+            # Deep copy to prevent external modifications
+            self._storage[label] = json.loads(json.dumps(snapshot))
+            self.logger.debug(f"Saved version '{label}' to memory")
+    
+    def get(self, label: str) -> Optional[Dict[str, Any]]:
+        """Retrieve snapshot from memory."""
+        with self._lock:
+            snapshot = self._storage.get(label)
+            if snapshot:
+                # Return deep copy to prevent external modifications
+                return json.loads(json.dumps(snapshot))
+            return None
+    
+    def list_all(self) -> List[Dict[str, Any]]:
+        """List all snapshots in memory."""
+        with self._lock:
+            # Return metadata only (without full graph data)
+            metadata_list = []
+            for label, snapshot in self._storage.items():
+                metadata = {
+                    "label": snapshot.get("label"),
+                    "timestamp": snapshot.get("timestamp"),
+                    "author": snapshot.get("author"),
+                    "description": snapshot.get("description"),
+                    "checksum": snapshot.get("checksum"),
+                    "entity_count": len(snapshot.get("entities", [])),
+                    "relationship_count": len(snapshot.get("relationships", []))
+                }
+                metadata_list.append(metadata)
+            return metadata_list
+    
+    def exists(self, label: str) -> bool:
+        """Check if version exists in memory."""
+        with self._lock:
+            return label in self._storage
+    
+    def delete(self, label: str) -> bool:
+        """Delete version from memory."""
+        with self._lock:
+            if label in self._storage:
+                del self._storage[label]
+                self.logger.debug(f"Deleted version '{label}' from memory")
+                return True
+            return False
+
+
+class SQLiteVersionStorage(VersionStorage):
+    """
+    SQLite-based persistent version storage implementation.
+    
+    This implementation stores version data in a SQLite database file,
+    providing persistence across process restarts.
+    """
+    
+    def __init__(self, storage_path: str):
+        """
+        Initialize SQLite storage.
+        
+        Args:
+            storage_path: Path to SQLite database file
+        """
+        self.storage_path = Path(storage_path)
+        self._lock = threading.RLock()
+        self.logger = get_logger("sqlite_storage")
+        
+        # Create directory if it doesn't exist
+        self.storage_path.parent.mkdir(parents=True, exist_ok=True)
+        
+        # Initialize database
+        self._init_database()
+    
+    def _init_database(self) -> None:
+        """Initialize SQLite database schema."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("""
+                    CREATE TABLE IF NOT EXISTS versions (
+                        label TEXT PRIMARY KEY,
+                        timestamp TEXT NOT NULL,
+                        author TEXT NOT NULL,
+                        description TEXT NOT NULL,
+                        checksum TEXT NOT NULL,
+                        snapshot_data TEXT NOT NULL,
+                        created_at TEXT NOT NULL
+                    )
+                """)
+                conn.commit()
+                self.logger.debug(f"Initialized SQLite database at {self.storage_path}")
+            finally:
+                conn.close()
+    
+    def save(self, snapshot: Dict[str, Any]) -> None:
+        """Save snapshot to SQLite database."""
+        label = snapshot.get("label")
+        if not label:
+            raise ValidationError("Snapshot must have a 'label' field")
+        
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                
+                # Check if version already exists
+                cursor.execute("SELECT label FROM versions WHERE label = ?", (label,))
+                if cursor.fetchone():
+                    raise ValidationError(f"Version '{label}' already exists")
+                
+                # Insert new version
+                cursor.execute("""
+                    INSERT INTO versions 
+                    (label, timestamp, author, description, checksum, snapshot_data, created_at)
+                    VALUES (?, ?, ?, ?, ?, ?, ?)
+                """, (
+                    label,
+                    snapshot.get("timestamp", ""),
+                    snapshot.get("author", ""),
+                    snapshot.get("description", ""),
+                    snapshot.get("checksum", ""),
+                    json.dumps(snapshot),
+                    datetime.now().isoformat()
+                ))
+                
+                conn.commit()
+                self.logger.debug(f"Saved version '{label}' to SQLite database")
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to save version to database: {e}")
+            finally:
+                conn.close()
+    
+    def get(self, label: str) -> Optional[Dict[str, Any]]:
+        """Retrieve snapshot from SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("""
+                    SELECT snapshot_data FROM versions WHERE label = ?
+                """, (label,))
+                
+                row = cursor.fetchone()
+                if not row:
+                    return None
+                
+                return json.loads(row[0])
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to retrieve version from database: {e}")
+            finally:
+                conn.close()
+    
+    def list_all(self) -> List[Dict[str, Any]]:
+        """List all snapshots in SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("""
+                    SELECT snapshot_data FROM versions ORDER BY timestamp DESC
+                """)
+                
+                metadata_list = []
+                for row in cursor.fetchall():
+                    snapshot = json.loads(row[0])
+                    
+                    metadata = {
+                        "label": snapshot.get("label"),
+                        "timestamp": snapshot.get("timestamp"),
+                        "author": snapshot.get("author"),
+                        "description": snapshot.get("description"),
+                        "checksum": snapshot.get("checksum"),
+                        "entity_count": len(snapshot.get("entities", [])),
+                        "relationship_count": len(snapshot.get("relationships", []))
+                    }
+                    metadata_list.append(metadata)
+                
+                return metadata_list
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to list versions from database: {e}")
+            finally:
+                conn.close()
+    
+    def exists(self, label: str) -> bool:
+        """Check if version exists in SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("SELECT 1 FROM versions WHERE label = ?", (label,))
+                return cursor.fetchone() is not None
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to check version existence: {e}")
+            finally:
+                conn.close()
+    
+    def delete(self, label: str) -> bool:
+        """Delete version from SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("DELETE FROM versions WHERE label = ?", (label,))
+                deleted = cursor.rowcount > 0
+                conn.commit()
+                
+                if deleted:
+                    self.logger.debug(f"Deleted version '{label}' from SQLite database")
+                
+                return deleted
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to delete version from database: {e}")
+            finally:
+                conn.close()
+
+
+def compute_checksum(data: Dict[str, Any]) -> str:
+    """
+    Compute SHA-256 checksum for version data.
+    
+    Args:
+        data: Dictionary containing version data
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+    """
+    # Create a deterministic JSON representation
+    json_str = json.dumps(data, sort_keys=True, separators=(',', ':'))
+    return hashlib.sha256(json_str.encode('utf-8')).hexdigest()
+
+
+def verify_checksum(snapshot: Dict[str, Any]) -> bool:
+    """
+    Verify the integrity of a snapshot using its checksum.
+    
+    Args:
+        snapshot: Snapshot dictionary with checksum field
+        
+    Returns:
+        True if checksum is valid, False otherwise
+    """
+    stored_checksum = snapshot.get("checksum")
+    if not stored_checksum:
+        return False
+    
+    # Create copy without checksum for verification
+    data_copy = snapshot.copy()
+    data_copy.pop("checksum", None)
+    
+    computed_checksum = compute_checksum(data_copy)
+    return stored_checksum == computed_checksum

semantica/conflicts/conflicts_provenance.py

@@ -0,0 +1,64 @@
+"""
+Provenance-enabled wrapper for conflict detection with unified backend.
+
+This module provides unified provenance backend integration for SourceTracker
+while maintaining 100% backward compatibility.
+
+Usage:
+    from semantica.conflicts.conflicts_provenance import SourceTrackerWithUnifiedBackend
+    
+    tracker = SourceTrackerWithUnifiedBackend()
+    tracker.track_property_source(entity_id, property_name, value, source)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Dict, Any, List
+
+
+class SourceTrackerWithUnifiedBackend:
+    """SourceTracker using unified provenance backend."""
+    
+    def __init__(self, **config):
+        """Initialize with unified backend or fallback to legacy."""
+        from .source_tracker import SourceTracker
+        
+        try:
+            from semantica.provenance import ProvenanceManager
+            self._unified_manager = ProvenanceManager()
+            self._use_unified = True
+        except ImportError:
+            self._use_unified = False
+        
+        self._original_tracker = SourceTracker(**config)
+    
+    def track_property_source(self, entity_id: str, property_name: str, value: Any, source: Any, **metadata):
+        """Track property source with unified backend."""
+        if self._use_unified:
+            from semantica.provenance import SourceReference
+            
+            source_ref = SourceReference(
+                document=source.document if hasattr(source, 'document') else str(source),
+                page=getattr(source, 'page', None),
+                section=getattr(source, 'section', None),
+                confidence=getattr(source, 'confidence', 1.0)
+            )
+            
+            self._unified_manager.track_property_source(
+                entity_id=entity_id,
+                property_name=property_name,
+                value=value,
+                source=source_ref,
+                **metadata
+            )
+        else:
+            self._original_tracker.track_property_source(
+                entity_id, property_name, value, source, **metadata
+            )
+    
+    def __getattr__(self, name):
+        return getattr(self._original_tracker, name)
+
+
+__all__ = ['SourceTrackerWithUnifiedBackend']

semantica/context/context_provenance.py

@@ -0,0 +1,54 @@
+"""
+Provenance-enabled wrapper for context management.
+
+Usage:
+    from semantica.context.context_provenance import ContextManagerWithProvenance
+    
+    ctx = ContextManagerWithProvenance(provenance=True)
+    ctx.add_context("context data", source="doc1.pdf")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Any
+import uuid
+
+
+class ContextManagerWithProvenance:
+    """Context manager with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        """Initialize context manager with optional provenance."""
+        from .context_manager import ContextManager
+        
+        self.provenance = provenance
+        self._context_manager = ContextManager(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def add_context(self, context: Any, source: Optional[str] = None, **kwargs):
+        """Add context with provenance tracking."""
+        result = self._context_manager.add_context(context, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"context_{uuid.uuid4().hex[:8]}",
+                source=source or "context_manager",
+                entity_type="context",
+                metadata={"context_preview": str(context)[:100]}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._context_manager, name)
+
+
+__all__ = ['ContextManagerWithProvenance']

semantica/context/context_retriever.py

@@ -1411,7 +1411,7 @@ Instructions:
 Answer:"""
 
         try:
-            response = llm_provider.generate(prompt, temperature=0.3)
+            response = llm_provider.generate(prompt)
             return response
         except Exception as e:
             self.logger.warning(f"LLM generation failed: {e}")

semantica/deduplication/deduplication_provenance.py

@@ -0,0 +1,60 @@
+"""
+Provenance-enabled wrapper for deduplication.
+
+Tracks: duplicates found, merge operations, deduplication strategy
+
+Usage:
+    from semantica.deduplication.deduplication_provenance import DeduplicatorWithProvenance
+    
+    dedup = DeduplicatorWithProvenance(provenance=True)
+    unique_items = dedup.deduplicate(items)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import List, Any
+import uuid
+
+
+class DeduplicatorWithProvenance:
+    """Deduplicator with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .deduplicator import Deduplicator
+        
+        self.provenance = provenance
+        self._deduplicator = Deduplicator(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def deduplicate(self, items: List[Any], source: str = None, **kwargs):
+        """Deduplicate items with provenance tracking."""
+        unique_items = self._deduplicator.deduplicate(items, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            duplicates_found = len(items) - len(unique_items)
+            self._prov_manager.track_entity(
+                entity_id=f"dedup_{uuid.uuid4().hex[:8]}",
+                source=source or "deduplication",
+                entity_type="deduplication_operation",
+                metadata={
+                    "input_count": len(items),
+                    "output_count": len(unique_items),
+                    "duplicates_removed": duplicates_found
+                }
+            )
+        
+        return unique_items
+    
+    def __getattr__(self, name):
+        return getattr(self._deduplicator, name)
+
+
+__all__ = ['DeduplicatorWithProvenance']

semantica/embeddings/embeddings_provenance.py

@@ -0,0 +1,59 @@
+"""
+Provenance-enabled wrappers for embedding generation.
+
+Tracks: model, dimensions, input texts, embedding vectors
+
+Usage:
+    from semantica.embeddings.embeddings_provenance import EmbeddingGeneratorWithProvenance
+    
+    embedder = EmbeddingGeneratorWithProvenance(provenance=True)
+    embeddings = embedder.embed(["text1", "text2"])
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import List
+import uuid
+
+
+class EmbeddingGeneratorWithProvenance:
+    """Embedding generator with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .embedding_generator import EmbeddingGenerator
+        
+        self.provenance = provenance
+        self._generator = EmbeddingGenerator(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def embed(self, texts: List[str], source: str = None, **kwargs):
+        """Generate embeddings with provenance tracking."""
+        embeddings = self._generator.embed(texts, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"embed_{uuid.uuid4().hex[:8]}",
+                source=source or "embedding_generation",
+                entity_type="embeddings",
+                metadata={
+                    "model": getattr(self._generator, 'model', 'unknown'),
+                    "dimensions": len(embeddings[0]) if embeddings else 0,
+                    "count": len(embeddings)
+                }
+            )
+        
+        return embeddings
+    
+    def __getattr__(self, name):
+        return getattr(self._generator, name)
+
+
+__all__ = ['EmbeddingGeneratorWithProvenance']

semantica/export/export_provenance.py

@@ -0,0 +1,58 @@
+"""
+Provenance-enabled wrappers for export operations.
+
+Tracks: export format, destination, timestamp, data exported
+
+Usage:
+    from semantica.export.export_provenance import JSONExporterWithProvenance
+    
+    exporter = JSONExporterWithProvenance(provenance=True)
+    exporter.export(data, "output.json")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class ExporterWithProvenance:
+    """Base exporter with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .exporter import Exporter
+        
+        self.provenance = provenance
+        self._exporter = Exporter(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def export(self, data: Any, destination: str, **kwargs):
+        """Export data with provenance tracking."""
+        result = self._exporter.export(data, destination, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"export_{uuid.uuid4().hex[:8]}",
+                source="export_operation",
+                entity_type="export",
+                metadata={
+                    "destination": destination,
+                    "format": kwargs.get('format', 'unknown')
+                }
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._exporter, name)
+
+
+__all__ = ['ExporterWithProvenance']

semantica/graph_store/graph_store_provenance.py

@@ -0,0 +1,56 @@
+"""
+Provenance-enabled wrapper for graph storage.
+
+Tracks: nodes added, edges created
+
+Usage:
+    from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+    
+    store = GraphStoreWithProvenance(provenance=True)
+    store.add_node(node, source="doc1.pdf")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class GraphStoreWithProvenance:
+    """Graph store with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .graph_store import GraphStore
+        
+        self.provenance = provenance
+        self._store = GraphStore(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def add_node(self, node: Any, source: str = None, **kwargs):
+        """Add node with provenance tracking."""
+        result = self._store.add_node(node, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            node_id = getattr(node, 'id', f"node_{uuid.uuid4().hex[:8]}")
+            self._prov_manager.track_entity(
+                entity_id=node_id,
+                source=source or "graph_store",
+                entity_type="graph_node",
+                metadata={"properties": getattr(node, 'properties', {})}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._store, name)
+
+
+__all__ = ['GraphStoreWithProvenance']

semantica/ingest/__init__.py

@@ -86,6 +86,7 @@ Main Classes:
     - RepoIngestor: Git repository processing
     - EmailIngestor: Email protocol handling
     - DBIngestor: Database export handling
+    - OntologyIngestor: Ontology file processing
     - MethodRegistry: Registry for custom ingestion methods
     - IngestConfig: Configuration manager for ingest module
 
@@ -98,6 +99,7 @@ Convenience Functions:
     - ingest_repository: Repository ingestion wrapper
     - ingest_email: Email ingestion wrapper
     - ingest_database: Database ingestion wrapper
+    - ingest_ontology: Ontology ingestion wrapper
 
 
 Example Usage:
@@ -134,6 +136,7 @@ from .methods import (
     ingest_feed,
     ingest_file,
     ingest_mcp,
+    ingest_ontology,
     ingest_repository,
     ingest_stream,
     ingest_web,
@@ -166,6 +169,8 @@ from .web_ingestor import (
     WebIngestor,
 )
 
+from .ontology_ingestor import OntologyData, OntologyIngestor
+
 __all__ = [
     # File ingestion
     "FileIngestor",
@@ -216,6 +221,9 @@ __all__ = [
     "MCPClient",
     "MCPResource",
     "MCPTool",
+    # Ontology ingestion
+    "OntologyIngestor",
+    "OntologyData",
     # Registry and Methods
     "MethodRegistry",
     "method_registry",
@@ -227,6 +235,7 @@ __all__ = [
     "ingest_repository",
     "ingest_email",
     "ingest_database",
+    "ingest_ontology",
     "ingest_mcp",
     "get_ingest_method",
     "list_available_methods",

semantica/ingest/feed_ingestor.py

@@ -359,9 +359,14 @@ class FeedParser:
 
             return parser.parse(date_string)
         except (ImportError, OSError):
+            # If dateutil isn't available, fall through to raising ValueError
             pass
+        except Exception as e:
+            # If dateutil fails to parse, raise ValueError to signal invalid input
+            raise ValueError(f"Invalid date format: {date_string}") from e
 
-        return None
+        # No known formats matched and dateutil is unavailable; raise ValueError
+        raise ValueError(f"Invalid date format: {date_string}")
 
     def validate_feed(self, feed_data: FeedData) -> bool:
         """

semantica/ingest/ingest_provenance.py

@@ -0,0 +1,67 @@
+"""
+Provenance-enabled wrappers for document ingestion.
+
+Tracks: file paths, pages, metadata, ingestion timestamps
+
+Usage:
+    from semantica.ingest.ingest_provenance import PDFIngestorWithProvenance
+    
+    ingestor = PDFIngestorWithProvenance(provenance=True)
+    docs = ingestor.ingest("document.pdf")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, List
+import uuid
+
+
+class IngestProvenanceMixin:
+    """Mixin for ingest provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **kwargs):
+        self.provenance = provenance
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+
+
+class PDFIngestorWithProvenance(IngestProvenanceMixin):
+    """PDF ingestor with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .pdf_ingestor import PDFIngestor
+        
+        IngestProvenanceMixin.__init__(self, provenance=provenance)
+        self._ingestor = PDFIngestor(**config)
+    
+    def ingest(self, file_path: str, **kwargs):
+        """Ingest PDF with provenance tracking."""
+        docs = self._ingestor.ingest(file_path, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            for doc in docs:
+                doc_id = getattr(doc, 'id', f"doc_{uuid.uuid4().hex[:8]}")
+                self._prov_manager.track_entity(
+                    entity_id=doc_id,
+                    source=file_path,
+                    entity_type="document",
+                    metadata={
+                        "file_type": "pdf",
+                        "pages": getattr(doc, 'page_count', None)
+                    }
+                )
+        
+        return docs
+    
+    def __getattr__(self, name):
+        return getattr(self._ingestor, name)
+
+
+__all__ = ['PDFIngestorWithProvenance', 'IngestProvenanceMixin']

semantica/ingest/methods.py

@@ -150,6 +150,7 @@ from .email_ingestor import EmailData, EmailIngestor
 from .feed_ingestor import FeedData, FeedIngestor
 from .file_ingestor import FileIngestor, FileObject
 from .mcp_ingestor import MCPData, MCPIngestor
+from .ontology_ingestor import OntologyData, OntologyIngestor
 from .registry import method_registry
 from .repo_ingestor import RepoIngestor
 from .stream_ingestor import StreamIngestor, StreamProcessor
@@ -537,6 +538,66 @@ def ingest_email(
         raise
 
 
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], method: str = "file", **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source (convenience function).
+
+    This is a user-friendly wrapper that ingests ontologies using the specified method.
+
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+            - "file": Single file ingestion
+            - "directory": Directory ingestion with recursive scanning
+        **kwargs: Additional options passed to OntologyIngestor
+
+    Returns:
+        OntologyData, List[OntologyData] with ingestion results
+
+    Examples:
+        >>> from semantica.ingest.methods import ingest_ontology
+        >>> ontology = ingest_ontology("ontology.ttl")
+        >>> ontologies = ingest_ontology("./ontologies", method="directory")
+    """
+    # Check for custom method in registry
+    custom_method = method_registry.get("ontology", method)
+    if custom_method and custom_method != ingest_ontology:
+        try:
+            return custom_method(source, **kwargs)
+        except Exception as e:
+            logger.warning(
+                f"Custom method {method} failed: {e}, falling back to default"
+            )
+
+    try:
+        # Get config
+        config = ingest_config.get_method_config("ontology")
+        config.update(kwargs)
+
+        ingestor = OntologyIngestor(**config)
+
+        source_path = str(source) if isinstance(source, (str, Path)) else None
+        
+        if method == "file" and source_path:
+            if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+            return ingestor.ingest_ontology(source_path, **kwargs)
+        elif method == "directory" and source_path:
+             recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
+             return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
+        else:
+             # Default: try as file
+             if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+             return ingestor.ingest_ontology(str(source), **kwargs)
+
+    except Exception as e:
+        logger.error(f"Failed to ingest ontology: {e}")
+        raise
+
+
 def ingest_database(
     source: Union[str, Dict[str, Any]], method: Optional[str] = None, **kwargs
 ) -> Union[TableData, List[TableData], Dict[str, Any]]:
@@ -769,6 +830,7 @@ def ingest(
             - "repo": Repository ingestion
             - "email": Email ingestion
             - "db": Database ingestion
+            - "ontology": Ontology ingestion
         method: Optional specific ingestion method
         **kwargs: Additional options passed to ingestor
 
@@ -802,6 +864,8 @@ def ingest(
                 ("git@", "https://github.com", "https://gitlab.com")
             ):
                 source_type = "repo"
+            elif source_str.endswith((".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")):
+                source_type = "ontology"
             else:
                 source_type = "file"
         else:
@@ -830,6 +894,8 @@ def ingest(
             raise ProcessingError("Email ingestion requires configuration dictionary")
     elif source_type == "db":
         return {"data": ingest_database(sources, method=method, **kwargs)}
+    elif source_type == "ontology":
+        return {"ontology": ingest_ontology(sources, method=method or "file", **kwargs)}
     elif source_type == "mcp":
         return {"data": ingest_mcp(sources, method=method or "resources", **kwargs)}
     else:
@@ -909,5 +975,8 @@ method_registry.register("mcp", "default", ingest_mcp)
 method_registry.register("mcp", "resources", ingest_mcp)
 method_registry.register("mcp", "tools", ingest_mcp)
 method_registry.register("mcp", "all", ingest_mcp)
+method_registry.register("ontology", "default", ingest_ontology)
+method_registry.register("ontology", "file", ingest_ontology)
+method_registry.register("ontology", "directory", ingest_ontology)
 method_registry.register("ingest", "default", ingest)
 method_registry.register("ingest", "unified", ingest)

semantica/ingest/ontology_ingestor.py

@@ -0,0 +1,392 @@
+"""
+Ontology Ingestion Module
+
+This module provides capabilities to ingest external ontologies from files (OWL, RDF, TTL, etc.)
+and convert them into Semantica's internal ontology dictionary format.
+
+Supported Formats:
+    - Turtle (.ttl): Terse RDF Triple Language. A concise, human-readable
+      format for representing RDF graphs. Commonly used for writing
+      ontologies by hand.
+    - RDF/XML (.rdf, .owl): The XML serialization of RDF. The standard
+      format for OWL (Web Ontology Language) ontologies and often used
+      for data interchange.
+    - JSON-LD (.jsonld): JSON for Linked Data. A lightweight Linked Data
+      format that is easy for humans to read and for machines to parse
+      and generate. Ideal for web-based applications.
+    - N-Triples (.nt): A line-based, plain text format for encoding an
+      RDF graph. Each line represents a single triple. Very simple to
+      parse but verbose.
+    - Notation3 (.n3): A superset of Turtle that adds features like logic
+      and rules.
+
+Key Features:
+    - Support for multiple RDF formats (Turtle, RDF/XML, JSON-LD, N3, NT)
+    - Automatic parsing using rdflib
+    - Conversion to Semantica ontology structure
+    - Batch processing of ontology files
+    - Extraction of classes, properties, and metadata
+
+Example Usage:
+    >>> from semantica.ingest import OntologyIngestor
+    >>> ingestor = OntologyIngestor()
+    >>> ontology = ingestor.ingest_ontology("my_ontology.ttl")
+"""
+
+import os
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Union
+
+import rdflib
+from rdflib import RDF, RDFS, OWL, Graph
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from ..utils.progress_tracker import get_progress_tracker
+
+
+@dataclass
+class OntologyData:
+    """Ontology data representation."""
+
+    data: Dict[str, Any]
+    source_path: str
+    format: str
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    ingested_at: datetime = field(default_factory=datetime.now)
+
+
+class OntologyIngestor:
+    """
+    Ontology ingestion handler.
+
+    This class parses OWL/RDF files and converts them to Semantica's ontology dictionary format.
+    """
+
+    def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
+        """
+        Initialize ontology ingestor.
+
+        Args:
+            config: Optional configuration dictionary
+            **kwargs: Additional configuration parameters
+        """
+        self.logger = get_logger("ontology_ingestor")
+        self.progress = get_progress_tracker()
+        self.config = config or {}
+        self.config.update(kwargs)
+
+    def ingest_ontology(self, file_path: Union[str, Path], format: Optional[str] = None, **kwargs) -> OntologyData:
+        """
+        Ingest an ontology file.
+
+        Args:
+            file_path: Path to the ontology file (string or Path object)
+            format: Optional format hint (e.g., 'turtle', 'xml'). If None, rdflib guesses.
+            **kwargs: Additional arguments for rdflib parsing
+
+        Returns:
+            OntologyData object containing the parsed ontology and metadata
+        """
+        file_path = Path(file_path)
+
+        # Track file ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(file_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Ontology: {file_path.name}",
+        )
+
+        try:
+            # Validate file exists
+            if not file_path.exists():
+                raise ValidationError(f"File not found: {file_path}")
+
+            self.progress.update_tracking(tracking_id, message="Parsing RDF graph...")
+            g = Graph()
+            
+            # Use provided format or let rdflib guess based on extension
+            parse_kwargs = kwargs.copy()
+            if format:
+                parse_kwargs['format'] = format
+            
+            try:
+                g.parse(file_path, **parse_kwargs)
+            except Exception as e:
+                # Fallback: try to guess format from extension if not provided and initial parse failed
+                if not format:
+                    ext = os.path.splitext(file_path)[1].lower()
+                    fmt_map = {
+                        '.ttl': 'turtle',
+                        '.owl': 'xml', # OWL is often XML
+                        '.rdf': 'xml',
+                        '.jsonld': 'json-ld',
+                        '.n3': 'n3',
+                        '.nt': 'nt'
+                    }
+                    guessed_fmt = fmt_map.get(ext)
+                    if guessed_fmt:
+                        self.logger.info(f"Retrying with guessed format: {guessed_fmt}")
+                        g.parse(file_path, format=guessed_fmt, **kwargs)
+                    else:
+                        raise e
+                else:
+                    raise e
+
+            self.progress.update_tracking(tracking_id, message="Converting to internal format...")
+            
+            # Determine format for metadata
+            used_format = format
+            if not used_format:
+                ext = os.path.splitext(file_path)[1].lower()
+                fmt_map = {
+                    '.ttl': 'turtle',
+                    '.owl': 'xml',
+                    '.rdf': 'xml',
+                    '.jsonld': 'json-ld',
+                    '.n3': 'n3',
+                    '.nt': 'nt'
+                }
+                used_format = fmt_map.get(ext, 'unknown')
+
+            ontology_dict = self._convert_to_dict(g, source_path=str(file_path), format=used_format)
+
+            ontology_data = OntologyData(
+                data=ontology_dict,
+                source_path=str(file_path),
+                format=used_format,
+                metadata=ontology_dict.get("metadata", {}).copy()
+            )
+
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Successfully ingested ontology from {file_path}",
+            )
+            
+            return ontology_data
+
+        except Exception as e:
+            self.logger.error(f"Failed to ingest ontology: {str(e)}")
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise ProcessingError(f"Failed to ingest ontology: {str(e)}") from e
+
+    def ingest_directory(self, directory_path: Union[str, Path], recursive: bool = True, **kwargs) -> List[OntologyData]:
+        """
+        Ingest all ontology files in a directory.
+
+        Args:
+            directory_path: Path to the directory (string or Path object)
+            recursive: Whether to search recursively
+            **kwargs: Additional arguments
+
+        Returns:
+            List of OntologyData objects
+        """
+        directory_path = Path(directory_path)
+        ontologies = []
+        extensions = {'.ttl', '.owl', '.rdf', '.jsonld', '.n3', '.nt'}
+        
+        # Track directory ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(directory_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Directory: {directory_path.name}",
+        )
+
+        try:
+            if not directory_path.exists():
+                raise ValidationError(f"Directory not found: {directory_path}")
+                
+            if not directory_path.is_dir():
+                raise ValidationError(f"Path is not a directory: {directory_path}")
+
+            files_to_process = []
+            for root, _, files in os.walk(directory_path):
+                for file in files:
+                    ext = os.path.splitext(file)[1].lower()
+                    if ext in extensions:
+                        files_to_process.append(os.path.join(root, file))
+                
+                if not recursive:
+                    break
+            
+            total_files = len(files_to_process)
+            self.progress.update_tracking(
+                tracking_id, message=f"Processing {total_files} ontology files"
+            )
+
+            for idx, file_path in enumerate(files_to_process, 1):
+                try:
+                    ont_data = self.ingest_ontology(file_path, **kwargs)
+                    ontologies.append(ont_data)
+                    
+                    self.progress.update_progress(
+                        tracking_id,
+                        processed=idx,
+                        total=total_files,
+                        message=f"Processing {idx}/{total_files}: {Path(file_path).name}"
+                    )
+                except Exception as e:
+                    self.logger.warning(f"Skipping {file_path}: {e}")
+            
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Ingested {len(ontologies)} ontologies",
+            )
+            return ontologies
+
+        except Exception as e:
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
+    def _convert_to_dict(self, graph: Graph, source_path: str, format: str = "unknown") -> Dict[str, Any]:
+        """
+        Convert rdflib Graph to Semantica ontology dictionary.
+
+        Args:
+            graph: Parsed rdflib Graph
+            source_path: Source file path
+            format: Format of the ontology file
+
+        Returns:
+            Ontology dictionary
+        """
+        ontology = {
+            "uri": "",
+            "name": os.path.basename(source_path),
+            "version": "1.0",
+            "classes": [],
+            "properties": [],
+            "metadata": {
+                "source_path": source_path,
+                "ingested_at": datetime.now().isoformat(),
+                "format": format
+            }
+        }
+
+        # 1. Extract Ontology Metadata
+        for s, p, o in graph.triples((None, RDF.type, OWL.Ontology)):
+            ontology["uri"] = str(s)
+            
+            # Try to find label/comment/versionInfo
+            for _, _, label in graph.triples((s, RDFS.label, None)):
+                ontology["name"] = str(label)
+                
+            for _, _, comment in graph.triples((s, RDFS.comment, None)):
+                ontology["description"] = str(comment)
+                
+            for _, _, version in graph.triples((s, OWL.versionInfo, None)):
+                ontology["version"] = str(version)
+            
+            # Break after first ontology definition found (usually only one per file)
+            break
+            
+        # 2. Extract Classes
+        classes = {}
+        # Union of owl:Class and rdfs:Class
+        class_types = [OWL.Class, RDFS.Class]
+        for c_type in class_types:
+            for s, p, o in graph.triples((None, RDF.type, c_type)):
+                if isinstance(s, rdflib.BNode):
+                    continue # Skip blank nodes for now
+                    
+                uri = str(s)
+                if uri not in classes:
+                    cls_def = {
+                        "uri": uri,
+                        "name": self._get_local_name(uri),
+                        "type": "class"
+                    }
+                    
+                    # Add label/comment
+                    label = graph.value(s, RDFS.label)
+                    if label:
+                        cls_def["label"] = str(label)
+                        cls_def["name"] = str(label) # Prefer label as name if available? Or keep URI fragment? 
+                        # Keeping local name from URI is safer for internal IDs, label for display. 
+                        # But Semantica seems to use "name" for the identifier in some examples.
+                        # Let's keep name as local name or label if simple.
+                    
+                    comment = graph.value(s, RDFS.comment)
+                    if comment:
+                        cls_def["description"] = str(comment)
+                        
+                    # Superclasses
+                    parents = []
+                    for _, _, parent in graph.triples((s, RDFS.subClassOf, None)):
+                        if isinstance(parent, rdflib.URIRef):
+                            parents.append(str(parent))
+                    if parents:
+                        cls_def["parents"] = parents
+                        
+                    classes[uri] = cls_def
+        
+        ontology["classes"] = list(classes.values())
+
+        # 3. Extract Properties
+        properties = {}
+        # Object Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.ObjectProperty)):
+            self._add_property(graph, s, "object", properties)
+            
+        # Datatype Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.DatatypeProperty)):
+            self._add_property(graph, s, "data", properties)
+            
+        # RDF Properties (generic)
+        for s, p, o in graph.triples((None, RDF.type, RDF.Property)):
+            if str(s) not in properties: # Don't overwrite if already found as specific type
+                self._add_property(graph, s, "annotation", properties) # Default to annotation or generic
+        
+        ontology["properties"] = list(properties.values())
+        
+        return ontology
+
+    def _add_property(self, graph: Graph, subject: rdflib.term.Node, prop_type: str, properties_dict: Dict):
+        if isinstance(subject, rdflib.BNode):
+            return
+
+        uri = str(subject)
+        if uri in properties_dict:
+            return
+            
+        prop_def = {
+            "uri": uri,
+            "name": self._get_local_name(uri),
+            "type": prop_type
+        }
+        
+        label = graph.value(subject, RDFS.label)
+        if label:
+            prop_def["label"] = str(label)
+            
+        comment = graph.value(subject, RDFS.comment)
+        if comment:
+            prop_def["description"] = str(comment)
+            
+        # Domain and Range
+        domain = graph.value(subject, RDFS.domain)
+        if domain and isinstance(domain, rdflib.URIRef):
+            prop_def["domain"] = str(domain)
+            
+        range_val = graph.value(subject, RDFS.range)
+        if range_val and isinstance(range_val, rdflib.URIRef):
+            prop_def["range"] = str(range_val)
+            
+        properties_dict[uri] = prop_def
+
+    def _get_local_name(self, uri: str) -> str:
+        """Extract local name from URI."""
+        if '#' in uri:
+            return uri.split('#')[-1]
+        return uri.split('/')[-1]

semantica/ingest/pandas_ingestor.py

@@ -25,6 +25,8 @@ Example Usage:
 """
 
 import json
+import csv
+import chardet
 from dataclasses import dataclass, field
 from datetime import datetime
 from pathlib import Path
@@ -193,23 +195,9 @@ class PandasIngestor:
     def from_csv(
         self,
         file_path: Union[str, Path],
+        chunksize: Optional[int] = None,
         **pandas_options,
     ) -> PandasData:
-        """
-        Ingest data from CSV file.
-
-        This method reads a CSV file using pandas and ingests it as a DataFrame.
-
-        Args:
-            file_path: Path to CSV file
-            **pandas_options: Additional options passed to pd.read_csv()
-
-        Returns:
-            PandasData: Ingested data object
-
-        Raises:
-            ProcessingError: If CSV reading fails
-        """
         file_path = Path(file_path)
 
         if not file_path.exists():
@@ -223,15 +211,104 @@ class PandasIngestor:
         )
 
         try:
-            # Read CSV with pandas
-            dataframe = pd.read_csv(file_path, **pandas_options)
 
-            self.progress_tracker.update_tracking(
-                tracking_id, message="CSV read successfully, processing DataFrame..."
+            # ---------- Encoding Detection ----------
+            with open(file_path, "rb") as f:
+                raw = f.read(100_000)
+                encoding_info = chardet.detect(raw)
+            encoding = encoding_info.get("encoding") or "utf-8"
+
+        
+            # ---------- Delimiter & Header Detection ----------
+            with open(file_path, "r", encoding=encoding, errors="replace") as f:
+                sample = f.read(10000)
+                sniffer = csv.Sniffer()
+
+                try:
+                    dialect = sniffer.sniff(sample, delimiters=[",", ";", "\t", "|"])
+                    delimiter = dialect.delimiter
+                    quotechar = dialect.quotechar
+                except Exception:
+                    delimiter = ","
+                    quotechar = '"'
+
+                # Header handling: default to True (treat first row as header)
+                # unless user explicitly overrides via pandas_options['header'].
+                has_header = True
+                header_opt = pandas_options.get("header", None)
+                if header_opt is None:
+                    has_header = True
+                elif header_opt == 0 or header_opt == "infer":
+                    has_header = True
+                else:
+                    # Any explicit non-header setting (e.g., None or int>0) implies no header
+                    try:
+                        has_header = False if header_opt is None or int(header_opt) != 0 else True
+                    except Exception:
+                        has_header = False
+
+
+            skipped_rows = 0
+            dataframes = []
+
+            # ---------- CSV Reading (Chunked if needed) ----------
+            # Preserve explicit header setting (including None) if user provided it.
+            has_explicit_header = "header" in pandas_options
+            explicit_header = pandas_options.pop("header", None) if has_explicit_header else None
+            header_arg = explicit_header if has_explicit_header else (0 if has_header else None)
+
+            reader = pd.read_csv(
+                file_path,
+                sep=delimiter,
+                encoding=encoding,
+                encoding_errors="replace",
+                quoting=csv.QUOTE_MINIMAL,
+                header=header_arg,
+                quotechar=quotechar,
+                escapechar="\\",
+                engine="python",
+                on_bad_lines="warn",
+                chunksize=chunksize,
+                **pandas_options,
             )
 
-            # Ingest the DataFrame
-            return self.ingest_dataframe(dataframe, **pandas_options)
+            if chunksize:
+                for chunk in reader:
+                    dataframes.append(chunk)
+            else:
+                dataframes.append(reader)
+
+            dataframe = pd.concat(dataframes, ignore_index=True)
+
+            self.progress_tracker.update_tracking(
+                tracking_id,
+                message="CSV parsed successfully, ingesting DataFrame...",
+            )
+
+            # ---------- Ingest ----------
+            pandas_data = self.ingest_dataframe(dataframe)
+
+            # ---------- Metadata ----------
+            pandas_data.metadata.update(
+                {
+                    "source": "csv",
+                    "file": str(file_path),
+                    "detected_encoding": encoding,
+                    "encoding_confidence": encoding_info.get("confidence"),
+                    "detected_delimiter": delimiter,
+                    "header_detected": has_header,
+                    "chunksize": chunksize,
+                    "malformed_rows_skipped": skipped_rows,
+                }
+            )
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"CSV ingestion completed: {pandas_data.row_count} rows",
+            )
+
+            return pandas_data
 
         except Exception as e:
             self.progress_tracker.stop_tracking(

semantica/kg/graph_builder.py

@@ -161,7 +161,14 @@ class GraphBuilder:
             }
             all_relationships.append(rel_dict)
         elif isinstance(item, dict):
-            # Detect and normalize Entity objects inside dict
+            if "source_id" in item and "source" not in item:
+                item["source"] = item["source_id"]
+            if "target_id" in item and "target" not in item:
+                item["target"] = item["target_id"]
+            if "subject" in item and "source" not in item:
+                item["source"] = item["subject"]
+            if "object" in item and "target" not in item:
+                item["target"] = item["object"]
             if "source" in item and not isinstance(item["source"], str):
                 src = item["source"]
                 item["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -347,6 +354,21 @@ class GraphBuilder:
         elif not isinstance(sources, list):
             sources = [sources]
 
+        # Count input relationships for warning if all are dropped
+        input_relationships_count = 0
+        if isinstance(source_dict, dict):
+            rels = source_dict.get("relationships", [])
+            if isinstance(rels, list):
+                input_relationships_count += len(rels)
+            elif rels is not None:
+                input_relationships_count += 1
+        if explicit_relationships:
+            for rel_item in explicit_relationships:
+                if isinstance(rel_item, list):
+                    input_relationships_count += len(rel_item)
+                else:
+                    input_relationships_count += 1
+
         # Track graph building
         build_start_time = time.time()
         
@@ -468,11 +490,12 @@ class GraphBuilder:
                         pipeline_id=pipeline_id,
                     )
                     
-                    # Check if relationships are already in dictionary format
                     sample_rel = relationships_list[0] if relationships_list else None
                     is_dict_format = isinstance(sample_rel, dict) and (
-                        "source" in sample_rel and "target" in sample_rel
-                    ) and not hasattr(sample_rel, "__dict__") # Ensure it's not a class instance
+                        ("source" in sample_rel and "target" in sample_rel)
+                        or ("source_id" in sample_rel and "target_id" in sample_rel)
+                        or ("subject" in sample_rel and "object" in sample_rel)
+                    ) and not hasattr(sample_rel, "__dict__")
                     
                     if is_dict_format:
                         # Fast path: directly append dictionaries after normalizing source/target
@@ -481,8 +504,15 @@ class GraphBuilder:
                             batch = relationships_list[i:i + batch_size]
                             for item in batch:
                                 if isinstance(item, dict):
-                                    # Normalize source/target if they are objects
                                     rel_dict = item.copy()
+                                    if "source_id" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["source_id"]
+                                    if "target_id" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["target_id"]
+                                    if "subject" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["subject"]
+                                    if "object" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["object"]
                                     if "source" in rel_dict and not isinstance(rel_dict["source"], str):
                                         src = rel_dict["source"]
                                         rel_dict["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -571,6 +601,14 @@ class GraphBuilder:
                     f"Entity resolution complete: {len(all_entities)} -> {len(resolved_entities)} unique entities"
                 )
 
+            if input_relationships_count > 0 and len(all_relationships) == 0:
+                warning_msg = (
+                    f"All relationships were dropped during graph building: "
+                    f"{input_relationships_count} input relationships, 0 in final graph"
+                )
+                self.logger.warning(warning_msg)
+                print(f"Warning: {warning_msg}")
+
             # Build graph structure
             print("Building graph structure...")
             structure_start = time.time()
@@ -682,6 +720,14 @@ class GraphBuilder:
             )
             raise
 
+    def build_single_source(
+        self,
+        kg_data: Dict[str, Any],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Dict[str, Any]:
+        return self.build(kg_data, pipeline_id=pipeline_id, **options)
+
     def add_temporal_edge(
         self,
         graph,

semantica/kg/provenance_tracker.py

@@ -1,19 +1,29 @@
 """
-Provenance Tracking Module
+Provenance Tracking Module (Enhanced with Unified Backend)
 
 This module provides comprehensive source tracking and lineage capabilities
 for the Semantica framework, enabling tracking of data origins and evolution
 for knowledge graph entities and relationships.
 
+IMPORTANT: This module now uses the unified semantica.provenance.ProvenanceManager
+backend for enhanced W3C PROV-O compliance and audit-grade tracking. All existing
+APIs remain 100% backward compatible.
+
+For new code, consider using the unified API:
+    >>> from semantica.provenance import ProvenanceManager
+    >>> prov_mgr = ProvenanceManager()
+
 Key Features:
     - Entity provenance tracking (source, timestamp, metadata)
     - Relationship provenance tracking
     - Lineage retrieval (complete provenance history)
     - Source aggregation (multiple sources per entity)
     - Temporal tracking (first seen, last updated)
+    - W3C PROV-O compliance (when using unified backend)
+    - Audit-grade integrity verification
 
 Main Classes:
-    - ProvenanceTracker: Main provenance tracking engine
+    - ProvenanceTracker: Main provenance tracking engine (backward compatible wrapper)
 
 Example Usage:
     >>> from semantica.kg import ProvenanceTracker
@@ -32,6 +42,13 @@ from typing import Any, Dict, List, Optional
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
 
+# Import unified provenance manager
+try:
+    from ..provenance import ProvenanceManager as UnifiedProvenanceManager
+    UNIFIED_AVAILABLE = True
+except ImportError:
+    UNIFIED_AVAILABLE = False
+
 
 class ProvenanceTracker:
     """
@@ -75,6 +92,10 @@ class ProvenanceTracker:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        self._use_unified = UNIFIED_AVAILABLE
+        if self._use_unified:
+            self._unified_manager = UnifiedProvenanceManager()
+
         self.logger.debug("Provenance tracker initialized")
 
     def track_entity(
@@ -89,10 +110,33 @@ class ProvenanceTracker:
 
         Args:
             entity_id: Entity identifier
-            source: Source identifier (e.g., "file_1", "api_endpoint_2")
+            source: Source identifier (e.g., "file_1", "api_endpoint_2", DOI)
             metadata: Optional metadata dictionary (e.g., confidence scores,
                      extraction methods, etc.)
         """
+        if self._use_unified:
+            # Delegate to unified manager
+            try:
+                self._unified_manager.track_entity(
+                    entity_id=entity_id,
+                    source=source,
+                    metadata=metadata
+                )
+            except Exception as e:
+                self.logger.warning(f"Unified tracking failed, using fallback: {e}")
+                self._track_entity_legacy(entity_id, source, metadata)
+        else:
+            # Use legacy implementation
+            self._track_entity_legacy(entity_id, source, metadata)
+        
+        self.logger.debug(
+            f"Tracked provenance for entity {entity_id} from source {source}"
+        )
+    
+    def _track_entity_legacy(
+        self, entity_id: str, source: str, metadata: Optional[Dict[str, Any]] = None
+    ) -> None:
+        """Legacy entity tracking implementation."""
         if entity_id not in self.provenance_data:
             self.provenance_data[entity_id] = {
                 "sources": [],
@@ -115,10 +159,6 @@ class ProvenanceTracker:
         if metadata:
             self.provenance_data[entity_id]["metadata"].update(metadata)
 
-        self.logger.debug(
-            f"Tracked provenance for entity {entity_id} from source {source}"
-        )
-
     def track_relationship(
         self,
         relationship_id: str,
@@ -175,9 +215,20 @@ class ProvenanceTracker:
                 - timestamp: ISO format timestamp
                 - metadata: Source metadata dictionary
         """
+        if self._use_unified:
+            # Get from unified manager
+            try:
+                return self._unified_manager.get_all_sources(entity_id)
+            except Exception as e:
+                self.logger.warning(f"Unified retrieval failed, using fallback: {e}")
+                return self._get_all_sources_legacy(entity_id)
+        else:
+            return self._get_all_sources_legacy(entity_id)
+    
+    def _get_all_sources_legacy(self, entity_id: str) -> List[Dict[str, Any]]:
+        """Legacy get all sources implementation."""
         if entity_id not in self.provenance_data:
             return []
-
         return self.provenance_data[entity_id].get("sources", [])
 
     def get_lineage(self, entity_id: str) -> Dict[str, Any]:
@@ -192,14 +243,38 @@ class ProvenanceTracker:
 
         Returns:
             dict: Complete lineage information containing:
-                - sources: List of all source entries
+                - sources: List of all source entries (legacy format)
                 - first_seen: ISO timestamp of first source
                 - last_updated: ISO timestamp of most recent source
                 - metadata: Aggregated metadata dictionary
+                - lineage_chain: Complete lineage chain (when using unified backend)
         """
+        if self._use_unified:
+            # Get from unified manager
+            try:
+                lineage = self._unified_manager.get_lineage(entity_id)
+                if not lineage:
+                    return {}
+                
+                # Convert to legacy format for backward compatibility
+                legacy_format = {
+                    "sources": self._unified_manager.get_all_sources(entity_id),
+                    "first_seen": lineage.get("first_seen"),
+                    "last_updated": lineage.get("last_updated"),
+                    "metadata": lineage.get("metadata", {}),  # Include metadata from lineage
+                    "lineage_chain": lineage.get("lineage_chain", [])
+                }
+                return legacy_format
+            except Exception as e:
+                self.logger.warning(f"Unified retrieval failed, using fallback: {e}")
+                return self._get_lineage_legacy(entity_id)
+        else:
+            return self._get_lineage_legacy(entity_id)
+    
+    def _get_lineage_legacy(self, entity_id: str) -> Dict[str, Any]:
+        """Legacy get lineage implementation."""
         if entity_id not in self.provenance_data:
             return {}
-
         return self.provenance_data[entity_id].copy()
 
     def get_provenance(self, entity_id: str) -> Optional[Dict[str, Any]]:
@@ -216,7 +291,18 @@ class ProvenanceTracker:
             dict: Complete provenance information (same as get_lineage()),
                   or None if entity is not tracked
         """
-        return self.provenance_data.get(entity_id)
+        if self._use_unified:
+            try:
+                prov = self._unified_manager.get_provenance(entity_id)
+                if not prov:
+                    return None
+                # Return in legacy format
+                return self.get_lineage(entity_id)
+            except Exception as e:
+                self.logger.warning(f"Unified retrieval failed, using fallback: {e}")
+                return self.provenance_data.get(entity_id)
+        else:
+            return self.provenance_data.get(entity_id)
 
     def track_entities_batch(
         self,

semantica/kg/temporal_query.py

@@ -631,39 +631,47 @@ class TemporalPatternDetector:
 
 class TemporalVersionManager:
     """
-    Temporal version management engine.
+    Enhanced temporal version management engine with persistent storage.
 
-    This class provides version/snapshot management capabilities for knowledge
-    graphs, enabling creation of temporal versions, version comparison, and
-    version history tracking.
+    This class provides comprehensive version/snapshot management capabilities for knowledge
+    graphs, including persistent storage, detailed change tracking, and audit trails.
 
     Features:
-        - Version snapshot creation
-        - Version comparison
-        - Version history tracking
-        - Automatic snapshotting (planned)
-        - Version rollback (planned)
+        - Persistent snapshot storage (SQLite or in-memory)
+        - Detailed change tracking with entity-level diffs
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+        - Version comparison with backward compatibility
+        - Input validation and security features
 
     Example Usage:
+        >>> # In-memory storage
         >>> manager = TemporalVersionManager()
-        >>> version = manager.create_version(graph, version_label="v1.0")
-        >>> comparison = manager.compare_versions(version1, version2)
+        >>> # Persistent storage
+        >>> manager = TemporalVersionManager(storage_path="versions.db")
+        >>> snapshot = manager.create_snapshot(graph, "v1.0", 
+        ...     author="alice@company.com", description="Initial release")
+        >>> versions = manager.list_versions()
+        >>> diff = manager.compare_versions("v1.0", "v1.1")
     """
 
     def __init__(
         self,
+        storage_path: Optional[str] = None,
         snapshot_interval: Optional[int] = None,
         auto_snapshot: bool = False,
         version_strategy: str = "timestamp",
         **config,
     ):
         """
-        Initialize temporal version manager.
+        Initialize enhanced temporal version manager.
 
-        Sets up the version manager with snapshot configuration and versioning
-        strategy.
+        Sets up the version manager with storage backend, snapshot configuration,
+        and versioning strategy.
 
         Args:
+            storage_path: Path to SQLite database file for persistent storage.
+                         If None, uses in-memory storage (default: None)
             snapshot_interval: Interval for automatic snapshots in seconds
                              (optional, auto_snapshot must be True)
             auto_snapshot: Enable automatic snapshots (default: False)
@@ -671,11 +679,23 @@ class TemporalVersionManager:
                 - "timestamp": Use timestamps for version labels (default)
                 - "incremental": Use incremental version numbers (planned)
                 - "semantic": Use semantic versioning (planned)
-            **config: Additional configuration options (unused)
+            **config: Additional configuration options
         """
+        from semantica.change_management import ChangeLogEntry, VersionStorage, InMemoryVersionStorage, SQLiteVersionStorage
+        from ..utils.logging import get_logger
+        
         self.snapshot_interval = snapshot_interval
         self.auto_snapshot = auto_snapshot
         self.version_strategy = version_strategy
+        self.logger = get_logger("temporal_version_manager")
+        
+        # Initialize storage backend
+        if storage_path:
+            self.storage = SQLiteVersionStorage(storage_path)
+            self.logger.info(f"Initialized with SQLite storage: {storage_path}")
+        else:
+            self.storage = InMemoryVersionStorage()
+            self.logger.info("Initialized with in-memory storage")
 
     def create_version(
         self,
@@ -722,37 +742,282 @@ class TemporalVersionManager:
 
     def compare_versions(
         self,
-        version1: Dict[str, Any],
-        version2: Dict[str, Any],
+        v1_label_or_dict,
+        v2_label_or_dict,
         comparison_metrics: Optional[List[str]] = None,
         **options,
     ) -> Dict[str, Any]:
         """
-        Compare two graph versions.
+        Compare two graph versions with detailed entity-level differences.
 
-        This method compares two version snapshots and calculates differences
-        in entities and relationships.
+        This method compares two version snapshots and calculates detailed differences
+        in entities and relationships, maintaining backward compatibility.
 
         Args:
-            version1: First version snapshot dictionary
-            version2: Second version snapshot dictionary
+            v1_label_or_dict: First version (label string or snapshot dict)
+            v2_label_or_dict: Second version (label string or snapshot dict)
             comparison_metrics: List of metrics to calculate (optional, unused)
             **options: Additional comparison options (unused)
 
         Returns:
-            dict: Version comparison results containing:
-                - version1: Label of first version
-                - version2: Label of second version
-                - entities_added: Change in entity count (version2 - version1)
-                - relationships_added: Change in relationship count (version2 - version1)
+            dict: Detailed version comparison results containing:
+                - summary: Backward-compatible summary counts
+                - entities_added: List of added entities
+                - entities_removed: List of removed entities  
+                - entities_modified: List of modified entities with changes
+                - relationships_added: List of added relationships
+                - relationships_removed: List of removed relationships
+                - relationships_modified: List of modified relationships
         """
-        comparison = {
+        from ..utils.exceptions import ValidationError
+        
+        # Handle both label strings and snapshot dictionaries
+        if isinstance(v1_label_or_dict, str):
+            version1 = self.storage.get(v1_label_or_dict)
+            if not version1:
+                raise ValidationError(f"Version not found: {v1_label_or_dict}")
+        else:
+            version1 = v1_label_or_dict
+            
+        if isinstance(v2_label_or_dict, str):
+            version2 = self.storage.get(v2_label_or_dict)
+            if not version2:
+                raise ValidationError(f"Version not found: {v2_label_or_dict}")
+        else:
+            version2 = v2_label_or_dict
+        
+        # Compute detailed diff
+        detailed_diff = self._compute_detailed_diff(version1, version2)
+        
+        # Maintain backward compatibility with summary
+        summary = {
+            "entities_added": len(detailed_diff["entities_added"]),
+            "entities_removed": len(detailed_diff["entities_removed"]),
+            "entities_modified": len(detailed_diff["entities_modified"]),
+            "relationships_added": len(detailed_diff["relationships_added"]),
+            "relationships_removed": len(detailed_diff["relationships_removed"]),
+            "relationships_modified": len(detailed_diff["relationships_modified"])
+        }
+        
+        return {
             "version1": version1.get("label", "unknown"),
             "version2": version2.get("label", "unknown"),
-            "entities_added": len(version2.get("entities", []))
-            - len(version1.get("entities", [])),
-            "relationships_added": len(version2.get("relationships", []))
-            - len(version1.get("relationships", [])),
+            "summary": summary,
+            **detailed_diff
         }
 
-        return comparison
+    def create_snapshot(
+        self, 
+        graph: Dict[str, Any], 
+        version_label: str, 
+        author: str, 
+        description: str,
+        **options
+    ) -> Dict[str, Any]:
+        """
+        Create and store snapshot with checksum and metadata.
+        
+        Args:
+            graph: Knowledge graph dict with "entities" and "relationships"
+            version_label: Version string (e.g., "v1.0")
+            author: Email address of the change author
+            description: Change description (max 500 chars)
+            **options: Additional options
+            
+        Returns:
+            dict: Snapshot with metadata and checksum
+            
+        Raises:
+            ValidationError: If input validation fails
+            ProcessingError: If storage operation fails
+        """
+        from ..change_management import ChangeLogEntry, compute_checksum
+        from datetime import datetime
+        
+        # Validate inputs
+        change_entry = ChangeLogEntry(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description
+        )
+        
+        # Create snapshot
+        snapshot = {
+            "label": version_label,
+            "timestamp": change_entry.timestamp,
+            "author": change_entry.author,
+            "description": change_entry.description,
+            "entities": graph.get("entities", []).copy(),
+            "relationships": graph.get("relationships", []).copy(),
+            "metadata": options.get("metadata", {})
+        }
+        
+        # Compute and add checksum
+        snapshot["checksum"] = compute_checksum(snapshot)
+        
+        # Store snapshot
+        self.storage.save(snapshot)
+        
+        self.logger.info(f"Created snapshot '{version_label}' by {author}")
+        return snapshot
+    
+    def list_versions(self) -> List[Dict[str, Any]]:
+        """
+        List all version snapshots.
+        
+        Returns:
+            List of version metadata dictionaries
+        """
+        return self.storage.list_all()
+    
+    def get_version(self, label: str) -> Optional[Dict[str, Any]]:
+        """
+        Retrieve specific version by label.
+        
+        Args:
+            label: Version label to retrieve
+            
+        Returns:
+            Snapshot dictionary or None if not found
+        """
+        return self.storage.get(label)
+    
+    def verify_checksum(self, snapshot: Dict[str, Any]) -> bool:
+        """
+        Verify the integrity of a snapshot using its checksum.
+        
+        Args:
+            snapshot: Snapshot dictionary with checksum field
+            
+        Returns:
+            True if checksum is valid, False otherwise
+        """
+        from ..change_management import verify_checksum
+        return verify_checksum(snapshot)
+    
+    def _compute_detailed_diff(self, version1: Dict[str, Any], version2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute detailed entity and relationship differences between versions.
+        
+        Args:
+            version1: First version snapshot
+            version2: Second version snapshot
+            
+        Returns:
+            Dict with detailed diff information
+        """
+        entities1 = {e.get("id", str(i)): e for i, e in enumerate(version1.get("entities", []))}
+        entities2 = {e.get("id", str(i)): e for i, e in enumerate(version2.get("entities", []))}
+        
+        relationships1 = {self._relationship_key(r): r for r in version1.get("relationships", [])}
+        relationships2 = {self._relationship_key(r): r for r in version2.get("relationships", [])}
+        
+        # Entity differences
+        entity_ids1 = set(entities1.keys())
+        entity_ids2 = set(entities2.keys())
+        
+        entities_added = [entities2[eid] for eid in entity_ids2 - entity_ids1]
+        entities_removed = [entities1[eid] for eid in entity_ids1 - entity_ids2]
+        
+        entities_modified = []
+        for eid in entity_ids1 & entity_ids2:
+            if entities1[eid] != entities2[eid]:
+                changes = self._compute_entity_changes(entities1[eid], entities2[eid])
+                entities_modified.append({
+                    "id": eid,
+                    "before": entities1[eid],
+                    "after": entities2[eid],
+                    "changes": changes
+                })
+        
+        # Relationship differences
+        rel_keys1 = set(relationships1.keys())
+        rel_keys2 = set(relationships2.keys())
+        
+        relationships_added = [relationships2[key] for key in rel_keys2 - rel_keys1]
+        relationships_removed = [relationships1[key] for key in rel_keys1 - rel_keys2]
+        
+        relationships_modified = []
+        for key in rel_keys1 & rel_keys2:
+            if relationships1[key] != relationships2[key]:
+                changes = self._compute_relationship_changes(relationships1[key], relationships2[key])
+                relationships_modified.append({
+                    "key": key,
+                    "before": relationships1[key],
+                    "after": relationships2[key],
+                    "changes": changes
+                })
+        
+        return {
+            "entities_added": entities_added,
+            "entities_removed": entities_removed,
+            "entities_modified": entities_modified,
+            "relationships_added": relationships_added,
+            "relationships_removed": relationships_removed,
+            "relationships_modified": relationships_modified
+        }
+    
+    def _relationship_key(self, relationship: Dict[str, Any]) -> str:
+        """
+        Generate a unique key for a relationship.
+        
+        Args:
+            relationship: Relationship dictionary
+            
+        Returns:
+            Unique string key for the relationship
+        """
+        source = relationship.get("source", "")
+        target = relationship.get("target", "")
+        rel_type = relationship.get("type", relationship.get("relationship", ""))
+        return f"{source}|{rel_type}|{target}"
+    
+    def _compute_entity_changes(self, entity1: Dict[str, Any], entity2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute changes between two entity versions.
+        
+        Args:
+            entity1: Original entity
+            entity2: Modified entity
+            
+        Returns:
+            Dictionary of changes
+        """
+        changes = {}
+        
+        # Check all keys from both entities
+        all_keys = set(entity1.keys()) | set(entity2.keys())
+        
+        for key in all_keys:
+            val1 = entity1.get(key)
+            val2 = entity2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes
+    
+    def _compute_relationship_changes(self, rel1: Dict[str, Any], rel2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute changes between two relationship versions.
+        
+        Args:
+            rel1: Original relationship
+            rel2: Modified relationship
+            
+        Returns:
+            Dictionary of changes
+        """
+        changes = {}
+        
+        # Check all keys from both relationships
+        all_keys = set(rel1.keys()) | set(rel2.keys())
+        
+        for key in all_keys:
+            val1 = rel1.get(key)
+            val2 = rel2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes

semantica/llms/llms_provenance.py

@@ -0,0 +1,386 @@
+"""
+Provenance-enabled wrappers for LLM providers.
+
+This module provides provenance tracking for all LLM operations:
+- Groq LLM
+- OpenAI LLM
+- HuggingFace LLM
+- LiteLLM
+
+Tracks: model name, tokens (prompt/completion), cost, latency, prompts, responses
+
+All classes wrap the original LLM providers and add optional provenance tracking
+without modifying existing functionality.
+
+Usage:
+    from semantica.llms.llms_provenance import (
+        GroqLLMWithProvenance,
+        OpenAILLMWithProvenance
+    )
+    
+    # Enable provenance tracking
+    llm = GroqLLMWithProvenance(provenance=True)
+    response = llm.generate("What is artificial intelligence?")
+    
+    # Provenance automatically tracks:
+    # - Model used
+    # - Token counts
+    # - API costs
+    # - Latency
+    # - Prompt and response previews
+
+Features:
+    - Zero breaking changes - works exactly like original LLM classes
+    - Opt-in provenance via provenance=True parameter
+    - Tracks all API calls with complete metadata
+    - Cost tracking for budget monitoring
+    - Performance monitoring (latency)
+    - Graceful degradation if provenance module unavailable
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Dict, Any
+import time
+import uuid
+
+
+class LLMProvenanceMixin:
+    """
+    Mixin to add provenance tracking to any LLM provider.
+    
+    This mixin provides common provenance infrastructure for tracking
+    LLM API calls including tokens, costs, and performance metrics.
+    """
+    
+    def __init__(self, provenance: bool = False, **kwargs):
+        """
+        Initialize LLM provenance tracking.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **kwargs: Additional arguments passed to parent class
+        """
+        self.provenance = provenance
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                # Graceful degradation if provenance module not available
+                self.provenance = False
+    
+    def _track_llm_call(
+        self,
+        call_id: str,
+        prompt: str,
+        response: Any,
+        **metadata
+    ) -> None:
+        """
+        Track LLM API call with provenance.
+        
+        Args:
+            call_id: Unique identifier for this API call
+            prompt: Input prompt
+            response: LLM response
+            **metadata: Additional metadata (tokens, cost, latency, etc.)
+        """
+        if self.provenance and self._prov_manager:
+            # Extract response text
+            response_text = response
+            if hasattr(response, 'text'):
+                response_text = response.text
+            elif hasattr(response, 'content'):
+                response_text = response.content
+            elif not isinstance(response, str):
+                response_text = str(response)
+            
+            self._prov_manager.track_entity(
+                entity_id=call_id,
+                source=f"{self.__class__.__name__}_api",
+                entity_type="llm_generation",
+                metadata={
+                    "model": getattr(self, 'model', 'unknown'),
+                    "prompt_preview": prompt[:200] if len(prompt) > 200 else prompt,
+                    "response_preview": response_text[:200] if len(str(response_text)) > 200 else str(response_text),
+                    **metadata
+                }
+            )
+
+
+class GroqLLMWithProvenance(LLMProvenanceMixin):
+    """
+    Groq LLM with provenance tracking.
+    
+    Wraps the original GroqLLM and tracks all API calls with complete metadata.
+    
+    Example:
+        >>> llm = GroqLLMWithProvenance(provenance=True, model="llama-3.1-70b")
+        >>> response = llm.generate("Explain quantum computing")
+        >>> # API call is tracked with model, tokens, cost, latency
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize Groq LLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original GroqLLM
+        """
+        from .groq_llm import GroqLLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = GroqLLM(**config)
+        self.model = getattr(self._llm, 'model', 'groq')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response (same format as original GroqLLM)
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            # Extract token counts if available
+            prompt_tokens = None
+            completion_tokens = None
+            total_cost = None
+            
+            if hasattr(response, 'usage'):
+                prompt_tokens = getattr(response.usage, 'prompt_tokens', None)
+                completion_tokens = getattr(response.usage, 'completion_tokens', None)
+            
+            if hasattr(response, 'cost'):
+                total_cost = response.cost
+            
+            self._track_llm_call(
+                call_id=f"groq_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_tokens=(prompt_tokens + completion_tokens) if (prompt_tokens and completion_tokens) else None,
+                total_cost=total_cost,
+                latency_seconds=elapsed,
+                temperature=kwargs.get('temperature'),
+                max_tokens=kwargs.get('max_tokens'),
+                top_p=kwargs.get('top_p')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+class OpenAILLMWithProvenance(LLMProvenanceMixin):
+    """
+    OpenAI LLM with provenance tracking.
+    
+    Wraps the original OpenAILLM and tracks all API calls.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize OpenAI LLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original OpenAILLM
+        """
+        from .openai_llm import OpenAILLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = OpenAILLM(**config)
+        self.model = getattr(self._llm, 'model', 'openai')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            # Extract token counts if available
+            prompt_tokens = None
+            completion_tokens = None
+            total_cost = None
+            
+            if hasattr(response, 'usage'):
+                prompt_tokens = getattr(response.usage, 'prompt_tokens', None)
+                completion_tokens = getattr(response.usage, 'completion_tokens', None)
+            
+            if hasattr(response, 'cost'):
+                total_cost = response.cost
+            
+            self._track_llm_call(
+                call_id=f"openai_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_tokens=(prompt_tokens + completion_tokens) if (prompt_tokens and completion_tokens) else None,
+                total_cost=total_cost,
+                latency_seconds=elapsed,
+                temperature=kwargs.get('temperature'),
+                max_tokens=kwargs.get('max_tokens')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+class HuggingFaceLLMWithProvenance(LLMProvenanceMixin):
+    """
+    HuggingFace LLM with provenance tracking.
+    
+    Wraps the original HuggingFaceLLM and tracks all generations.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize HuggingFace LLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original HuggingFaceLLM
+        """
+        from .huggingface_llm import HuggingFaceLLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = HuggingFaceLLM(**config)
+        self.model = getattr(self._llm, 'model', 'huggingface')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            self._track_llm_call(
+                call_id=f"hf_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                latency_seconds=elapsed,
+                max_length=kwargs.get('max_length'),
+                temperature=kwargs.get('temperature')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+class LiteLLMWithProvenance(LLMProvenanceMixin):
+    """
+    LiteLLM with provenance tracking.
+    
+    Wraps the original LiteLLM and tracks all API calls across providers.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize LiteLLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original LiteLLM
+        """
+        from .lite_llm import LiteLLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = LiteLLM(**config)
+        self.model = getattr(self._llm, 'model', 'litellm')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            # LiteLLM provides unified response format
+            prompt_tokens = None
+            completion_tokens = None
+            total_cost = None
+            
+            if hasattr(response, 'usage'):
+                prompt_tokens = getattr(response.usage, 'prompt_tokens', None)
+                completion_tokens = getattr(response.usage, 'completion_tokens', None)
+            
+            if hasattr(response, '_hidden_params') and 'response_cost' in response._hidden_params:
+                total_cost = response._hidden_params['response_cost']
+            
+            self._track_llm_call(
+                call_id=f"lite_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_cost=total_cost,
+                latency_seconds=elapsed,
+                provider=kwargs.get('provider')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+# Convenience exports
+__all__ = [
+    'GroqLLMWithProvenance',
+    'OpenAILLMWithProvenance',
+    'HuggingFaceLLMWithProvenance',
+    'LiteLLMWithProvenance',
+    'LLMProvenanceMixin',
+]

semantica/normalize/normalize_provenance.py

@@ -0,0 +1,53 @@
+"""
+Provenance-enabled wrapper for normalization.
+
+Usage:
+    from semantica.normalize.normalize_provenance import NormalizerWithProvenance
+    
+    normalizer = NormalizerWithProvenance(provenance=True)
+    normalized_data = normalizer.normalize(data)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class NormalizerWithProvenance:
+    """Normalizer with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .normalizer import Normalizer
+        
+        self.provenance = provenance
+        self._normalizer = Normalizer(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def normalize(self, data: Any, source: str = None, **kwargs):
+        """Normalize data with provenance tracking."""
+        result = self._normalizer.normalize(data, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"normalize_{uuid.uuid4().hex[:8]}",
+                source=source or "normalization",
+                entity_type="normalized_data",
+                metadata={"method": kwargs.get('method', 'default')}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._normalizer, name)
+
+
+__all__ = ['NormalizerWithProvenance']

semantica/normalize/text_normalizer.py

@@ -562,15 +562,19 @@ class SpecialCharacterProcessor:
         Returns:
             str: Text with normalized punctuation marks
         """
-        # Normalize quotes
-        text = re.sub(r'["""]', '"', text)
-        text = re.sub(r"[''']", "'", text)
+        # Replace common smart punctuation with ASCII equivalents
+        replacements = {
+            "\u2018": "'",   # Left single quotation mark
+            "\u2019": "'",   # Right single quotation mark
+            "\u201C": '"',   # Left double quotation mark
+            "\u201D": '"',   # Right double quotation mark
+            "\u2013": "-",   # En dash
+            "\u2014": "--",  # Em dash
+            "\u2026": "...", # Ellipsis
+        }
 
-        # Normalize dashes
-        text = re.sub(r"[–—]", "-", text)
-
-        # Normalize ellipsis
-        text = text.replace("…", "...")
+        for old, new in replacements.items():
+            text = text.replace(old, new)
 
         return text
 

semantica/ontology/__init__.py

@@ -109,12 +109,14 @@ Convenience Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory
 
 Example Usage:
-    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator
+    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator, ingest_ontology
     >>> # Using convenience functions
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
     >>> # Using classes directly
     >>> from semantica.ontology import OntologyGenerator, ClassInferrer, PropertyGenerator
     >>> generator = OntologyGenerator(base_uri="https://example.org/ontology/")
@@ -154,7 +156,10 @@ from .property_generator import PropertyGenerator
 from .registry import MethodRegistry, method_registry
 from .requirements_spec import RequirementsSpec, RequirementsSpecManager
 from .reuse_manager import ReuseDecision, ReuseManager
-from .version_manager import OntologyVersion, VersionManager
+# VersionManager and OntologyVersion moved to change_management module
+# Import them directly from there: from semantica.change_management import VersionManager, OntologyVersion
+from semantica.ingest import OntologyData, OntologyIngestor
+from .methods import ingest_ontology
 
 __all__ = [
     # Main generators
@@ -180,8 +185,7 @@ __all__ = [
     # Management
     "ReuseManager",
     "ReuseDecision",
-    "VersionManager",
-    "OntologyVersion",
+    # VersionManager and OntologyVersion moved to change_management module
     "NamespaceManager",
     "NamingConventions",
     "ModuleManager",
@@ -200,4 +204,7 @@ __all__ = [
     # Configuration
     "OntologyConfig",
     "ontology_config",
+    "ingest_ontology",
+    "OntologyData",
+    "OntologyIngestor",
 ]

semantica/ontology/llm_generator.py

@@ -41,7 +41,7 @@ class LLMOntologyGenerator:
 
         try:
             result = self.provider.generate_structured(
-                prompt, model=self.model or options.get("model"), temperature=options.get("temperature", 0.2)
+                prompt, model=self.model or options.get("model"), temperature=options.get("temperature")
             )
         except Exception as e:
             self.progress.update_tracking(tracking_id, message="LLM generation failed")

semantica/ontology/methods.py

@@ -111,15 +111,19 @@ Main Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory (via semantica.ingest)
 
 Example Usage:
-    >>> from semantica.ontology.methods import generate_ontology, infer_classes
+    >>> from semantica.ontology.methods import generate_ontology, infer_classes, ingest_ontology
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
 """
 
-from typing import Any, Callable, Dict, List, Optional
+from typing import Any, Callable, Dict, List, Optional, Union
+from pathlib import Path
 
+from semantica.ingest import ingest_ontology as _ingest_ontology, OntologyData
 from .registry import method_registry
 
 
@@ -172,4 +176,23 @@ def list_available_methods(task: Optional[str] = None) -> Dict[str, List[str]]:
     return method_registry.list_all(task)
 
 
-pass
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], 
+    method: str = "file", 
+    **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source.
+    
+    This is a convenience wrapper around semantica.ingest.ingest_ontology.
+    
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+        **kwargs: Additional options
+        
+    Returns:
+        OntologyData or List[OntologyData]
+    """
+    return _ingest_ontology(source, method=method, **kwargs)
+

semantica/ontology/ontology_provenance.py

@@ -0,0 +1,53 @@
+"""
+Provenance-enabled wrapper for ontology operations.
+
+Usage:
+    from semantica.ontology.ontology_provenance import OntologyManagerWithProvenance
+    
+    ontology = OntologyManagerWithProvenance(provenance=True)
+    ontology.add_concept(concept, source="ontology.owl")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class OntologyManagerWithProvenance:
+    """Ontology manager with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .ontology_manager import OntologyManager
+        
+        self.provenance = provenance
+        self._manager = OntologyManager(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def add_concept(self, concept: Any, source: str = None, **kwargs):
+        """Add concept with provenance tracking."""
+        result = self._manager.add_concept(concept, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"concept_{uuid.uuid4().hex[:8]}",
+                source=source or "ontology",
+                entity_type="ontology_concept",
+                metadata={"concept_name": str(concept)}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._manager, name)
+
+
+__all__ = ['OntologyManagerWithProvenance']

semantica/ontology/ontology_usage.md

@@ -42,6 +42,18 @@ classes = inferrer.infer_classes(entities, build_hierarchy=True)
 properties = prop_gen.infer_properties(entities, relationships, classes)
 ```
 
+### Ingesting Ontologies
+
+```python
+from semantica.ingest import OntologyIngestor
+
+# Create ingestor
+ingestor = OntologyIngestor()
+
+# Ingest ontology
+ontology_data = ingestor.ingest_ontology("ontology.ttl")
+```
+
 ## Ontology Generation
 
 ### Basic Ontology Generation
@@ -115,6 +127,49 @@ ontology = engine.from_data(
 )
 ```
 
+## Ontology Ingestion
+
+### Basic Ingestion
+
+Ingest existing ontologies from files (Turtle, RDF/XML, JSON-LD, etc.) into `OntologyData` objects.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a single file
+ontology_data = ingest_ontology("path/to/ontology.ttl")
+
+print(f"Source: {ontology_data.source_path}")
+print(f"Format: {ontology_data.format}")
+print(f"Data keys: {ontology_data.data.keys()}")
+```
+
+### Ingesting Directories
+
+Ingest all ontology files in a directory recursively.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a directory
+ontologies = ingest_ontology("path/to/ontologies_dir/")
+
+for ont in ontologies:
+    print(f"Ingested: {ont.source_path} ({ont.format})")
+```
+
+### Unified Ingestion Interface
+
+You can also use the unified `semantica.ingest` interface.
+
+```python
+from semantica.ingest import ingest
+
+# Ingest as "ontology" source type
+result = ingest("path/to/ontology.ttl", source_type="ontology")
+ontology_data = result["ontology"]
+```
+
 ## Class Inference
 
 ### Basic Class Inference

semantica/ontology/reuse_manager.py

@@ -363,3 +363,90 @@ class ReuseManager:
     def list_known_ontologies(self) -> List[str]:
         """List known ontology URIs."""
         return list(self.known_ontologies.keys())
+
+    def merge_ontology_data(
+        self, target: Dict[str, Any], source: Dict[str, Any], **options
+    ) -> Dict[str, Any]:
+        """
+        Merge source ontology data into target ontology.
+
+        Merges classes, properties, and metadata from source to target.
+        Handles deduplication based on URI and name.
+
+        Args:
+            target: Target ontology dictionary (modified in-place)
+            source: Source ontology dictionary
+            **options: Merge options:
+                - overwrite: Whether to overwrite existing elements (default: False)
+                - merge_metadata: Whether to merge metadata (default: True)
+
+        Returns:
+            Merged target ontology
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="ontology",
+            submodule="ReuseManager",
+            message=f"Merging ontology {source.get('name', 'unknown')} into {target.get('name', 'unknown')}",
+        )
+
+        try:
+            overwrite = options.get("overwrite", False)
+            
+            # Helper to merge lists of dicts (classes/properties)
+            def merge_lists(target_list, source_list, key_field="uri"):
+                existing_keys = {item.get(key_field): i for i, item in enumerate(target_list) if item.get(key_field)}
+                
+                for item in source_list:
+                    key = item.get(key_field)
+                    if not key:
+                        # Fallback to name if URI missing
+                        key = item.get("name")
+                        
+                    if key in existing_keys:
+                        if overwrite:
+                            target_list[existing_keys[key]] = item
+                    else:
+                        target_list.append(item)
+                        if key:
+                            existing_keys[key] = len(target_list) - 1
+
+            # Merge Classes
+            if "classes" in source:
+                if "classes" not in target:
+                    target["classes"] = []
+                merge_lists(target["classes"], source["classes"])
+
+            # Merge Properties
+            if "properties" in source:
+                if "properties" not in target:
+                    target["properties"] = []
+                merge_lists(target["properties"], source["properties"])
+
+            # Merge Metadata
+            if options.get("merge_metadata", True) and "metadata" in source:
+                if "metadata" not in target:
+                    target["metadata"] = {}
+                # Update with source metadata, preserving target's specific fields if needed
+                # Here we just update
+                target["metadata"].update(source["metadata"])
+                
+            # Merge Imports
+            if "imports" in source:
+                if "imports" not in target:
+                    target["imports"] = []
+                for imp in source["imports"]:
+                    if imp not in target["imports"]:
+                        target["imports"].append(imp)
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Merged ontology data successfully",
+            )
+            return target
+
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise

semantica/parse/parse_provenance.py

@@ -0,0 +1,58 @@
+"""
+Provenance-enabled wrappers for parsing operations.
+
+Tracks: file parsed, format, structure, parsing method
+
+Usage:
+    from semantica.parse.parse_provenance import JSONParserWithProvenance
+    
+    parser = JSONParserWithProvenance(provenance=True)
+    data = parser.parse("data.json")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class ParserWithProvenance:
+    """Base parser with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .parser import Parser
+        
+        self.provenance = provenance
+        self._parser = Parser(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def parse(self, file_path: str, **kwargs):
+        """Parse file with provenance tracking."""
+        data = self._parser.parse(file_path, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"parse_{uuid.uuid4().hex[:8]}",
+                source=file_path,
+                entity_type="parsed_data",
+                metadata={
+                    "file_path": file_path,
+                    "format": kwargs.get('format', 'unknown')
+                }
+            )
+        
+        return data
+    
+    def __getattr__(self, name):
+        return getattr(self._parser, name)
+
+
+__all__ = ['ParserWithProvenance']

semantica/pipeline/pipeline_builder.py

@@ -32,12 +32,14 @@ License: MIT
 
 from dataclasses import dataclass, field
 from enum import Enum
-from typing import Any, Callable, Dict, List, Optional, Union
+from typing import Any, Callable, Dict, List, Optional, Union, TYPE_CHECKING
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
-from .pipeline_validator import PipelineValidator
+
+if TYPE_CHECKING:
+    from .pipeline_validator import PipelineValidator
 
 
 class StepStatus(Enum):
@@ -104,6 +106,8 @@ class PipelineBuilder:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        from .pipeline_validator import PipelineValidator
+
         self.validator = PipelineValidator(**self.config)
         self.steps: List[PipelineStep] = []
         self.step_registry: Dict[str, Callable] = {}

semantica/pipeline/pipeline_provenance.py

@@ -0,0 +1,67 @@
+"""
+Provenance-enabled wrapper for pipeline execution.
+
+This module provides provenance tracking for end-to-end pipeline workflows,
+capturing all steps, inputs, outputs, and transformations.
+
+Usage:
+    from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
+    
+    pipeline = PipelineWithProvenance(provenance=True)
+    result = pipeline.run(data)
+    # Tracks all pipeline steps with complete lineage
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Any, Dict, List
+import uuid
+import time
+
+
+class PipelineWithProvenance:
+    """Pipeline executor with complete provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        """Initialize pipeline with optional provenance."""
+        from .pipeline import Pipeline
+        
+        self.provenance = provenance
+        self._pipeline = Pipeline(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def run(self, data: Any, source: Optional[str] = None, **kwargs):
+        """Run pipeline with provenance tracking."""
+        pipeline_id = f"pipeline_{uuid.uuid4().hex[:8]}"
+        start_time = time.time()
+        
+        result = self._pipeline.run(data, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=pipeline_id,
+                source=source or "pipeline_execution",
+                entity_type="pipeline_run",
+                metadata={
+                    "steps": len(self._pipeline.steps) if hasattr(self._pipeline, 'steps') else 0,
+                    "duration_seconds": elapsed,
+                    "status": "completed"
+                }
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._pipeline, name)
+
+
+__all__ = ['PipelineWithProvenance']

semantica/provenance/__init__.py

@@ -0,0 +1,69 @@
+"""
+Provenance Tracking Module for Semantica
+
+Provides audit-grade provenance tracking for high-stakes domains requiring
+complete traceability (blue finance, healthcare, legal, pharma).
+
+This module consolidates and enhances provenance tracking from:
+- kg.ProvenanceTracker (entity/relationship tracking)
+- split.ProvenanceTracker (chunk tracking)
+- conflicts.SourceTracker (source tracking)
+
+Key Features:
+    - W3C PROV-O compliant tracking
+    - End-to-end lineage (doc → chunk → entity → KG → query → response)
+    - Bridge axiom translation chains (L1 → L2 → L3)
+    - Audit-grade source tracking (DOI + page + quote)
+    - Zero breaking changes (opt-in only)
+    - No new dependencies (stdlib only)
+
+Example Usage:
+    >>> # Enable provenance tracking
+    >>> from semantica.semantic_extract import NERExtractor
+    >>> ner = NERExtractor(provenance=True)
+    >>> entities = ner.extract("Steve Jobs founded Apple.")
+    >>> 
+    >>> # Trace lineage
+    >>> from semantica.provenance import ProvenanceManager
+    >>> prov_mgr = ProvenanceManager()
+    >>> lineage = prov_mgr.get_lineage(entities[0].id)
+    >>> 
+    >>> # Track with source details
+    >>> prov_mgr.track_entity(
+    ...     entity_id="entity_1",
+    ...     source="DOI:10.1371/journal.pone.0023601",
+    ...     source_location="Figure 2",
+    ...     source_quote="Total fish biomass increased by 463%...",
+    ...     confidence=0.92
+    ... )
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from .schemas import ProvenanceEntry, SourceReference
+from .storage import ProvenanceStorage, InMemoryStorage, SQLiteStorage
+from .manager import ProvenanceManager
+from .integrity import compute_checksum, verify_checksum
+
+__all__ = [
+    # Core schemas
+    "ProvenanceEntry",
+    "SourceReference",
+    
+    # Storage backends
+    "ProvenanceStorage",
+    "InMemoryStorage",
+    "SQLiteStorage",
+    
+    # Manager
+    "ProvenanceManager",
+    
+    # Utilities
+    "compute_checksum",
+    "verify_checksum",
+]
+
+__version__ = "1.0.0"
+__author__ = "Semantica Team"
+__description__ = "Audit-Grade Provenance Tracking for Semantica"

semantica/provenance/bridge_axiom.py

@@ -0,0 +1,435 @@
+"""
+Bridge Axiom Translation Chain Tracking
+
+This module provides bridge axiom tracking for multi-layer provenance chains,
+enabling translation from one domain to another across all high-stakes domains.
+
+Supported Domain Translations:
+    - Ecological → Financial (Blue Finance, Natural Capital)
+    - Clinical → Diagnostic (Healthcare, Medical Research)
+    - Evidence → Legal Conclusion (Legal, Forensic)
+    - Research Data → Drug Efficacy (Pharmaceutical, Clinical Trials)
+    - Raw Data → Financial Metrics (Finance, Risk Assessment)
+    - Sensor Data → Security Threat (Intelligence, Cybersecurity)
+    - Biological Data → Biomedical Insights (Biomedical Research)
+    - Asset Data → Portfolio Risk (Asset Management)
+
+Features:
+    - Bridge axiom definition and tracking
+    - Translation chain provenance
+    - Coefficient source tracking (DOI + page + quote)
+    - Multi-layer lineage tracing
+    - Confidence propagation
+
+Examples:
+    >>> # Blue Finance: Ecological → Financial
+    >>> ba_finance = BridgeAxiom(
+    ...     axiom_id="BA-FINANCE-001",
+    ...     name="biomass_tourism_elasticity",
+    ...     rule="1% biomass increase → 0.346% tourism revenue increase",
+    ...     coefficient=0.346,
+    ...     source_doi="10.1038/s41586-021-03371-z",
+    ...     input_domain="ecological",
+    ...     output_domain="financial"
+    ... )
+    >>> 
+    >>> # Healthcare: Clinical Observation → Diagnosis Probability
+    >>> ba_health = BridgeAxiom(
+    ...     axiom_id="BA-HEALTH-001",
+    ...     name="fever_influenza_correlation",
+    ...     rule="Fever >38°C increases influenza probability by 0.65",
+    ...     coefficient=0.65,
+    ...     source_doi="10.1001/jama.2020.12345",
+    ...     input_domain="clinical_observation",
+    ...     output_domain="diagnostic_probability"
+    ... )
+    >>> 
+    >>> # Legal: Evidence Strength → Conviction Probability
+    >>> ba_legal = BridgeAxiom(
+    ...     axiom_id="BA-LEGAL-001",
+    ...     name="dna_match_conviction",
+    ...     rule="DNA match increases conviction probability by 0.95",
+    ...     coefficient=0.95,
+    ...     source_doi="10.1016/j.forsciint.2019.12345",
+    ...     input_domain="forensic_evidence",
+    ...     output_domain="legal_conclusion"
+    ... )
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from dataclasses import dataclass, field
+from typing import Optional, Dict, Any, List
+from datetime import datetime
+import uuid
+
+
+@dataclass
+class BridgeAxiom:
+    """
+    Bridge axiom for domain translation with provenance.
+    
+    Represents a rule that translates data from one domain to another,
+    with complete source tracking for audit-grade provenance.
+    
+    Attributes:
+        axiom_id: Unique axiom identifier (e.g., "BA-001")
+        name: Human-readable axiom name
+        rule: Rule description in natural language
+        coefficient: Numeric coefficient for translation
+        source_doi: DOI of source paper/document
+        source_page: Page/table/figure in source
+        source_quote: Direct quote supporting the coefficient
+        confidence: Confidence score (0.0-1.0)
+        input_domain: Input domain (e.g., "ecological")
+        output_domain: Output domain (e.g., "financial")
+        metadata: Additional metadata
+    
+    Examples:
+        >>> # Blue Finance
+        >>> ba_finance = BridgeAxiom(
+        ...     axiom_id="BA-FINANCE-001",
+        ...     name="biomass_tourism_elasticity",
+        ...     rule="1% biomass increase → 0.346% tourism revenue increase",
+        ...     coefficient=0.346,
+        ...     source_doi="10.1038/s41586-021-03371-z",
+        ...     input_domain="ecological",
+        ...     output_domain="financial"
+        ... )
+        >>> 
+        >>> # Healthcare
+        >>> ba_health = BridgeAxiom(
+        ...     axiom_id="BA-HEALTH-001",
+        ...     name="symptom_diagnosis_correlation",
+        ...     rule="Symptom X increases diagnosis Y probability by 0.75",
+        ...     coefficient=0.75,
+        ...     source_doi="10.1001/jama.2020.12345",
+        ...     input_domain="clinical_symptom",
+        ...     output_domain="diagnosis"
+        ... )
+        >>> 
+        >>> # Pharmaceutical
+        >>> ba_pharma = BridgeAxiom(
+        ...     axiom_id="BA-PHARMA-001",
+        ...     name="dosage_efficacy_relationship",
+        ...     rule="10mg increase → 0.15 efficacy improvement",
+        ...     coefficient=0.15,
+        ...     source_doi="10.1056/NEJMoa2020123",
+        ...     input_domain="drug_dosage",
+        ...     output_domain="clinical_efficacy"
+        ... )
+    """
+    
+    axiom_id: str
+    name: str
+    rule: str
+    coefficient: float
+    source_doi: str
+    source_page: str
+    source_quote: Optional[str] = None
+    confidence: float = 1.0
+    input_domain: str = "unknown"
+    output_domain: str = "unknown"
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def apply(
+        self,
+        input_entity: str,
+        input_value: float,
+        prov_manager: Optional[Any] = None,
+        **kwargs
+    ) -> Dict[str, Any]:
+        """
+        Apply bridge axiom to input value with provenance tracking.
+        
+        Args:
+            input_entity: Input entity identifier
+            input_value: Input value to transform
+            prov_manager: ProvenanceManager instance (optional)
+            **kwargs: Additional parameters
+            
+        Returns:
+            Dictionary with result and provenance information
+            
+        Example:
+            >>> result = ba.apply(
+            ...     input_entity="cabo_pulmo_biomass",
+            ...     input_value=463,
+            ...     prov_manager=prov_mgr
+            ... )
+            >>> print(result["output_value"])
+            160.098
+        """
+        # Calculate output value
+        output_value = input_value * self.coefficient
+        
+        # Generate output entity ID
+        output_entity = f"{input_entity}_transformed_{self.axiom_id}"
+        
+        # Track provenance if manager provided
+        if prov_manager:
+            try:
+                # Track the bridge axiom application
+                prov_manager.track_entity(
+                    entity_id=output_entity,
+                    source=self.source_doi,
+                    entity_type="bridge_axiom_result",
+                    activity_id=f"bridge_axiom_application_{self.axiom_id}",
+                    source_location=self.source_page,
+                    source_quote=self.source_quote,
+                    confidence=self.confidence,
+                    metadata={
+                        "axiom_id": self.axiom_id,
+                        "axiom_name": self.name,
+                        "rule": self.rule,
+                        "coefficient": self.coefficient,
+                        "input_entity": input_entity,
+                        "input_value": input_value,
+                        "output_value": output_value,
+                        "input_domain": self.input_domain,
+                        "output_domain": self.output_domain,
+                        **kwargs
+                    }
+                )
+            except Exception:
+                pass  # Graceful failure
+        
+        return {
+            "axiom_id": self.axiom_id,
+            "axiom_name": self.name,
+            "input_entity": input_entity,
+            "input_value": input_value,
+            "output_entity": output_entity,
+            "output_value": output_value,
+            "coefficient": self.coefficient,
+            "confidence": self.confidence,
+            "source_doi": self.source_doi,
+            "source_page": self.source_page,
+            "input_domain": self.input_domain,
+            "output_domain": self.output_domain
+        }
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert bridge axiom to dictionary."""
+        return {
+            "axiom_id": self.axiom_id,
+            "name": self.name,
+            "rule": self.rule,
+            "coefficient": self.coefficient,
+            "source_doi": self.source_doi,
+            "source_page": self.source_page,
+            "source_quote": self.source_quote,
+            "confidence": self.confidence,
+            "input_domain": self.input_domain,
+            "output_domain": self.output_domain,
+            "metadata": self.metadata
+        }
+
+
+@dataclass
+class TranslationChain:
+    """
+    Multi-layer translation chain with complete provenance.
+    
+    Tracks a complete translation from source data through multiple
+    bridge axioms to final output (e.g., L1 → L2 → L3).
+    
+    Attributes:
+        chain_id: Unique chain identifier
+        layers: List of layer dictionaries
+        confidence: Overall confidence score
+        metadata: Additional metadata
+    
+    Example:
+        >>> chain = TranslationChain(
+        ...     chain_id="chain_001",
+        ...     layers=[
+        ...         {"layer": "L1", "type": "ecological", "value": 463},
+        ...         {"layer": "L2", "type": "bridge_axiom", "axiom": "BA-001"},
+        ...         {"layer": "L3", "type": "financial", "value": 29.27}
+        ...     ]
+        ... )
+    """
+    
+    chain_id: str
+    layers: List[Dict[str, Any]] = field(default_factory=list)
+    confidence: float = 1.0
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def add_layer(
+        self,
+        layer_name: str,
+        layer_type: str,
+        value: Any,
+        source: Optional[str] = None,
+        **kwargs
+    ) -> None:
+        """
+        Add a layer to the translation chain.
+        
+        Args:
+            layer_name: Layer name (e.g., "L1", "L2", "L3")
+            layer_type: Layer type (e.g., "ecological", "bridge_axiom", "financial")
+            value: Layer value
+            source: Source document/DOI
+            **kwargs: Additional layer metadata
+        """
+        layer = {
+            "layer": layer_name,
+            "type": layer_type,
+            "value": value,
+            "source": source,
+            "timestamp": datetime.utcnow().isoformat(),
+            **kwargs
+        }
+        self.layers.append(layer)
+    
+    def get_layer(self, layer_name: str) -> Optional[Dict[str, Any]]:
+        """
+        Get a specific layer by name.
+        
+        Args:
+            layer_name: Layer name to retrieve
+            
+        Returns:
+            Layer dictionary or None
+        """
+        for layer in self.layers:
+            if layer.get("layer") == layer_name:
+                return layer
+        return None
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert translation chain to dictionary."""
+        return {
+            "chain_id": self.chain_id,
+            "layers": self.layers,
+            "confidence": self.confidence,
+            "metadata": self.metadata
+        }
+
+
+def create_translation_chain(
+    input_data: Dict[str, Any],
+    bridge_axioms: List[BridgeAxiom],
+    prov_manager: Optional[Any] = None
+) -> TranslationChain:
+    """
+    Create a complete translation chain through multiple bridge axioms.
+    
+    Args:
+        input_data: Input data dictionary with 'entity_id' and 'value'
+        bridge_axioms: List of BridgeAxiom objects to apply in sequence
+        prov_manager: ProvenanceManager instance (optional)
+        
+    Returns:
+        TranslationChain object with complete provenance
+        
+    Example:
+        >>> input_data = {
+        ...     "entity_id": "cabo_pulmo_biomass",
+        ...     "value": 463,
+        ...     "source": "DOI:10.1371/journal.pone.0023601"
+        ... }
+        >>> axioms = [ba_001, ba_002]
+        >>> chain = create_translation_chain(input_data, axioms, prov_mgr)
+    """
+    chain_id = str(uuid.uuid4())
+    chain = TranslationChain(chain_id=chain_id)
+    
+    # Add L1 (input layer)
+    chain.add_layer(
+        layer_name="L1",
+        layer_type="input",
+        value=input_data.get("value"),
+        source=input_data.get("source"),
+        entity_id=input_data.get("entity_id")
+    )
+    
+    # Apply bridge axioms sequentially
+    current_value = input_data.get("value")
+    current_entity = input_data.get("entity_id")
+    
+    for i, axiom in enumerate(bridge_axioms):
+        # Apply axiom
+        result = axiom.apply(
+            input_entity=current_entity,
+            input_value=current_value,
+            prov_manager=prov_manager
+        )
+        
+        # Add bridge axiom layer
+        chain.add_layer(
+            layer_name=f"L{i+2}_axiom",
+            layer_type="bridge_axiom",
+            value=axiom.coefficient,
+            source=axiom.source_doi,
+            axiom_id=axiom.axiom_id,
+            axiom_name=axiom.name,
+            rule=axiom.rule
+        )
+        
+        # Update for next iteration
+        current_value = result["output_value"]
+        current_entity = result["output_entity"]
+        
+        # Update chain confidence (minimum of all confidences)
+        chain.confidence = min(chain.confidence, axiom.confidence)
+    
+    # Add final output layer
+    chain.add_layer(
+        layer_name=f"L{len(bridge_axioms)+2}",
+        layer_type="output",
+        value=current_value,
+        entity_id=current_entity
+    )
+    
+    return chain
+
+
+def trace_translation_chain(
+    chain: TranslationChain,
+    prov_manager: Any
+) -> Dict[str, Any]:
+    """
+    Trace complete provenance for a translation chain.
+    
+    Args:
+        chain: TranslationChain object
+        prov_manager: ProvenanceManager instance
+        
+    Returns:
+        Dictionary with complete provenance trace
+        
+    Example:
+        >>> trace = trace_translation_chain(chain, prov_mgr)
+        >>> print(trace["layers"])
+    """
+    trace = {
+        "chain_id": chain.chain_id,
+        "layers": [],
+        "confidence": chain.confidence,
+        "provenance": []
+    }
+    
+    for layer in chain.layers:
+        layer_trace = {
+            "layer": layer.get("layer"),
+            "type": layer.get("type"),
+            "value": layer.get("value"),
+            "source": layer.get("source")
+        }
+        
+        # Get provenance for entities in this layer
+        entity_id = layer.get("entity_id")
+        if entity_id:
+            try:
+                lineage = prov_manager.get_lineage(entity_id)
+                layer_trace["provenance"] = lineage
+            except Exception:
+                pass
+        
+        trace["layers"].append(layer_trace)
+    
+    return trace

semantica/provenance/integrity.py

@@ -0,0 +1,178 @@
+"""
+Integrity Verification Utilities
+
+This module provides utilities for data integrity verification using
+SHA-256 checksums, ensuring provenance data has not been tampered with.
+
+Features:
+    - SHA-256 checksum computation
+    - Checksum verification
+    - Data integrity validation
+    - Tamper detection
+
+Compliance:
+    - FDA 21 CFR Part 11 (electronic records)
+    - SOX (Sarbanes-Oxley)
+    - HIPAA (healthcare data integrity)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import hashlib
+from typing import Any, Dict, Optional
+from .schemas import ProvenanceEntry
+
+
+def compute_checksum(entry: ProvenanceEntry) -> str:
+    """
+    Compute SHA-256 checksum for a provenance entry.
+    
+    Creates a deterministic checksum based on critical provenance fields
+    to detect any tampering or corruption of provenance data.
+    
+    Args:
+        entry: ProvenanceEntry to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+        
+    Example:
+        >>> entry = ProvenanceEntry(
+        ...     entity_id="entity_123",
+        ...     entity_type="entity",
+        ...     activity_id="extraction",
+        ...     source_document="DOI:10.1371/..."
+        ... )
+        >>> checksum = compute_checksum(entry)
+        >>> print(checksum)
+        'a3b2c1d4e5f6...'
+    """
+    # Concatenate critical fields for checksum
+    data = (
+        f"{entry.entity_id}"
+        f"{entry.entity_type}"
+        f"{entry.activity_id}"
+        f"{entry.source_document}"
+        f"{entry.timestamp}"
+        f"{entry.confidence}"
+    )
+    
+    return hashlib.sha256(data.encode('utf-8')).hexdigest()
+
+
+def verify_checksum(entry: ProvenanceEntry, expected_checksum: Optional[str] = None) -> bool:
+    """
+    Verify checksum for a provenance entry.
+    
+    Computes the current checksum and compares it with the expected checksum
+    to detect tampering or corruption.
+    
+    Args:
+        entry: ProvenanceEntry to verify
+        expected_checksum: Expected checksum (uses entry.checksum if None)
+        
+    Returns:
+        True if checksum matches, False otherwise
+        
+    Example:
+        >>> entry = ProvenanceEntry(...)
+        >>> entry.checksum = compute_checksum(entry)
+        >>> is_valid = verify_checksum(entry)
+        >>> print(is_valid)
+        True
+    """
+    if expected_checksum is None:
+        expected_checksum = entry.checksum
+    
+    if expected_checksum is None:
+        return False
+    
+    current_checksum = compute_checksum(entry)
+    return current_checksum == expected_checksum
+
+
+def compute_data_checksum(data: str) -> str:
+    """
+    Compute SHA-256 checksum for arbitrary data.
+    
+    Args:
+        data: String data to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+        
+    Example:
+        >>> checksum = compute_data_checksum("some data")
+        >>> print(checksum)
+        'a3b2c1d4e5f6...'
+    """
+    return hashlib.sha256(data.encode('utf-8')).hexdigest()
+
+
+def verify_data_checksum(data: str, expected_checksum: str) -> bool:
+    """
+    Verify checksum for arbitrary data.
+    
+    Args:
+        data: String data to verify
+        expected_checksum: Expected checksum
+        
+    Returns:
+        True if checksum matches, False otherwise
+        
+    Example:
+        >>> data = "some data"
+        >>> checksum = compute_data_checksum(data)
+        >>> is_valid = verify_data_checksum(data, checksum)
+        >>> print(is_valid)
+        True
+    """
+    current_checksum = compute_data_checksum(data)
+    return current_checksum == expected_checksum
+
+
+def compute_dict_checksum(data: Dict[str, Any]) -> str:
+    """
+    Compute SHA-256 checksum for dictionary data.
+    
+    Sorts keys to ensure deterministic checksum computation.
+    
+    Args:
+        data: Dictionary data to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+        
+    Example:
+        >>> data = {"key1": "value1", "key2": "value2"}
+        >>> checksum = compute_dict_checksum(data)
+        >>> print(checksum)
+        'a3b2c1d4e5f6...'
+    """
+    # Sort keys for deterministic checksum
+    sorted_items = sorted(data.items())
+    data_str = str(sorted_items)
+    return hashlib.sha256(data_str.encode('utf-8')).hexdigest()
+
+
+def verify_dict_checksum(data: Dict[str, Any], expected_checksum: str) -> bool:
+    """
+    Verify checksum for dictionary data.
+    
+    Args:
+        data: Dictionary data to verify
+        expected_checksum: Expected checksum
+        
+    Returns:
+        True if checksum matches, False otherwise
+        
+    Example:
+        >>> data = {"key1": "value1", "key2": "value2"}
+        >>> checksum = compute_dict_checksum(data)
+        >>> is_valid = verify_dict_checksum(data, checksum)
+        >>> print(is_valid)
+        True
+    """
+    current_checksum = compute_dict_checksum(data)
+    return current_checksum == expected_checksum

semantica/provenance/manager.py

@@ -0,0 +1,574 @@
+"""
+Unified Provenance Manager
+
+This module provides the central ProvenanceManager class that consolidates
+provenance tracking from multiple Semantica modules:
+    - kg.ProvenanceTracker (entity/relationship tracking)
+    - split.ProvenanceTracker (chunk tracking)
+    - conflicts.SourceTracker (source tracking)
+
+The ProvenanceManager provides a unified API for all provenance operations
+while maintaining backward compatibility with existing tracker interfaces.
+
+Features:
+    - W3C PROV-O compliant tracking
+    - Entity and relationship tracking
+    - Chunk provenance tracking
+    - Source and property tracking
+    - Complete lineage tracing
+    - Multiple storage backends
+    - Integrity verification
+    - Batch operations
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, List, Dict, Any
+from datetime import datetime
+
+from .schemas import ProvenanceEntry, SourceReference, PropertySource
+from .storage import ProvenanceStorage, InMemoryStorage, SQLiteStorage
+from .integrity import compute_checksum
+
+
+class ProvenanceManager:
+    """
+    Unified provenance tracking manager.
+    
+    Consolidates and enhances provenance tracking from:
+    - kg.ProvenanceTracker: Entity/relationship tracking with temporal info
+    - split.ProvenanceTracker: Chunk tracking with parent-child relationships
+    - conflicts.SourceTracker: Source tracking with credibility scores
+    
+    Example:
+        >>> # Basic usage
+        >>> prov_mgr = ProvenanceManager()
+        >>> prov_mgr.track_entity("entity_1", source="doc_1")
+        >>> 
+        >>> # With persistent storage
+        >>> prov_mgr = ProvenanceManager(storage_path="provenance.db")
+        >>> 
+        >>> # Trace lineage
+        >>> lineage = prov_mgr.get_lineage("entity_1")
+    """
+    
+    def __init__(
+        self,
+        storage: Optional[ProvenanceStorage] = None,
+        storage_path: Optional[str] = None
+    ):
+        """
+        Initialize provenance manager.
+        
+        Args:
+            storage: Custom storage backend (optional)
+            storage_path: Path to SQLite database (optional, uses in-memory if None)
+        """
+        if storage:
+            self.storage = storage
+        elif storage_path:
+            self.storage = SQLiteStorage(storage_path)
+        else:
+            self.storage = InMemoryStorage()
+    
+    # === Entity Tracking (from kg.ProvenanceTracker) ===
+    
+    def track_entity(
+        self,
+        entity_id: str,
+        source: str,
+        metadata: Optional[Dict[str, Any]] = None,
+        **kwargs
+    ) -> ProvenanceEntry:
+        """
+        Track entity provenance (kg.ProvenanceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            source: Source identifier (document ID, DOI, file path)
+            metadata: Optional metadata dictionary
+            **kwargs: Additional fields (confidence, source_location, etc.)
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> prov_mgr.track_entity(
+            ...     entity_id="entity_1",
+            ...     source="DOI:10.1371/journal.pone.0023601",
+            ...     metadata={"confidence": 0.92}
+            ... )
+        """
+        # Validate entity_id
+        if entity_id is None:
+            raise ValueError("entity_id cannot be None")
+        if not isinstance(entity_id, str):
+            raise TypeError(f"entity_id must be a string, got {type(entity_id).__name__}")
+        
+        if not isinstance(entity_id, str):
+            raise TypeError(f"entity_id must be a string, got {type(entity_id).__name__}")
+        
+        if not isinstance(entity_id, str):
+            raise TypeError(f"entity_id must be a string, got {type(entity_id).__name__}")
+        
+        # Check if entity already exists
+        existing = self.storage.retrieve(entity_id)
+        parent_id = kwargs.get("parent_entity_id")
+        
+        # If source is a known entity, link it as parent (unless parent already set)
+        if not parent_id and source and isinstance(source, str):
+            try:
+                # Check if source exists in storage
+                # trace_lineage is cheaper than retrieve for just checking existence? Or retrieve?
+                # retrieve returns the *latest* entry for that ID
+                source_entity = self.storage.retrieve(source)
+                if source_entity:
+                    parent_id = source
+            except Exception:
+                pass
+
+        # If entity exists, preserve history by archiving the old state
+        if existing:
+            # Create a history entry for the previous state
+            # Use timestamp or counter for uniqueness
+            import copy
+            history_entry = copy.deepcopy(existing)
+            history_id = f"{entity_id}:v:{existing.last_updated}"
+            
+            # Ensure unique ID if update happens same second
+            if self.storage.retrieve(history_id):
+                 history_id = f"{history_id}:{datetime.utcnow().microsecond}"
+            
+            history_entry.entity_id = history_id
+            
+            # Store the history entry
+            try:
+                self.storage.store(history_entry)
+                # Link new entry to this history entry
+                parent_id = history_id
+            except Exception:
+                pass # If history archiving fails, proceed with update but lose history (graceful degradation)
+
+        entry = ProvenanceEntry(
+            entity_id=entity_id,
+            entity_type=kwargs.get("entity_type", "entity"),
+            activity_id=kwargs.get("activity_id", "entity_tracking"),
+            source_document=source,
+            source_location=kwargs.get("source_location"),
+            source_quote=kwargs.get("source_quote"),
+            confidence=kwargs.get("confidence", 1.0),
+            metadata=metadata or {},
+            first_seen=existing.first_seen if existing else datetime.utcnow().isoformat(),
+            last_updated=datetime.utcnow().isoformat(),
+            parent_entity_id=parent_id  # Link to history or explicit parent
+        )
+        
+        # Compute checksum for integrity
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass  # Graceful failure - don't break main functionality
+        
+        return entry
+    
+    def track_relationship(
+        self,
+        relationship_id: str,
+        source: str,
+        metadata: Optional[Dict[str, Any]] = None,
+        **kwargs
+    ) -> ProvenanceEntry:
+        """
+        Track relationship provenance (kg.ProvenanceTracker compatible).
+        
+        Args:
+            relationship_id: Relationship identifier
+            source: Source identifier
+            metadata: Optional metadata dictionary
+            **kwargs: Additional fields
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> prov_mgr.track_relationship(
+            ...     relationship_id="rel_1",
+            ...     source="doc_1",
+            ...     metadata={"type": "founded"}
+            ... )
+        """
+        entry = ProvenanceEntry(
+            entity_id=relationship_id,
+            entity_type="relationship",
+            activity_id=kwargs.get("activity_id", "relationship_tracking"),
+            source_document=source,
+            source_location=kwargs.get("source_location"),
+            confidence=kwargs.get("confidence", 1.0),
+            metadata=metadata or {},
+            first_seen=datetime.utcnow().isoformat(),
+            last_updated=datetime.utcnow().isoformat()
+        )
+        
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass
+        
+        return entry
+    
+    # === Chunk Tracking (from split.ProvenanceTracker) ===
+    
+    def track_chunk(
+        self,
+        chunk_id: str,
+        source_document: str,
+        source_path: Optional[str] = None,
+        start_index: int = 0,
+        end_index: int = 0,
+        parent_chunk_id: Optional[str] = None,
+        **metadata
+    ) -> ProvenanceEntry:
+        """
+        Track chunk provenance (split.ProvenanceTracker compatible).
+        
+        Args:
+            chunk_id: Chunk identifier
+            source_document: Source document identifier
+            source_path: Path to source document
+            start_index: Start character index
+            end_index: End character index
+            parent_chunk_id: Parent chunk ID (if chunk was split)
+            **metadata: Additional metadata
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> prov_mgr.track_chunk(
+            ...     chunk_id="chunk_1",
+            ...     source_document="doc_1",
+            ...     source_path="/path/to/doc.pdf",
+            ...     start_index=0,
+            ...     end_index=500
+            ... )
+        """
+        entry = ProvenanceEntry(
+            entity_id=chunk_id,
+            entity_type="chunk",
+            activity_id="chunking",
+            source_document=source_document,
+            source_location=source_path,
+            start_index=start_index,
+            end_index=end_index,
+            parent_entity_id=parent_chunk_id,
+            metadata=metadata,
+            timestamp=datetime.utcnow().isoformat()
+        )
+        
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass
+        
+        return entry
+    
+    # === Source Tracking (from conflicts.SourceTracker) ===
+    
+    def track_property_source(
+        self,
+        entity_id: str,
+        property_name: str,
+        value: Any,
+        source: SourceReference,
+        **metadata
+    ) -> ProvenanceEntry:
+        """
+        Track property source (conflicts.SourceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            property_name: Property name
+            value: Property value
+            source: SourceReference object
+            **metadata: Additional metadata
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> source = SourceReference(
+            ...     document="DOI:10.1038/...",
+            ...     page=4,
+            ...     confidence=0.92
+            ... )
+            >>> prov_mgr.track_property_source(
+            ...     entity_id="entity_1",
+            ...     property_name="biomass_increase",
+            ...     value="463%",
+            ...     source=source
+            ... )
+        """
+        entry = ProvenanceEntry(
+            entity_id=f"{entity_id}_{property_name}",
+            entity_type="property",
+            activity_id="property_tracking",
+            source_document=source.document,
+            source_location=f"page_{source.page}" if source.page else source.section,
+            confidence=source.confidence,
+            credibility=source.metadata.get("credibility"),
+            metadata={
+                "entity_id": entity_id,
+                "property_name": property_name,
+                "value": value,
+                **metadata,
+                **source.metadata
+            },
+            timestamp=datetime.utcnow().isoformat()
+        )
+        
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass
+        
+        return entry
+    
+    # === Batch Operations ===
+    
+    def track_entities_batch(
+        self,
+        entities: List[Dict[str, Any]],
+        source: str,
+        **metadata
+    ) -> int:
+        """
+        Track multiple entities in batch.
+        
+        Args:
+            entities: List of entity dictionaries with 'id' key
+            source: Source identifier
+            **metadata: Metadata to apply to all entities
+            
+        Returns:
+            Number of entities tracked
+            
+        Example:
+            >>> entities = [
+            ...     {"id": "entity_1", "confidence": 0.9},
+            ...     {"id": "entity_2", "confidence": 0.85}
+            ... ]
+            >>> count = prov_mgr.track_entities_batch(entities, "doc_1")
+        """
+        tracked_count = 0
+        
+        for entity in entities:
+            entity_id = entity.get("id") or entity.get("entity_id")
+            if not entity_id:
+                continue
+            
+            entity_metadata = {**metadata, **entity.get("metadata", {})}
+            
+            try:
+                self.track_entity(entity_id, source, entity_metadata)
+                tracked_count += 1
+            except Exception:
+                pass  # Continue with other entities
+        
+        return tracked_count
+    
+    def track_chunks_batch(
+        self,
+        chunks: List[Dict[str, Any]],
+        source_document: str,
+        source_path: Optional[str] = None,
+        **metadata
+    ) -> int:
+        """
+        Track multiple chunks in batch.
+        
+        Args:
+            chunks: List of chunk dictionaries
+            source_document: Source document identifier
+            source_path: Path to source document
+            **metadata: Metadata to apply to all chunks
+            
+        Returns:
+            Number of chunks tracked
+        """
+        tracked_count = 0
+        
+        for chunk in chunks:
+            chunk_id = chunk.get("id") or chunk.get("chunk_id")
+            if not chunk_id:
+                continue
+            
+            try:
+                self.track_chunk(
+                    chunk_id=chunk_id,
+                    source_document=source_document,
+                    source_path=source_path,
+                    start_index=chunk.get("start_index", 0),
+                    end_index=chunk.get("end_index", 0),
+                    parent_chunk_id=chunk.get("parent_chunk_id"),
+                    **{**metadata, **chunk.get("metadata", {})}
+                )
+                tracked_count += 1
+            except Exception:
+                pass
+        
+        return tracked_count
+    
+    # === Lineage Retrieval ===
+    
+    def get_lineage(self, entity_id: str) -> Dict[str, Any]:
+        """
+        Get complete lineage for an entity.
+        
+        Compatible with all existing tracker interfaces.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            Dictionary containing lineage information including metadata
+            
+        Example:
+            >>> lineage = prov_mgr.get_lineage("entity_1")
+            >>> print(lineage["source_documents"])
+            ['DOI:10.1371/...', 'doc_2']
+            >>> print(lineage["metadata"])
+            {'text': 'Apple Inc.', 'label': 'ORG'}
+        """
+        lineage_entries = self.storage.trace_lineage(entity_id)
+        
+        if not lineage_entries:
+            return {}
+        
+        # Aggregate metadata from all lineage entries
+        # Most recent entry's metadata takes precedence
+        aggregated_metadata = {}
+        for entry in lineage_entries:
+            if entry.metadata:
+                meta = entry.metadata
+                if isinstance(meta, str):
+                    try:
+                        import json
+                        meta = json.loads(meta)
+                    except (json.JSONDecodeError, TypeError):
+                        pass
+                
+                if isinstance(meta, dict):
+                    aggregated_metadata.update(meta)
+        
+        return {
+            "entity_id": entity_id,
+            "lineage_chain": [entry.to_dict() for entry in lineage_entries],
+            "source_documents": list(set(
+                e.source_document for e in lineage_entries 
+                if e.source_document
+            )),
+            "first_seen": min(
+                (e.first_seen for e in lineage_entries if e.first_seen),
+                default=None
+            ),
+            "last_updated": max(
+                (e.last_updated for e in lineage_entries if e.last_updated),
+                default=None
+            ),
+            "entity_count": len(lineage_entries),
+            "metadata": aggregated_metadata  # Add metadata key
+        }
+    
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage and return raw entries.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        return self.storage.trace_lineage(entity_id)
+    
+    def get_all_sources(self, entity_id: str) -> List[Dict[str, Any]]:
+        """
+        Get all sources for an entity (kg.ProvenanceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of source dictionaries
+        """
+        lineage_entries = self.storage.trace_lineage(entity_id)
+        
+        sources = []
+        for entry in lineage_entries:
+            if entry.source_document:
+                sources.append({
+                    "source": entry.source_document,
+                    "location": entry.source_location,
+                    "timestamp": entry.timestamp,
+                    "confidence": entry.confidence,
+                    "metadata": entry.metadata
+                })
+        
+        return sources
+    
+    def get_provenance(self, entity_id: str) -> Optional[Dict[str, Any]]:
+        """
+        Get provenance for entity (kg.ProvenanceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            Provenance dictionary or None
+        """
+        entry = self.storage.retrieve(entity_id)
+        if entry:
+            return entry.to_dict()
+        return None
+    
+    # === Utility Methods ===
+    
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        return self.storage.clear()
+    
+    def get_statistics(self) -> Dict[str, Any]:
+        """
+        Get provenance statistics.
+        
+        Returns:
+            Dictionary with statistics
+        """
+        all_entries = self.storage.retrieve_all()
+        
+        entity_types = {}
+        for entry in all_entries:
+            entity_types[entry.entity_type] = entity_types.get(entry.entity_type, 0) + 1
+        
+        return {
+            "total_entries": len(all_entries),
+            "entity_types": entity_types,
+            "unique_sources": len(set(
+                e.source_document for e in all_entries 
+                if e.source_document
+            ))
+        }

semantica/provenance/schemas.py

@@ -0,0 +1,277 @@
+"""
+W3C PROV-O Compliant Provenance Schemas
+
+This module provides dataclasses for provenance tracking that comply with
+W3C PROV-O (Provenance Ontology) standards while consolidating functionality
+from existing Semantica provenance trackers.
+
+Consolidates:
+    - kg.ProvenanceTracker: Entity/relationship tracking with temporal info
+    - split.ProvenanceInfo: Chunk tracking with parent-child relationships
+    - conflicts.SourceReference: Source tracking with credibility scores
+
+W3C PROV-O Mapping:
+    - ProvenanceEntry.entity_id → prov:Entity
+    - ProvenanceEntry.activity_id → prov:Activity
+    - ProvenanceEntry.agent_id → prov:Agent
+    - ProvenanceEntry.parent_entity_id → prov:wasDerivedFrom
+    - ProvenanceEntry.used_entities → prov:used
+    - ProvenanceEntry.timestamp → prov:generatedAtTime
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from dataclasses import dataclass, field
+from typing import Optional, List, Dict, Any
+from datetime import datetime
+
+
+@dataclass
+class ProvenanceEntry:
+    """
+    W3C PROV-O compliant provenance entry.
+    
+    This unified schema consolidates provenance tracking from:
+    - kg.ProvenanceTracker (entity/relationship tracking)
+    - split.ProvenanceInfo (chunk tracking)
+    - conflicts.SourceReference (source tracking)
+    
+    Attributes:
+        entity_id: Unique identifier for the entity (prov:Entity)
+        entity_type: Type of entity (entity, chunk, relationship, property, etc.)
+        activity_id: Activity that generated this entity (prov:Activity)
+        agent_id: Agent responsible for the activity (prov:Agent)
+        source_document: Source document identifier (DOI, file path, URL)
+        source_location: Location within source (page, figure, char range)
+        source_quote: Direct quote from source (for audit trail)
+        timestamp: When this provenance entry was created (prov:generatedAtTime)
+        first_seen: When entity was first tracked (from kg.ProvenanceTracker)
+        last_updated: When entity was last updated (from kg.ProvenanceTracker)
+        confidence: Confidence score for this provenance entry (0.0-1.0)
+        checksum: SHA-256 checksum for integrity verification
+        parent_entity_id: Parent entity ID (prov:wasDerivedFrom)
+        used_entities: List of entities used to create this entity (prov:used)
+        start_index: Start character index (from split.ProvenanceInfo)
+        end_index: End character index (from split.ProvenanceInfo)
+        credibility: Source credibility score (from conflicts.SourceTracker)
+        metadata: Additional metadata dictionary
+        version: Provenance schema version
+    
+    Example:
+        >>> entry = ProvenanceEntry(
+        ...     entity_id="entity_123",
+        ...     entity_type="named_entity",
+        ...     activity_id="ner_extraction",
+        ...     source_document="DOI:10.1371/journal.pone.0023601",
+        ...     source_location="Figure 2",
+        ...     source_quote="Total fish biomass increased by 463%",
+        ...     confidence=0.92
+        ... )
+    """
+    
+    # W3C PROV-O core entities
+    entity_id: str
+    entity_type: str
+    activity_id: str
+    agent_id: str = "semantica"
+    
+    # Audit-grade source tracking
+    source_document: str = ""
+    source_location: Optional[str] = None
+    source_quote: Optional[str] = None
+    
+    # Temporal tracking (from kg.ProvenanceTracker)
+    timestamp: str = field(default_factory=lambda: datetime.utcnow().isoformat())
+    first_seen: Optional[str] = None
+    last_updated: Optional[str] = None
+    
+    # Quality metrics
+    confidence: float = 1.0
+    checksum: Optional[str] = None
+    
+    # Chain of custody (W3C PROV-O)
+    parent_entity_id: Optional[str] = None
+    used_entities: List[str] = field(default_factory=list)
+    
+    # Chunk-specific fields (from split.ProvenanceInfo)
+    start_index: Optional[int] = None
+    end_index: Optional[int] = None
+    
+    # Source credibility (from conflicts.SourceTracker)
+    credibility: Optional[float] = None
+    
+    # Metadata
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    version: str = "1.0"
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Convert provenance entry to dictionary.
+        
+        Returns:
+            Dictionary representation of provenance entry
+        """
+        return {
+            "entity_id": self.entity_id,
+            "entity_type": self.entity_type,
+            "activity_id": self.activity_id,
+            "agent_id": self.agent_id,
+            "source_document": self.source_document,
+            "source_location": self.source_location,
+            "source_quote": self.source_quote,
+            "timestamp": self.timestamp,
+            "first_seen": self.first_seen,
+            "last_updated": self.last_updated,
+            "confidence": self.confidence,
+            "checksum": self.checksum,
+            "parent_entity_id": self.parent_entity_id,
+            "used_entities": self.used_entities,
+            "start_index": self.start_index,
+            "end_index": self.end_index,
+            "credibility": self.credibility,
+            "metadata": self.metadata,
+            "version": self.version,
+        }
+    
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ProvenanceEntry":
+        """
+        Create provenance entry from dictionary.
+        
+        Args:
+            data: Dictionary containing provenance data
+            
+        Returns:
+            ProvenanceEntry instance
+        """
+        return cls(**data)
+
+
+@dataclass
+class SourceReference:
+    """
+    Source reference for provenance tracking.
+    
+    Compatible with conflicts.SourceReference for backward compatibility.
+    
+    Attributes:
+        document: Document identifier (DOI, file path, URL)
+        page: Page number within document
+        section: Section identifier within document
+        line: Line number within document
+        timestamp: When this source was accessed/created
+        confidence: Confidence score for this source (0.0-1.0)
+        metadata: Additional metadata dictionary
+    
+    Example:
+        >>> source = SourceReference(
+        ...     document="DOI:10.1038/s41586-021-03371-z",
+        ...     page=4,
+        ...     section="Table S4",
+        ...     confidence=0.92
+        ... )
+    """
+    
+    document: str
+    page: Optional[int] = None
+    section: Optional[str] = None
+    line: Optional[int] = None
+    timestamp: Optional[datetime] = None
+    confidence: float = 1.0
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Convert source reference to dictionary.
+        
+        Returns:
+            Dictionary representation of source reference
+        """
+        return {
+            "document": self.document,
+            "page": self.page,
+            "section": self.section,
+            "line": self.line,
+            "timestamp": self.timestamp.isoformat() if self.timestamp else None,
+            "confidence": self.confidence,
+            "metadata": self.metadata,
+        }
+    
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "SourceReference":
+        """
+        Create source reference from dictionary.
+        
+        Args:
+            data: Dictionary containing source reference data
+            
+        Returns:
+            SourceReference instance
+        """
+        if "timestamp" in data and isinstance(data["timestamp"], str):
+            data["timestamp"] = datetime.fromisoformat(data["timestamp"])
+        return cls(**data)
+
+
+@dataclass
+class PropertySource:
+    """
+    Property source information for conflict tracking.
+    
+    Compatible with conflicts.PropertySource for backward compatibility.
+    
+    Attributes:
+        property_name: Name of the property
+        value: Value of the property
+        sources: List of source references
+        entity_id: Entity this property belongs to
+        metadata: Additional metadata dictionary
+    
+    Example:
+        >>> prop_source = PropertySource(
+        ...     property_name="biomass_increase",
+        ...     value="463%",
+        ...     sources=[source_ref],
+        ...     entity_id="cabo_pulmo_mpa"
+        ... )
+    """
+    
+    property_name: str
+    value: Any
+    sources: List[SourceReference] = field(default_factory=list)
+    entity_id: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Convert property source to dictionary.
+        
+        Returns:
+            Dictionary representation of property source
+        """
+        return {
+            "property_name": self.property_name,
+            "value": self.value,
+            "sources": [s.to_dict() for s in self.sources],
+            "entity_id": self.entity_id,
+            "metadata": self.metadata,
+        }
+    
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "PropertySource":
+        """
+        Create property source from dictionary.
+        
+        Args:
+            data: Dictionary containing property source data
+            
+        Returns:
+            PropertySource instance
+        """
+        if "sources" in data:
+            data["sources"] = [
+                SourceReference.from_dict(s) if isinstance(s, dict) else s
+                for s in data["sources"]
+            ]
+        return cls(**data)

semantica/provenance/storage.py

@@ -0,0 +1,489 @@
+"""
+Provenance Storage Backends
+
+This module provides storage backends for provenance tracking, including
+in-memory and persistent SQLite storage with W3C PROV-O compliance.
+
+Storage Backends:
+    - InMemoryStorage: Fast in-memory storage for development/testing
+    - SQLiteStorage: Persistent SQLite storage for production use
+
+Features:
+    - W3C PROV-O compliant schema
+    - Lineage tracing with BFS traversal
+    - Efficient entity retrieval
+    - Type-based filtering
+    - Integrity verification support
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from abc import ABC, abstractmethod
+from typing import List, Optional, Dict, Any
+import sqlite3
+import json
+from collections import deque
+
+from .schemas import ProvenanceEntry
+
+
+class ProvenanceStorage(ABC):
+    """
+    Abstract storage interface for provenance tracking.
+    
+    All storage backends must implement these methods to ensure
+    consistent provenance tracking across different storage types.
+    """
+    
+    @abstractmethod
+    def store(self, entry: ProvenanceEntry) -> None:
+        """
+        Store a provenance entry.
+        
+        Args:
+            entry: ProvenanceEntry to store
+        """
+        pass
+    
+    @abstractmethod
+    def retrieve(self, entity_id: str) -> Optional[ProvenanceEntry]:
+        """
+        Retrieve a provenance entry by entity ID.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            ProvenanceEntry if found, None otherwise
+        """
+        pass
+    
+    @abstractmethod
+    def retrieve_all(self, entity_type: Optional[str] = None) -> List[ProvenanceEntry]:
+        """
+        Retrieve all provenance entries, optionally filtered by type.
+        
+        Args:
+            entity_type: Optional entity type filter
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        pass
+    
+    @abstractmethod
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage for an entity.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects in lineage chain
+        """
+        pass
+    
+    @abstractmethod
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        pass
+
+
+class InMemoryStorage(ProvenanceStorage):
+    """
+    Fast in-memory storage for provenance tracking.
+    
+    Suitable for:
+        - Development and testing
+        - Short-lived processes
+        - Small to medium datasets
+        - When persistence is not required
+    
+    Features:
+        - O(1) entity retrieval
+        - BFS lineage tracing
+        - Type-based filtering
+        - No external dependencies
+    
+    Example:
+        >>> storage = InMemoryStorage()
+        >>> storage.store(entry)
+        >>> lineage = storage.trace_lineage("entity_123")
+    """
+    
+    def __init__(self):
+        """Initialize in-memory storage."""
+        self._entries: Dict[str, ProvenanceEntry] = {}
+    
+    def store(self, entry: ProvenanceEntry) -> None:
+        """
+        Store a provenance entry in memory.
+        
+        Args:
+            entry: ProvenanceEntry to store
+        """
+        self._entries[entry.entity_id] = entry
+    
+    def retrieve(self, entity_id: str) -> Optional[ProvenanceEntry]:
+        """
+        Retrieve a provenance entry by entity ID.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            ProvenanceEntry if found, None otherwise
+        """
+        return self._entries.get(entity_id)
+    
+    def retrieve_all(self, entity_type: Optional[str] = None) -> List[ProvenanceEntry]:
+        """
+        Retrieve all provenance entries, optionally filtered by type.
+        
+        Args:
+            entity_type: Optional entity type filter
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        if entity_type:
+            return [
+                entry for entry in self._entries.values()
+                if entry.entity_type == entity_type
+            ]
+        return list(self._entries.values())
+    
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage using BFS traversal.
+        
+        Traces both parent entities (wasDerivedFrom) and used entities
+        to build complete provenance chain.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects in lineage chain
+        """
+        lineage = []
+        visited = set()
+        queue = deque([entity_id])
+        
+        while queue:
+            current_id = queue.popleft()
+            
+            if current_id in visited:
+                continue
+            
+            visited.add(current_id)
+            entry = self.retrieve(current_id)
+            
+            if entry:
+                lineage.append(entry)
+                
+                # Add parent entity to queue
+                if entry.parent_entity_id:
+                    queue.append(entry.parent_entity_id)
+                
+                # Add used entities to queue
+                for used_id in entry.used_entities:
+                    if used_id not in visited:
+                        queue.append(used_id)
+        
+        return lineage
+    
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        count = len(self._entries)
+        self._entries.clear()
+        return count
+
+
+class SQLiteStorage(ProvenanceStorage):
+    """
+    Persistent SQLite storage for provenance tracking.
+    
+    Suitable for:
+        - Production use
+        - Long-term provenance tracking
+        - Large datasets
+        - Audit trail requirements
+        - Regulatory compliance
+    
+    Features:
+        - W3C PROV-O compliant schema
+        - Persistent storage
+        - Efficient indexing
+        - Transaction support
+        - Integrity verification
+    
+    Example:
+        >>> storage = SQLiteStorage("provenance.db")
+        >>> storage.store(entry)
+        >>> lineage = storage.trace_lineage("entity_123")
+    """
+    
+    def __init__(self, db_path: str = "provenance.db"):
+        """
+        Initialize SQLite storage.
+        
+        Args:
+            db_path: Path to SQLite database file
+        """
+        self.db_path = db_path
+        self._init_db()
+    
+    def _init_db(self) -> None:
+        """Create tables with W3C PROV-O compliant schema."""
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        cursor.execute("""
+            CREATE TABLE IF NOT EXISTS provenance (
+                entity_id TEXT PRIMARY KEY,
+                entity_type TEXT NOT NULL,
+                activity_id TEXT NOT NULL,
+                agent_id TEXT DEFAULT 'semantica',
+                source_document TEXT,
+                source_location TEXT,
+                source_quote TEXT,
+                timestamp TEXT NOT NULL,
+                first_seen TEXT,
+                last_updated TEXT,
+                confidence REAL DEFAULT 1.0,
+                checksum TEXT,
+                parent_entity_id TEXT,
+                used_entities TEXT,
+                start_index INTEGER,
+                end_index INTEGER,
+                credibility REAL,
+                metadata TEXT,
+                version TEXT DEFAULT '1.0'
+            )
+        """)
+        
+        # Create indexes for efficient querying
+        cursor.execute("""
+            CREATE INDEX IF NOT EXISTS idx_entity_type 
+            ON provenance(entity_type)
+        """)
+        
+        cursor.execute("""
+            CREATE INDEX IF NOT EXISTS idx_source_document 
+            ON provenance(source_document)
+        """)
+        
+        cursor.execute("""
+            CREATE INDEX IF NOT EXISTS idx_parent_entity 
+            ON provenance(parent_entity_id)
+        """)
+        
+        conn.commit()
+        conn.close()
+    
+    def store(self, entry: ProvenanceEntry) -> None:
+        """
+        Store a provenance entry in SQLite database.
+        
+        Args:
+            entry: ProvenanceEntry to store
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            cursor.execute("""
+                INSERT OR REPLACE INTO provenance VALUES (
+                    ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
+                )
+            """, (
+                entry.entity_id,
+                entry.entity_type,
+                entry.activity_id,
+                entry.agent_id,
+                entry.source_document,
+                entry.source_location,
+                entry.source_quote,
+                entry.timestamp,
+                entry.first_seen,
+                entry.last_updated,
+                entry.confidence,
+                entry.checksum,
+                entry.parent_entity_id,
+                json.dumps(entry.used_entities),
+                entry.start_index,
+                entry.end_index,
+                entry.credibility,
+                json.dumps(entry.metadata),
+                entry.version
+            ))
+            
+            conn.commit()
+        finally:
+            conn.close()
+    
+    def retrieve(self, entity_id: str) -> Optional[ProvenanceEntry]:
+        """
+        Retrieve a provenance entry by entity ID.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            ProvenanceEntry if found, None otherwise
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            cursor.execute("""
+                SELECT * FROM provenance WHERE entity_id = ?
+            """, (entity_id,))
+            
+            row = cursor.fetchone()
+            if not row:
+                return None
+            
+            return self._row_to_entry(row)
+        finally:
+            conn.close()
+    
+    def retrieve_all(self, entity_type: Optional[str] = None) -> List[ProvenanceEntry]:
+        """
+        Retrieve all provenance entries, optionally filtered by type.
+        
+        Args:
+            entity_type: Optional entity type filter
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            if entity_type:
+                cursor.execute("""
+                    SELECT * FROM provenance WHERE entity_type = ?
+                """, (entity_type,))
+            else:
+                cursor.execute("SELECT * FROM provenance")
+            
+            rows = cursor.fetchall()
+            return [self._row_to_entry(row) for row in rows]
+        finally:
+            conn.close()
+    
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage using BFS traversal.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects in lineage chain
+        """
+        lineage = []
+        visited = set()
+        queue = deque([entity_id])
+        
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            while queue:
+                current_id = queue.popleft()
+                
+                if current_id in visited:
+                    continue
+                
+                visited.add(current_id)
+                
+                cursor.execute("""
+                    SELECT * FROM provenance WHERE entity_id = ?
+                """, (current_id,))
+                
+                row = cursor.fetchone()
+                if row:
+                    entry = self._row_to_entry(row)
+                    lineage.append(entry)
+                    
+                    # Add parent entity to queue
+                    if entry.parent_entity_id:
+                        queue.append(entry.parent_entity_id)
+                    
+                    # Add used entities to queue
+                    for used_id in entry.used_entities:
+                        if used_id not in visited:
+                            queue.append(used_id)
+            
+            return lineage
+        finally:
+            conn.close()
+    
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            cursor.execute("SELECT COUNT(*) FROM provenance")
+            count = cursor.fetchone()[0]
+            
+            cursor.execute("DELETE FROM provenance")
+            conn.commit()
+            
+            return count
+        finally:
+            conn.close()
+    
+    def _row_to_entry(self, row: tuple) -> ProvenanceEntry:
+        """
+        Convert database row to ProvenanceEntry.
+        
+        Args:
+            row: Database row tuple
+            
+        Returns:
+            ProvenanceEntry object
+        """
+        return ProvenanceEntry(
+            entity_id=row[0],
+            entity_type=row[1],
+            activity_id=row[2],
+            agent_id=row[3],
+            source_document=row[4] or "",
+            source_location=row[5],
+            source_quote=row[6],
+            timestamp=row[7],
+            first_seen=row[8],
+            last_updated=row[9],
+            confidence=row[10],
+            checksum=row[11],
+            parent_entity_id=row[12],
+            used_entities=json.loads(row[13]) if row[13] else [],
+            start_index=row[14],
+            end_index=row[15],
+            credibility=row[16],
+            metadata=json.loads(row[17]) if row[17] else {},
+            version=row[18]
+        )

semantica/reasoning/reasoning_provenance.py

@@ -0,0 +1,58 @@
+"""
+Provenance-enabled wrappers for reasoning operations.
+
+Tracks: premises, conclusions, inference rules, confidence scores
+
+Usage:
+    from semantica.reasoning.reasoning_provenance import ReasoningEngineWithProvenance
+    
+    reasoner = ReasoningEngineWithProvenance(provenance=True)
+    result = reasoner.infer(premises)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class ReasoningEngineWithProvenance:
+    """Reasoning engine with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .reasoning_engine import ReasoningEngine
+        
+        self.provenance = provenance
+        self._engine = ReasoningEngine(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def infer(self, premises: Any, source: str = None, **kwargs):
+        """Perform inference with provenance tracking."""
+        result = self._engine.infer(premises, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"inference_{uuid.uuid4().hex[:8]}",
+                source=source or "reasoning_engine",
+                entity_type="inference",
+                metadata={
+                    "premises_count": len(premises) if hasattr(premises, '__len__') else 1,
+                    "confidence": getattr(result, 'confidence', None)
+                }
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._engine, name)
+
+
+__all__ = ['ReasoningEngineWithProvenance']

semantica/semantic_extract/cache.py

@@ -0,0 +1,184 @@
+"""
+Result Caching Module
+
+This module provides caching mechanisms for extraction results to avoid redundant
+computations and API calls. It implements an LRU (Least Recently Used) cache
+with Time-To-Live (TTL) support.
+
+Key Features:
+    - LRU Caching: Evicts least recently used items when cache is full
+    - TTL Support: Expires items after a configurable duration
+    - Namespaced Caching: Separate caches for entities, relations, and triplets
+    - Hash-based Keys: Uses stable hashing for text and parameters
+
+Classes:
+    - ExtractionCache: Main cache manager
+    - CacheItem: Container for cached data with metadata
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import time
+import hashlib
+import json
+from collections import OrderedDict
+from typing import Any, Dict, Optional, Union, List
+from threading import Lock
+
+from ..utils.logging import get_logger
+
+class CacheItem:
+    """Container for cached data."""
+    def __init__(self, value: Any, ttl: Optional[int] = None):
+        self.value = value
+        self.timestamp = time.time()
+        self.ttl = ttl
+
+    def is_expired(self) -> bool:
+        """Check if item has expired."""
+        if self.ttl is None:
+            return False
+        return time.time() - self.timestamp > self.ttl
+
+class ExtractionCache:
+    """
+    LRU Cache for extraction results.
+    Thread-safe implementation.
+    """
+    def __init__(self, max_size: int = 1000, ttl: int = 3600):
+        """
+        Initialize the cache.
+        
+        Args:
+            max_size: Maximum number of items to store per namespace
+            ttl: Time to live in seconds (default 1 hour)
+        """
+        self.max_size = max_size
+        self.ttl = ttl
+        self._caches: Dict[str, OrderedDict] = {
+            "entities": OrderedDict(),
+            "relations": OrderedDict(),
+            "triplets": OrderedDict()
+        }
+        self._locks: Dict[str, Lock] = {
+            "entities": Lock(),
+            "relations": Lock(),
+            "triplets": Lock()
+        }
+        self.logger = get_logger("extraction_cache")
+        self.enabled = True
+
+    def _generate_key(self, text: str, **params) -> str:
+        """
+        Generate a stable cache key based on text and parameters.
+        
+        Note: Sensitive parameters like 'api_key' are excluded from the cache key
+        to prevent security risks and ensure cache sharing where appropriate.
+        """
+        # Filter out sensitive keys
+        sensitive_keys = {'api_key', 'token', 'password', 'secret', 'auth', 'authorization'}
+        filtered_params = {k: v for k, v in params.items() if k.lower() not in sensitive_keys}
+
+        # Create a stable string representation of params
+        # Sort keys to ensure consistent ordering
+        param_str = json.dumps(filtered_params, sort_keys=True, default=str)
+        
+        # Combine text and params
+        content = f"{text}|{param_str}"
+        
+        # Return hash (SHA-256 for better security than MD5)
+        return hashlib.sha256(content.encode('utf-8')).hexdigest()
+
+    def get(self, namespace: str, text: str, **params) -> Optional[Any]:
+        """
+        Retrieve item from cache.
+        
+        Args:
+            namespace: Cache namespace ("entities", "relations", "triplets")
+            text: Input text used for extraction
+            **params: Extraction parameters used
+            
+        Returns:
+            Cached result or None if not found/expired
+        """
+        if not self.enabled:
+            return None
+
+        if namespace not in self._caches:
+            return None
+
+        key = self._generate_key(text, **params)
+        
+        with self._locks[namespace]:
+            cache = self._caches[namespace]
+            if key in cache:
+                item = cache[key]
+                
+                # Check expiration
+                if item.is_expired():
+                    del cache[key]
+                    return None
+                
+                # Move to end (mark as recently used)
+                cache.move_to_end(key)
+                return item.value
+        
+        return None
+
+    def set(self, namespace: str, text: str, value: Any, **params) -> None:
+        """
+        Add item to cache.
+        
+        Args:
+            namespace: Cache namespace
+            text: Input text
+            value: Result to cache
+            **params: Extraction parameters
+        """
+        if not self.enabled:
+            return
+
+        if namespace not in self._caches:
+            self.logger.warning(f"Unknown cache namespace: {namespace}")
+            return
+
+        key = self._generate_key(text, **params)
+        item = CacheItem(value, self.ttl)
+        
+        with self._locks[namespace]:
+            cache = self._caches[namespace]
+            
+            # If key exists, update and move to end
+            if key in cache:
+                cache.move_to_end(key)
+            
+            cache[key] = item
+            
+            # Evict if full
+            if len(cache) > self.max_size:
+                cache.popitem(last=False)  # Remove first (least recently used)
+
+    def clear(self, namespace: Optional[str] = None):
+        """Clear cache(s)."""
+        if namespace:
+            if namespace in self._caches:
+                with self._locks[namespace]:
+                    self._caches[namespace].clear()
+        else:
+            for ns in self._caches:
+                with self._locks[ns]:
+                    self._caches[ns].clear()
+
+    def get_stats(self) -> Dict[str, Dict[str, int]]:
+        """Get cache statistics."""
+        stats = {}
+        for ns, cache in self._caches.items():
+            stats[ns] = {
+                "size": len(cache),
+                "max_size": self.max_size
+            }
+        return stats
+
+# Global cache instance
+extraction_cache = ExtractionCache()

semantica/semantic_extract/config.py

@@ -40,8 +40,9 @@ License: MIT
 """
 
 import os
+import multiprocessing
 from pathlib import Path
-from typing import Dict, Optional
+from typing import Dict, Optional, Any
 
 from ..utils.logging import get_logger
 
@@ -53,9 +54,23 @@ class Config:
         """Initialize configuration manager."""
         self.logger = get_logger("config")
         self._configs: Dict[str, Dict] = {}
+        # Default optimization settings
+        self._configs["optimization"] = {
+            "enable_cache": True,
+            "cache_size": 1000,
+            "max_workers": 8,
+            "enable_batching": True,
+            "batch_size": 10,
+            "max_tokens_per_batch": 2000
+        }
         self._load_config_file(config_file)
         self._load_env_vars()
 
+    def get_optimization_config(self) -> Dict:
+        """Get optimization configuration."""
+        return self._configs.get("optimization", {})
+
+
     def _load_config_file(self, config_file: Optional[str]):
         """Load configuration from file."""
         if config_file and Path(config_file).exists():
@@ -114,6 +129,66 @@ class Config:
             return self._configs[provider].get("api_key")
         return os.getenv(f"{provider.upper()}_API_KEY")
 
+    def get(self, key: str, default: Any = None) -> Any:
+        """
+        Get configuration value by key.
+        Searches in top-level configs and optimization settings.
+        """
+        # 1. Check top-level keys
+        if key in self._configs:
+            return self._configs[key]
+            
+        # 2. Check optimization settings (common keys)
+        if "optimization" in self._configs and key in self._configs["optimization"]:
+            return self._configs["optimization"][key]
+            
+        # 3. Handle specific mapping for optimization keys
+        # Map cache_enabled -> enable_cache if needed
+        if key == "cache_enabled":
+            return self._configs.get("optimization", {}).get("enable_cache", default)
+            
+        return default
+
 
 # Global config instance
 config = Config()
+
+
+def resolve_max_workers(
+    explicit: Optional[int] = None,
+    local_config: Optional[Dict[str, Any]] = None,
+    methods: Optional[Any] = None,
+) -> int:
+    def to_int(val: Any, default: int) -> int:
+        try:
+            return int(val)
+        except Exception:
+            return default
+
+    if isinstance(methods, str):
+        normalized_methods = [methods]
+    elif isinstance(methods, (list, tuple, set)):
+        normalized_methods = [m for m in methods if isinstance(m, str)]
+    else:
+        normalized_methods = []
+
+    if explicit is not None:
+        value = to_int(explicit, 1)
+    elif local_config and "max_workers" in local_config:
+        value = to_int(local_config.get("max_workers", 1), 1)
+    else:
+        value = to_int(config.get("max_workers", 5), 5)
+
+    if "ml" in normalized_methods and explicit is None and not (local_config and "max_workers" in local_config):
+        value = 1
+
+    if value < 1:
+        value = 1
+
+    cpu_count = multiprocessing.cpu_count() or 1
+    if value > cpu_count:
+        value = cpu_count
+    if value > 32:
+        value = 32
+
+    return value

semantica/semantic_extract/coreference_resolver.py

@@ -240,6 +240,12 @@ class CoreferenceResolver:
             self.progress_tracker.stop_tracking(
                 tracking_id, status="failed", message=str(e)
             )
+            verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+            if verbose_mode:
+                import sys
+                print(f"    [CoreferenceResolver] ERROR: Resolution failed: {e}", flush=True, file=sys.stderr)
+                import traceback
+                traceback.print_exc(file=sys.stderr)
             raise
 
     def resolve(

semantica/semantic_extract/event_detector.py

@@ -85,68 +85,59 @@ class Event:
 class EventDetector:
     """Event detection and extraction handler."""
 
-    def __init__(
-        self,
-        event_types: Optional[List[str]] = None,
-        extract_participants: bool = True,
-        extract_location: bool = True,
-        extract_time: bool = True,
-        method: Union[str, List[str]] = None,
-        config=None,
-        **kwargs
-    ):
+    def __init__(self, method: str = "llm", **config):
         """
         Initialize event detector.
 
         Args:
-            event_types: Specific event types to detect (e.g., ["launch", "acquisition"])
-            extract_participants: Whether to extract event participants
-            extract_location: Whether to extract event locations
-            extract_time: Whether to extract temporal information
-            method: Extraction method(s) for underlying NER/relation extractors.
-                   Can be passed to ner_method and relation_method in config.
-            config: Legacy config dict (deprecated, use kwargs)
-            **kwargs: Configuration options:
-                - ner_method: Method for NER extraction (if entities need to be extracted)
-                - relation_method: Method for relation extraction (if relations need to be extracted)
-                - Other options passed to sub-components
+            method: Extraction method ("llm", "pattern")
+            **config: Configuration options
         """
         self.logger = get_logger("event_detector")
-        self.config = config or {}
-        self.config.update(kwargs)
+        self.config = config
+        self.method = method
         self.progress_tracker = get_progress_tracker()
+        
         # Ensure progress tracker is enabled
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
-        # Store parameters
-        self.event_types_filter = event_types
-        self.extract_participants = extract_participants
-        self.extract_location = extract_location
-        self.extract_time = extract_time
+        # Initialize components
+        self.event_classifier = EventClassifier(**config)
+        self.temporal_processor = TemporalEventProcessor(**config)
+
+        # Configure extraction options
+        self.extract_participants = config.get("extract_participants", True)
+        self.extract_location = config.get("extract_location", True)
+        self.extract_time = config.get("extract_time", True)
+        self.event_types_filter = config.get("event_types", [])
+
+        # Define event patterns
+        self.event_patterns = {
+            "acquisition": r"\b(acquired|acquisition|buying|bought|merger|merged)\b",
+            "partnership": r"\b(partnered|partnership|collaborate|collaboration)\b",
+            "launch": r"\b(launch|launched|releasing|released|unveil|unveiled)\b",
+            "investment": r"\b(invest|invested|investment|funding|raised)\b",
+            "legal": r"\b(sue|sued|lawsuit|litigation|legal action)\b",
+        }
+        
+        # Pre-compile location patterns
+        self.location_patterns = [
+            re.compile(r"in\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)"),
+            re.compile(r"at\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)"),
+        ]
+        
+        # Pre-compile time patterns
+        self.time_patterns = [
+            re.compile(r"on\s+([A-Z][a-z]+\s+\d{1,2},?\s+\d{4})"),
+            re.compile(r"in\s+(\d{4})"),
+            re.compile(r"(\d{1,2}[/-]\d{1,2}[/-]\d{2,4})"),
+        ]
 
-        # Store method for passing to extractors if needed
         if method is not None:
             self.config["ner_method"] = method
             self.config["relation_method"] = method
 
-        self.event_classifier = EventClassifier(**self.config.get("classifier", {}))
-        self.temporal_processor = TemporalEventProcessor(
-            **self.config.get("temporal", {})
-        )
-        self.relationship_extractor = EventRelationshipExtractor(
-            **self.config.get("relationship", {})
-        )
-
-        # Event patterns
-        self.event_patterns = {
-            "founded": r"founded|created|established",
-            "acquired": r"acquired|bought|purchased",
-            "launched": r"launched|released|introduced",
-            "announced": r"announced|declared|stated",
-            "meeting": r"met|meeting|conference|summit",
-        }
-
     def extract(
         self,
         text: Union[str, List[str], List[Dict[str, Any]]],
@@ -175,9 +166,10 @@ class EventDetector:
             )
 
             try:
-                results = []
+                results = [None] * len(text)  # Pre-allocate to maintain order
                 total_items = len(text)
                 total_events_count = 0
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -193,33 +185,77 @@ class EventDetector:
                     message=f"Starting batch detection... 0/{total_items} (remaining: {total_items})"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=[self.config.get("ner_method"), self.config.get("relation_method"), self.config.get("method")],
+                )
+
+                def process_item(idx, item):
+                    try:
+                        # Prepare arguments for single item
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                        
+                        # Detect
+                        events = self.detect_events(doc_text, **kwargs)
+
+                        # Add provenance metadata
+                        for event in events:
+                            if event.metadata is None:
+                                event.metadata = {}
+                            event.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                event.metadata["document_id"] = item["id"]
+                        
+                        return idx, events
+                    except Exception as e:
+                        self.logger.error(f"Error processing item {idx}: {e}")
+                        # Return empty list on failure to continue processing
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
                     
-                    # Detect
-                    events = self.detect_events(doc_text, **kwargs)
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {}
+                        for idx, item in enumerate(text):
+                            future = executor.submit(process_item, idx, item)
+                            future_to_idx[future] = idx
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, events = future.result()
+                            results[idx] = events
+                            total_events_count += len(events)
+                            processed_count += 1
+                            
+                            # Update progress
+                            if processed_count % update_interval == 0 or processed_count == total_items:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, events = process_item(idx, item)
+                        results[idx] = events
+                        total_events_count += len(events)
+                        processed_count += 1
 
-                    # Add provenance metadata
-                    for event in events:
-                        if event.metadata is None:
-                            event.metadata = {}
-                        event.metadata["batch_index"] = idx
-                        if isinstance(item, dict) and "id" in item:
-                            event.metadata["document_id"] = item["id"]
-
-                    results.append(events)
-                    total_events_count += len(events)
-
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
-                        )
+                        # Update progress
+                        if processed_count % update_interval == 0 or processed_count == total_items:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
+                            )
 
                 self.progress_tracker.stop_tracking(
                     tracking_id,
@@ -238,17 +274,26 @@ class EventDetector:
             # Single item
             return self.detect_events(text, **kwargs)
 
-    def detect_events(self, text: str, **options) -> List[Event]:
+    def detect_events(
+        self,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Event], List[List[Event]]]:
         """
         Detect events in text content.
 
         Args:
             text: Input text
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Detection options
 
         Returns:
             list: List of detected events
         """
+        if isinstance(text, list):
+            return self.extract(text, pipeline_id=pipeline_id, **options)
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="EventDetector",

semantica/semantic_extract/llm_extraction.py

@@ -93,7 +93,7 @@ class LLMExtraction:
             **config: Configuration options:
                 - model: Model name (default depends on provider)
                 - api_key: API key (from environment if not provided)
-                - temperature: Temperature for generation
+                - temperature: Temperature for generation (None = use model's default)
         """
         self.logger = get_logger("llm_extraction")
         self.config = config
@@ -104,11 +104,16 @@ class LLMExtraction:
 
         self.provider_name = provider
         self.model = config.get("model")
-        self.temperature = config.get("temperature", 0.3)
+        self.temperature = config.get("temperature")  # None = use model default
 
         # Initialize provider using new system
         try:
-            self.provider = create_provider(provider, **config)
+            # Sanitize config: remove api_key if it's None/empty to allow fallback
+            provider_config = config.copy()
+            if "api_key" in provider_config and not provider_config["api_key"]:
+                del provider_config["api_key"]
+                
+            self.provider = create_provider(provider, **provider_config)
         except Exception as e:
             self.logger.warning(f"Failed to initialize {provider} provider: {e}")
             self.provider = None

semantica/semantic_extract/ner_extractor.py

@@ -178,9 +178,11 @@ class NERExtractor:
             )
             
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
                 total_entities_count = 0
+                processed_count = 0
+                
                 # Update more frequently: every 1% or at least every 10 items, but always update for small datasets
                 if total_items <= 10:
                     update_interval = 1  # Update every item for small datasets
@@ -188,15 +190,22 @@ class NERExtractor:
                     update_interval = max(1, min(10, total_items // 100))
                 
                 # Initial progress update - ALWAYS show this
-                remaining = total_items
                 self.progress_tracker.update_progress(
                     tracking_id,
                     processed=0,
                     total=total_items,
-                    message=f"Starting batch extraction... 0/{total_items} (remaining: {remaining})"
+                    message=f"Starting batch extraction... 0/{total_items}"
                 )
                 
-                for idx, item in enumerate(text, 1):
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=self.method,
+                )
+                
+                # Helper function for single item processing
+                def process_item(idx, item):
                     try:
                         current_entities = []
                         if isinstance(item, dict) and "content" in item:
@@ -214,30 +223,69 @@ class NERExtractor:
                         for ent in current_entities:
                             if ent.metadata is None:
                                 ent.metadata = {}
-                            ent.metadata["batch_index"] = idx - 1
+                            ent.metadata["batch_index"] = idx
                             if isinstance(item, dict) and "id" in item:
                                 ent.metadata["document_id"] = item["id"]
                         
-                        results.append(current_entities)
-                        total_entities_count += len(current_entities)
-                    except Exception:
-                        results.append([])
+                        return idx, current_entities
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {idx}: {e}")
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
                     
-                    remaining = total_items - idx
-                    # Update progress: always update for small datasets, or at intervals for large ones
-                    should_update = (
-                        idx % update_interval == 0 or 
-                        idx == total_items or 
-                        idx == 1 or
-                        total_items <= 10  # Always update for small datasets
-                    )
-                    if should_update:
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx,
-                            total=total_items,
-                            message=f"Processing documents... {idx}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit all tasks
+                        future_to_idx = {
+                            executor.submit(process_item, idx, item): idx 
+                            for idx, item in enumerate(text)
+                        }
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, entities = future.result()
+                            results[idx] = entities
+                            total_entities_count += len(entities)
+                            processed_count += 1
+                            
+                            # Update progress
+                            should_update = (
+                                processed_count % update_interval == 0 or 
+                                processed_count == total_items or 
+                                processed_count == 1 or
+                                total_items <= 10
+                            )
+                            if should_update:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing documents... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, entities = process_item(idx, item)
+                        results[idx] = entities
+                        total_entities_count += len(entities)
+                        processed_count += 1
+                        
+                        # Update progress
+                        should_update = (
+                            processed_count % update_interval == 0 or 
+                            processed_count == total_items or 
+                            processed_count == 1 or
+                            total_items <= 10
                         )
+                        if should_update:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing documents... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                            )
                 
                 self.progress_tracker.stop_tracking(
                     tracking_id,
@@ -253,12 +301,18 @@ class NERExtractor:
         else:
             return self.extract_entities(text, **kwargs)
 
-    def extract_entities(self, text: str, **options) -> List[Entity]:
+    def extract_entities(
+        self,
+        text: Union[str, List[Dict[str, Any]], List[str]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Entity], List[List[Entity]]]:
         """
         Extract named entities from text.
 
         Args:
             text: Input text
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Extraction options:
                 - entity_types: Filter by entity types (list)
                 - min_confidence: Minimum confidence threshold
@@ -267,6 +321,9 @@ class NERExtractor:
         Returns:
             list: List of extracted entities
         """
+        if isinstance(text, list):
+            return self.extract(text, pipeline_id=pipeline_id, **options)
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="NERExtractor",
@@ -307,8 +364,11 @@ class NERExtractor:
                     # Prepare method-specific options
                     method_options = all_options.copy()
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", self.huggingface_model
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model") 
+                            or self.huggingface_model
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -318,14 +378,13 @@ class NERExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key

semantica/semantic_extract/providers.py

@@ -103,6 +103,12 @@ class BaseProvider:
         """Check if provider is available."""
         return True
 
+    def _add_if_set(self, target: dict, source: dict, *keys: str) -> None:
+        """Add keys from source to target only if their values are not None."""
+        for key in keys:
+            if key in source and source[key] is not None:
+                target[key] = source[key]
+
     def generate(self, prompt: str, **kwargs) -> str:
         """Generate text - must be implemented."""
         raise NotImplementedError
@@ -243,86 +249,157 @@ class BaseProvider:
                 mode = instructor.Mode.TOOLS  # Default mode
                 
                 if provider_name == "OpenAIProvider" and self.client:
-                    client = instructor.from_openai(self.client)
-                elif provider_name == "AnthropicProvider" and self.client:
-                    client = instructor.from_anthropic(self.client)
-                elif provider_name == "GeminiProvider" and self.client:
-                    client = instructor.from_gemini(
-                        self.client, 
-                        mode=instructor.Mode.GEMINI_JSON
-                    )
-                elif provider_name == "GroqProvider" and self.client:
-                    # Try using from_groq if available (newer instructor versions)
-                    if hasattr(instructor, "from_groq"):
-                        client = instructor.from_groq(self.client, mode=instructor.Mode.JSON)
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"openai/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_openai(self.client)
                     else:
-                        # Fallback: Create OpenAI client pointing to Groq
-                        # This avoids the "Client should be an instance of openai.OpenAI" warning
+                        client = instructor.from_openai(self.client)
+                elif provider_name == "AnthropicProvider" and self.client:
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"anthropic/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_anthropic(self.client)
+                    else:
+                        client = instructor.from_anthropic(self.client)
+                elif provider_name == "GeminiProvider" and self.client:
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"gemini/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = instructor.from_gemini(
+                                self.client, 
+                                mode=instructor.Mode.GEMINI_JSON
+                            )
+                    else:
+                        client = instructor.from_gemini(
+                            self.client, 
+                            mode=instructor.Mode.GEMINI_JSON
+                        )
+                elif provider_name == "GroqProvider" and self.client:
+                    # Try using from_provider which is recommended for Groq in latest instructor
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"groq/{kwargs.get('model', self.model)}", 
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = None
+
+                    if not client:
+                        # Try using from_groq if available (newer instructor versions)
+                        if hasattr(instructor, "from_groq"):
+                            client = instructor.from_groq(self.client, mode=instructor.Mode.JSON)
+                        else:
+                            # Fallback: Create OpenAI client pointing to Groq
+                            # This avoids the "Client should be an instance of openai.OpenAI" warning
+                            try:
+                                from openai import OpenAI
+                                # Fix: Use self.api_key instead of self.client.api_key
+                                groq_client = OpenAI(
+                                    base_url="https://api.groq.com/openai/v1",
+                                    api_key=self.api_key,
+                                )
+                                client = instructor.from_openai(groq_client, mode=instructor.Mode.JSON)
+                            except Exception:
+                                # Last resort: try passing the groq client directly
+                                client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
+                elif provider_name == "OllamaProvider":
+                    # Try from_provider for Ollama if available
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"ollama/{kwargs.get('model', self.model)}",
+                            )
+                        except Exception:
+                            client = None
+                    
+                    if not client:
+                        # Create OpenAI-compatible client for Ollama
                         try:
                             from openai import OpenAI
-                            groq_client = OpenAI(
-                                base_url="https://api.groq.com/openai/v1",
-                                api_key=self.client.api_key,
+                            # Ollama typically runs on localhost:11434/v1
+                            base_url = getattr(self, "base_url", "http://localhost:11434")
+                            if not base_url.endswith("/v1"):
+                                base_url = f"{base_url.rstrip('/')}/v1"
+                            
+                            ollama_client = OpenAI(
+                                base_url=base_url,
+                                api_key="ollama", # required but unused
                             )
-                            client = instructor.from_openai(groq_client, mode=instructor.Mode.JSON)
-                        except Exception:
-                            # Last resort: try passing the groq client directly
-                            client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
-                elif provider_name == "OllamaProvider":
-                    # Create OpenAI-compatible client for Ollama
-                    try:
-                        from openai import OpenAI
-                        # Ollama typically runs on localhost:11434/v1
-                        base_url = getattr(self, "base_url", "http://localhost:11434")
-                        if not base_url.endswith("/v1"):
-                            base_url = f"{base_url.rstrip('/')}/v1"
-                        
-                        ollama_client = OpenAI(
-                            base_url=base_url,
-                            api_key="ollama", # required but unused
-                        )
-                        client = instructor.from_openai(ollama_client, mode=instructor.Mode.JSON)
-                    except ImportError:
-                        pass
+                            client = instructor.from_openai(ollama_client, mode=instructor.Mode.JSON)
+                        except ImportError:
+                            pass
                 elif provider_name == "DeepSeekProvider" and self.client:
-                    # DeepSeek is OpenAI compatible
-                    # We need to wrap the underlying client if it exposes the OpenAI interface
-                    # or create a new OpenAI client if self.client is a deepseek.Client (which might be just a wrapper)
-                    # Assuming deepseek.Client is compatible or we can use OpenAI client
-                    try:
-                        # DeepSeek usually works with standard OpenAI client
-                        # If self.client is deepseek.Client, check if we can wrap it
-                        # Otherwise create a new OpenAI client
-                        from openai import OpenAI
-                        if isinstance(self.client, OpenAI):
-                             client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
-                        else:
-                             # Try creating fresh client
-                             ds_client = OpenAI(
-                                 api_key=self.api_key, 
-                                 base_url="https://api.deepseek.com"
-                             )
-                             client = instructor.from_openai(ds_client, mode=instructor.Mode.JSON)
-                    except Exception:
-                        pass
+                    # Try from_provider for DeepSeek
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            client = instructor.from_provider(
+                                provider=f"deepseek/{kwargs.get('model', self.model)}",
+                                api_key=self.api_key
+                            )
+                        except Exception:
+                            client = None
+
+                    if not client:
+                        # DeepSeek is OpenAI compatible
+                        try:
+                            from openai import OpenAI
+                            if isinstance(self.client, OpenAI):
+                                 client = instructor.from_openai(self.client, mode=instructor.Mode.JSON)
+                            else:
+                                 # Try creating fresh client
+                                 ds_client = OpenAI(
+                                     api_key=self.api_key, 
+                                     base_url="https://api.deepseek.com"
+                                 )
+                                 client = instructor.from_openai(ds_client, mode=instructor.Mode.JSON)
+                        except Exception:
+                            pass
+
+                # Global LiteLLM support - if litellm is passed in kwargs or config
+                if not client and (kwargs.get("litellm") or self.config.get("litellm")):
+                    if hasattr(instructor, "from_provider"):
+                        try:
+                            # Format for litellm in instructor is litellm/model_name
+                            provider_model = kwargs.get("model", self.model)
+                            litellm_provider = f"litellm/{provider_model}"
+                            client = instructor.from_provider(litellm_provider, api_key=self.api_key)
+                        except Exception:
+                            pass
 
                 if client:
                     # Map generate arguments to client arguments
                     # Instructor standardizes on chat.completions.create for OpenAI/Groq/Anthropic/Gemini
-                    
                     create_kwargs = {
                         "model": kwargs.get("model", self.model),
                         "messages": [{"role": "user", "content": prompt}],
                         "response_model": schema,
                         "max_retries": max_retries,
-                        "temperature": kwargs.get("temperature", 0.1), # Low temp for structured
+                        "temperature": kwargs.get("temperature") if kwargs.get("temperature") is not None else 0.1,
                     }
-                    
-                    # Add provider-specific params
+                    self._add_if_set(create_kwargs, kwargs, "max_tokens", "max_completion_tokens",
+                                     "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user", "top_k")
+
                     if provider_name == "GroqProvider":
                         create_kwargs["response_format"] = {"type": "json_object"}
                     
                     response = client.chat.completions.create(**create_kwargs)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [BaseProvider.generate_typed] Typed response received via instructor ({provider_name}).", flush=True, file=sys.stdout)
                     return response
             except Exception as e:
                 self.logger.warning(f"Instructor generation failed ({e}), falling back to manual repair loop.")
@@ -464,11 +541,14 @@ class OpenAIProvider(BaseProvider):
                 "OpenAI client not initialized. Set OPENAI_API_KEY or pass api_key."
             )
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+        }
+        self._add_if_set(create_kwargs, kwargs, "temperature", "max_completion_tokens", "max_tokens",
+                         "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user")
+
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -476,12 +556,15 @@ class OpenAIProvider(BaseProvider):
         if not self.client:
             raise ProcessingError("OpenAI client not initialized.")
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            response_format={"type": "json_object"},
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "response_format": {"type": "json_object"},
+        }
+        self._add_if_set(create_kwargs, kwargs, "temperature", "max_completion_tokens", "max_tokens",
+                         "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user")
+
+        response = self.client.chat.completions.create(**create_kwargs)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -499,26 +582,40 @@ class GeminiProvider(BaseProvider):
         self.api_key = api_key or config.get_api_key("gemini")
         self.model = model
         self.client = None
+        self._use_new_genai = False
         self._init_client()
 
     def _init_client(self):
-        """Initialize Gemini client."""
         try:
-            import google.generativeai as genai
-
+            from google import genai as new_genai
             if self.api_key:
-                genai.configure(api_key=self.api_key)
-                self.client = genai.GenerativeModel(self.model)
-        except (ImportError, OSError):
+                self.client = new_genai.Client(api_key=self.api_key)
+                self._use_new_genai = True
+                return
+        except Exception:
+            pass
+        try:
+            import google.generativeai as old_genai
+            if self.api_key:
+                old_genai.configure(api_key=self.api_key)
+                self.client = old_genai.GenerativeModel(self.model)
+                self._use_new_genai = False
+        except Exception:
             self.client = None
-            self.logger.warning(
-                "google-generativeai library not installed. Install with: pip install semantica[llm-gemini]"
-            )
+            self.logger.warning("Gemini SDK not installed. Install with: pip install semantica[llm-gemini]")
 
     def is_available(self) -> bool:
         """Check if provider is available."""
         return self.client is not None
 
+    def _resp_text(self, resp: Any) -> str:
+        if hasattr(resp, "text"):
+            return getattr(resp, "text")
+        try:
+            return resp.candidates[0].content.parts[0].text
+        except Exception:
+            return str(resp)
+
     def generate(self, prompt: str, **kwargs) -> str:
         """Generate text from prompt."""
         if not self.client:
@@ -526,23 +623,39 @@ class GeminiProvider(BaseProvider):
                 "Gemini client not initialized. Set GEMINI_API_KEY or pass api_key."
             )
 
-        response = self.client.generate_content(
-            prompt, generation_config={"temperature": kwargs.get("temperature", 0.3)}
-        )
-        return response.text
+        config = {}
+        self._add_if_set(config, kwargs, "temperature", "top_p", "top_k", "stop_sequences", "candidate_count")
+        if "max_tokens" in kwargs:
+            config["max_output_tokens"] = kwargs["max_tokens"]
+
+        if self._use_new_genai:
+            resp = self.client.models.generate_content(
+                model=kwargs.get("model", self.model), contents=prompt, config=config or None
+            )
+            return self._resp_text(resp)
+        else:
+            response = self.client.generate_content(prompt, generation_config=config or None)
+            return self._resp_text(response)
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
         """Generate structured output."""
         if not self.client:
             raise ProcessingError("Gemini client not initialized.")
 
-        # Add JSON format instruction to prompt
         json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
-        response = self.client.generate_content(json_prompt)
-        try:
-            return self._parse_json(response.text)
-        except Exception as e:
-            raise ProcessingError(f"Failed to parse JSON from Gemini response: {e}")
+        if self._use_new_genai:
+            model = kwargs.get("model", self.model)
+            resp = self.client.models.generate_content(model=model, contents=json_prompt)
+            try:
+                return self._parse_json(self._resp_text(resp))
+            except Exception as e:
+                raise ProcessingError(f"Failed to parse JSON from Gemini response: {e}")
+        else:
+            response = self.client.generate_content(json_prompt)
+            try:
+                return self._parse_json(self._resp_text(response))
+            except Exception as e:
+                raise ProcessingError(f"Failed to parse JSON from Gemini response: {e}")
 
 
 class GroqProvider(BaseProvider):
@@ -621,11 +734,14 @@ class GroqProvider(BaseProvider):
                 "Groq client not initialized. Set GROQ_API_KEY or pass api_key."
             )
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+        }
+        self._add_if_set(create_kwargs, kwargs, "temperature", "max_completion_tokens", "max_tokens",
+                         "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "user")
+
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -634,16 +750,17 @@ class GroqProvider(BaseProvider):
             raise ProcessingError("Groq client not initialized.")
 
         # Groq requires 'json' in the prompt for json_object mode
-        json_prompt = prompt
-        if "json" not in prompt.lower():
-            json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
+        json_prompt = prompt if "json" in prompt.lower() else f"{prompt}\n\nReturn the response as valid JSON only."
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": json_prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-            response_format={"type": "json_object"},
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": json_prompt}],
+            "response_format": {"type": "json_object"},
+        }
+        self._add_if_set(create_kwargs, kwargs, "temperature", "max_completion_tokens", "max_tokens",
+                         "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "user")
+
+        response = self.client.chat.completions.create(**create_kwargs)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -690,11 +807,23 @@ class AnthropicProvider(BaseProvider):
                 "Anthropic client not initialized. Set ANTHROPIC_API_KEY or pass api_key."
             )
 
-        response = self.client.messages.create(
-            model=kwargs.get("model", self.model),
-            max_tokens=kwargs.get("max_tokens", 4096),
-            messages=[{"role": "user", "content": prompt}],
-        )
+        # Anthropic requires max_tokens. 
+        # We rely on kwargs, but fallback to 8192 (safe max for newer models) if not provided.
+        max_tokens = kwargs.get("max_tokens", 8192)
+        
+        # Prepare arguments
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "max_tokens": max_tokens,
+            "messages": [{"role": "user", "content": prompt}],
+        }
+        
+        # Pass through other common parameters
+        for param in ["temperature", "top_p", "top_k", "stop_sequences", "system", "metadata"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        response = self.client.messages.create(**create_kwargs)
         return response.content[0].text
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -703,11 +832,23 @@ class AnthropicProvider(BaseProvider):
             raise ProcessingError("Anthropic client not initialized.")
 
         json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
-        response = self.client.messages.create(
-            model=kwargs.get("model", self.model),
-            max_tokens=kwargs.get("max_tokens", 4096),
-            messages=[{"role": "user", "content": json_prompt}],
-        )
+        
+        # Anthropic requires max_tokens. 
+        max_tokens = kwargs.get("max_tokens", 8192)
+        
+        # Prepare arguments
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "max_tokens": max_tokens,
+            "messages": [{"role": "user", "content": json_prompt}],
+        }
+        
+        # Pass through other common parameters
+        for param in ["temperature", "top_p", "top_k", "stop_sequences", "system", "metadata"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+        
+        response = self.client.messages.create(**create_kwargs)
         try:
             return self._parse_json(response.content[0].text)
         except Exception as e:
@@ -751,6 +892,18 @@ class OllamaProvider(BaseProvider):
         """Check if provider is available."""
         return self.client is not None
 
+    def _build_options(self, kwargs: dict) -> Optional[dict]:
+        """Build Ollama options dict from kwargs."""
+        options = {}
+        self._add_if_set(options, kwargs, "temperature", "top_p", "top_k", "repeat_penalty", "seed")
+        if "max_tokens" in kwargs:
+            options["num_predict"] = kwargs["max_tokens"]
+        if "num_ctx" in kwargs:
+            options["num_ctx"] = kwargs["num_ctx"]
+        elif "context_window" in kwargs:
+            options["num_ctx"] = kwargs["context_window"]
+        return options or None
+
     def generate(self, prompt: str, **kwargs) -> str:
         """Generate text from prompt."""
         if not self.client:
@@ -761,7 +914,7 @@ class OllamaProvider(BaseProvider):
         response = self.client.generate(
             model=kwargs.get("model", self.model),
             prompt=prompt,
-            options={"temperature": kwargs.get("temperature", 0.3)},
+            options=self._build_options(kwargs),
         )
         return response.get("response", "")
 
@@ -774,7 +927,7 @@ class OllamaProvider(BaseProvider):
         response = self.client.generate(
             model=kwargs.get("model", self.model),
             prompt=json_prompt,
-            options={"temperature": kwargs.get("temperature", 0.3)},
+            options=self._build_options(kwargs),
         )
         try:
             return self._parse_json(response.get("response", "{}"))
@@ -808,21 +961,28 @@ class DeepSeekProvider(BaseProvider):
     def generate(self, prompt: str, **kwargs) -> str:
         if not self.client:
             raise ProcessingError("DeepSeek client not initialized. Set DEEPSEEK_API_KEY or pass api_key.")
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+        }
+        self._add_if_set(create_kwargs, kwargs, "temperature", "max_tokens")
+
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
+
     def generate_structured(self, prompt: str, **kwargs) -> Union[dict, list]:
         """Generate structured output."""
         if not self.client:
             raise ProcessingError("DeepSeek client not initialized.")
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+        }
+        self._add_if_set(create_kwargs, kwargs, "temperature", "max_tokens")
+
+        response = self.client.chat.completions.create(**create_kwargs)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -879,11 +1039,27 @@ class HuggingFaceLLMProvider(BaseProvider):
             raise ProcessingError("HuggingFace model not initialized.")
 
         inputs = self.tokenizer.encode(prompt, return_tensors="pt").to(self.device)
+        
+        # Use max_new_tokens if available, otherwise fallback to max_length with a safe default
+        generate_kwargs = {
+            "temperature": kwargs.get("temperature", 0.7),
+            "do_sample": True,
+        }
+        
+        if "max_new_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_new_tokens"]
+        elif "max_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_tokens"]
+            
+        # Support legacy max_length if explicitly provided
+        if "max_length" in kwargs:
+            generate_kwargs["max_length"] = kwargs["max_length"]
+            # Remove max_new_tokens if max_length is set to avoid conflict
+            generate_kwargs.pop("max_new_tokens", None)
+
         outputs = self.model.generate(
             inputs,
-            max_length=kwargs.get("max_length", 100),
-            temperature=kwargs.get("temperature", 0.7),
-            do_sample=True,
+            **generate_kwargs
         )
         generated_text = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
         # Remove the original prompt from the response
@@ -924,25 +1100,32 @@ class HuggingFaceModelLoader:
         # Import torch at method level to ensure it's available
         import torch
         
-        cache_key = f"{model_name}_ner"
+        # Include aggregation_strategy in cache key
+        agg_strategy = kwargs.get("aggregation_strategy", "simple")
+        cache_key = f"{model_name}_ner_{agg_strategy}"
         if cache_key in self._cache:
             return self._cache[cache_key]
 
         try:
             from transformers import pipeline
+        except ImportError:
+            raise ImportError(
+                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
+            )
 
+        try:
             nlp = pipeline(
                 "ner",
                 model=model_name,
                 device=self.device if torch.cuda.is_available() else -1,
-                aggregation_strategy="simple",
+                aggregation_strategy=agg_strategy,
+                tokenizer=kwargs.get("tokenizer") # Allow custom tokenizer
             )
             self._cache[cache_key] = nlp
             return nlp
-        except (ImportError, OSError):
-            raise ImportError(
-                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
-            )
+        except OSError as e:
+            self.logger.error(f"Failed to load NER model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load NER model {model_name}: {e}")
             raise
@@ -957,19 +1140,31 @@ class HuggingFaceModelLoader:
             return self._cache[cache_key]
 
         try:
-            from transformers import pipeline
-
-            nlp = pipeline(
-                "text-classification",
-                model=model_name,
-                device=self.device if torch.cuda.is_available() else -1,
-            )
-            self._cache[cache_key] = nlp
-            return nlp
-        except (ImportError, OSError):
+            from transformers import pipeline, AutoTokenizer
+        except ImportError:
             raise ImportError(
                 "transformers library not installed. Install with: pip install semantica[models-huggingface]"
             )
+
+        try:
+            # Allow custom tokenizer
+            tokenizer = kwargs.get("tokenizer")
+            if not tokenizer and kwargs.get("tokenizer_name"):
+                tokenizer = AutoTokenizer.from_pretrained(kwargs.get("tokenizer_name"))
+
+            pipeline_kwargs = {
+                "model": model_name,
+                "device": self.device if torch.cuda.is_available() else -1,
+            }
+            if tokenizer:
+                pipeline_kwargs["tokenizer"] = tokenizer
+
+            nlp = pipeline("text-classification", **pipeline_kwargs)
+            self._cache[cache_key] = nlp
+            return nlp
+        except OSError as e:
+            self.logger.error(f"Failed to load relation model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load relation model {model_name}: {e}")
             raise
@@ -982,18 +1177,27 @@ class HuggingFaceModelLoader:
 
         try:
             from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, pipeline
+        except ImportError:
+            raise ImportError(
+                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
+            )
 
-            tokenizer = AutoTokenizer.from_pretrained(model_name)
+        try:
+            # Allow custom tokenizer
+            tokenizer = kwargs.get("tokenizer")
+            if not tokenizer:
+                tokenizer_name = kwargs.get("tokenizer_name", model_name)
+                tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
+            
             model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
             model.to(self.device)
 
             nlp = {"tokenizer": tokenizer, "model": model, "device": self.device}
             self._cache[cache_key] = nlp
             return nlp
-        except (ImportError, OSError):
-            raise ImportError(
-                "transformers library not installed. Install with: pip install semantica[models-huggingface]"
-            )
+        except OSError as e:
+            self.logger.error(f"Failed to load triplet model '{model_name}': {e}")
+            raise ValueError(f"Could not load HuggingFace model '{model_name}'. Check if model name is correct. Error: {e}")
         except Exception as e:
             self.logger.error(f"Failed to load triplet model {model_name}: {e}")
             raise
@@ -1002,50 +1206,199 @@ class HuggingFaceModelLoader:
         """Extract entities using loaded model."""
         return model(text)
 
-    def extract_relations(self, model, text: str, entities: List) -> List[Dict]:
-        """Extract relations using loaded model."""
-        # This would need to be customized based on the model architecture
-        return model(text)
+    def extract_relations(self, model, text: str, entities: List, **kwargs) -> List[Dict]:
+        """
+        Extract relations using loaded model.
+        Iterates through entity pairs and classifies the relationship.
+        """
+        results = []
+        
+        # Sort entities by position
+        sorted_entities = sorted(entities, key=lambda e: e.start_char)
+        
+        # Marker configuration
+        subj_start = kwargs.get("subj_start_marker", "<subj>")
+        subj_end = kwargs.get("subj_end_marker", "</subj>")
+        obj_start = kwargs.get("obj_start_marker", "<obj>")
+        obj_end = kwargs.get("obj_end_marker", "</obj>")
+        
+        # Iterate through all pairs
+        import itertools
+        for i, e1 in enumerate(sorted_entities):
+            for e2 in sorted_entities:
+                if e1 == e2:
+                    continue
+                
+                # Check distance (optional optimization)
+                # if abs(e1.start_char - e2.start_char) > 200: continue
+                
+                # Format text with markers
+                # Strategy: [CLS] text with <subj>...</subj> and <obj>...</obj> [SEP]
+                # We need to insert markers into the original text
+                
+                # Create a copy of text with markers inserted
+                # We need to handle offsets correctly. 
+                # Simplest way: reconstruct string pieces
+                
+                p1_start, p1_end = e1.start_char, e1.end_char
+                p2_start, p2_end = e2.start_char, e2.end_char
+                
+                if p1_start < p2_start:
+                    formatted_text = (
+                        text[:p1_start] + 
+                        f"{subj_start} " + text[p1_start:p1_end] + f" {subj_end}" + 
+                        text[p1_end:p2_start] + 
+                        f"{obj_start} " + text[p2_start:p2_end] + f" {obj_end}" + 
+                        text[p2_end:]
+                    )
+                else:
+                     formatted_text = (
+                        text[:p2_start] + 
+                        f"{obj_start} " + text[p2_start:p2_end] + f" {obj_end}" + 
+                        text[p2_end:p1_start] + 
+                        f"{subj_start} " + text[p1_start:p1_end] + f" {subj_end}" + 
+                        text[p1_end:]
+                    )
+                
+                # Predict
+                try:
+                    # Pipeline returns [{'label': 'LABEL', 'score': 0.99}]
+                    prediction = model(formatted_text, top_k=1)
+                    
+                    if prediction:
+                        res = prediction[0] if isinstance(prediction, list) else prediction
+                        if isinstance(res, list): res = res[0] # top_k=1 returns list of dicts
+                        
+                        label = res.get("label")
+                        score = res.get("score")
+                        
+                        # Filter "no_relation" or low confidence
+                        if label != "no_relation" and score > kwargs.get("threshold", 0.5):
+                            results.append({
+                                "subject": e1,
+                                "object": e2,
+                                "relation": label,
+                                "score": score
+                            })
+                except Exception as e:
+                    self.logger.warning(f"Relation prediction failed for pair {e1.text}-{e2.text}: {e}")
+                    
+        return results
 
-    def extract_triplets(self, model, text: str) -> List[Dict]:
+    def extract_triplets(self, model, text: str, **kwargs) -> List[Dict]:
         """Extract triplets using loaded model."""
         tokenizer = model["tokenizer"]
         model_obj = model["model"]
         device = model["device"]
 
+        # Use kwargs for max_length, default to 512 for input and 128 for output if not specified
+        max_input_length = kwargs.get("max_input_length", 512)
+        max_length = kwargs.get("max_length", 128)
+        
+        generate_kwargs = {"max_length": max_length}
+        if "max_new_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_new_tokens"]
+            
+        # Pass other generation args including beams and penalties
+        for param in ["num_beams", "temperature", "top_p", "top_k", "do_sample", 
+                      "length_penalty", "repetition_penalty"]:
+            if param in kwargs:
+                generate_kwargs[param] = kwargs[param]
+
         inputs = tokenizer(
-            text, return_tensors="pt", truncation=True, max_length=512
+            text, return_tensors="pt", truncation=True, max_length=max_input_length
         ).to(device)
-        outputs = model_obj.generate(**inputs, max_length=128)
-        decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
+        
+        outputs = model_obj.generate(**inputs, **generate_kwargs)
+        # Allow controlling skip_special_tokens (important for REBEL which uses special tokens for delimiters)
+        skip_special_tokens = kwargs.get("skip_special_tokens", True)
+        decoded = tokenizer.decode(outputs[0], skip_special_tokens=skip_special_tokens)
 
-        # Parse decoded output (format depends on model)
-        # This is a placeholder - actual parsing would depend on model output format
         return [{"triplet": decoded}]
 
 
-def create_provider(name: str, **kwargs) -> BaseProvider:
-    """Create provider - checks registry for custom providers."""
-    # Check registry first
-    custom_provider = provider_registry.get(name)
-    if custom_provider:
-        return custom_provider(**kwargs)
+class ProviderPool:
+    """Pool for reusing provider instances."""
+    
+    def __init__(self):
+        self._providers: Dict[str, BaseProvider] = {}
+        self.logger = get_logger("provider_pool")
 
-    # Built-in providers
-    builtin = {
-        "openai": OpenAIProvider,
-        "gemini": GeminiProvider,
-        "groq": GroqProvider,
-        "anthropic": AnthropicProvider,
-        "ollama": OllamaProvider,
-        "huggingface_llm": HuggingFaceLLMProvider,
-        "deepseek": DeepSeekProvider,
-    }
+    def get(self, name: str, **kwargs) -> BaseProvider:
+        """Get or create a provider instance."""
+        # Create a cache key from name and kwargs
+        # Filter out non-hashable items or volatile args if any
+        # For now, we assume kwargs are configuration options that should match
+        
+        # Helper to make dict hashable
+        def make_hashable(value):
+            if isinstance(value, dict):
+                return tuple(sorted((k, make_hashable(v)) for k, v in value.items()))
+            elif isinstance(value, list):
+                return tuple(make_hashable(v) for v in value)
+            return value
 
-    provider_class = builtin.get(name.lower())
-    if not provider_class:
-        raise ValueError(
-            f"Unknown provider: {name}. Register custom provider or use built-in: {list(builtin.keys())}"
-        )
+        key_parts = [name]
+        for k, v in sorted(kwargs.items()):
+            # Skip some keys if they shouldn't affect pooling? 
+            # For now, all init args matter for the instance identity.
+            key_parts.append((k, make_hashable(v)))
+            
+        key = str(tuple(key_parts))
+        
+        if key in self._providers:
+            return self._providers[key]
+            
+        self.logger.debug(f"Creating new provider instance for {name}")
+        provider = self._create_provider(name, **kwargs)
+        self._providers[key] = provider
+        return provider
+    
+    def _create_provider(self, name: str, **kwargs) -> BaseProvider:
+        """Internal creation logic."""
+        # Check registry first
+        custom_provider = provider_registry.get(name)
+        if custom_provider:
+            return custom_provider(**kwargs)
 
-    return provider_class(**kwargs)
+        # Built-in providers
+        builtin = {
+            "openai": OpenAIProvider,
+            "gemini": GeminiProvider,
+            "groq": GroqProvider,
+            "anthropic": AnthropicProvider,
+            "ollama": OllamaProvider,
+            "huggingface_llm": HuggingFaceLLMProvider,
+            "deepseek": DeepSeekProvider,
+        }
+
+        provider_class = builtin.get(name.lower())
+        if not provider_class:
+            raise ValueError(
+                f"Unknown provider: {name}. Register custom provider or use built-in: {list(builtin.keys())}"
+            )
+
+        return provider_class(**kwargs)
+
+    def clear(self):
+        """Clear the provider pool."""
+        self._providers.clear()
+
+
+# Global provider pool
+_provider_pool = ProviderPool()
+
+
+def create_provider(name: str, use_pool: bool = True, **kwargs) -> BaseProvider:
+    """
+    Create provider - checks registry for custom providers.
+    
+    Args:
+        name: Provider name
+        use_pool: Whether to use the provider pool (default: True)
+        **kwargs: Provider arguments
+    """
+    if use_pool:
+        return _provider_pool.get(name, **kwargs)
+        
+    return _provider_pool._create_provider(name, **kwargs)

semantica/semantic_extract/relation_extractor.py

@@ -201,10 +201,12 @@ class RelationExtractor:
             )
             
             try:
-                results = []
                 # Ensure lists are same length
                 min_len = min(len(text), len(entities))
+                results = [None] * min_len
                 total_relations_count = 0
+                processed_count = 0
+                
                 # Update more frequently: every 1% or at least every 10 items, but always update for small datasets
                 if min_len <= 10:
                     update_interval = 1  # Update every item for small datasets
@@ -212,58 +214,101 @@ class RelationExtractor:
                     update_interval = max(1, min(10, min_len // 100))
                 
                 # Initial progress update - ALWAYS show this
-                remaining = min_len
                 self.progress_tracker.update_progress(
                     tracking_id,
                     processed=0,
                     total=min_len,
-                    message=f"Starting batch extraction... 0/{min_len} (remaining: {remaining})"
+                    message=f"Starting batch extraction... 0/{min_len}"
                 )
                 
-                for i in range(min_len):
-                    doc_item = text[i]
-                    ent_item = entities[i]
-                    
-                    doc_text = ""
-                    if isinstance(doc_item, dict) and "content" in doc_item:
-                        doc_text = doc_item["content"]
-                    elif isinstance(doc_item, str):
-                        doc_text = doc_item
-                    else:
-                        doc_text = str(doc_item)
-                    
-                    # Ensure ent_item is a list of entities
-                    if not isinstance(ent_item, list):
-                        ent_item = [] # Should not happen if entities is List[List[Entity]]
-                    
-                    current_relations = self.extract_relations(doc_text, ent_item, **kwargs)
-                    
-                    # Add provenance metadata
-                    for rel in current_relations:
-                        if rel.metadata is None:
-                            rel.metadata = {}
-                        rel.metadata["batch_index"] = i
-                        if isinstance(doc_item, dict) and "id" in doc_item:
-                            rel.metadata["document_id"] = doc_item["id"]
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=self.method,
+                )
 
-                    results.append(current_relations)
-                    total_relations_count += len(current_relations)
+                def process_item(i, doc_item, ent_item):
+                    try:
+                        doc_text = ""
+                        if isinstance(doc_item, dict) and "content" in doc_item:
+                            doc_text = doc_item["content"]
+                        elif isinstance(doc_item, str):
+                            doc_text = doc_item
+                        else:
+                            doc_text = str(doc_item)
+                        
+                        # Ensure ent_item is a list of entities
+                        if not isinstance(ent_item, list):
+                            ent_item = [] # Should not happen if entities is List[List[Entity]]
+                        
+                        current_relations = self.extract_relations(doc_text, ent_item, **kwargs)
+                        
+                        # Add provenance metadata
+                        for rel in current_relations:
+                            if rel.metadata is None:
+                                rel.metadata = {}
+                            rel.metadata["batch_index"] = i
+                            if isinstance(doc_item, dict) and "id" in doc_item:
+                                rel.metadata["document_id"] = doc_item["id"]
+                        
+                        return i, current_relations
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {i}: {e}")
+                        return i, []
+
+                if max_workers > 1:
+                    import concurrent.futures
                     
-                    remaining = min_len - (i + 1)
-                    # Update progress: always update for small datasets, or at intervals for large ones
-                    should_update = (
-                        (i + 1) % update_interval == 0 or 
-                        (i + 1) == min_len or 
-                        i == 0 or
-                        min_len <= 10  # Always update for small datasets
-                    )
-                    if should_update:
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=i + 1,
-                            total=min_len,
-                            message=f"Processing documents... {i + 1}/{min_len} (remaining: {remaining}) - Extracted {total_relations_count} relations so far"
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {
+                            executor.submit(process_item, i, text[i], entities[i]): i
+                            for i in range(min_len)
+                        }
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            i, relations = future.result()
+                            results[i] = relations
+                            total_relations_count += len(relations)
+                            processed_count += 1
+                            
+                            should_update = (
+                                processed_count % update_interval == 0 or 
+                                processed_count == min_len or 
+                                processed_count == 1 or
+                                min_len <= 10
+                            )
+                            if should_update:
+                                remaining = min_len - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=min_len,
+                                    message=f"Processing documents... {processed_count}/{min_len} (remaining: {remaining}) - Extracted {total_relations_count} relations so far"
+                                )
+                else:
+                    # Sequential processing
+                    for i in range(min_len):
+                        _, relations = process_item(i, text[i], entities[i])
+                        results[i] = relations
+                        total_relations_count += len(relations)
+                        processed_count += 1
+                        
+                        should_update = (
+                            processed_count % update_interval == 0 or 
+                            processed_count == min_len or 
+                            processed_count == 1 or
+                            min_len <= 10
                         )
+                        if should_update:
+                            remaining = min_len - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=min_len,
+                                message=f"Processing documents... {processed_count}/{min_len} (remaining: {remaining}) - Extracted {total_relations_count} relations so far"
+                            )
                 
                 self.progress_tracker.stop_tracking(
                     tracking_id,
@@ -284,14 +329,19 @@ class RelationExtractor:
             return []
 
     def extract_relations(
-        self, text: str, entities: List[Entity], **options
-    ) -> List[Relation]:
+        self,
+        text: Union[str, List[Dict[str, Any]], List[str]],
+        entities: Union[List[Entity], List[List[Entity]]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Relation], List[List[Relation]]]:
         """
         Extract relations between entities.
 
         Args:
             text: Input text
             entities: List of extracted entities
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Extraction options:
                 - method: Override method (if not set in __init__)
                 - min_confidence: Minimum confidence threshold
@@ -300,7 +350,18 @@ class RelationExtractor:
         Returns:
             list: List of extracted relations
         """
-        from .methods import get_relation_method
+        if isinstance(text, list):
+            if entities is None:
+                entities_batch = [[] for _ in range(len(text))]
+            elif isinstance(entities, list) and (not entities):
+                entities_batch = [[] for _ in range(len(text))]
+            elif isinstance(entities, list) and all(isinstance(e, Entity) for e in entities):
+                entities_batch = [entities for _ in range(len(text))]
+            else:
+                entities_batch = entities
+            return self.extract(text, entities_batch, pipeline_id=pipeline_id, **options)
+
+        from .methods import get_relation_method, match_entity
 
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
@@ -347,8 +408,12 @@ class RelationExtractor:
                         method_options["relation_types"] = relation_types
 
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", all_options.get("model")
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model")
+                            or self.config.get("huggingface_model") 
+                            or self.config.get("model")
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -358,14 +423,13 @@ class RelationExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key
@@ -379,6 +443,12 @@ class RelationExtractor:
                     if verbose_mode and method_name == "llm":
                         import sys
                         print(f"    [RelationExtractor] Processing with {method_name}...", flush=True, file=sys.stdout)
+                        print(f"    [RelationExtractor Debug] method_options keys: {list(method_options.keys())}", flush=True, file=sys.stdout)
+                        if "api_key" in method_options:
+                            masked = method_options["api_key"][:4] + "..." if method_options["api_key"] else "None"
+                            print(f"    [RelationExtractor Debug] api_key present: {masked}", flush=True, file=sys.stdout)
+                        else:
+                            print(f"    [RelationExtractor Debug] api_key NOT present", flush=True, file=sys.stdout)
                     
                     relations = method_func(text, entities, **method_options)
                     
@@ -421,6 +491,11 @@ class RelationExtractor:
 
                 except Exception as e:
                     self.logger.warning(f"Method {method_name} failed: {e}")
+                    if verbose_mode:
+                        import sys
+                        print(f"    [RelationExtractor] ERROR: Method {method_name} failed: {e}", flush=True, file=sys.stderr)
+                        import traceback
+                        traceback.print_exc(file=sys.stderr)
                     continue
 
             # Use first successful method or combine
@@ -487,11 +562,9 @@ class RelationExtractor:
         self, text: str, entities: List[Entity]
     ) -> List[Relation]:
         """Extract relations using pattern matching."""
+        from .methods import match_entity
         relations = []
 
-        # Create entity lookup by text
-        entity_map = {e.text.lower(): e for e in entities}
-
         # Check each relation pattern
         for relation_type, patterns in self.relation_patterns.items():
             for pattern in patterns:
@@ -499,8 +572,8 @@ class RelationExtractor:
                     subject_text = match.group("subject").strip()
                     object_text = match.group("object").strip()
 
-                    subject_entity = entity_map.get(subject_text.lower())
-                    object_entity = entity_map.get(object_text.lower())
+                    subject_entity = match_entity(subject_text, entities)
+                    object_entity = match_entity(object_text, entities)
 
                     if subject_entity and object_entity:
                         # Get context around the match

semantica/semantic_extract/semantic_analyzer.py

@@ -148,8 +148,9 @@ class SemanticAnalyzer:
             )
 
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -165,40 +166,83 @@ class SemanticAnalyzer:
                     message=f"Starting batch analysis... 0/{total_items} (remaining: {total_items})"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
-                    
-                    # Analyze
-                    analysis = self.analyze_semantics(doc_text, **kwargs)
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                )
 
-                    # Add provenance metadata
-                    analysis["batch_index"] = idx
-                    if isinstance(item, dict) and "id" in item:
-                        analysis["document_id"] = item["id"]
-                        
-                    # Also inject into semantic roles if present
-                    if "semantic_roles" in analysis:
-                        for role in analysis["semantic_roles"]:
-                            # role is a dict here because analyze_semantics converts it
-                            if "metadata" not in role:
-                                role["metadata"] = {}
-                            
-                            role["metadata"]["batch_index"] = idx
-                            if isinstance(item, dict) and "id" in item:
-                                role["metadata"]["document_id"] = item["id"]
+                def process_item(idx, item):
+                    try:
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                        analysis = self.analyze_semantics(doc_text, **kwargs)
 
-                    results.append(analysis)
+                        analysis["batch_index"] = idx
+                        if isinstance(item, dict) and "id" in item:
+                            analysis["document_id"] = item["id"]
 
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining})"
+                        if "semantic_roles" in analysis:
+                            for role in analysis["semantic_roles"]:
+                                if "metadata" not in role:
+                                    role["metadata"] = {}
+
+                                role["metadata"]["batch_index"] = idx
+                                if isinstance(item, dict) and "id" in item:
+                                    role["metadata"]["document_id"] = item["id"]
+
+                        return idx, analysis
+                    except Exception as e:
+                        self.logger.warning(f"Failed to analyze item {idx}: {e}")
+                        return idx, {"error": str(e), "batch_index": idx}
+
+                if max_workers > 1:
+                    import concurrent.futures
+
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        future_to_idx = {
+                            executor.submit(process_item, idx, item): idx
+                            for idx, item in enumerate(text)
+                        }
+
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, analysis = future.result()
+                            results[idx] = analysis
+                            processed_count += 1
+
+                            should_update = (
+                                processed_count % update_interval == 0
+                                or processed_count == total_items
+                                or processed_count == 1
+                                or total_items <= 10
+                            )
+                            if should_update:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                                )
+                else:
+                    for idx, item in enumerate(text):
+                        _, analysis = process_item(idx, item)
+                        results[idx] = analysis
+                        processed_count += 1
+
+                        should_update = (
+                            processed_count % update_interval == 0
+                            or processed_count == total_items
+                            or processed_count == 1
+                            or total_items <= 10
                         )
+                        if should_update:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                            )
 
                 self.progress_tracker.stop_tracking(
                     tracking_id,

semantica/semantic_extract/semantic_extract_provenance.py

@@ -0,0 +1,411 @@
+"""
+Provenance-enabled wrappers for semantic extraction.
+
+This module provides provenance tracking for all semantic extraction operations:
+- Named Entity Recognition (NER)
+- Relation Extraction
+- Event Detection
+- Coreference Resolution
+- Triplet Extraction
+
+All classes wrap the original extractors and add optional provenance tracking
+without modifying existing functionality.
+
+Usage:
+    from semantica.semantic_extract.semantic_extract_provenance import (
+        NERExtractorWithProvenance,
+        RelationExtractorWithProvenance,
+        EventDetectorWithProvenance
+    )
+    
+    # Enable provenance tracking
+    ner = NERExtractorWithProvenance(provenance=True)
+    entities = ner.extract("Steve Jobs founded Apple.", source="document.pdf")
+    
+    # Provenance is automatically tracked for each extracted entity
+    # Access via: ner._prov_manager.get_lineage(entity_id)
+
+Features:
+    - Zero breaking changes - works exactly like original classes
+    - Opt-in provenance via provenance=True parameter
+    - Tracks: entity text, labels, confidence, source documents
+    - Complete lineage tracing
+    - Graceful degradation if provenance module unavailable
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, List, Dict, Any
+import uuid
+
+
+class ProvenanceMixin:
+    """
+    Mixin to add provenance tracking to any extractor class.
+    
+    This mixin provides the common provenance infrastructure that can be
+    added to any extraction class without modifying its core functionality.
+    """
+    
+    def __init__(self, provenance: bool = False, **kwargs):
+        """
+        Initialize provenance tracking.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **kwargs: Additional arguments passed to parent class
+        """
+        self.provenance = provenance
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                # Graceful degradation if provenance module not available
+                self.provenance = False
+    
+    def _track_extraction(
+        self,
+        entity_id: str,
+        source: str,
+        entity_type: str,
+        **metadata
+    ) -> None:
+        """
+        Track extraction with provenance.
+        
+        Args:
+            entity_id: Unique identifier for extracted entity
+            source: Source document or text
+            entity_type: Type of entity (e.g., 'named_entity', 'relation')
+            **metadata: Additional metadata to track
+        """
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=entity_id,
+                source=source,
+                entity_type=entity_type,
+                metadata=metadata
+            )
+
+
+class NERExtractorWithProvenance(ProvenanceMixin):
+    """
+    Named Entity Recognition extractor with provenance tracking.
+    
+    Wraps the original NERExtractor and adds optional provenance tracking
+    for all extracted entities.
+    
+    Example:
+        >>> ner = NERExtractorWithProvenance(provenance=True)
+        >>> entities = ner.extract("Steve Jobs founded Apple.", source="doc1.pdf")
+        >>> # Each entity is tracked with source, confidence, and metadata
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize NER extractor with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original NERExtractor
+        """
+        from .ner_extractor import NERExtractor
+        
+        ProvenanceMixin.__init__(self, provenance=provenance)
+        self._extractor = NERExtractor(**config)
+    
+    def extract(self, text: str, source: Optional[str] = None, **kwargs):
+        """
+        Extract named entities with provenance tracking.
+        
+        Args:
+            text: Input text to extract entities from
+            source: Source document identifier (for provenance)
+            **kwargs: Additional arguments for extraction
+            
+        Returns:
+            List of extracted entities (same as original NERExtractor)
+        """
+        entities = self._extractor.extract(text, **kwargs)
+        
+        if self.provenance:
+            for entity in entities:
+                entity_id = getattr(entity, 'id', None)
+                if not entity_id:
+                    entity_id = f"entity_{uuid.uuid4().hex[:8]}"
+                    try:
+                        entity.id = entity_id
+                    except AttributeError:
+                        pass
+                
+                self._track_extraction(
+                    entity_id=entity_id,
+                    source=source or text[:100],
+                    entity_type="named_entity",
+                    text=entity.text,
+                    label=entity.label,
+                    confidence=getattr(entity, 'confidence', 1.0),
+                    start=entity.start,
+                    end=entity.end
+                )
+        
+        return entities
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped extractor."""
+        return getattr(self._extractor, name)
+
+
+class RelationExtractorWithProvenance(ProvenanceMixin):
+    """
+    Relation extractor with provenance tracking.
+    
+    Wraps the original RelationExtractor and tracks all extracted relations.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize relation extractor with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original RelationExtractor
+        """
+        from .relation_extractor import RelationExtractor
+        
+        ProvenanceMixin.__init__(self, provenance=provenance)
+        self._extractor = RelationExtractor(**config)
+    
+    def extract(self, text: str, source: Optional[str] = None, **kwargs):
+        """
+        Extract relations with provenance tracking.
+        
+        Args:
+            text: Input text to extract relations from
+            source: Source document identifier (for provenance)
+            **kwargs: Additional arguments for extraction
+            
+        Returns:
+            List of extracted relations
+        """
+        relations = self._extractor.extract(text, **kwargs)
+        
+        if self.provenance:
+            for relation in relations:
+                relation_id = getattr(relation, 'id', None)
+                if not relation_id:
+                    relation_id = f"rel_{uuid.uuid4().hex[:8]}"
+                    try:
+                        relation.id = relation_id
+                    except AttributeError:
+                        pass
+                
+                self._track_extraction(
+                    entity_id=relation_id,
+                    source=source or text[:100],
+                    entity_type="relation",
+                    subject=relation.subject,
+                    predicate=relation.predicate,
+                    object=relation.object,
+                    confidence=getattr(relation, 'confidence', 1.0)
+                )
+        
+        return relations
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped extractor."""
+        return getattr(self._extractor, name)
+
+
+class EventDetectorWithProvenance(ProvenanceMixin):
+    """
+    Event detector with provenance tracking.
+    
+    Wraps the original EventDetector and tracks all detected events.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize event detector with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original EventDetector
+        """
+        from .event_detector import EventDetector
+        
+        ProvenanceMixin.__init__(self, provenance=provenance)
+        self._detector = EventDetector(**config)
+    
+    def detect(self, text: str, source: Optional[str] = None, **kwargs):
+        """
+        Detect events with provenance tracking.
+        
+        Args:
+            text: Input text to detect events from
+            source: Source document identifier (for provenance)
+            **kwargs: Additional arguments for detection
+            
+        Returns:
+            List of detected events
+        """
+        events = self._detector.detect(text, **kwargs)
+        
+        if self.provenance:
+            for event in events:
+                event_id = getattr(event, 'id', None)
+                if not event_id:
+                    event_id = f"event_{uuid.uuid4().hex[:8]}"
+                    try:
+                        event.id = event_id
+                    except AttributeError:
+                        pass
+                
+                self._track_extraction(
+                    entity_id=event_id,
+                    source=source or text[:100],
+                    entity_type="event",
+                    event_type=event.type,
+                    trigger=event.trigger,
+                    confidence=getattr(event, 'confidence', 1.0)
+                )
+        
+        return events
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped detector."""
+        return getattr(self._detector, name)
+
+
+class CoreferenceResolverWithProvenance(ProvenanceMixin):
+    """
+    Coreference resolver with provenance tracking.
+    
+    Wraps the original CoreferenceResolver and tracks coreference chains.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize coreference resolver with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original CoreferenceResolver
+        """
+        from .coreference_resolver import CoreferenceResolver
+        
+        ProvenanceMixin.__init__(self, provenance=provenance)
+        self._resolver = CoreferenceResolver(**config)
+    
+    def resolve(self, text: str, source: Optional[str] = None, **kwargs):
+        """
+        Resolve coreferences with provenance tracking.
+        
+        Args:
+            text: Input text to resolve coreferences
+            source: Source document identifier (for provenance)
+            **kwargs: Additional arguments for resolution
+            
+        Returns:
+            Coreference chains
+        """
+        chains = self._resolver.resolve(text, **kwargs)
+        
+        if self.provenance:
+            for chain in chains:
+                chain_id = getattr(chain, 'id', None)
+                if not chain_id:
+                    chain_id = f"coref_{uuid.uuid4().hex[:8]}"
+                    try:
+                        chain.id = chain_id
+                    except AttributeError:
+                        pass
+                
+                self._track_extraction(
+                    entity_id=chain_id,
+                    source=source or text[:100],
+                    entity_type="coreference_chain",
+                    mentions=len(chain.mentions) if hasattr(chain, 'mentions') else 0
+                )
+        
+        return chains
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped resolver."""
+        return getattr(self._resolver, name)
+
+
+class TripletExtractorWithProvenance(ProvenanceMixin):
+    """
+    Triplet extractor with provenance tracking.
+    
+    Wraps the original TripletExtractor and tracks all extracted triplets.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize triplet extractor with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original TripletExtractor
+        """
+        from .triplet_extractor import TripletExtractor
+        
+        ProvenanceMixin.__init__(self, provenance=provenance)
+        self._extractor = TripletExtractor(**config)
+    
+    def extract(self, text: str, source: Optional[str] = None, **kwargs):
+        """
+        Extract triplets with provenance tracking.
+        
+        Args:
+            text: Input text to extract triplets from
+            source: Source document identifier (for provenance)
+            **kwargs: Additional arguments for extraction
+            
+        Returns:
+            List of extracted triplets
+        """
+        triplets = self._extractor.extract(text, **kwargs)
+        
+        if self.provenance:
+            for triplet in triplets:
+                triplet_id = getattr(triplet, 'id', None)
+                if not triplet_id:
+                    triplet_id = f"triplet_{uuid.uuid4().hex[:8]}"
+                    try:
+                        triplet.id = triplet_id
+                    except AttributeError:
+                        pass
+                
+                self._track_extraction(
+                    entity_id=triplet_id,
+                    source=source or text[:100],
+                    entity_type="triplet",
+                    subject=triplet.subject,
+                    predicate=triplet.predicate,
+                    object=triplet.object,
+                    confidence=getattr(triplet, 'confidence', 1.0)
+                )
+        
+        return triplets
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped extractor."""
+        return getattr(self._extractor, name)
+
+
+# Convenience exports
+__all__ = [
+    'NERExtractorWithProvenance',
+    'RelationExtractorWithProvenance',
+    'EventDetectorWithProvenance',
+    'CoreferenceResolverWithProvenance',
+    'TripletExtractorWithProvenance',
+    'ProvenanceMixin',
+]

semantica/semantic_extract/semantic_extract_usage.md

@@ -41,6 +41,7 @@ print(f"Extracted {len(entities)} entities and {len(relations)} relations")
 All extractors support batch processing for high-throughput extraction. You can pass a list of strings or a list of dictionaries (with `content` and `id` keys).
 
 **Features:**
+- **Parallel Processing**: Multi-threaded extraction for high throughput (control via `max_workers`).
 - **Progress Tracking**: Automatically shows a progress bar for large batches.
 - **Provenance Metadata**: Each extracted item includes `batch_index` and `document_id` in its `metadata`.
 
@@ -52,9 +53,13 @@ documents = [
     {"id": "doc_2", "content": "Microsoft Corporation was founded by Bill Gates."}
 ]
 
-extractor = NERExtractor()
+# Initialize with parallel processing enabled
+extractor = NERExtractor(max_workers=4)
 batch_results = extractor.extract(documents)
 
+# OR override during extraction call
+# batch_results = extractor.extract(documents, max_workers=8)
+
 for i, doc_entities in enumerate(batch_results):
     print(f"Document {i} entities:")
     for entity in doc_entities:
@@ -108,9 +113,17 @@ extractor = NERExtractor(method="ml")
 entities = extractor.extract(text)
 print(f"ML method: {len(entities)} entities")
 
-# HuggingFace model extraction
+# HuggingFace model extraction (Bring Your Own Model)
 extractor = NERExtractor(method="huggingface")
-entities = extractor.extract(text, model="dslim/bert-base-NER")
+
+# Use a specific model and aggregation strategy at runtime
+# Runtime options override configuration defaults
+entities = extractor.extract(
+    text, 
+    model="dslim/bert-base-NER", 
+    aggregation_strategy="max", # Options: "simple", "first", "average", "max"
+    device="cpu" # or "cuda"
+)
 print(f"HuggingFace method: {len(entities)} entities")
 
 # LLM-based extraction with advanced options
@@ -120,9 +133,22 @@ entities = extractor.extract(
     provider="openai", 
     model="gpt-4",
     silent_fail=False,      # Raise ProcessingError on failure (default)
-    max_text_length=4000     # Auto-chunking for long text
+    max_text_length=4000,   # Auto-chunking for long text (default: 64k for major providers)
+    max_tokens=4096,        # Explicitly control generation output length
+    temperature=0.0
 )
 print(f"LLM method: {len(entities)} entities")
+
+# Groq extraction with long context support
+# Groq defaults to 64k chunking limit for models like llama-3.3-70b
+groq_extractor = NERExtractor(method="llm")
+groq_entities = groq_extractor.extract(
+    text,
+    provider="groq",
+    model="llama-3.3-70b-versatile",
+    max_tokens=8000 # Passed directly to Groq API
+)
+print(f"Groq method: {len(groq_entities)} entities")
 ```
 
 ### Using NERExtractor Directly
@@ -211,9 +237,18 @@ relations = extractor.extract(text, entities=entities)
 extractor = RelationExtractor(method="cooccurrence")
 relations = extractor.extract(text, entities=entities)
 
-# HuggingFace model
+# HuggingFace model (Bring Your Own Model)
 extractor = RelationExtractor(method="huggingface")
-relations = extractor.extract(text, entities=entities, model="microsoft/DialoGPT-medium")
+
+# Use a sequence classification model trained for relations
+# The extractor automatically formats input with entity markers: 
+# "Steve Jobs founded Apple" -> "<subj> Steve Jobs </subj> founded <obj> Apple </obj>"
+relations = extractor.extract(
+    text, 
+    entities=entities, 
+    model="semantica/relation-model-v1", # Replace with your model ID
+    device="cpu"
+)
 
 # LLM-based relation extraction
 extractor = RelationExtractor(method="llm")
@@ -221,6 +256,8 @@ relations = extractor.extract(
     text, 
     entities=entities, 
     provider="openai",
+    model="gpt-4",
+    max_tokens=2048, # Increased output limit for many relations
     silent_fail=True  # Return empty list if extraction fails
 )
 ```
@@ -274,9 +311,16 @@ triplets = extractor.extract_triplets(text)
 extractor = TripletExtractor(method="rules")
 triplets = extractor.extract_triplets(text)
 
-# HuggingFace model
+# HuggingFace model (Seq2Seq / REBEL)
 extractor = TripletExtractor(method="huggingface")
-triplets = extractor.extract_triplets(text, model="t5-base")
+
+# Use a Seq2Seq model like REBEL for end-to-end triplet extraction
+# This method generates triplets directly from text without needing separate NER/RE steps
+triplets = extractor.extract_triplets(
+    text, 
+    model="Babelscape/rebel-large",
+    device="cpu"
+)
 
 # LLM-based triplet extraction
 extractor = TripletExtractor(method="llm")
@@ -284,7 +328,8 @@ triplets = extractor.extract_triplets(
     text, 
     provider="openai", 
     model="gpt-4",
-    max_text_length=2000  # Force chunking for long text
+    max_text_length=64000, # Large default chunk size supported
+    max_tokens=4096 # Ensure enough tokens for all triplets
 )
 ```
 

semantica/semantic_extract/semantic_network_extractor.py

@@ -149,6 +149,9 @@ class SemanticNetworkExtractor:
             self.config["ner_method"] = method
             self.config["relation_method"] = method
 
+        self._ner_extractor = None
+        self._relation_extractor = None
+
     def extract(
         self,
         text: Union[str, List[str], List[Dict[str, Any]]],
@@ -181,8 +184,9 @@ class SemanticNetworkExtractor:
             )
 
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -198,54 +202,113 @@ class SemanticNetworkExtractor:
                     message=f"Starting batch extraction... 0/{total_items} (remaining: {total_items})"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
-                    
-                    doc_entities = None
-                    if entities and isinstance(entities, list) and idx < len(entities):
-                        doc_entities = entities[idx]
-                    
-                    doc_relations = None
-                    if relations and isinstance(relations, list) and idx < len(relations):
-                        doc_relations = relations[idx]
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                )
 
-                    # Extract
-                    network = self.extract_network(
-                        doc_text, 
-                        entities=doc_entities, 
-                        relations=doc_relations, 
-                        **kwargs
-                    )
-
-                    # Add provenance metadata to nodes and edges
-                    batch_meta = {"batch_index": idx}
-                    if isinstance(item, dict) and "id" in item:
-                        batch_meta["document_id"] = item["id"]
-                    
-                    # Update network metadata
-                    network.metadata.update(batch_meta)
-                    
-                    # Update nodes metadata
-                    for node in network.nodes:
-                        node.metadata.update(batch_meta)
+                def process_item(idx, item, doc_entities, doc_relations):
+                    try:
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
                         
-                    # Update edges metadata
-                    for edge in network.edges:
-                        edge.metadata.update(batch_meta)
-
-                    results.append(network)
-
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining})"
+                        # Extract
+                        network = self.extract_network(
+                            doc_text, 
+                            entities=doc_entities, 
+                            relations=doc_relations, 
+                            **kwargs
                         )
 
+                        # Add provenance metadata to nodes and edges
+                        batch_meta = {"batch_index": idx}
+                        if isinstance(item, dict) and "id" in item:
+                            batch_meta["document_id"] = item["id"]
+                        
+                        # Update network metadata
+                        network.metadata.update(batch_meta)
+                        
+                        # Update nodes metadata
+                        for node in network.nodes:
+                            node.metadata.update(batch_meta)
+                            
+                        # Update edges metadata
+                        for edge in network.edges:
+                            edge.metadata.update(batch_meta)
+                            
+                        return idx, network
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {idx}: {e}")
+                        verbose_mode = kwargs.get("verbose", False) or self.config.get("verbose", False)
+                        if verbose_mode:
+                            import sys
+                            print(f"    [SemanticNetworkExtractor] ERROR: Batch item {idx} failed: {e}", flush=True, file=sys.stderr)
+                        return idx, None
+
+                if max_workers > 1:
+                    import concurrent.futures
+                    
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {}
+                        for idx, item in enumerate(text):
+                            doc_entities = None
+                            if entities and isinstance(entities, list) and idx < len(entities):
+                                doc_entities = entities[idx]
+                            
+                            doc_relations = None
+                            if relations and isinstance(relations, list) and idx < len(relations):
+                                doc_relations = relations[idx]
+                                
+                            future = executor.submit(process_item, idx, item, doc_entities, doc_relations)
+                            future_to_idx[future] = idx
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, network = future.result()
+                            if network:
+                                results[idx] = network
+                            
+                            processed_count += 1
+                            
+                            # Update progress
+                            if (processed_count) % update_interval == 0 or (processed_count) == total_items:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        doc_entities = None
+                        if entities and isinstance(entities, list) and idx < len(entities):
+                            doc_entities = entities[idx]
+                        
+                        doc_relations = None
+                        if relations and isinstance(relations, list) and idx < len(relations):
+                            doc_relations = relations[idx]
+
+                        _, network = process_item(idx, item, doc_entities, doc_relations)
+                        if network:
+                            results[idx] = network
+
+                        processed_count += 1
+                        
+                        # Update progress
+                        if (processed_count) % update_interval == 0 or (processed_count) == total_items:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining})"
+                            )
+
+                # Filter out None results if any failed
+                results = [r for r in results if r is not None]
+
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
@@ -265,11 +328,12 @@ class SemanticNetworkExtractor:
 
     def extract_network(
         self,
-        text: str,
-        entities: Optional[List[Entity]] = None,
-        relations: Optional[List[Relation]] = None,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        entities: Optional[Union[List[Entity], List[List[Entity]]]] = None,
+        relations: Optional[Union[List[Relation], List[List[Relation]]]] = None,
+        pipeline_id: Optional[str] = None,
         **options,
-    ) -> SemanticNetwork:
+    ) -> Union[SemanticNetwork, List[SemanticNetwork]]:
         """
         Extract semantic network from text.
 
@@ -282,6 +346,23 @@ class SemanticNetworkExtractor:
         Returns:
             SemanticNetwork: Extracted semantic network
         """
+        if isinstance(text, list):
+            entities_batch = entities
+            if entities is not None and isinstance(entities, list) and (not entities or all(isinstance(e, Entity) for e in entities)):
+                entities_batch = [entities for _ in range(len(text))] if entities else [[] for _ in range(len(text))]
+
+            relations_batch = relations
+            if relations is not None and isinstance(relations, list) and (not relations or all(isinstance(r, Relation) for r in relations)):
+                relations_batch = [relations for _ in range(len(text))] if relations else [[] for _ in range(len(text))]
+
+            return self.extract(
+                text,
+                entities=entities_batch,
+                relations=relations_batch,
+                pipeline_id=pipeline_id,
+                **options,
+            )
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="SemanticNetworkExtractor",
@@ -301,15 +382,16 @@ class SemanticNetworkExtractor:
                 # Pass method if specified
                 if "ner_method" in self.config:
                     ner_config["method"] = self.config["ner_method"]
-                ner = NERExtractor(
-                    **ner_config,
-                    **{
-                        k: v
-                        for k, v in self.config.items()
-                        if k not in ["ner", "relation"]
-                    },
-                )
-                entities = ner.extract_entities(text, **options)
+                if self._ner_extractor is None:
+                    self._ner_extractor = NERExtractor(
+                        **ner_config,
+                        **{
+                            k: v
+                            for k, v in self.config.items()
+                            if k not in ["ner", "relation"]
+                        },
+                    )
+                entities = self._ner_extractor.extract_entities(text, **options)
 
             # Extract relations if not provided
             if relations is None:
@@ -320,15 +402,16 @@ class SemanticNetworkExtractor:
                 # Pass method if specified
                 if "relation_method" in self.config:
                     rel_config["method"] = self.config["relation_method"]
-                rel_extractor = RelationExtractor(
-                    **rel_config,
-                    **{
-                        k: v
-                        for k, v in self.config.items()
-                        if k not in ["ner", "relation"]
-                    },
-                )
-                relations = rel_extractor.extract_relations(text, entities, **options)
+                if self._relation_extractor is None:
+                    self._relation_extractor = RelationExtractor(
+                        **rel_config,
+                        **{
+                            k: v
+                            for k, v in self.config.items()
+                            if k not in ["ner", "relation"]
+                        },
+                    )
+                relations = self._relation_extractor.extract_relations(text, entities, **options)
 
             # Build network
             total_steps = 2  # Create nodes, create edges
@@ -355,6 +438,12 @@ class SemanticNetworkExtractor:
             self.progress_tracker.stop_tracking(
                 tracking_id, status="failed", message=str(e)
             )
+            verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+            if verbose_mode:
+                import sys
+                print(f"    [SemanticNetworkExtractor] ERROR: Extraction failed: {e}", flush=True, file=sys.stderr)
+                import traceback
+                traceback.print_exc(file=sys.stderr)
             raise
 
     def _build_network(

semantica/semantic_extract/triplet_extractor.py

@@ -143,6 +143,13 @@ class TripletExtractor:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        if method is not None:
+            self.config["ner_method"] = method
+            self.config["relation_method"] = method
+
+        self._ner_extractor = None
+        self._relation_extractor = None
+
         # Store parameters
         self.triplet_types = triplet_types
         self.include_temporal = include_temporal
@@ -191,9 +198,10 @@ class TripletExtractor:
             )
 
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
                 total_triplets_count = 0
+                processed_count = 0
                 
                 # Determine update interval
                 if total_items <= 10:
@@ -206,50 +214,91 @@ class TripletExtractor:
                     tracking_id,
                     processed=0,
                     total=total_items,
-                    message=f"Starting batch extraction... 0/{total_items} (remaining: {total_items})"
+                    message=f"Starting batch extraction... 0/{total_items}"
                 )
 
-                for idx, item in enumerate(text):
-                    # Prepare arguments for single item
-                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
-                    
-                    doc_entities = None
-                    if entities and isinstance(entities, list) and idx < len(entities):
-                        doc_entities = entities[idx]
-                    
-                    doc_relations = None
-                    if relations and isinstance(relations, list) and idx < len(relations):
-                        doc_relations = relations[idx]
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=self.method,
+                )
 
-                    # Extract
-                    current_triplets = self.extract_triplets(
-                        doc_text, 
-                        entities=doc_entities, 
-                        relations=doc_relations, 
-                        **kwargs
-                    )
+                def process_item(idx, item):
+                    try:
+                        # Prepare arguments for single item
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                        
+                        doc_entities = None
+                        if entities and isinstance(entities, list) and idx < len(entities):
+                            doc_entities = entities[idx]
+                        
+                        doc_relations = None
+                        if relations and isinstance(relations, list) and idx < len(relations):
+                            doc_relations = relations[idx]
 
-                    # Add provenance metadata
-                    for triplet in current_triplets:
-                        if triplet.metadata is None:
-                            triplet.metadata = {}
-                        triplet.metadata["batch_index"] = idx
-                        if isinstance(item, dict) and "id" in item:
-                            triplet.metadata["document_id"] = item["id"]
-
-                    results.append(current_triplets)
-                    total_triplets_count += len(current_triplets)
-
-                    # Update progress
-                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
-                        remaining = total_items - (idx + 1)
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx + 1,
-                            total=total_items,
-                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining}) - Extracted {total_triplets_count} triplets"
+                        # Extract
+                        current_triplets = self.extract_triplets(
+                            doc_text, 
+                            entities=doc_entities, 
+                            relations=doc_relations, 
+                            **kwargs
                         )
 
+                        # Add provenance metadata
+                        for triplet in current_triplets:
+                            if triplet.metadata is None:
+                                triplet.metadata = {}
+                            triplet.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                triplet.metadata["document_id"] = item["id"]
+                        
+                        return idx, current_triplets
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {idx}: {e}")
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
+                    
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {
+                            executor.submit(process_item, idx, item): idx 
+                            for idx, item in enumerate(text)
+                        }
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, triplets = future.result()
+                            results[idx] = triplets
+                            total_triplets_count += len(triplets)
+                            processed_count += 1
+                            
+                            if processed_count % update_interval == 0 or processed_count == total_items:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_triplets_count} triplets"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, triplets = process_item(idx, item)
+                        results[idx] = triplets
+                        total_triplets_count += len(triplets)
+                        processed_count += 1
+                        
+                        if processed_count % update_interval == 0 or processed_count == total_items:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_triplets_count} triplets"
+                            )
+
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
@@ -269,11 +318,12 @@ class TripletExtractor:
 
     def extract_triplets(
         self,
-        text: str,
-        entities: Optional[List[Entity]] = None,
-        relations: Optional[List[Relation]] = None,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        entities: Optional[Union[List[Entity], List[List[Entity]]]] = None,
+        relations: Optional[Union[List[Relation], List[List[Relation]]]] = None,
+        pipeline_id: Optional[str] = None,
         **options,
-    ) -> List[Triplet]:
+    ) -> Union[List[Triplet], List[List[Triplet]]]:
         """
         Extract RDF triplets from text.
 
@@ -281,11 +331,29 @@ class TripletExtractor:
             text: Input text
             entities: Pre-extracted entities (optional)
             relations: Pre-extracted relations (optional)
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Extraction options
 
         Returns:
             list: List of extracted triplets
         """
+        if isinstance(text, list):
+            entities_batch = entities
+            if entities is not None and isinstance(entities, list) and (not entities or all(isinstance(e, Entity) for e in entities)):
+                entities_batch = [entities for _ in range(len(text))] if entities else [[] for _ in range(len(text))]
+
+            relations_batch = relations
+            if relations is not None and isinstance(relations, list) and (not relations or all(isinstance(r, Relation) for r in relations)):
+                relations_batch = [relations for _ in range(len(text))] if relations else [[] for _ in range(len(text))]
+
+            return self.extract(
+                text,
+                entities=entities_batch,
+                relations=relations_batch,
+                pipeline_id=pipeline_id,
+                **options,
+            )
+
         from .methods import get_triplet_method
 
         tracking_id = self.progress_tracker.start_tracking(
@@ -298,27 +366,62 @@ class TripletExtractor:
             from .ner_extractor import NERExtractor
             from .relation_extractor import RelationExtractor
 
-            # Extract entities if not provided
-            if entities is None:
-                self.progress_tracker.update_tracking(
-                    tracking_id, message="Extracting entities..."
-                )
-                ner = NERExtractor(**self.config.get("ner", {}))
-                entities = ner.extract_entities(text)
-
-            # Extract relations if not provided
-            if relations is None:
-                self.progress_tracker.update_tracking(
-                    tracking_id, message="Extracting relations..."
-                )
-                rel_extractor = RelationExtractor(**self.config.get("relation", {}))
-                relations = rel_extractor.extract_relations(text, entities)
-
             # Use method-based extraction
             methods = options.get("method", self.method)
             if isinstance(methods, str):
                 methods = [methods]
 
+            # Determine if we need to extract entities/relations based on method
+            # HuggingFace (Seq2Seq) does not need pre-extracted entities/relations
+            needs_entities_relations = any(m not in ["huggingface"] for m in methods)
+
+            # Extract entities if not provided
+            if entities is None and needs_entities_relations:
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Extracting entities..."
+                )
+                if self._ner_extractor is None:
+                    ner_config = self.config.get("ner", {})
+                    if "ner_method" in self.config:
+                        ner_config = {**ner_config, "method": self.config["ner_method"]}
+                    
+                    # Filter out 'model' and 'huggingface_model' from shared config
+                    # to prevent passing triplet model to NER extractor
+                    shared_config = {
+                        k: v
+                        for k, v in self.config.items()
+                        if k not in ["ner", "relation", "validator", "serializer", "quality", "model", "huggingface_model"]
+                    }
+
+                    self._ner_extractor = NERExtractor(
+                        **ner_config,
+                        **shared_config,
+                    )
+                entities = self._ner_extractor.extract_entities(text)
+
+            # Extract relations if not provided
+            if relations is None and needs_entities_relations:
+                self.progress_tracker.update_tracking(
+                    tracking_id, message="Extracting relations..."
+                )
+                if self._relation_extractor is None:
+                    rel_config = self.config.get("relation", {})
+                    if "relation_method" in self.config:
+                        rel_config = {**rel_config, "method": self.config["relation_method"]}
+                    
+                    # Filter out 'model' and 'huggingface_model' from shared config
+                    shared_config = {
+                        k: v
+                        for k, v in self.config.items()
+                        if k not in ["ner", "relation", "validator", "serializer", "quality", "model", "huggingface_model"]
+                    }
+
+                    self._relation_extractor = RelationExtractor(
+                        **rel_config,
+                        **shared_config,
+                    )
+                relations = self._relation_extractor.extract_relations(text, entities)
+
             triplet_types = options.get("triplet_types", self.triplet_types)
             
             # Merge config with options
@@ -360,8 +463,12 @@ class TripletExtractor:
                         method_options["triplet_types"] = triplet_types
 
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", all_options.get("model")
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model")
+                            or self.config.get("huggingface_model") 
+                            or self.config.get("model")
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -371,18 +478,28 @@ class TripletExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key
 
+                    # Print progress if verbose mode is enabled (only for LLM method to avoid spam)
+                    verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+                    if verbose_mode and method_name == "llm":
+                        import sys
+                        print(f"    [TripletExtractor] Processing with {method_name}...", flush=True, file=sys.stdout)
+                        if "api_key" in method_options:
+                             masked = method_options["api_key"][:4] + "..." if method_options["api_key"] else "None"
+                             print(f"    [TripletExtractor Debug] api_key present: {masked}", flush=True, file=sys.stdout)
+                        else:
+                             print(f"    [TripletExtractor Debug] api_key NOT present", flush=True, file=sys.stdout)
+
                     triplets = method_func(
                         text,
                         entities=entities,
@@ -390,6 +507,11 @@ class TripletExtractor:
                         **method_options,
                     )
 
+                    # Print result count if verbose (only for LLM method)
+                    if verbose_mode and method_name == "llm" and len(triplets) > 0:
+                        import sys
+                        print(f"    [TripletExtractor] Extracted {len(triplets)} triplets", flush=True, file=sys.stdout)
+
                     # Apply weighted scoring if triplet_types are provided
                     if triplet_types:
                         try:
@@ -425,6 +547,12 @@ class TripletExtractor:
 
                 except Exception as e:
                     self.logger.warning(f"Method {method_name} failed: {e}")
+                    verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+                    if verbose_mode:
+                        import sys
+                        print(f"    [TripletExtractor] ERROR: Method {method_name} failed: {e}", flush=True, file=sys.stderr)
+                        import traceback
+                        traceback.print_exc(file=sys.stderr)
                     continue
 
             # Use first successful method or fallback to relation conversion

semantica/split/provenance_tracker.py

@@ -1,9 +1,17 @@
 """
-Provenance Tracker Module
+Provenance Tracker Module (Enhanced with Unified Backend)
 
 This module provides comprehensive source tracking for document chunks,
 maintaining data lineage and traceability throughout the chunking process.
 
+IMPORTANT: This module now uses the unified semantica.provenance.ProvenanceManager
+backend for enhanced W3C PROV-O compliance and audit-grade tracking. All existing
+APIs remain 100% backward compatible.
+
+For new code, consider using the unified API:
+    >>> from semantica.provenance import ProvenanceManager
+    >>> prov_mgr = ProvenanceManager()
+
 Key Features:
     - Chunk source tracking
     - Document lineage management
@@ -11,9 +19,11 @@ Key Features:
     - Chunk linking and relationships
     - Provenance export
     - Version tracking support
+    - W3C PROV-O compliance (when using unified backend)
+    - Audit-grade integrity verification
 
 Main Classes:
-    - ProvenanceTracker: Main provenance tracking coordinator
+    - ProvenanceTracker: Main provenance tracking coordinator (backward compatible wrapper)
     - ProvenanceInfo: Provenance information representation dataclass
 
 Example Usage:
@@ -36,6 +46,13 @@ from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
 from .semantic_chunker import Chunk
 
+# Import unified provenance manager
+try:
+    from ..provenance import ProvenanceManager as UnifiedProvenanceManager
+    UNIFIED_AVAILABLE = True
+except ImportError:
+    UNIFIED_AVAILABLE = False
+
 
 @dataclass
 class ProvenanceInfo:
@@ -53,7 +70,12 @@ class ProvenanceInfo:
 
 
 class ProvenanceTracker:
-    """Provenance tracker for chunk source tracking."""
+    """
+    Provenance tracker for chunk source tracking (Enhanced with Unified Backend).
+    
+    This class now wraps semantica.provenance.ProvenanceManager for enhanced
+    W3C PROV-O compliance while maintaining 100% API compatibility.
+    """
 
     def __init__(self, **config):
         """
@@ -63,6 +85,7 @@ class ProvenanceTracker:
             **config: Configuration options:
                 - store_metadata: Store chunk metadata (default: True)
                 - track_versions: Track version history (default: False)
+                - storage_path: Path to SQLite database (optional)
         """
         self.logger = get_logger("provenance_tracker")
         self.config = config
@@ -74,9 +97,22 @@ class ProvenanceTracker:
         self.store_metadata = config.get("store_metadata", True)
         self.track_versions = config.get("track_versions", False)
 
-        # In-memory store (could be replaced with database)
-        self._provenance_store: Dict[str, ProvenanceInfo] = {}
-        self._chunk_registry: Dict[str, str] = {}  # chunk_id -> provenance_id
+        # Determine whether to use unified backend
+        use_unified = config.get("use_unified", True) and UNIFIED_AVAILABLE
+        
+        if use_unified:
+            # Use unified provenance manager
+            storage_path = config.get("storage_path")
+            self._unified_manager = UnifiedProvenanceManager(storage_path=storage_path)
+            self._use_unified = True
+            self.logger.debug("Chunk provenance tracker initialized with unified backend")
+        else:
+            # Fallback to legacy in-memory storage
+            self._unified_manager = None
+            self._use_unified = False
+            self._provenance_store: Dict[str, ProvenanceInfo] = {}
+            self._chunk_registry: Dict[str, str] = {}  # chunk_id -> provenance_id
+            self.logger.debug("Chunk provenance tracker initialized with legacy backend")
 
     def track_chunk(
         self,
@@ -107,6 +143,43 @@ class ProvenanceTracker:
             except AttributeError:
                 pass  # Chunk might be immutable
         
+        if self._use_unified:
+            # Delegate to unified manager
+            try:
+                chunk_metadata = {**chunk.metadata, **metadata, "chunk_size": len(chunk.text)} if self.store_metadata else metadata
+                self._unified_manager.track_chunk(
+                    chunk_id=chunk_id,
+                    source_document=source_document,
+                    source_path=source_path,
+                    start_index=chunk.start_index,
+                    end_index=chunk.end_index,
+                    parent_chunk_id=parent_chunk_id,
+                    **chunk_metadata
+                )
+                return chunk_id  # Return chunk_id for compatibility
+            except Exception as e:
+                self.logger.warning(f"Unified tracking failed, using fallback: {e}")
+                return self._track_chunk_legacy(chunk, source_document, source_path, parent_chunk_id, **metadata)
+        else:
+            return self._track_chunk_legacy(chunk, source_document, source_path, parent_chunk_id, **metadata)
+    
+    def _track_chunk_legacy(
+        self,
+        chunk: Chunk,
+        source_document: str,
+        source_path: Optional[str] = None,
+        parent_chunk_id: Optional[str] = None,
+        **metadata,
+    ) -> str:
+        """Legacy chunk tracking implementation."""
+        chunk_id = getattr(chunk, "id", None)
+        if not chunk_id:
+            chunk_id = str(uuid4())
+            try:
+                chunk.id = chunk_id
+            except AttributeError:
+                pass
+        
         provenance_id = str(uuid4())
 
         provenance_info = ProvenanceInfo(
@@ -214,6 +287,31 @@ class ProvenanceTracker:
         Returns:
             ProvenanceInfo: Provenance information or None
         """
+        if self._use_unified:
+            try:
+                prov = self._unified_manager.get_provenance(chunk_id)
+                if prov:
+                    # Convert to ProvenanceInfo for backward compatibility
+                    return ProvenanceInfo(
+                        chunk_id=chunk_id,
+                        source_document=prov.get("source_document", ""),
+                        source_path=prov.get("source_location"),
+                        start_index=prov.get("start_index", 0),
+                        end_index=prov.get("end_index", 0),
+                        parent_chunk_id=prov.get("parent_entity_id"),
+                        metadata=prov.get("metadata", {}),
+                        version=prov.get("version", "1.0"),
+                        timestamp=prov.get("timestamp")
+                    )
+                return None
+            except Exception as e:
+                self.logger.warning(f"Unified retrieval failed, using fallback: {e}")
+                return self._get_provenance_legacy(chunk_id)
+        else:
+            return self._get_provenance_legacy(chunk_id)
+    
+    def _get_provenance_legacy(self, chunk_id: str) -> Optional[ProvenanceInfo]:
+        """Legacy get provenance implementation."""
         provenance_id = self._chunk_registry.get(chunk_id)
         if provenance_id:
             return self._provenance_store.get(provenance_id)
@@ -245,11 +343,38 @@ class ProvenanceTracker:
         Returns:
             list: Lineage chain (oldest to newest)
         """
+        if self._use_unified:
+            try:
+                lineage_entries = self._unified_manager.trace_lineage(chunk_id)
+                # Convert to ProvenanceInfo list for backward compatibility
+                return [
+                    ProvenanceInfo(
+                        chunk_id=entry.entity_id,
+                        source_document=entry.source_document,
+                        source_path=entry.source_location,
+                        start_index=entry.start_index or 0,
+                        end_index=entry.end_index or 0,
+                        parent_chunk_id=entry.parent_entity_id,
+                        metadata=entry.metadata,
+                        version=entry.version,
+                        timestamp=entry.timestamp
+                    )
+                    for entry in lineage_entries
+                    if entry.entity_type == "chunk"
+                ]
+            except Exception as e:
+                self.logger.warning(f"Unified lineage retrieval failed, using fallback: {e}")
+                return self._get_chunk_lineage_legacy(chunk_id)
+        else:
+            return self._get_chunk_lineage_legacy(chunk_id)
+    
+    def _get_chunk_lineage_legacy(self, chunk_id: str) -> List[ProvenanceInfo]:
+        """Legacy get chunk lineage implementation."""
         lineage = []
         current_chunk_id = chunk_id
 
         while current_chunk_id:
-            provenance = self.get_provenance(current_chunk_id)
+            provenance = self._get_provenance_legacy(current_chunk_id)
             if not provenance:
                 break
 

semantica/triplet_store/jena_store.py

@@ -132,7 +132,7 @@ class JenaStore:
         )
 
         try:
-            if not self.graph:
+            if self.graph is None:
                 self.progress_tracker.stop_tracking(
                     tracking_id, status="failed", message="Graph not initialized"
                 )
@@ -175,14 +175,14 @@ class JenaStore:
         return self.add_triplets([triplet], **options)
 
     def get_triplets(
-        self,
-        subject: Optional[str] = None,
-        predicate: Optional[str] = None,
-        object: Optional[str] = None,
-        **options,
+            self,
+            subject: Optional[str] = None,
+            predicate: Optional[str] = None,
+            object: Optional[str] = None,
+            **options,
     ) -> List[Triplet]:
         """Get triplets matching criteria."""
-        if not self.graph:
+        if self.graph is None:
             return []
 
         try:
@@ -218,7 +218,7 @@ class JenaStore:
 
     def delete_triplet(self, triplet: Triplet, **options) -> Dict[str, Any]:
         """Delete triplet."""
-        if not self.graph:
+        if self.graph is None:
             raise ProcessingError("Graph not initialized")
 
         try:
@@ -273,7 +273,7 @@ class JenaStore:
         Returns:
             Query results
         """
-        if not self.graph:
+        if self.graph is None:
             raise ProcessingError("Graph not initialized")
 
         try:
@@ -319,7 +319,7 @@ class JenaStore:
         Returns:
             Serialized RDF string
         """
-        if not self.graph:
+        if self.graph is None:
             return ""
 
         try:

semantica/triplet_store/triplet_store_provenance.py

@@ -0,0 +1,59 @@
+"""
+Provenance-enabled wrapper for triplet storage.
+
+Tracks: triplets stored
+
+Usage:
+    from semantica.triplet_store.triplet_store_provenance import TripletStoreWithProvenance
+    
+    store = TripletStoreWithProvenance(provenance=True)
+    store.add_triplet(subject, predicate, object, source="kg.json")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class TripletStoreWithProvenance:
+    """Triplet store with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .triplet_store import TripletStore
+        
+        self.provenance = provenance
+        self._store = TripletStore(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def add_triplet(self, subject: Any, predicate: Any, obj: Any, source: str = None, **kwargs):
+        """Add triplet with provenance tracking."""
+        result = self._store.add_triplet(subject, predicate, obj, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"triplet_{uuid.uuid4().hex[:8]}",
+                source=source or "triplet_store",
+                entity_type="triplet",
+                metadata={
+                    "subject": str(subject),
+                    "predicate": str(predicate),
+                    "object": str(obj)
+                }
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._store, name)
+
+
+__all__ = ['TripletStoreWithProvenance']

semantica/utils/progress_tracker.py

@@ -34,6 +34,7 @@ License: MIT
 """
 
 import inspect
+import os
 import sys
 import threading
 import time
@@ -46,6 +47,13 @@ from typing import Any, Callable, Dict, List, Optional, Tuple, Union
 
 from .logging import get_logger
 
+DISABLE_JUPYTER_PROGRESS = os.getenv("SEMANTICA_DISABLE_JUPYTER_PROGRESS", "").strip().lower() in (
+    "1",
+    "true",
+    "yes",
+    "on",
+)
+
 # Try to import IPython for Jupyter support
 try:
     from IPython import get_ipython
@@ -1017,6 +1025,7 @@ class ProgressTracker:
 
         # Detect environment - will be checked dynamically
         self.is_jupyter = self._detect_jupyter()
+        self.disable_jupyter_progress = DISABLE_JUPYTER_PROGRESS
 
         # Create displays
         self.displays: List[ProgressDisplay] = []
@@ -1024,7 +1033,7 @@ class ProgressTracker:
         # Always try Jupyter first if available, fallback to console
         if IPYTHON_AVAILABLE:
             # Try to detect Jupyter - if available, use it
-            if self.is_jupyter:
+            if self.is_jupyter and not self.disable_jupyter_progress:
                 self.displays.append(JupyterProgressDisplay(use_emoji=use_emoji))
             # Also add console as fallback for immediate feedback
             self.displays.append(
@@ -1213,7 +1222,11 @@ class ProgressTracker:
         if IPYTHON_AVAILABLE and not self.is_jupyter:
             self.is_jupyter = self._detect_jupyter()
             # If Jupyter is now detected and we don't have a Jupyter display, add it
-            if self.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays):
+            if (
+                self.is_jupyter
+                and not self.disable_jupyter_progress
+                and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays)
+            ):
                 # Insert Jupyter display at the beginning for priority
                 self.displays.insert(0, JupyterProgressDisplay(use_emoji=self.use_emoji))
 
@@ -1341,7 +1354,11 @@ class ProgressTracker:
         if IPYTHON_AVAILABLE and not self.is_jupyter:
             self.is_jupyter = self._detect_jupyter()
             # If Jupyter is now detected and we don't have a Jupyter display, add it
-            if self.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays):
+            if (
+                self.is_jupyter
+                and not self.disable_jupyter_progress
+                and not any(isinstance(d, JupyterProgressDisplay) for d in self.displays)
+            ):
                 # Insert Jupyter display at the beginning for priority
                 self.displays.insert(0, JupyterProgressDisplay(use_emoji=self.use_emoji))
 
@@ -1526,7 +1543,11 @@ def get_progress_tracker() -> ProgressTracker:
     if IPYTHON_AVAILABLE and not _global_tracker.is_jupyter:
         _global_tracker.is_jupyter = _global_tracker._detect_jupyter()
         # If Jupyter is now detected and we don't have a Jupyter display, add it
-        if _global_tracker.is_jupyter and not any(isinstance(d, JupyterProgressDisplay) for d in _global_tracker.displays):
+        if (
+            _global_tracker.is_jupyter
+            and not _global_tracker.disable_jupyter_progress
+            and not any(isinstance(d, JupyterProgressDisplay) for d in _global_tracker.displays)
+        ):
             # Insert Jupyter display at the beginning for priority
             _global_tracker.displays.insert(0, JupyterProgressDisplay(use_emoji=_global_tracker.use_emoji))
     

semantica/vector_store/__init__.py

@@ -3,7 +3,7 @@ Vector Store Management Module
 
 This module provides comprehensive vector storage and retrieval capabilities for the
 Semantica framework, including support for multiple vector store backends (FAISS,
-Weaviate, Qdrant, Milvus), hybrid search combining vector similarity and
+Weaviate, Qdrant, Pinecone, Milvus), hybrid search combining vector similarity and
 metadata filtering, metadata management, and namespace isolation.
 
 Algorithms Used:
@@ -58,6 +58,7 @@ Supported Backends:
     - FAISS: In-memory/local disk (Facebook AI Similarity Search)
     - Weaviate: Cloud/Self-hosted (Schema-aware vector database)
     - Qdrant: Cloud/Self-hosted (Vector database for the next generation of AI)
+    - Pinecone: Cloud-managed (Managed vector database service)
     - Milvus: Cloud/Self-hosted (Highly scalable vector database)
     - InMemory: Simple list-based storage for testing/small datasets
 
@@ -73,7 +74,7 @@ Dependencies:
     - pymilvus
 
 Key Features:
-    - Multi-backend vector store support (FAISS, Weaviate, Qdrant, Milvus)
+    - Multi-backend vector store support (FAISS, Weaviate, Qdrant, Pinecone, Milvus)
     - Vector indexing and similarity search
     - Metadata indexing and filtering
     - Hybrid search combining vector and metadata queries
@@ -91,6 +92,7 @@ Main Classes:
     - FAISSStore: FAISS integration for local vector storage
     - WeaviateStore: Weaviate vector database integration
     - QdrantStore: Qdrant vector database integration
+    - PineconeStore: Pinecone vector database integration
     - MilvusStore: Milvus vector database integration
     - HybridSearch: Hybrid vector and metadata search
     - MetadataStore: Metadata indexing and management
@@ -145,6 +147,7 @@ from .methods import (
 )
 from .milvus_store import MilvusStore, MilvusClient, MilvusCollection, MilvusSearch
 from .namespace_manager import Namespace, NamespaceManager
+from .pinecone_store import PineconeStore, PineconeClient, PineconeIndex, PineconeSearch
 from .qdrant_store import QdrantStore, QdrantClient, QdrantCollection, QdrantSearch
 from .registry import MethodRegistry, method_registry
 from .vector_store import VectorIndexer, VectorManager, VectorRetriever, VectorStore
@@ -181,6 +184,11 @@ __all__ = [
     "MilvusClient",
     "MilvusCollection",
     "MilvusSearch",
+    # Pinecone
+    "PineconeStore",
+    "PineconeClient",
+    "PineconeIndex",
+    "PineconeSearch",
     # Hybrid search
     "HybridSearch",
     "MetadataFilter",