Fix(provenance): Resolve metadata crash and broken lineage chains

- Fix: Handle stringified JSON in get_lineage metadata aggregation to prevent ValueError.
- Fix: Auto-detect and link source as parent_entity_id in 	rack_entity to ensure cross-module lineage continuity.
- Verified: 	est_cross_module_lineage passed.

.github/FUNDING.yml

@@ -1,8 +1,3 @@
 # Funding options for Semantica
-# Uncomment and add your usernames/links below
-
-# github: [username]
-# patreon: username
-# ko_fi: username
-# custom: ["https://your-funding-page.com"]
+github: Hawksight-AI
 

.github/SUPPORT.md

@@ -32,6 +32,8 @@ For enterprise support, custom development, or consulting services:
 
 ## Sponsorship
 
+### Sponsor this project
+
 Support Semantica development:
 - [GitHub Sponsors](https://github.com/sponsors/Hawksight-AI)
 

.gitignore

@@ -61,6 +61,7 @@ wheels/
 .installed.cfg
 *.egg
 MANIFEST
+.python-version
 
 # IDE
 .vscode/

.pre-commit-config.yaml

@@ -5,6 +5,7 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-yaml
+        exclude: 'neptune-setup\.yaml$'
       - id: check-json
       - id: check-toml
       - id: check-added-large-files
@@ -49,9 +50,15 @@ repos:
     hooks:
       - id: yamllint
         args: ['-d', '{extends: default, rules: {line-length: {max: 120}}}']
+        exclude: 'neptune-setup\.yaml$'
+
+  - repo: https://github.com/aws-cloudformation/cfn-lint
+    rev: v1.43.3
+    hooks:
+      - id: cfn-lint
+        files: 'neptune-setup\.yaml$'
 
   # Removed slow hooks for faster development:
   # - mypy: Type checking (can be run manually or in CI)
   # - bandit: Security scanning (can be run separately)
   # - pytest: Testing (should be run manually, not on every commit)
-

CHANGELOG.md

@@ -7,28 +7,312 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added / Changed
+
+- **W3C PROV-O Compliant Provenance Tracking** (#254, #246):
+  - Comprehensive provenance tracking system with W3C PROV-O compliance across all 17 Semantica modules
+  - **Core Module**: `ProvenanceManager`, W3C PROV-O schemas, storage backends (InMemory, SQLite), SHA-256 integrity verification
+  - **Module Integrations**: Semantic Extract, LLMs (Groq, OpenAI, HuggingFace, LiteLLM), Pipeline, Context, Ingest, Embeddings, Graph/Vector/Triplet stores, Reasoning, Conflicts, Deduplication, Export, Parse, Normalize, Ontology, Visualization
+  - **Features**: Complete lineage tracking (Document → Chunk → Entity → Relationship → Graph), LLM tracking (tokens, costs, latency), source tracking, bridge axioms for domain transformations
+  - **Compliance Infrastructure**: W3C PROV-O, FDA 21 CFR Part 11, SOX, HIPAA, TNFD
+  - **Testing**: 237 tests covering core functionality, all 17 module integrations, edge cases, backward compatibility
+  - **Design**: Opt-in with `provenance=False` by default, zero breaking changes, no new dependencies
+  - Contributed by @KaifAhmad1
+
+- **Enhanced Change Management Module** (#248, #243):
+  - Enterprise-grade version control for knowledge graphs and ontologies with persistent storage and audit trails
+  - **Core Classes**: `TemporalVersionManager` (KG versioning), `OntologyVersionManager` (ontology versioning), `ChangeLogEntry` (metadata)
+  - **Storage**: SQLite (persistent) and in-memory backends with thread-safe operations
+  - **Features**: SHA-256 checksums, detailed entity/relationship diffs, structural ontology comparison, email validation
+  - **Compliance Infrastructure**: HIPAA, SOX, FDA 21 CFR Part 11 with immutable audit trails
+  - **Testing**: 104 tests (100% pass) - unit, integration, compliance, performance, edge cases
+  - **Performance**: 17.6ms for 10k entities, 510+ ops/sec concurrent, handles 5k+ entity graphs
+  - **Migration**: Backward compatible, simplified class names, zero external dependencies
+  - Contributed by @KaifAhmad1
+
+- CSV Ingestion Enhancements (PR #244 by @saloni0318)
+  - Auto-detect CSV encoding (chardet) and delimiter (csv.Sniffer)
+  - Tolerant decoding and malformed-row handling (`on_bad_lines='warn'`)
+  - Optional chunked reading for large files; metadata tracks detected values
+  - Expanded unit tests covering delimiters, quoted/multiline fields, header overrides, chunks, and NaN preservation
+
+- Tests: Comprehensive units for TextNormalizer (PR #242 by @ZohaibHassan16)
+  - Added focused test coverage for TextNormalizer behavior across inputs
+
+- Tests: Register integration mark and tidy ingest test warnings (PR #241 by @KaifAhmad1)
+  - Introduced integration test marker and reduced noisy warnings in ingest tests
+
+- **Ingest Unit Tests** (#239, #232):
+  - Comprehensive unit tests for ingestion modules (file, web, and feed ingestors)
+  - **Coverage**: File scanning (local/cloud S3/GCS/Azure), web ingestion (URL/sitemap/robots.txt), RSS/Atom feed parsing
+  - **Testing**: 998 lines of test code with mocked external dependencies for fast, isolated execution
+  - **Results**: file_ingestor (86%), web_ingestor (86%), feed_ingestor (80%) coverage
+  - Covers happy paths, edge cases, and error handling
+  - Contributed by @Mohammed2372
+
 ### Fixed
-- Fixed `TypeError: unhashable type: 'Entity'` in `GraphAnalyzer` when processing graphs with raw `Entity` objects or dictionaries in relationships (#159).
-- Robustified ID extraction across `CentralityCalculator`, `CommunityDetector`, and `ConnectivityAnalyzer` to handle various entity formats.
-- Improved `Entity` class hashability and equality logic in `utils/types.py`.
-- Added end-to-end integration test suite for Knowledge Graph pipeline validation (GraphBuilder -> EntityResolver -> GraphAnalyzer).
+
+- **Temperature Compatibility Fix** (#256, #252):
+  - Fixed hardcoded `temperature=0.3` that broke compatibility with models requiring specific temperature values (e.g., gpt-5-mini)
+  - Added `_add_if_set` helper method to `BaseProvider` that only passes parameters when explicitly set
+  - When `temperature=None`, parameter is omitted allowing APIs to use model defaults
+  - Updated all 5 providers: OpenAI, Groq, Gemini, Ollama, DeepSeek
+  - Reduced code by ~85 lines with cleaner parameter handling
+  - Comprehensive test coverage added (10 temperature tests, all passing)
+  - Backward compatible - no breaking changes
+  - Contributed by @F0rt1s and @IGES-Institut
+
+- **JenaStore Empty Graph Bug** (#257, #258):
+  - Fixed `ProcessingError: Graph not initialized` when operating on empty (but initialized) graphs
+  - Replaced implicit `if not self.graph:` checks with explicit `if self.graph is None:` validation in 5 methods (`add_triplets`, `get_triplets`, `delete_triplet`, `execute_sparql`, `serialize`)
+  - Properly distinguishes `None` (uninitialized) from empty graphs (initialized with 0 triplets)
+  - Unblocks benchmarking suite, fresh deployments, and testing workflows
+  - Contributed by @ZohaibHassan16
+
+## [0.2.5] - 2026-01-27
+
+### Added
+- **Pinecone Vector Store Support**:
+    - Implemented native Pinecone support (`PineconeStore`) with full CRUD capabilities.
+    - Added support for serverless and pod-based indexes, namespaces, and metadata filtering.
+    - Integrated with `VectorStore` unified interface and registry.
+    - (Closes #219, Resolves #220)
+- **Configurable LLM Retry Logic**:
+    - Exposed `max_retries` parameter in `NERExtractor`, `RelationExtractor`, `TripletExtractor` and low-level extraction methods (`extract_entities_llm`, `extract_relations_llm`, `extract_triplets_llm`).
+    - Defaults to 3 retries to prevent infinite loops during JSON validation failures or API timeouts.
+    - Propagated retry configuration through chunked processing helpers to ensure consistent behavior for long documents.
+    - Updated `03_Earnings_Call_Analysis.ipynb` to use `max_retries=3` by default.
+
+### Added
+- **Bring Your Own Model (BYOM) Support**:
+    - Enabled full support for custom Hugging Face models in `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Added support for custom tokenizers in `HuggingFaceModelLoader` to handle models with non-standard tokenization requirements.
+    - Implemented robust fallback logic for model selection: runtime options (`extract(model=...)`) now correctly override configuration defaults.
+- **Enhanced NER Implementation**:
+    - Added configurable aggregation strategies (`simple`, `first`, `average`, `max`) to `extract_entities_huggingface` for better sub-word token handling.
+    - Implemented robust IOB/BILOU parsing to reconstruct entities from raw model outputs when structured output is unavailable.
+    - Added confidence scoring for aggregated entities.
+- **Relation Extraction Improvements**:
+    - Implemented standard entity marker technique (wrapping subject/object with `<subj>`, `<obj>` tags) in `extract_relations_huggingface` for compatibility with sequence classification models.
+    - Added structured output parsing to convert raw model predictions into validated `Relation` objects.
+- **Triplet Extraction Completion**:
+    - Added specialized parsing for Seq2Seq models (e.g., REBEL) in `extract_triplets_huggingface` to generate structured triplets directly from text.
+    - Implemented post-processing logic to clean and validate generated triplets.
+
+### Fixed
+- **LLM Extraction Stability**:
+    - Fixed infinite retry loops in `BaseProvider` by strictly enforcing `max_retries` limit during structured output generation.
+    - Resolved stuck execution in earnings call analysis notebooks when using smaller models (e.g., Llama 3 8B) that frequently produce invalid JSON.
+- **Model Parameter Precedence**:
+    - Fixed issue where configuration defaults took precedence over runtime arguments in Hugging Face extractors. Runtime options now correctly override config values.
+- **Import Handling**:
+    - Fixed circular import issues in test suites by implementing robust mocking strategies.
+
+## [0.2.4] - 2026-01-22
+
+### Added
+- **Ontology Ingestion Module**:
+    - Implemented `OntologyIngestor` in `semantica.ingest` for parsing RDF/OWL files (Turtle, RDF/XML, JSON-LD, N3) into standardized `OntologyData` objects.
+    - Added `ingest_ontology` convenience function and integrated it into the unified `ingest(source_type="ontology")` interface.
+    - Added recursive directory scanning support for batch ontology ingestion.
+    - Exposed ingestion tools in `semantica.ontology` for better discoverability.
+    - Added `OntologyData` dataclass for consistent metadata handling (source path, format, timestamps).
+- **Documentation**:
+    - **Ontology Usage Guide**: Updated `ontology_usage.md` with comprehensive examples for single-file and directory ingestion.
+    - **API Reference**: Updated `ontology.md` with `OntologyIngestor` class documentation and method details.
+- **Tests**:
+    - **Comprehensive Test Suite**: Added `tests/ingest/test_ontology_ingestor.py` covering all supported formats, error handling, and unified interface integration.
+    - **Demo Script**: Added `examples/demo_ontology_ingest.py` for end-to-end usage demonstration.
+
+## [0.2.3] - 2026-01-20
+
+### Fixed
+- **LLM Relation Extraction Parsing**:
+    - Fixed relation extraction returning zero relations despite successful API calls to Groq and other providers
+    - Normalized typed responses from instructor/OpenAI/Groq to consistent dict format before parsing
+    - Added structured JSON fallback when typed generation yields zero relations to avoid silent empty outputs
+    - Removed acceptance of extra kwargs (`max_tokens`, `max_entities_prompt`) from relation extraction internals
+    - Filtered kwargs passed to provider LLM calls to only `temperature` and `verbose`
+- **API Parameter Handling**:
+    - Limited kwargs forwarded in chunked extraction helper to prevent parameter leakage
+    - Ensured minimal, safe parameters are passed to provider calls
+- **Pipeline Circular Import (Issues #192, #193)**:
+    - Fixed circular import between `pipeline_builder` and `pipeline_validator` triggered during `semantica.pipeline` import
+    - Lazy-loaded `PipelineValidator` inside `PipelineBuilder.__init__` and guarded type hints with `TYPE_CHECKING`
+    - Ensured `from semantica.deduplication import DuplicateDetector` no longer fails even when pipeline module is imported
+- **JupyterLab Progress Output (Issue #181)**:
+    - Added `SEMANTICA_DISABLE_JUPYTER_PROGRESS` environment variable to disable rich Jupyter/Colab progress tables
+    - When enabled, progress falls back to console-style output, preventing infinite scrolling and JupyterLab out-of-memory errors
+
+### Added
+- **Comprehensive Test Suite**:
+-    - Added unit tests (`tests/test_relations_llm.py`) with mocked LLM provider covering both typed and structured response paths
+-    - Added integration tests (`tests/integration/test_relations_groq.py`) for real Groq API calls with environment variable API key
+-    - Tests validate relation extraction completion and result parsing across different response formats
+- **Amazon Neptune Dev Environment**:
+-    - Added CloudFormation template (`cookbook/introduction/neptune-setup.yaml`) to provision a dev Neptune cluster with public endpoint and IAM auth enabled
+-    - Documented deployment, cost estimates, and IAM User vs IAM Role best practices in `cookbook/introduction/21_Amazon_Neptune_Store.ipynb`
+-    - Added `cfn-lint` to `.pre-commit-config.yaml` for validating CloudFormation templates while excluding `neptune-setup.yaml` from generic YAML linters
+- **Vector Store High-Performance Ingestion**:
+-    - Added `VectorStore.add_documents` for high-throughput ingestion with automatic embedding generation, batching, and parallel processing
+-    - Added `VectorStore.embed_batch` helper for generating embeddings for lists of texts without immediately storing them
+-    - Enabled default parallel ingestion in `VectorStore` with `max_workers=6` for common workloads
+-    - Added dedicated documentation page `docs/vector_store_usage.md` describing high-performance vector store usage and configuration
+-    - Added `tests/vector_store/test_vector_store_parallel.py` covering parallel vs sequential performance, error handling, and edge cases for `add_documents` and `embed_batch`
+
+### Changed
+- **Relation Extraction API**:
+-    - Simplified parameter interface by removing unused kwargs that were previously ignored
+-    - Improved error handling and verbose logging for debugging relation extraction issues
+-    - Enhanced robustness of post-response parsing across different LLM providers
+- **Vector Store Defaults and Examples**:
+-    - Standardized `VectorStore` default concurrency to `max_workers=6` for parallel ingestion
+-    - Updated vector store reference documentation and usage guides to rely on implicit defaults instead of requiring manual `max_workers` configuration in examples
+
+
+## [0.2.2] - 2026-01-15
+
+### Added
+- **Parallel Extraction Engine**:
+    - Implemented high-throughput parallel batch processing across all core extractors (`NERExtractor`, `RelationExtractor`, `TripletExtractor`, `EventDetector`, `SemanticNetworkExtractor`) using `concurrent.futures.ThreadPoolExecutor`.
+    - Added `max_workers` configuration parameter (default: 1) to all extractor `extract()` methods, allowing users to tune concurrency based on available CPU cores or API rate limits.
+    - **Parallel Chunking**: Implemented parallel processing for large document chunking in `_extract_entities_chunked` and `_extract_relations_chunked`, significantly reducing latency for long-form text analysis.
+    - **Thread-Safe Progress Tracking**: Enhanced `ProgressTracker` to handle concurrent updates from multiple threads without race conditions during batch processing.
+- **Semantic Extract Performance & Regression**:
+    - Added edge-case regression suite covering max worker defaults, LLM prompt entity filtering, and extractor reuse.
+    - Added a runnable real-use-case benchmark script for batch latency across `NERExtractor`, `RelationExtractor`, `TripletExtractor`, `EventDetector`, `SemanticAnalyzer`, and `SemanticNetworkExtractor`.
+    - Added Groq LLM smoke tests that exercise LLM-based entities/relations/triplets when `GROQ_API_KEY` is available via environment configuration.
+
+### Security
+- **Credential Sanitization**:
+    - Removed hardcoded API keys from 8 cookbook notebooks to prevent secret leakage.
+    - Enforced environment variable usage for `GROQ_API_KEY` across all examples.
+- **Secure Caching**:
+    - Updated `ExtractionCache` to exclude sensitive parameters (e.g., `api_key`, `token`, `password`) from cache key generation, preventing secret leakage and enabling safe cache sharing.
+    - Upgraded cache key hashing algorithm from MD5 to **SHA-256** for enhanced collision resistance and security.
+
+### Changed
+- **Gemini SDK Migration**:
+    - Migrated `GeminiProvider` to use the new `google-genai` SDK (v0.1.0+) to address deprecation warnings.
+    - Implemented graceful fallback to `google.generativeai` for backward compatibility.
+- **Dependency Resolution**:
+    - Pinned `opentelemetry-api` and `opentelemetry-sdk` to `1.37.0` to resolve pip conflicts.
+    - Updated `protobuf` and `grpcio` constraints for better stability.
+- **Entity Filtering Scope**:
+    - Removed entity filtering from non-LLM extraction flows to avoid accuracy regressions.
+    - Applied entity downselection only to LLM relation prompt construction, while matching returned entities against the full original entity list.
+- **Batch Concurrency Defaults**:
+    - Standardized `max_workers` defaulting across `semantic_extract` and tuned for low-latency: ML-backed methods default to single-worker, while pattern/regex/rules/LLM/huggingface methods use a higher parallelism default capped by CPU.
+    - Raised the global `optimization.max_workers` default to 8 for better throughput on batch workloads.
+
+### Performance
+- **Bottleneck Optimization (GitHub Issue #186)**:
+    - **Resolved Bottleneck #1 (Sequential Processing)**: Replaced sequential `for` loops with parallel execution for both document-level batches and intra-document chunks.
+    - **Performance Gains**: Achieved **~1.89x speedup** in real-world extraction scenarios (tested with Groq `llama-3.3-70b-versatile` on standard datasets).
+    - **Initialization Optimization**: Refactored test suite to use class-level `setUpClass` for LLM provider initialization, eliminating redundant API client creation overhead.
+- **Low-Latency Entity Matching**:
+    - Avoided heavyweight embedding stack imports on common matches by improving fast matching heuristics and short-circuiting before embedding similarity.
+    - Optimized entity matching to prioritize exact/substring/word-boundary matches and only fall back to embedding similarity when needed, reducing CPU overhead in LLM relation/triplet mapping.
+
+
+## [0.2.1] - 2026-01-12
+
+### Fixed
+- **LLM Output Stability (Bug #176)**:
+    - Fixed incomplete JSON output issues by correctly propagating `max_tokens` parameter in `extract_relations_llm`.
+    - Implemented automatic error handling that halves chunk sizes and retries when LLM context or output limits are exceeded.
+    - Fixed `AttributeError` in provider integration by ensuring consistent parameter passing via `**kwargs`.
+- **Constraint Relaxations**:
+    - Removed hardcoded `max_length` constraints from `Entity`, `Relation`, and `Triplet` classes to support long-form semantic extraction (e.g., long descriptions or names).
 - Fixed orchestrator lazy property initialization and configuration normalization logic in `Orchestrator`.
 - Resolved `AssertionError` in orchestrator tests by aligning test mocks with production component usage.
-- Fixed dependency compatibility issues by pinning `protobuf==4.25.3` and `grpcio==1.67.1`.
+- Fixed dependency compatibility issues by pinning `protobuf>=5.29.1,<7.0` and `grpcio>=1.71.2`.
 - Added missing dependencies `GitPython` and `chardet` to `pyproject.toml`.
 - Verified and aligned `FileObject.text` property usage in GraphRAG notebooks for consistent content decoding.
 
+### Changed
+- **Chunking Defaults**:
+    - Increased default `max_text_length` for auto-chunking to **64,000 characters** (from 32k/16k) for OpenAI, Anthropic, Gemini, Groq, and DeepSeek providers.
+    - Unified chunking logic across `extract_entities_llm`, `extract_relations_llm`, and `extract_triplets_llm`.
+- **Groq Support**:
+    - Standardized Groq provider defaults to use `llama-3.3-70b-versatile` with a 64k context window.
+    - Added native support for `max_tokens` and `max_completion_tokens` to prevent output truncation.
+
 ### Added
+- **Testing**:
+    - Added `tests/reproduce_issue_176.py` to validate `max_tokens` propagation and chunking behavior across all extractors.
+
+
+## [0.2.0] - 2026-01-10
+
+### Added
+- **Amazon Neptune Support**:
+    - Added `AmazonNeptuneStore` providing Amazon Neptune graph database integration via Bolt protocol and OpenCypher.
+    - Implemented `NeptuneAuthTokenManager` extending Neo4j AuthManager for AWS IAM SigV4 signing with automatic token refresh.
+    - Added robust connection handling: retry logic with backoff for transient errors (signature expired, connection closed) and driver recreation.
+    - Added `graph-amazon-neptune` optional dependency group (boto3, neo4j).
+    - Comprehensive test suite covering all GraphStore interface methods.
+- **Docling Integration**:
+    - Added `DoclingParser` in `semantica.parse` for high-fidelity document parsing using the Docling library.
+    - Supports multi-format parsing (PDF, DOCX, PPTX, XLSX, HTML, images) with superior table extraction and structure understanding.
+    - Implemented as a standalone parser supporting local execution, OCR, and multiple export formats (Markdown, HTML, JSON).
+- **Robust Extraction Fallbacks**:
+    - Implemented comprehensive fallback chains ("ML/LLM" -> "Pattern" -> "Last Resort") across `NERExtractor`, `RelationExtractor`, and `TripletExtractor` to prevent empty result lists.
+    - Added "Last Resort" pattern matching in `NERExtractor` to identify capitalized words as generic entities when all other methods fail.
+    - Added "Last Resort" adjacency-based relation extraction in `RelationExtractor` to create weak connections between adjacent entities if no relations are found.
+    - Added fallback logic in `TripletExtractor` to convert relations to triplets or use rule-based extraction if standard methods fail.
+- **Provenance & Tracking**:
+    - Added count tracking to batch processing logs in `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Added `batch_index` and `document_id` to the metadata of all extracted entities, relations, triplets, semantic roles, and clusters for better traceability.
 - **Semantic Extract Improvements**:
     - Introduced `auto-chunking` for long text processing in LLM extraction methods (`extract_entities_llm`, `extract_relations_llm`, `extract_triplets_llm`).
     - Added `silent_fail` parameter to LLM extraction methods for configurable error handling.
     - Implemented robust JSON parsing and automatic retry logic (3 attempts with exponential backoff) in `BaseProvider` for all LLM providers.
     - Enhanced `GroqProvider` with better diagnostics and connectivity testing.
     - Added comprehensive entity, relation, and triplet deduplication for chunked extraction.
+    - Added `semantica/semantic_extract/schemas.py` with canonical Pydantic models for consistent structured output.
+- **Testing**:
+    - Added comprehensive robustness test suite `tests/semantic_extract/test_robustness_fallback.py` for validating extraction fallbacks and metadata propagation.
+    - Added comprehensive unit test suite `tests/embeddings/test_model_switching.py` for verifying dynamic model transitions and dimension updates.
+    - Added end-to-end integration test suite for Knowledge Graph pipeline validation (GraphBuilder -> EntityResolver -> GraphAnalyzer).
+- **Other**:
+    - Added missing dependencies `GitPython` and `chardet` to `pyproject.toml`.
+    - Robustified ID extraction across `CentralityCalculator`, `CommunityDetector`, and `ConnectivityAnalyzer` to handle various entity formats.
+    - Improved `Entity` class hashability and equality logic in `utils/types.py`.
+
+### Changed
+- **Deduplication & Conflict Logic**:
+    - Removed internal deduplication logic from `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Removed consistency/conflict checking from `ExtractionValidator` to defer to dedicated `semantica/conflicts` module.
+    - Removed `_deduplicate_*` methods from `semantica/semantic_extract/methods.py`.
+- **Batch Processing & Consistency**:
+    - Standardized batch processing across all extractors (`NERExtractor`, `RelationExtractor`, `TripletExtractor`, `SemanticNetworkExtractor`, `EventDetector`, `SemanticAnalyzer`, `CoreferenceResolver`) using a unified `extract`/`analyze`/`resolve` method pattern with progress tracking.
+    - Added provenance metadata (`batch_index`, `document_id`) to `SemanticNetwork` nodes/edges, `Event` objects, `SemanticRole` results, `CoreferenceChain` mentions, and `SemanticCluster` (tracking source `document_ids`).
+    - Updated `SemanticClusterer.cluster` and `SemanticAnalyzer.cluster_semantically` to accept list of dictionaries (with `content` and `id` keys) for better document tracking during clustering.
+    - Removed legacy `check_triplet_consistency` from `TripletExtractor`.
+    - Removed `validate_consistency` and `_check_consistency` from `ExtractionValidator`.
+- **Weighted Scoring**:
+    - Clarified weighted confidence scoring (50% Method Confidence + 50% Type Similarity) in comments.
+    - Explicitly labeled "Type Similarity" as "user-provided" in code comments to remove ambiguity.
+- **Refactoring**:
+    - Fixed orchestrator lazy property initialization and configuration normalization logic in `Orchestrator`.
+    - Verified and aligned `FileObject.text` property usage in GraphRAG notebooks for consistent content decoding.
 
 ### Fixed
-- Fixed a bug in `TripletExtractor` where the `validate_triplets` method was shadowed by an internal attribute.
-- Fixed incorrect `TextSplitter` import path in the `semantic_extract.methods` module.
+- **Critical Fixes**:
+    - Resolved `NameError` in `extraction_validator.py` by adding missing `Union` import.
+    - Resolved issues where extractors would return empty lists for valid input text when primary extraction methods failed.
+    - Fixed metadata initialization issue in batch processing where `batch_index` and `document_id` were occasionally missing from extracted items.
+    - Ensured `LLMExtraction` methods (`enhance_entities`, `enhance_relations`) return original input instead of failing or returning empty results when LLM providers are unavailable.
+- **Component Fixes**:
+    - Fixed model switching bug in `TextEmbedder` where internal state was not cleared, preventing dynamic updates between `fastembed` and `sentence_transformers` (#160).
+    - Implemented model-intrinsic embedding dimension detection in `TextEmbedder` to ensure consistency between models and vector databases.
+    - Updated `set_model` to properly refresh configuration and dimensions during model switches.
+    - Fixed `TypeError: unhashable type: 'Entity'` in `GraphAnalyzer` when processing graphs with raw `Entity` objects or dictionaries in relationships (#159).
+    - Resolved `AssertionError` in orchestrator tests by aligning test mocks with production component usage.
+    - Fixed dependency compatibility issues by pinning `protobuf==4.25.3` and `grpcio==1.67.1`.
+    - Fixed a bug in `TripletExtractor` where the `validate_triplets` method was shadowed by an internal attribute.
+    - Fixed incorrect `TextSplitter` import path in the `semantic_extract.methods` module.
 
 ## [0.1.1] - 2026-01-05
 

CONTRIBUTING.md

@@ -1,306 +1,266 @@
 # Contributing to Semantica
 
-Thank you for your interest in contributing to Semantica! This document provides guidelines and instructions for contributing to the project.
+Thank you for your interest in contributing! Every contribution, no matter how small, is valuable. 🎉
 
-## Table of Contents
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-- [Code of Conduct](#code-of-conduct)
-- [Getting Started](#getting-started)
-- [Development Setup](#development-setup)
-- [Code Style Guidelines](#code-style-guidelines)
-- [Testing Requirements](#testing-requirements)
-- [Commit Message Conventions](#commit-message-conventions)
-- [Pull Request Process](#pull-request-process)
-- [Documentation Standards](#documentation-standards)
-- [Types of Contributions](#types-of-contributions)
-- [Getting Help](#getting-help)
+> **New to contributing?** Start with a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue) or join our [Discord](https://discord.gg/vqRt2qbx) community.
 
-## Code of Conduct
+---
 
-This project adheres to a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the maintainers.
+## 🚀 Quick Start
 
-## Getting Started
+1. Find a [`good first issue`](https://github.com/Hawksight-AI/semantica/labels/good%20first%20issue)
+2. [Fork Semantica](https://github.com/Hawksight-AI/semantica/fork) & clone the repository
+3. Make your changes
+4. Submit a pull request!
 
-1. **Fork the repository** on GitHub
-2. **Clone your fork** locally:
-   ```bash
-   git clone https://github.com/your-username/semantica.git
-   cd semantica
-   ```
-3. **Add the upstream remote**:
-   ```bash
-   git remote add upstream https://github.com/Hawksight-AI/semantica.git
-   ```
+**Need help?** Join [Discord](https://discord.gg/vqRt2qbx) or [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
 
-## Development Setup
+---
 
-### Prerequisites
+## 🎯 Ways to Contribute
 
-- Python 3.8 or higher (3.9+ recommended)
-- pip package manager
-- Git
+### 💻 Code
 
-### Installation
+**What you can do:**
+- Fix bugs
+- Add new features
+- Improve code quality (add type hints, docstrings, improve error messages)
+- Optimize performance
 
-1. **Create a virtual environment** (recommended):
-   ```bash
-   python -m venv venv
-   source venv/bin/activate  # On Windows: venv\Scripts\activate
-   ```
+**Where:** `semantica/` directory
 
-2. **Install the project in editable mode with dev dependencies**:
-   ```bash
-   pip install -e ".[dev]"
-   ```
+**Good first issues:** Add docstrings, type hints, or improve error messages
 
-3. **Install pre-commit hooks**:
-   ```bash
-   pre-commit install
-   ```
+---
 
-### Verify Installation
+### 📝 Documentation
+
+**What you can do:**
+- Fix typos and grammar errors
+- Improve clarity and readability
+- Add code examples and tutorials
+- Create new cookbook notebooks
+- Improve API documentation (docstrings)
+- Create troubleshooting guides
+- Update installation instructions
+- Add missing documentation
+
+**Where:** `README.md`, `docs/`, `cookbook/`, docstrings in code
+
+**Good first issues:** Fix typos, add examples, create cookbook tutorials, improve docstrings
+
+**Documentation formatting:**
+- Use clear, concise language
+- Include code examples where helpful
+- Follow markdown best practices
+- Use proper headings hierarchy
+- Add links to related sections
+- Include screenshots for UI-related docs
+
+---
+
+### 🧪 Testing
+
+**What you can do:**
+- Add unit tests
+- Improve test coverage
+- Add integration tests
+
+**Where:** `tests/` directory
+
+**Good first issues:** Add tests for specific functions or classes
+
+---
+
+### 🐛 Bug Reports
+
+**What:** Report bugs you find
+
+**How:** Use the [bug report template](https://github.com/Hawksight-AI/semantica/issues/new?template=bug_report.md)
+
+**Include:** Description, steps to reproduce, expected vs actual behavior, environment details
+
+---
+
+### 💡 Feature Requests
+
+**What:** Suggest new features or improvements
+
+**How:** Use the [feature request template](https://github.com/Hawksight-AI/semantica/issues/new?template=feature_request.md)
+
+**Include:** Problem statement, proposed solution, use cases
+
+---
+
+### 🎨 Cookbook & Examples
+
+**What:** Create tutorials and examples
+
+**Where:** `cookbook/` directory
+
+**Examples:** Create new notebooks, add examples, improve existing tutorials
+
+---
+
+### 💬 Community Support
+
+**What:** Help others in the community
+
+**Where:** [Discord](https://discord.gg/vqRt2qbx), [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions)
+
+**Examples:** Answer questions, review PRs, share your projects
+
+---
+
+### 🎓 Educational Content
+
+**What:** Create educational materials
+
+**Examples:** Blog posts, video tutorials, talks, workshops, case studies
+
+---
+
+### 🔧 Other Contributions
+
+- **Design & Graphics:** Logos, diagrams, visualizations
+- **Tools & Integrations:** CLI tools, integrations with other frameworks
+- **Infrastructure:** CI/CD improvements, Docker optimization
+- **Security:** Report security vulnerabilities (privately)
+
+---
+
+## 📋 Getting Started
+
+### 1. Fork & Clone
+
+First, [fork Semantica](https://github.com/Hawksight-AI/semantica/fork) on GitHub, then:
 
 ```bash
-python -c "import semantica; print(semantica.__version__)"
-pytest --version
-black --version
+git clone https://github.com/your-username/semantica.git
+cd semantica
+git remote add upstream https://github.com/Hawksight-AI/semantica.git
 ```
 
-## Code Style Guidelines
-
-We use several tools to maintain code quality and consistency:
-
-### Formatting
-
-- **Black**: Code formatting (line length: 88)
-  ```bash
-  black semantica/
-  ```
-
-- **isort**: Import sorting
-  ```bash
-  isort semantica/
-  ```
-
-### Linting
-
-- **flake8**: Style guide enforcement
-  ```bash
-  flake8 semantica/
-  ```
-
-- **mypy**: Static type checking
-  ```bash
-  mypy semantica/
-  ```
-
-### Running All Checks
+### 2. Set Up Environment
 
 ```bash
-# Format code
-black semantica/ tests/
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate  # Windows: venv\Scripts\activate
 
-# Sort imports
-isort semantica/ tests/
+# Install dev dependencies
+pip install -e ".[dev]"
 
-# Lint
-flake8 semantica/ tests/
-
-# Type check
-mypy semantica/
+# Install pre-commit hooks (optional)
+pre-commit install
 ```
 
-Or use pre-commit hooks (automatically runs on commit):
-```bash
-pre-commit run --all-files
-```
-
-## Testing Requirements
-
-### Running Tests
+### 3. Create Branch
 
 ```bash
-# Run all tests
-pytest
-
-# Run with coverage
-pytest --cov=semantica --cov-report=html
-
-# Run specific test file
-pytest tests/test_specific.py
-
-# Run with verbose output
-pytest -v
+git checkout -b feature/your-feature-name
+# or
+git checkout -b fix/bug-description
 ```
 
-### Test Coverage
+### 4. Make Changes
 
-- Minimum coverage: **80%**
-- Critical modules: **90%+**
-- Coverage reports are generated in `htmlcov/`
+- Follow code style (see below)
+- Add tests for new features
+- Update documentation
 
-### Writing Tests
+### 5. Run Checks
 
-- Follow pytest conventions
-- Use descriptive test names
-- Include docstrings for complex tests
-- Test both success and failure cases
-- Use fixtures for common setup
-
-Example:
-```python
-def test_entity_extraction():
-    """Test basic entity extraction functionality."""
-    from semantica.semantic_extract import NamedEntityRecognizer
-    
-    ner = NamedEntityRecognizer()
-    entities = ner.extract("Apple Inc. was founded by Steve Jobs.")
-    
-    assert len(entities) > 0
-    assert any(e.text == "Apple Inc." for e in entities)
+```bash
+pytest                          # Run tests
+black semantica/ tests/        # Format code
+isort semantica/ tests/         # Sort imports
+flake8 semantica/ tests/        # Lint
 ```
 
-## Commit Message Conventions
+Or use pre-commit hooks: `pre-commit run --all-files`
 
-We follow [Conventional Commits](https://www.conventionalcommits.org/) specification:
+### 6. Commit & Push
 
-### Format
-
-```
-<type>(<scope>): <subject>
-
-<body>
-
-<footer>
+```bash
+git commit -m "feat(module): add new feature"
+git push origin feature/your-feature-name
 ```
 
-### Types
+Then create a pull request on GitHub!
 
-- `feat`: New feature
-- `fix`: Bug fix
-- `docs`: Documentation changes
-- `style`: Code style changes (formatting, etc.)
-- `refactor`: Code refactoring
-- `test`: Adding or updating tests
-- `chore`: Maintenance tasks
-- `perf`: Performance improvements
-- `ci`: CI/CD changes
+---
 
-### Examples
+## 📐 Code Style
+
+We use automated tools:
+
+| Tool     | Purpose                    | Command                    |
+|----------|----------------------------|----------------------------|
+| **Black** | Code formatting            | `black semantica/ tests/` |
+| **isort** | Import sorting             | `isort semantica/ tests/` |
+| **flake8** | Style enforcement          | `flake8 semantica/ tests/` |
+| **mypy** | Type checking              | `mypy semantica/`          |
+
+**Run all:** `black semantica/ tests/ && isort semantica/ tests/ && flake8 semantica/ tests/ && mypy semantica/`
+
+---
+
+## 🧪 Testing
+
+```bash
+pytest                          # Run all tests
+pytest --cov=semantica         # With coverage
+pytest tests/test_file.py      # Specific file
+```
+
+**Coverage goal:** 80% minimum, 90%+ for critical modules
+
+---
+
+## 📝 Commit Messages
+
+Use [Conventional Commits](https://www.conventionalcommits.org/):
 
 ```
 feat(kg): add temporal graph support
-
-Add support for temporal knowledge graphs with version tracking
-and time-based queries.
-
-Closes #123
+fix(parse): handle empty PDF files
+docs(readme): add installation guide
+test(extract): add unit tests
 ```
 
-```
-fix(parse): handle empty PDF files gracefully
+**Types:** `feat`, `fix`, `docs`, `test`, `refactor`, `perf`, `style`, `chore`
 
-Previously, empty PDF files would cause a crash. Now they return
-an empty document with appropriate warnings.
+---
 
-Fixes #456
-```
+## ✅ PR Checklist
 
-## Pull Request Process
-
-### Before Submitting
-
-1. **Update your fork**:
-   ```bash
-   git fetch upstream
-   git checkout main
-   git merge upstream/main
-   ```
-
-2. **Create a feature branch**:
-   ```bash
-   git checkout -b feature/your-feature-name
-   # or
-   git checkout -b fix/bug-description
-   ```
-
-3. **Make your changes** and commit following our conventions
-
-4. **Run all checks**:
-   ```bash
-   pytest
-   black semantica/ tests/
-   isort semantica/ tests/
-   flake8 semantica/ tests/
-   mypy semantica/
-   ```
-
-5. **Push to your fork**:
-   ```bash
-   git push origin feature/your-feature-name
-   ```
-
-### PR Checklist
+Before submitting:
 
 - [ ] Code follows style guidelines
 - [ ] Tests pass locally
-- [ ] New tests added for new features
+- [ ] New tests added (if applicable)
 - [ ] Documentation updated
 - [ ] Commit messages follow conventions
 - [ ] No merge conflicts
-- [ ] PR description is clear and complete
 
-### PR Description Template
+---
 
-```markdown
-## Description
-Brief description of changes
+## 📖 Documentation Standards
 
-## Type of Change
-- [ ] Bug fix
-- [ ] New feature
-- [ ] Breaking change
-- [ ] Documentation update
+### Code Documentation (Docstrings)
 
-## Related Issues
-Closes #123
-Related to #456
+**Format:** Use Google-style docstrings
 
-## Testing
-- [ ] Tests pass locally
-- [ ] Added new tests
-- [ ] Updated existing tests
-
-## Checklist
-- [ ] Code follows style guidelines
-- [ ] Self-review completed
-- [ ] Comments added for complex code
-- [ ] Documentation updated
-- [ ] No new warnings generated
-```
-
-## Documentation Standards
-
-### Code Documentation
-
-- Use Google-style docstrings
-- Include type hints
-- Document all public functions and classes
-- Include examples for complex functions
-
-Example:
 ```python
-def extract_entities(
-    text: str,
-    model: str = "transformer",
-    confidence_threshold: float = 0.7
-) -> List[Entity]:
+def extract_entities(text: str, model: str = "transformer") -> List[Entity]:
     """Extract named entities from text.
     
     Args:
         text: Input text to process
         model: NER model to use (default: "transformer")
-        confidence_threshold: Minimum confidence score (default: 0.7)
     
     Returns:
         List of extracted Entity objects
@@ -309,92 +269,98 @@ def extract_entities(
         ValueError: If text is empty or model is invalid
     
     Example:
-        >>> ner = NamedEntityRecognizer()
+        >>> from semantica.semantic_extract import NERExtractor
+        >>> ner = NERExtractor(method="ml", model="en_core_web_sm")
         >>> entities = ner.extract("Apple Inc. was founded in 1976.")
         >>> len(entities)
         2
     """
-    ...
 ```
 
-### Documentation Files
+### Markdown Documentation Formatting
 
-- Update relevant documentation in `docs/`
-- Add examples to cookbook if applicable
-- Update API reference if adding new public APIs
-- Keep README.md up to date
+**General Guidelines:**
+- Use clear headings (H1 for title, H2 for main sections, H3 for subsections)
+- Keep paragraphs short and focused
+- Use bullet points for lists
+- Add code blocks with syntax highlighting
+- Include links to related documentation
 
-## Types of Contributions
+**Code Blocks:**
+- Use triple backticks with language identifier: ` ```python `, ` ```bash `
+- Include comments in code examples
+- Show expected output when helpful
 
-### 💻 Code Contributions
+**Examples:**
 
-- **Bug Fixes**: Resolving issues reported in the issue tracker.
-- **New Features**: Implementing new capabilities (please discuss via an issue first!).
-- **Refactoring**: Improving code structure and maintainability without changing behavior.
-- **Algorithm Optimization**: Improving the efficiency of graph algorithms and vector search.
+```markdown
+## Section Title
 
-#### ⚡ Performance and Latency
-We deeply value efficiency. Contributions that make Semantica faster and lighter are highly appreciated!
+Brief introduction paragraph.
 
-- **Latency Reduction**: Optimize critical paths and RAG pipeline response times.
-- **Memory Optimization**: Reduce graph/vector processing memory footprint.
-- **Throughput**: Improve operations per second (bulk ingestion, parallel queries).
-- **Benchmarks**: Add performance benchmarks to track regressions.
-- **Async/Concurrency**: Enhance asynchronous execution and concurrency.
+### Subsection
 
-### 📚 Documentation Contributions
+- Bullet point 1
+- Bullet point 2
 
-- Fix typos and grammar
-- Improve clarity
-- Add examples
-- Create tutorials
-- Translate documentation
+**Code example:**
 
-### Testing Contributions
+```python
+from semantica import SomeClass
 
-- Add test coverage
-- Improve test quality
-- Add integration tests
-- Performance benchmarks
+instance = SomeClass()
+result = instance.method()
+```
 
-### Other Contributions
+**Note:** Additional context or warnings.
+```
 
-- Answer questions in discussions
-- Help with issues
-- Review pull requests
-- Share use cases
-- Report bugs
-- Suggest features
+**Best Practices:**
+- Start with an overview/introduction
+- Use consistent terminology
+- Include "See also" links
+- Add examples for complex concepts
+- Keep formatting consistent across docs
 
-## Getting Help
+---
 
-### Communication Channels
+## 🆘 Getting Help
 
-- **GitHub Discussions**: General questions and discussions
-- **GitHub Issues**: Bug reports and feature requests
-- **Discord**: Real-time chat and community support
+- 💬 [Discord](https://discord.gg/vqRt2qbx) - Real-time chat
+- 💭 [GitHub Discussions](https://github.com/Hawksight-AI/semantica/discussions) - Q&A
+- 🐛 [GitHub Issues](https://github.com/Hawksight-AI/semantica/issues) - Bug reports
 
-### Before Asking for Help
+**Before asking:** Check existing documentation, search issues/discussions, review cookbook examples
 
-1. Check existing documentation
-2. Search GitHub issues and discussions
-3. Review code examples in cookbook
-4. Check FAQ in documentation
+---
 
-### Asking Good Questions
+## 🏆 Recognition
 
-- Provide context and environment details
-- Include code examples
-- Show what you've tried
-- Include error messages and logs
-- Be specific about what you need
-
-## Recognition
-
-Contributors are recognized in:
+All contributors are recognized in:
 - [CONTRIBUTORS.md](CONTRIBUTORS.md)
 - GitHub contributors page
-- Release notes for significant contributions
+- Release notes
 
-Thank you for contributing to Semantica! 🎉
+We follow the [all-contributors](https://allcontributors.org) specification!
 
+---
+
+## 📜 Code of Conduct
+
+This project follows a [Code of Conduct](CODE_OF_CONDUCT.md). Be respectful and inclusive.
+
+---
+
+## 📚 Resources
+
+- [README.md](README.md) - Project overview
+- [Cookbook](cookbook/) - Tutorials and examples
+- [Documentation](docs/) - Comprehensive guides
+
+---
+
+**Thank you for contributing!** 🚀
+
+Every contribution matters - whether it's a single line of code, a typo fix, a helpful answer, or a bug report. We appreciate you! 🙏
+
+⭐ **Give us a Star** • 🍴 **[Fork Semantica](https://github.com/Hawksight-AI/semantica/fork)** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**

CONTRIBUTORS.md

@@ -4,44 +4,31 @@ Thank you to all the people who have contributed to Semantica! 🎉
 
 This project follows the [all-contributors](https://allcontributors.org) specification. Contributions of any kind are welcome!
 
-## How to Contribute
+⭐ **Give us a Star** • 🍴 **Fork us** • 💬 **Join our [Discord](https://discord.gg/vqRt2qbx)**
 
-We welcome contributions of all kinds! Whether you're:
-- Writing code
-- Improving documentation
-- Reporting bugs
-- Suggesting features
-- Answering questions
-- Reviewing pull requests
-- Sharing use cases
-- Creating examples
-
-All contributions are valuable and appreciated!
+---
 
 ## Contribution Types
 
+We recognize all types of contributions:
+
 - 💻 **Code**: Writing code, fixing bugs, implementing features
 - 📝 **Documentation**: Writing docs, tutorials, examples
+- 🧪 **Testing**: Writing tests, improving test coverage
 - 🐛 **Bug Reports**: Finding and reporting bugs
 - 💡 **Ideas**: Suggesting new features or improvements
 - 🎨 **Design**: UI/UX improvements, graphics, branding
-- 📖 **Examples**: Creating code examples and tutorials
-- 🔍 **Testing**: Writing tests, improving test coverage
-- 💬 **Answering Questions**: Helping others in discussions
-- 📢 **Talks**: Giving talks, presentations, workshops
-- 🌍 **Translation**: Translating documentation
+- 🎨 **Cookbook**: Creating tutorials and examples
+- 💬 **Community**: Answering questions, reviewing PRs
+- 🎓 **Education**: Blog posts, video tutorials, talks, workshops
 - 🔧 **Tools**: Creating tools, scripts, integrations
 - 📦 **Packaging**: Improving build, release, distribution
 - ⚠️ **Security**: Reporting security vulnerabilities
-- 🎓 **Education**: Teaching, mentoring, tutorials
-- 📹 **Video**: Creating video content, tutorials
-- 🎵 **Audio**: Podcasts, audio content
-- 📸 **Photography**: Screenshots, images
-- 🔬 **Research**: Research, analysis, studies
-- 💰 **Financial**: Sponsoring, funding
 - 🏗️ **Infrastructure**: CI/CD, hosting, infrastructure
 - 🚇 **Maintenance**: Maintenance, triage, project management
 
+---
+
 ## Contributors
 
 <!-- ALL-CONTRIBUTORS-LIST:START -->
@@ -50,48 +37,78 @@ All contributions are valuable and appreciated!
 
 <!-- ALL-CONTRIBUTORS-LIST:END -->
 
+---
+
 ## Recognition
 
-### Top Contributors
+All contributors are recognized in:
 
-Contributors are recognized based on their contributions to the project. Recognition includes:
+- This contributors list
+- [GitHub contributors page](https://github.com/Hawksight-AI/semantica/graphs/contributors)
+- Release notes for significant contributions
+- Community appreciation
 
-- Listing in this file
-- GitHub contributor statistics
-- Special mentions in release notes
-- Featured showcases for significant contributions
-
-### Hall of Fame
-
-Special recognition for exceptional contributions:
-
-- **Coming soon** - We'll feature outstanding contributors here!
+---
 
 ## How to Add Yourself
 
-If you've contributed to Semantica and want to be added to this list:
+### Automatic Recognition
 
-1. **Automatic**: If you've made a commit, you'll appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors)
-2. **Manual**: Open a PR adding yourself to this file, or use the [@all-contributors bot](https://allcontributors.org/docs/en/bot/usage)
+If you've made a commit, you'll automatically appear in [GitHub's contributors graph](https://github.com/Hawksight-AI/semantica/graphs/contributors).
 
-Example:
-```markdown
-- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
-```
+### Using All-Contributors Bot
 
-## All Contributors Bot
-
-We use the [all-contributors](https://allcontributors.org) bot to automatically recognize contributors. To add a contributor, comment on an issue or PR:
+Comment on any issue or PR with:
 
 ```
 @all-contributors please add @username for code, docs, bug
 ```
 
-## Thank You!
+**Examples:**
 
-Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community!
+```
+@all-contributors please add @johndoe for code
+@all-contributors please add @janedoe for docs, bug
+@all-contributors please add @devuser for code, test, maintenance
+```
+
+### Manual Addition
+
+Open a PR adding yourself to this file:
+
+```markdown
+- [Your Name](https://github.com/yourusername) - 💻 📝 🐛
+```
 
 ---
 
-**Want to contribute?** Check out our [Contributing Guide](CONTRIBUTING.md) to get started!
+## Contribution Type Codes
 
+When using the all-contributors bot, use these codes:
+
+- `code` - Code contributions
+- `doc` - Documentation
+- `test` - Testing
+- `bug` - Bug reports
+- `ideas` - Feature requests/ideas
+- `design` - Design work
+- `example` - Cookbook/examples
+- `question` - Answering questions
+- `talk` - Talks/presentations
+- `tool` - Tools/integrations
+- `packaging` - Packaging/distribution
+- `security` - Security reports
+- `infra` - Infrastructure
+- `maintenance` - Maintenance
+
+See [all-contributors specification](https://allcontributors.org/docs/en/emoji-key) for complete list.
+
+---
+
+## Thank You!
+
+Every contribution, no matter how small, helps make Semantica better. Thank you for being part of our community! 🙏
+
+**Want to contribute?** 
+
+⭐ Give us a Star • 🍴 [Fork us](https://github.com/Hawksight-AI/semantica/fork) • Check out our [Contributing Guide](CONTRIBUTING.md) to get started!

README.md

@@ -1,204 +1,237 @@
 <div align="center">
 
-<img src="semantica_logo.png" alt="Semantica Logo" width="450" height="auto">
+<img src="Semantica Updated Logo.png" alt="Semantica Logo" width="460"/>
 
 # 🧠 Semantica
+### Open-Source Semantic Layer & Knowledge Engineering Framework
 
-[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/)
-[![Monthly Downloads](https://img.shields.io/pypi/dm/semantica)](https://pypi.org/project/semantica/)
+[![PyPI](https://img.shields.io/pypi/v/semantica.svg)](https://pypi.org/project/semantica/)
 [![Total Downloads](https://static.pepy.tech/badge/semantica)](https://pepy.tech/project/semantica)
-[![Discord](https://img.shields.io/badge/Discord-Join%20Us-7289da?style=flat&logo=discord&logoColor=white)](https://discord.gg/pMHguUzG)
 [![CI](https://github.com/Hawksight-AI/semantica/workflows/CI/badge.svg)](https://github.com/Hawksight-AI/semantica/actions)
+[![Discord](https://img.shields.io/badge/Discord-Join-7289da?logo=discord&logoColor=white)](https://discord.gg/RgaGTj9J)
 
-<p align="center">
-    <a href="https://github.com/Hawksight-AI/semantica/stargazers">
-        <img src="https://img.shields.io/badge/Give%20a%20Star-%E2%AD%90-yellow?style=for-the-badge&labelColor=555555" alt="Give a Star">
-    </a>
-    &nbsp;&nbsp;
-    <a href="https://github.com/Hawksight-AI/semantica/fork">
-        <img src="https://img.shields.io/badge/Support%20Project-Fork%20Us-blue?style=for-the-badge&labelColor=555555" alt="Support Project">
-    </a>
-</p>
+### ⭐ Give us a Star • 🍴 Fork us • 💬 Join our Discord
 
-**Open Source Framework for Semantic Layer & Knowledge Engineering**
-
-> **Transform chaotic data into intelligent knowledge.**
-
-*The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.*
-
-**100% Open Source** • **MIT Licensed** • **Latest Version: 0.1.1** • **Production Ready** • **Community Driven**
-
-[**Discord**](https://discord.gg/pMHguUzG)
+> **Transform Chaos into Intelligence. Build AI systems that are explainable, traceable, and trustworthy — not black boxes.**
 
 </div>
 
-## What is Semantica?
-
-Semantica bridges the gap between raw data chaos and AI-ready knowledge. It's a **semantic intelligence platform** that transforms unstructured data into structured, queryable knowledge graphs powering GraphRAG, AI agents, and multi-agent systems.
-
-### What Makes Semantica Different?
-
-Unlike traditional approaches that process isolated documents and extract text into vectors, Semantica understands **semantic relationships across all content**, provides **automated ontology generation**, and builds a **unified semantic layer** with **production-grade QA**.
-
-| **Traditional Approaches** | **Semantica's Approach** |
-|:---------------------------|:-------------------------|
-| Process data as isolated documents | Understands semantic relationships across all content |
-| Extract text and store vectors | Builds knowledge graphs with meaningful connections |
-| Generic entity recognition | General-purpose ontology generation and validation |
-| Manual schema definition | Automatic semantic modeling from content patterns |
-| Disconnected data silos | Unified semantic layer across all data sources |
-| Basic quality checks | Production-grade QA with conflict detection & resolution |
 
 ---
 
-## 🎯 The Problem We Solve
+## 🚀 Why Semantica?
 
-### The Semantic Gap
+**Semantica** bridges the **semantic gap** between text similarity and true meaning. It's the **semantic intelligence layer** that makes your AI agents auditable, explainable, and trustworthy.
 
-Organizations today face a **fundamental mismatch** between how data exists and how AI systems need it.
-
-#### The Semantic Gap: Problem vs. Solution
-
-Organizations have **unstructured data** (PDFs, emails, logs), **messy data** (inconsistent formats, duplicates, conflicts), and **disconnected silos** (no shared context, missing relationships). AI systems need **clear rules** (formal ontologies), **structured entities** (validated, consistent), and **relationships** (semantic connections, context-aware reasoning).
-
-| **What Organizations Have** | **What AI Systems Require** |
-|:------------------------------|:------------------------------|
-| **Unstructured Data** | **Clear Rules** |
-| PDFs, emails, logs | Formal ontologies |
-| Mixed schemas | Graphs & Networks |
-| Conflicting facts | |
-| **Messy, Noisy Data** | **Structured Entities** |
-| Inconsistent formats | Validated entities |
-| Duplicate records | Domain Knowledge |
-| Missing relationships | |
-| **Disconnected, Siloed Data** | **Relationships** |
-| Data in separate systems | Semantic connections |
-| No shared context | Context-Aware Reasoning |
-| Isolated knowledge | |
-
-### **SEMANTICA FRAMEWORK**
-
-Semantica operates through three integrated layers that transform raw data into AI-ready knowledge:
-
-**Input Layer** — Universal ingestion from multiple data formats (PDFs, DOCX, HTML, JSON, CSV, databases, live feeds, APIs, streams, archives, multi-modal content) into a unified pipeline.
-
-**Semantic Layer** — Core intelligence engine performing entity extraction, relationship mapping, ontology generation, context engineering, and quality assurance. Includes **advanced entity deduplication** (Jaro-Winkler, disjoint property handling) to ensure a clean single source of truth.
-
-**Output Layer** — Production-ready knowledge graphs, vector embeddings, and validated ontologies that power GraphRAG systems, AI agents, and multi-agent systems.
-
-**Powers: GraphRAG, AI Agents, Multi-Agent Systems**
-
-
-
-
-
-### What Happens Without Semantics?
-
-**They Break** — Systems crash due to inconsistent formats and missing structure.
-
-**They Hallucinate** — AI models generate false information without semantic context to validate outputs.
-
-**They Fail Silently** — Systems return wrong answers without warnings, leading to bad decisions.
-
-**Why?** Systems have data — not semantics. They can't connect concepts, understand relationships, validate against domain rules, or detect conflicts.
+Perfect for **high-stakes domains** where mistakes have real consequences.
 
 ---
 
-## 💡 The Semantica Solution
+### ⚡ Get Started in 30 Seconds
 
-**Semantica** is an **open-source framework** that closes the semantic gap between real-world messy data and the structured semantic layers required by advanced AI systems — GraphRAG, agents, multi-agent systems, reasoning models, and more.
+```bash
+pip install semantica
+```
 
-### How Semantica Solves These Problems
+```python
+from semantica.semantic_extract import NERExtractor
+from semantica.kg import GraphBuilder
 
-**Efficient Embeddings** — Uses **FastEmbed** by default for high-performance, lightweight local embedding generation (faster than sentence-transformers).
+# Extract entities and build knowledge graph
+ner = NERExtractor(method="ml", model="en_core_web_sm")
+entities = ner.extract("Apple Inc. was founded by Steve Jobs in 1976.")
+kg = GraphBuilder().build({"entities": entities, "relationships": []})
 
-**Universal Data Ingestion** — Handles multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams) with unified pipeline, no custom parsers needed.
+print(f"Built KG with {len(kg.get('entities', []))} entities")
+```
 
-**Automated Semantic Extraction** — NER, relationship extraction, and triplet generation with LLM enhancement. Includes **auto-chunking** for long documents and **robust error handling** with automatic retry logic.
-
-**Knowledge Graph Construction** — Production-ready graphs with entity resolution, temporal support, and graph analytics. Queryable knowledge ready for AI applications.
-
-**GraphRAG Engine** — Hybrid vector + graph retrieval achieves 91% accuracy (30% improvement) via semantic search + graph traversal for multi-hop reasoning. Features LLM-generated responses grounded in knowledge graph context with reasoning traces. [See Comparison Benchmark](cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb)
-
-**AI Agent Context Engineering** — Persistent memory with RAG + knowledge graphs enables context maintenance, action validation, and structured knowledge access.
-
-**Automated Ontology Generation** — 6-stage LLM pipeline generates validated OWL ontologies with HermiT/Pellet validation, eliminating manual engineering.
-
-**Production-Grade QA** — Conflict detection, deduplication, quality scoring, and provenance tracking ensure trusted, production-ready knowledge graphs.
-
-**Pipeline Orchestration** — Flexible pipeline builder with parallel execution enables scalable processing via orchestrator-worker pattern.
-
-### Core Features at a Glance
-
-| **Feature Category** | **Capabilities** | **Key Benefits** |
-|:---------------------|:-----------------|:------------------|
-| **Data Ingestion** | Multiple formats (PDF, DOCX, HTML, JSON, CSV, databases, APIs, streams, archives) | Universal ingestion, no custom parsers needed |
-| **Semantic Extraction** | NER, relations, triplets, LLM enhancement, **auto-chunking** | Automated discovery with robust error handling |
-| **Knowledge Graphs** | Entity resolution, temporal support, graph analytics, query interface | Production-ready, queryable knowledge structures |
-| **Ontology Generation** | 6-stage LLM pipeline, OWL generation, HermiT/Pellet validation | Automated ontology creation from documents |
-| **GraphRAG** | Hybrid vector + graph retrieval, multi-hop reasoning, LLM-generated responses | 91% accuracy, 30% improvement over vector-only, reasoning traces |
-| **LLM Providers** | Unified interface to 100+ LLMs (Groq, OpenAI, HuggingFace, LiteLLM) | Clean imports, multiple providers, structured output |
-| **Agent Memory** | Persistent memory (Save/Load), Hybrid Retrieval (Vector+Graph), FastEmbed support | Context-aware agents with semantic understanding |
-| **Pipeline Orchestration** | Parallel execution, custom steps, orchestrator-worker pattern | Scalable, flexible data processing |
-| **Quality Assurance** | Conflict detection, deduplication, quality scoring, provenance | Trusted knowledge graphs ready for production |
+**[📖 Full Quick Start](#-quick-start)** • **[🍳 Cookbook Examples](#-semantica-cookbook)** • **[💬 Join Discord](https://discord.gg/RgaGTj9J)** • **[⭐ Star Us](https://github.com/Hawksight-AI/semantica)**
 
 ---
 
-## 👥 Who Is This For?
+## Core Value Proposition
 
-Semantica is designed for **developers, data engineers, and organizations** building the next generation of AI applications that require semantic understanding and knowledge graphs.
+| **Trustworthy** | **Explainable** | **Auditable** |
+|:------------------:|:------------------:|:-----------------:|
+| Conflict detection & validation | Transparent reasoning paths | Complete provenance tracking |
+| Rule-based governance | Entity relationships & ontologies | W3C PROV-O compliant lineage |
+| Production-grade QA | Multi-hop graph reasoning | Source tracking & integrity verification |
 
-### Who Uses Semantica
+---
 
-**AI/ML Engineers & Data Scientists** — Build GraphRAG systems, AI agents, and multi-agent systems.
+## Key Features & Benefits
 
-**Data Engineers** — Build scalable pipelines with semantic enrichment.
+### Not Just Another Agentic Framework
 
-**Knowledge Engineers & Ontologists** — Create knowledge graphs and ontologies with automated pipelines.
+**Semantica complements** LangChain, LlamaIndex, AutoGen, CrewAI, Google ADK, Agno, and other frameworks to enhance your agents with:
 
-**Enterprise Data Teams** — Unify semantic layers, improve data quality, resolve conflicts.
+| Feature | Benefit |
+|:--------|:--------|
+| **Auditable** | Complete provenance tracking with W3C PROV-O compliance |
+| **Explainable** | Transparent reasoning paths with entity relationships |
+| **Provenance-Aware** | End-to-end lineage from documents to responses |
+| **Validated** | Built-in conflict detection, deduplication, QA |
+| **Governed** | Rule-based validation and semantic consistency |
+| **Version Control** | Enterprise-grade change management with integrity verification |
 
-**Software & DevOps Engineers** — Build semantic APIs and infrastructure with production-ready SDK.
+### Perfect For High-Stakes Use Cases
 
-**Analysts & Researchers** — Transform data into queryable knowledge graphs for insights.
+| 🏥 **Healthcare** | 💰 **Finance** | ⚖️ **Legal** |
+|:-----------------:|:--------------:|:------------:|
+| Clinical decisions | Fraud detection | Evidence-backed research |
+| Drug interactions | Regulatory support | Contract analysis |
+| Patient safety | Risk assessment | Case law reasoning |
 
-**Security & Compliance Teams** — Threat intelligence, regulatory reporting, audit trails.
+| 🔒 **Cybersecurity** | 🏛️ **Government** | 🏭 **Infrastructure** | 🚗 **Autonomous** |
+|:-------------------:|:----------------:|:-------------------:|:-----------------:|
+| Threat attribution | Policy decisions | Power grids | Decision logs |
+| Incident response | Classified info | Transportation | Safety validation |
 
-**Product Teams & Startups** — Rapid prototyping of AI products and semantic features.
+### Powers Your AI Stack
+
+- **GraphRAG Systems** — Retrieval with graph reasoning and hybrid search
+- **AI Agents** — Trustworthy, accountable multi-agent systems with semantic memory
+- **Reasoning Models** — Explainable AI decisions with reasoning paths
+- **Enterprise AI** — Governed, auditable platforms that support compliance
+
+### Integrations
+
+- **Docling Support** — Document parsing with table extraction (PDF, DOCX, PPTX, XLSX)
+- **AWS Neptune** — Amazon Neptune graph database support with IAM authentication
+- **Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD)
+
+> **Built for environments where every answer must be explainable and governed.**
+
+
+---
+
+## 🚨 The Problem: The Semantic Gap
+
+### Most AI systems fail in high-stakes domains because they operate on **text similarity**, not **meaning**.
+
+### Understanding the Semantic Gap
+
+The **semantic gap** is the fundamental disconnect between what AI systems can process (text patterns, vector similarities) and what high-stakes applications require (semantic understanding, meaning, context, and relationships).
+
+**Traditional AI approaches:**
+- Rely on statistical patterns and text similarity
+- Cannot understand relationships between entities
+- Cannot reason about domain-specific rules
+- Cannot explain why decisions were made
+- Cannot trace back to original sources with confidence
+
+**High-stakes AI requires:**
+- Semantic understanding of entities and their relationships
+- Domain knowledge encoded as formal rules (ontologies)
+- Explainable reasoning paths
+- Source-level provenance
+- Conflict detection and resolution
+
+**Semantica bridges this gap** by providing a semantic intelligence layer that transforms unstructured data into validated, explainable, and auditable knowledge.
+
+### What Organizations Have vs What They Need
+
+| **Current State** | **Required for High-Stakes AI** |
+|:---------------------|:-----------------------------------|
+| PDFs, DOCX, emails, logs | Formal domain rules (ontologies) |
+| APIs, databases, streams | Structured and validated entities |
+| Conflicting facts and duplicates | Explicit semantic relationships |
+| Siloed systems with no lineage | **Explainable reasoning paths** |
+| | **Source-level provenance** |
+| | **Audit-ready compliance** |
+
+### The Cost of Missing Semantics
+
+- **Decisions cannot be explained** — No transparency in AI reasoning
+- **Errors cannot be traced** — No way to debug or improve
+- **Conflicts go undetected** — Contradictory information causes failures
+- **Compliance becomes impossible** — No audit trails for regulations
+
+**Trustworthy AI requires semantic accountability.**
+
+---
+
+## 🆚 Semantica vs Traditional RAG
+
+| Feature | Traditional RAG | Semantica |
+|:--------|:----------------|:----------|
+| **Reasoning** | ❌ Black-box answers | ✅ Explainable reasoning paths |
+| **Provenance** | ❌ No provenance | ✅ W3C PROV-O compliant lineage tracking |
+| **Search** | ⚠️ Vector similarity only | ✅ Semantic + graph reasoning |
+| **Quality** | ❌ No conflict handling | ✅ Explicit contradiction detection |
+| **Safety** | ⚠️ Unsafe for high-stakes | ✅ Designed for governed environments |
+| **Compliance** | ❌ No audit trails | ✅ Complete audit trails with integrity verification |
+
+---
+
+## 🧩 Semantica Architecture
+
+### 1️⃣ Input Layer — Governed Ingestion
+- 📄 **Multiple Formats** — PDFs, DOCX, HTML, JSON, CSV, Excel, PPTX
+- 🔧 **Docling Support** — Docling parser for table extraction
+- 💾 **Data Sources** — Databases, APIs, streams, archives, web content
+- 🎨 **Media Support** — Image parsing with OCR, audio/video metadata extraction
+- 📊 **Single Pipeline** — Unified ingestion with metadata and source tracking
+
+### 2️⃣ Semantic Layer — Trust & Reasoning Engine
+- 🔍 **Entity Extraction** — NER, normalization, classification
+- 🔗 **Relationship Discovery** — Triplet generation, semantic links
+- 📐 **Ontology Induction** — Automated domain rule generation
+- 🔄 **Deduplication** — Jaro-Winkler similarity, conflict resolution
+- ✅ **Quality Assurance** — Conflict detection, validation
+- 📊 **Provenance Tracking** — W3C PROV-O compliant lineage tracking across all modules
+- 🧠 **Reasoning Traces** — Explainable inference paths
+- 🔐 **Change Management** — Version control with audit trails, checksums, compliance support
+
+### 3️⃣ Output Layer — Auditable Knowledge Assets
+- 📊 **Knowledge Graphs** — Queryable, temporal, explainable
+- 📐 **OWL Ontologies** — HermiT/Pellet validated, custom ontology import support
+- 🔢 **Vector Embeddings** — FastEmbed by default
+- ☁️ **AWS Neptune** — Amazon Neptune graph database support
+- 🔍 **Provenance** — Every AI response links back to:
+  - 📄 Source documents
+  - 🏷️ Extracted entities & relations
+  - 📐 Ontology rules applied
+  - 🧠 Reasoning steps used
+
+---
+
+## 🏥 Built for High-Stakes Domains
+
+Designed for domains where **mistakes have real consequences** and **every decision must be accountable**:
+
+- **🏥 Healthcare & Life Sciences** — Clinical decision support, drug interaction analysis, medical literature reasoning, patient safety tracking
+- **💰 Finance & Risk** — Fraud detection, regulatory support (SOX, GDPR, MiFID II), credit risk assessment, algorithmic trading validation
+- **⚖️ Legal & Compliance** — Evidence-backed legal research, contract analysis, regulatory change tracking, case law reasoning
+- **🔒 Cybersecurity & Intelligence** — Threat attribution, incident response, security audit trails, intelligence analysis
+- **🏛️ Government & Defense** — Governed AI systems, policy decisions, classified information handling, defense intelligence
+- **🏭 Critical Infrastructure** — Power grid management, transportation safety, water treatment, emergency response
+- **🚗 Autonomous Systems** — Self-driving vehicles, drone navigation, robotics safety, industrial automation  
+
+---
+
+## 👥 Who Uses Semantica?
+
+- **🤖 AI / ML Engineers** — Building explainable GraphRAG & agents
+- **⚙️ Data Engineers** — Creating governed semantic pipelines
+- **📊 Knowledge Engineers** — Managing ontologies & KGs at scale
+- **🏢 Enterprise Teams** — Requiring trustworthy AI infrastructure
+- **🛡️ Risk & Compliance Teams** — Needing audit-ready systems  
 
 ---
 
 ## 📦 Installation
 
-> **✅ Available on PyPI!** Semantica is now published on PyPI. Install it with a single command: `pip install semantica`
-
-**Prerequisites:** Python 3.8+ (3.9+ recommended) • pip (latest version)
-
 ### Install from PyPI (Recommended)
 
 ```bash
-# Install latest version from PyPI
 pip install semantica
-
-# Or install with optional dependencies
+# or
 pip install semantica[all]
-
-# GitHub Workaround (if PyPI version has issues)
-pip install git+https://github.com/Hawksight-AI/semantica.git@main
-
-# Verify installation
-python -c "from semantica.parse import DoclingParser; DoclingParser(); print('✓ Semantica ready')"
 ```
 
-**Current Version:** [![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/) • [View on PyPI](https://pypi.org/project/semantica/)
-
-!!! info "Windows PyTorch Note"
-    If you encounter PyTorch DLL errors on Windows, ensure you have the [Microsoft Visual C++ Redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) installed. This is a common environment-specific issue with PyTorch on Windows and not a bug in Semantica.
-
-
-
 ### Install from Source (Development)
 
 ```bash
@@ -258,7 +291,7 @@ print(f" Ingested {len(sources)} sources")
 
 ### Document Parsing & Processing
 
-> **Multi-format parsing** • **Text normalization** • **Intelligent chunking**
+> **Multi-format parsing** • **Docling Support** • **Text normalization** • **Intelligent chunking**
 
 ```python
 from semantica.parse import DocumentParser, DoclingParser
@@ -269,11 +302,15 @@ from semantica.split import TextSplitter
 parser = DocumentParser()
 parsed = parser.parse("document.pdf", format="auto")
 
-# Enhanced parsing with Docling (recommended for complex layouts/tables)
+# Parsing with Docling (for complex layouts/tables)
 # Requires: pip install docling
-docling_parser = DoclingParser()
-docling_result = docling_parser.parse("complex_table.pdf")
-print(f"Extracted {len(docling_result.tables)} tables")
+docling_parser = DoclingParser(enable_ocr=True)
+result = docling_parser.parse("complex_table.pdf")
+
+print(f"Text (Markdown): {result['full_text'][:100]}...")
+print(f"Extracted {len(result['tables'])} tables")
+for i, table in enumerate(result['tables']):
+    print(f"Table {i+1} headers: {table.get('headers', [])}")
 
 # Normalize text
 normalizer = TextNormalizer()
@@ -356,7 +393,7 @@ results = vector_store.search(query="supply chain", top_k=5)
 
 ### Graph Store & Triplet Store
 
-> **Neo4j, FalkorDB support** • **SPARQL queries** • **RDF triplets**
+> **Neo4j, FalkorDB, Amazon Neptune** • **SPARQL queries** • **RDF triplets**
 
 ```python
 from semantica.graph_store import GraphStore
@@ -366,6 +403,24 @@ from semantica.triplet_store import TripletStore
 graph_store = GraphStore(backend="neo4j", uri="bolt://localhost:7687", user="neo4j", password="password")
 graph_store.add_nodes([{"id": "n1", "labels": ["Person"], "properties": {"name": "Alice"}}])
 
+# Amazon Neptune Graph Store (OpenCypher via HTTP with IAM Auth)
+neptune_store = GraphStore(
+    backend="neptune",
+    endpoint="your-cluster.us-east-1.neptune.amazonaws.com",
+    port=8182,
+    region="us-east-1",
+    iam_auth=True,  # Uses AWS credential chain (boto3, env vars, or IAM role)
+)
+
+# Node Operations
+neptune_store.add_nodes([
+    {"labels": ["Person"], "properties": {"id": "alice", "name": "Alice", "age": 30}},
+    {"labels": ["Person"], "properties": {"id": "bob", "name": "Bob", "age": 25}},
+])
+
+# Query Operations
+result = neptune_store.execute_query("MATCH (p:Person) RETURN p.name, p.age")
+
 # Triplet Store (Blazegraph, Jena, RDF4J)
 triplet_store = TripletStore(backend="blazegraph", endpoint="http://localhost:9999/blazegraph")
 triplet_store.add_triplet({"subject": "Alice", "predicate": "knows", "object": "Bob"})
@@ -376,19 +431,137 @@ results = triplet_store.execute_query("SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT
 
 ### Ontology Generation & Management
 
-> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation
+> **6-Stage LLM Pipeline** • Automatic OWL Generation • HermiT/Pellet Validation • **Custom Ontology Import** (OWL, RDF, Turtle, JSON-LD)
 
 ```python
 from semantica.ontology import OntologyGenerator
+from semantica.ingest import ingest_ontology
 
+# Generate ontology automatically
 generator = OntologyGenerator(llm_provider="openai", model="gpt-4")
 ontology = generator.generate_from_documents(sources=["domain_docs/"])
 
-print(f"Classes: {len(ontology.classes)}")
+# Or import your existing ontology
+custom_ontology = ingest_ontology("my_ontology.ttl")  # Supports OWL, RDF, Turtle, JSON-LD
+print(f"Classes: {len(custom_ontology.classes)}")
 ```
 
 [**Cookbook: Ontology**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/introduction/14_Ontology.ipynb)
 
+### Change Management & Version Control
+
+> **Version Control for Knowledge Graphs & Ontologies** • **SQLite & In-Memory Storage** • **SHA-256 Integrity Verification**
+
+```python
+from semantica.change_management import TemporalVersionManager, OntologyVersionManager
+
+# Knowledge Graph versioning with audit trails
+kg_manager = TemporalVersionManager(storage_path="kg_versions.db")
+
+# Create versioned snapshot
+snapshot = kg_manager.create_snapshot(
+    knowledge_graph,
+    version_label="v1.0",
+    author="user@company.com",
+    description="Initial patient record"
+)
+
+# Compare versions with detailed diffs
+diff = kg_manager.compare_versions("v1.0", "v2.0")
+print(f"Entities added: {diff['summary']['entities_added']}")
+print(f"Entities modified: {diff['summary']['entities_modified']}")
+
+# Verify data integrity
+is_valid = kg_manager.verify_checksum(snapshot)
+```
+
+**What We Provide:**
+- 🔐 **Persistent Storage** — SQLite and in-memory backends implemented
+- 📊 **Detailed Diffs** — Entity-level and relationship-level change tracking
+- ✅ **Data Integrity** — SHA-256 checksums with tamper detection
+- 📝 **Standardized Metadata** — ChangeLogEntry with author, timestamp, description
+- ⚡ **Performance Tested** — Benchmarked with 10k entities
+- 🧪 **Test Coverage** — 104 tests covering core functionality
+
+**Compliance Note:** Provides technical infrastructure (audit trails, checksums, temporal tracking) that supports compliance efforts for HIPAA, SOX, FDA 21 CFR Part 11. Organizations must implement additional policies and procedures for full regulatory compliance.
+
+[**Documentation: Change Management**](docs/reference/change_management.md) • [**Usage Guide**](semantica/change_management/change_management_usage.md)
+
+### Provenance Tracking — W3C PROV-O Compliant Lineage
+
+> **W3C PROV-O Implementation** • **17 Module Integrations** • **Opt-In Design** • **Zero Breaking Changes**
+
+**⚠️ Compliance Note:** Provides technical infrastructure for provenance tracking that supports compliance efforts. Organizations must implement additional policies, procedures, and controls for full regulatory compliance.
+
+```python
+from semantica.semantic_extract.semantic_extract_provenance import NERExtractorWithProvenance
+from semantica.llms.llms_provenance import GroqLLMWithProvenance
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+
+# Enable provenance tracking - just add provenance=True
+ner = NERExtractorWithProvenance(provenance=True)
+entities = ner.extract(
+    text="Apple Inc. was founded by Steve Jobs.",
+    source="biography.pdf"
+)
+
+# Track LLM calls with costs and latency
+llm = GroqLLMWithProvenance(provenance=True, model="llama-3.1-70b")
+response = llm.generate("Summarize the document")
+
+# Store in graph with complete lineage
+graph = GraphStoreWithProvenance(provenance=True)
+graph.add_node(entity, source="biography.pdf")
+
+# Retrieve complete provenance
+lineage = ner._prov_manager.get_lineage("entity_id")
+print(f"Source: {lineage['source']}")
+print(f"Lineage chain: {lineage['lineage_chain']}")
+```
+
+**What We Provide:**
+- ✅ **W3C PROV-O Implementation** — Data schemas implementing prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom
+- ✅ **17 Module Integrations** — Provenance-enabled versions of semantic extract, LLMs, pipeline, context, ingest, embeddings, reasoning, conflicts, deduplication, export, parse, normalize, ontology, visualization, graph/vector/triplet stores
+- ✅ **Opt-In Design** — Zero breaking changes, `provenance=False` by default
+- ✅ **Lineage Tracking** — Document → Chunk → Entity → Relationship → Graph lineage chains
+- ✅ **LLM Tracking** — Token counts, costs, and latency tracking for LLM calls
+- ✅ **Source Tracking Fields** — Document identifiers, page numbers, sections, and quote fields in schemas
+- ✅ **Storage Backends** — InMemoryStorage (fast) and SQLiteStorage (persistent) implemented
+- ✅ **Bridge Axioms** — BridgeAxiom and TranslationChain classes for domain transformations (L1 → L2 → L3)
+- ✅ **Integrity Verification** — SHA-256 checksum computation and verification functions
+- ✅ **No New Dependencies** — Uses Python stdlib only (sqlite3, json, dataclasses)
+
+**Supported Modules:**
+```python
+# Semantic Extract
+from semantica.semantic_extract.semantic_extract_provenance import (
+    NERExtractorWithProvenance, RelationExtractorWithProvenance, EventDetectorWithProvenance
+)
+
+# LLM Providers
+from semantica.llms.llms_provenance import (
+    GroqLLMWithProvenance, OpenAILLMWithProvenance, HuggingFaceLLMWithProvenance
+)
+
+# Storage & Processing
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+from semantica.vector_store.vector_store_provenance import VectorStoreWithProvenance
+from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
+
+# ... and 12 more modules
+```
+
+**High-Stakes Use Cases:**
+- 🏥 **Healthcare** — Clinical decision audit trails with source tracking
+- 💰 **Finance** — Fraud detection provenance with complete lineage
+- ⚖️ **Legal** — Evidence chain of custody with temporal tracking
+- 🔒 **Cybersecurity** — Threat attribution with relationship tracking
+- 🏛️ **Government** — Policy decision audit trails with integrity verification
+
+**Note:** Provenance tracking provides the *technical infrastructure* for compliance. Organizations must implement additional policies and procedures to meet specific regulatory requirements (HIPAA, SOX, FDA 21 CFR Part 11, etc.).
+
+[**Documentation: Provenance Tracking**](semantica/provenance/provenance_usage.md)
+
 ### Context Engineering & Memory Systems
 
 > **Persistent Memory** • **Context Graph** • **Context Retriever** • **Hybrid Retrieval (Vector + Graph)** • **Production Graph Store (Neo4j)** • **Entity Linking** • **Multi-Hop Reasoning**
@@ -454,7 +627,7 @@ reasoned_result = context.query_with_reasoning(
 
 ### Knowledge Graph-Powered RAG (GraphRAG)
 
-> **30% Accuracy Improvement** • Vector + Graph Hybrid Search • 91% Accuracy • **Multi-Hop Reasoning** • **LLM-Generated Responses**
+> **Vector + Graph Hybrid Search** • **Multi-Hop Reasoning** • **LLM-Generated Responses** • **Semantic Re-ranking**
 
 ```python
 from semantica.context import AgentContext
@@ -503,7 +676,7 @@ print(f"Confidence: {result['confidence']:.3f}")
 from semantica.llms import Groq, OpenAI, HuggingFaceLLM, LiteLLM
 import os
 
-# Groq - Fast inference
+# Groq
 groq = Groq(
     model="llama-3.1-8b-instant",
     api_key=os.getenv("GROQ_API_KEY")
@@ -533,7 +706,7 @@ structured = groq.generate_structured("Extract entities from: Apple Inc. was fou
 ```
 
 **Supported Providers:**
-- **Groq**: Fast inference with Llama models
+- **Groq**: Inference with Llama models
 - **OpenAI**: GPT-3.5, GPT-4, and other OpenAI models
 - **HuggingFace**: Local LLM inference with Transformers
 - **LiteLLM**: Unified interface to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, and more)
@@ -733,7 +906,7 @@ print(f"Found {len(results)} results")
 
 #### Cybersecurity
 - [**Real-Time Anomaly Detection**](cookbook/use_cases/cybersecurity/01_Real_Time_Anomaly_Detection.ipynb) - CVE RSS, Kafka streams, temporal KGs, sentence chunking
-- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, enhanced GraphRAG, deduplication
+- [**Threat Intelligence Hybrid RAG**](cookbook/use_cases/cybersecurity/02_Threat_Intelligence_Hybrid_RAG.ipynb) - Security RSS, entity-aware chunking, GraphRAG, deduplication
 
 #### Intelligence & Law Enforcement
 - [**Criminal Network Analysis**](cookbook/use_cases/intelligence/01_Criminal_Network_Analysis.ipynb) - OSINT RSS, deduplication, network centrality, graph analytics
@@ -750,12 +923,16 @@ print(f"Found {len(results)} results")
 
 ## 🔬 Advanced Features
 
+**Docling Integration** — Document parsing with table extraction for PDFs, DOCX, PPTX, and XLSX files. Supports OCR and multiple export formats.
+
+**AWS Neptune Support** — Amazon Neptune graph database integration with IAM authentication and OpenCypher queries.
+
+**Custom Ontology Import** — Import existing ontologies (OWL, RDF, Turtle, JSON-LD, N3) and extend Schema.org, FOAF, Dublin Core, or custom ontologies.
+
 **Incremental Updates** — Real-time stream processing with Kafka, RabbitMQ, Kinesis for live updates.
 
 **Multi-Language Support** — Process multiple languages with automatic detection.
 
-**Custom Ontology Import** — Import and extend Schema.org and custom ontologies.
-
 **Advanced Reasoning** — Forward/backward chaining, Rete-based pattern matching, and automated explanation generation.
 
 **Graph Analytics** — Centrality, community detection, path finding, temporal analysis.
@@ -766,20 +943,6 @@ print(f"Found {len(results)} results")
 
 [**See Advanced Examples**](https://github.com/Hawksight-AI/semantica/tree/main/cookbook/advanced) — Advanced extraction, graph analytics, reasoning, and more.
 
-## 🗺️ Roadmap
-
-### Q1 2026
-- [x] Core framework (v1.0)
-- [x] GraphRAG engine
-- [x] 6-stage ontology pipeline
-- [x] Advanced reasoning v2 (Rete, Forward/Backward Chaining)
-- [ ] Quality assurance features and Quality Assurance module
-- [ ] Enhanced multi-language support
-- [ ] Evals
-- [ ] Real-time streaming improvements
-
-### Q2 2026
-- [ ] Multi-modal processing
 
 ---
 
@@ -845,20 +1008,11 @@ git push origin feature/your-feature
 4. **Feature Requests** - [Request feature](https://github.com/Hawksight-AI/semantica/issues/new)
 
 
-###  Contributors
-
-<a href="https://github.com/Hawksight-AI/semantica/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=Hawksight-AI/semantica" alt="Contributors" />
-</a>
-
 ## 📜 License
 
 Semantica is licensed under the **MIT License** - see the [LICENSE](https://github.com/Hawksight-AI/semantica/blob/main/LICENSE) file for details.
 
-<div align="center">
-
 **Built by the Semantica Community**
 
-[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/pMHguUzG)
 
-</div>
+[GitHub](https://github.com/Hawksight-AI/semantica) • [Discord](https://discord.gg/RgaGTj9J)

RELEASE.md

@@ -26,10 +26,10 @@ Before releasing, ensure:
 
 The project uses GitHub Actions for automated releases to PyPI.
 
-1. **Tag the commit**: Create a new git tag for the version (e.g., `v0.1.1`).
+1.29. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.3`).
    ```bash
-   git tag -a v0.1.1 -m "Release v0.1.1"
-   git push origin v0.1.1
+   git tag -a v0.2.3 -m "Release v0.2.3"
+   git push origin v0.2.3
    ```
 2. **GitHub Action**: The `Release` workflow will automatically trigger, build the package, create a GitHub Release, and publish to PyPI using Trusted Publishing.
 

SECURITY.md

@@ -6,6 +6,10 @@ We actively support the following versions of Semantica with security updates:
 
 | Version | Supported          |
 | ------- | ------------------ |
+| 0.2.3   | :white_check_mark: |
+| 0.2.2   | :white_check_mark: |
+| 0.2.1   | :white_check_mark: |
+| 0.2.0   | :white_check_mark: |
 | 0.1.1   | :white_check_mark: |
 | 0.1.0   | :white_check_mark: |
 | < 0.1.0 | :x:                |

cookbook/introduction/21_Amazon_Neptune_Store.ipynb

@@ -0,0 +1,725 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Amazon Neptune Graph Store\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "This notebook covers the Amazon Neptune Database integration in Semantica. Amazon Neptune is a fully managed graph database service that supports both property graphs (via OpenCypher/Gremlin) and RDF graphs (via SPARQL).\n",
+    "\n",
+    "### Key Features\n",
+    "\n",
+    "- **IAM Authentication**: Secure access using AWS SigV4 signatures via AuthManager\n",
+    "- **OpenCypher Support**: Query using standard OpenCypher syntax\n",
+    "- **Bolt Protocol**: Uses Neo4j Bolt driver for efficient binary communication\n",
+    "- **Native ~id Support**: Leverages Neptune's native element ID handling\n",
+    "- **Full CRUD Operations**: Create, read, update, delete nodes and relationships\n",
+    "- **Automatic Retry**: Built-in retry logic with exponential backoff for transient errors\n",
+    "\n",
+    "### Prerequisites\n",
+    "\n",
+    "- An Amazon Neptune Database cluster\n",
+    "- AWS credentials configured (boto3, environment variables, or IAM role)\n",
+    "- Network access to your Neptune cluster (VPC, security groups)\n",
+    "\n",
+    "#### Quick Setup with CloudFormation\n",
+    "\n",
+    "If you don't have a Neptune cluster, use the provided CloudFormation template to create one with a public endpoint and IAM authentication:\n",
+    "\n",
+    "```bash\n",
+    "# Deploy the Neptune stack (takes ~15-20 minutes)\n",
+    "aws cloudformation create-stack \\\n",
+    "  --stack-name semantica-neptune \\\n",
+    "  --template-body file://neptune-setup.yaml \\\n",
+    "  --capabilities CAPABILITY_NAMED_IAM\n",
+    "\n",
+    "# Wait for stack creation to complete\n",
+    "aws cloudformation wait stack-create-complete --stack-name semantica-neptune\n",
+    "\n",
+    "# Get the outputs (endpoint, port, credentials)\n",
+    "aws cloudformation describe-stacks --stack-name semantica-neptune \\\n",
+    "  --query 'Stacks[0].Outputs' --output table\n",
+    "```\n",
+    "\n",
+    "The template creates:\n",
+    "- VPC with public subnets and Internet Gateway\n",
+    "- Neptune cluster (`db.t3.medium`) with IAM authentication enabled\n",
+    "- IAM user with least-privilege access for OpenCypher queries\n",
+    "- Security group allowing Bolt protocol (port 8182) access\n",
+    "\n",
+    "> ⚠️ **Security Note**: This template creates an IAM User with static access keys for simplicity in demo/test environments. For production use, we recommend IAM Roles (EC2 instance roles, ECS task roles, Lambda execution roles) which provide temporary credentials that are automatically rotated. The secret access key in the Cloudformation outputs is provided in plaintext to simplify initial setup - in production, use AWS Secrets Manager.\n",
+    "\n",
+    "**Outputs:**\n",
+    "- `NeptuneEndpoint` - Cluster hostname (use as `NEPTUNE_ENDPOINT`)\n",
+    "- `NeptunePort` - 8182 (use as `NEPTUNE_PORT`)\n",
+    "- `AwsAccessKeyId` - IAM user access key (use as `AWS_ACCESS_KEY_ID`)\n",
+    "- `AwsSecretAccessKey` - IAM user secret key in **plaintext** (use as `AWS_SECRET_ACCESS_KEY`)\n",
+    "- `AwsRegion` - Deployment region (use as `AWS_REGION`)\n",
+    "\n",
+    "**Cleanup:**\n",
+    "```bash\n",
+    "aws cloudformation delete-stack --stack-name semantica-neptune\n",
+    "```\n",
+    "\n",
+    "**Estimated Monthly Cost (approximately 100-105 USD/month at 100% utilization):**\n",
+    "\n",
+    "| Resource | Cost (USD) |\n",
+    "| --- | --- |\n",
+    "| Neptune db.t3.medium instance | ~96/month (0.132/hr) |\n",
+    "| Storage (10 GB) | ~1/month |\n",
+    "| I/O requests | ~1-5/month |\n",
+    "| Public IPv4 address | ~3.60/month (0.005/hr) |\n",
+    "| VPC, subnets, route tables, Internet Gateway, IAM | No Additional Charge |\n",
+    "\n",
+    "> **Free Tier**: New Neptune users get 30 days free (750 hours of db.t3.medium, 10M I/Os, 1 GB storage). Delete the stack when not in use to avoid charges.\n",
+    "\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Installation\n",
+    "\n",
+    "```bash\n",
+    "# Install Semantica with Neptune support\n",
+    "pip install semantica\n",
+    "\n",
+    "# Required dependencies (installed automatically)\n",
+    "pip install boto3 neo4j\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install semantica"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Configuration\n",
+    "\n",
+    "Set your Neptune cluster endpoint and AWS credentials. Replace the placeholder values with your actual configuration."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "# Neptune cluster configuration - REPLACE WITH YOUR VALUES\n",
+    "# (Get these from CloudFormation stack outputs)\n",
+    "os.environ[\"NEPTUNE_ENDPOINT\"] = \"your-cluster.us-east-1.neptune.amazonaws.com\"\n",
+    "os.environ[\"NEPTUNE_PORT\"] = \"8182\"\n",
+    "os.environ[\"AWS_REGION\"] = \"us-east-1\"\n",
+    "\n",
+    "# AWS credentials for IAM Authentication\n",
+    "# Option 1: IAM User (static credentials from CloudFormation template)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"AKIA...\"  # From AwsAccessKeyId output\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # From AwsSecretAccessKey output\n",
+    "# Note: No AWS_SESSION_TOKEN needed for IAM users\n",
+    "\n",
+    "# Option 2: IAM Role / Temporary credentials (e.g., STS AssumeRole, EC2 instance role)\n",
+    "# os.environ[\"AWS_ACCESS_KEY_ID\"] = \"ASIA...\"  # Temporary access key\n",
+    "# os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"...\"  # Temporary secret key\n",
+    "# os.environ[\"AWS_SESSION_TOKEN\"] = \"...\"      # REQUIRED for temporary credentials\n",
+    "\n",
+    "print(f\"Neptune Endpoint: {os.environ.get('NEPTUNE_ENDPOINT')}\")\n",
+    "print(f\"AWS Region: {os.environ.get('AWS_REGION')}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 1: Initialize Neptune Store\n",
+    "\n",
+    "Initialize a connection to your Amazon Neptune cluster with IAM authentication."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "from semantica.graph_store import GraphStore\n",
+    "\n",
+    "# Option 1: Using GraphStore factory (recommended)\n",
+    "neptune_store = GraphStore(\n",
+    "    backend=\"neptune\",\n",
+    "    endpoint=os.environ.get(\"NEPTUNE_ENDPOINT\"),\n",
+    "    port=int(os.environ.get(\"NEPTUNE_PORT\", 8182)),\n",
+    "    region=os.environ.get(\"AWS_REGION\", \"us-east-1\"),\n",
+    "    iam_auth=True,\n",
+    ")\n",
+    "\n",
+    "# Connect to Neptune\n",
+    "neptune_store.connect()\n",
+    "print(\"Connected to Amazon Neptune!\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Development/Testing Without IAM Auth\n",
+    "\n",
+    "For development or testing environments where IAM authentication is not required (e.g., Neptune notebooks or VPC-only access), you can disable IAM signing:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# For dev/test environments without IAM authentication\n",
+    "neptune_store_dev = GraphStore(\n",
+    "    backend=\"neptune\",\n",
+    "    endpoint=os.environ.get(\"NEPTUNE_ENDPOINT\"),\n",
+    "    port=int(os.environ.get(\"NEPTUNE_PORT\", 8182)),\n",
+    "    region=os.environ.get(\"AWS_REGION\", \"us-east-1\"),\n",
+    "    iam_auth=False,  # Disable IAM signing for dev/test\n",
+    ")\n",
+    "neptune_store_dev.connect()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Authentication Options\n",
+    "\n",
+    "IAM Authentication (recommended for production) automatically uses the AWS credential chain:\n",
+    "1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)\n",
+    "2. AWS credentials file (~/.aws/credentials)\n",
+    "3. IAM role (for EC2, Lambda, ECS)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 2: Node Operations\n",
+    "\n",
+    "### Creating Nodes\n",
+    "\n",
+    "Nodes represent entities in your graph. Each node can have:\n",
+    "- **ID**: A unique identifier (custom or auto-generated UUID)\n",
+    "- **Labels**: Categories/types (e.g., `Person`, `Company`)\n",
+    "- **Properties**: Key-value pairs (e.g., `{\"name\": \"Alice\", \"age\": 30}`)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create a single node with custom ID (id in properties)\n",
+    "alice = neptune_store.create_node(\n",
+    "    labels=[\"Person\"],\n",
+    "    properties={\"id\": \"alice\", \"name\": \"Alice\", \"age\": 30, \"role\": \"Engineer\"}\n",
+    ")\n",
+    "print(f\"Created node: {alice}\")\n",
+    "\n",
+    "# Create a node with auto-generated UUID (no id in properties)\n",
+    "bob = neptune_store.create_node(\n",
+    "    labels=[\"Person\"],\n",
+    "    properties={\"name\": \"Bob\", \"age\": 25, \"role\": \"Designer\"}\n",
+    ")\n",
+    "print(f\"Created node with UUID: {bob['id']}\")\n",
+    "\n",
+    "# Create a company node with auto-generated ID\n",
+    "acme = neptune_store.create_node(\n",
+    "    labels=[\"Company\"],\n",
+    "    properties={\"name\": \"Acme Corp\", \"industry\": \"Technology\", \"founded\": 2010}\n",
+    ")\n",
+    "print(f\"Created company: {acme}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Creating Multiple Nodes (Batch)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Batch create nodes for better performance\n",
+    "# Include 'id' in properties for custom IDs\n",
+    "nodes_data = [\n",
+    "    {\"labels\": [\"Person\"], \"properties\": {\"id\": \"charlie\", \"name\": \"Charlie\", \"age\": 35}},\n",
+    "    {\"labels\": [\"Person\"], \"properties\": {\"id\": \"diana\", \"name\": \"Diana\", \"age\": 28}},\n",
+    "    {\"labels\": [\"Location\"], \"properties\": {\"name\": \"San Francisco\", \"state\": \"CA\"}},\n",
+    "]\n",
+    "\n",
+    "created_nodes = neptune_store.create_nodes(nodes_data)\n",
+    "print(f\"Created {len(created_nodes)} nodes in batch\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Retrieving Nodes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get a specific node by ID\n",
+    "alice_node = neptune_store.get_node(node_id=\"alice\")\n",
+    "print(f\"Retrieved: {alice_node}\")\n",
+    "\n",
+    "# Get nodes by label\n",
+    "people = neptune_store.get_nodes(labels=[\"Person\"], limit=10)\n",
+    "print(f\"Found {len(people)} Person nodes:\")\n",
+    "for person in people:\n",
+    "    print(f\"  - {person.get('properties', {}).get('name')}\")\n",
+    "\n",
+    "# Get nodes by properties\n",
+    "engineers = neptune_store.get_nodes(\n",
+    "    labels=[\"Person\"],\n",
+    "    properties={\"role\": \"Engineer\"},\n",
+    "    limit=5\n",
+    ")\n",
+    "print(f\"Found {len(engineers)} engineers\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Updating Nodes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Update node properties (merge mode - default)\n",
+    "updated_alice = neptune_store.update_node(\n",
+    "    node_id=\"alice\",\n",
+    "    properties={\"age\": 31, \"department\": \"AI Research\"},\n",
+    "    merge=True\n",
+    ")\n",
+    "print(f\"Updated Alice: {updated_alice}\")\n",
+    "\n",
+    "# Replace all properties (merge=False)\n",
+    "# WARNING: This removes properties not in the update\n",
+    "replaced = neptune_store.update_node(\n",
+    "    node_id=\"charlie\",\n",
+    "    properties={\"name\": \"Charlie\", \"age\": 36},\n",
+    "    merge=False\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Deleting Nodes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Delete a node (with detach=True to also delete relationships)\n",
+    "deleted = neptune_store.delete_node(node_id=\"diana\", detach=True)\n",
+    "print(f\"Deleted diana: {deleted}\")\n",
+    "\n",
+    "# Without detach (fails if node has relationships)\n",
+    "# neptune_store.delete_node(node_id=\"alice\", detach=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 3: Relationship Operations\n",
+    "\n",
+    "### Creating Relationships\n",
+    "\n",
+    "Relationships connect nodes and represent connections between entities."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create a relationship between Alice and Acme\n",
+    "works_at = neptune_store.create_relationship(\n",
+    "    start_node_id=\"alice\",\n",
+    "    end_node_id=acme[\"id\"],\n",
+    "    rel_type=\"WORKS_AT\",\n",
+    "    properties={\"since\": 2020, \"position\": \"Senior Engineer\"}\n",
+    ")\n",
+    "print(f\"Created relationship: {works_at}\")\n",
+    "\n",
+    "# Create a KNOWS relationship between people\n",
+    "knows_rel = neptune_store.create_relationship(\n",
+    "    start_node_id=\"alice\",\n",
+    "    end_node_id=bob[\"id\"],\n",
+    "    rel_type=\"KNOWS\",\n",
+    "    properties={\"since\": 2019}\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Retrieving Relationships"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get all relationships for a node\n",
+    "alice_rels = neptune_store.get_relationships(node_id=\"alice\", direction=\"both\")\n",
+    "print(f\"Alice has {len(alice_rels)} relationships\")\n",
+    "\n",
+    "# Get outgoing relationships only\n",
+    "outgoing = neptune_store.get_relationships(node_id=\"alice\", direction=\"out\")\n",
+    "\n",
+    "# Filter by relationship type\n",
+    "works_rels = neptune_store.get_relationships(\n",
+    "    node_id=\"alice\",\n",
+    "    rel_type=\"WORKS_AT\",\n",
+    "    direction=\"out\"\n",
+    ")\n",
+    "print(f\"Alice's work relationships: {len(works_rels)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Deleting Relationships"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Delete a specific relationship by ID\n",
+    "if works_at.get(\"id\"):\n",
+    "    deleted = neptune_store.delete_relationship(rel_id=works_at[\"id\"])\n",
+    "    print(f\"Deleted relationship: {deleted}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 4: OpenCypher Queries\n",
+    "\n",
+    "Amazon Neptune supports OpenCypher queries via the Bolt protocol. Execute complex graph patterns using standard Cypher syntax."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Simple query\n",
+    "results = neptune_store.execute_query(\n",
+    "    \"MATCH (p:Person) RETURN p.name, p.age ORDER BY p.age\"\n",
+    ")\n",
+    "print(\"People in the graph:\")\n",
+    "for record in results.get(\"records\", []):\n",
+    "    print(f\"  - {record.get('p.name')}: {record.get('p.age')} years old\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Using parameters (safer and more efficient)\n",
+    "results = neptune_store.execute_query(\n",
+    "    \"MATCH (p:Person) WHERE p.age > $min_age RETURN p.name, p.age\",\n",
+    "    parameters={\"min_age\": 25}\n",
+    ")\n",
+    "print(f\"People over 25: {len(results.get('records', []))}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Find relationships between nodes\n",
+    "results = neptune_store.execute_query(\"\"\"\n",
+    "    MATCH (p:Person)-[r:WORKS_AT]->(c:Company)\n",
+    "    RETURN p.name as employee, c.name as company, r.since as start_year\n",
+    "\"\"\")\n",
+    "for record in results.get(\"records\", []):\n",
+    "    print(f\"{record['employee']} works at {record['company']} since {record['start_year']}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Count and aggregate\n",
+    "results = neptune_store.execute_query(\"\"\"\n",
+    "    MATCH (p:Person)\n",
+    "    RETURN count(p) as total, avg(p.age) as avg_age, max(p.age) as max_age\n",
+    "\"\"\")\n",
+    "stats = results.get(\"records\", [{}])[0]\n",
+    "print(f\"Total: {stats.get('total')}, Avg Age: {stats.get('avg_age'):.1f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 5: Graph Analytics\n",
+    "\n",
+    "### Get Neighbors\n",
+    "\n",
+    "Traverse the graph to find connected nodes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get immediate neighbors (depth=1)\n",
+    "neighbors = neptune_store.get_neighbors(\n",
+    "    node_id=\"alice\",\n",
+    "    direction=\"both\",\n",
+    "    depth=1\n",
+    ")\n",
+    "print(f\"Alice's direct neighbors: {len(neighbors)}\")\n",
+    "\n",
+    "# Get neighbors up to 2 hops away\n",
+    "extended = neptune_store.get_neighbors(\n",
+    "    node_id=\"alice\",\n",
+    "    direction=\"out\",\n",
+    "    depth=2\n",
+    ")\n",
+    "print(f\"Nodes within 2 hops: {len(extended)}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Shortest Path\n",
+    "\n",
+    "Find the shortest path between two nodes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Find shortest path\n",
+    "path = neptune_store.shortest_path(\n",
+    "    start_node_id=\"alice\",\n",
+    "    end_node_id=\"charlie\",\n",
+    "    max_depth=5\n",
+    ")\n",
+    "\n",
+    "if path:\n",
+    "    print(\"Path found!\")\n",
+    "    print(f\"  Length: {path.get('length')}\")\n",
+    "    print(f\"  Nodes: {len(path.get('nodes', []))}\")\n",
+    "    print(f\"  Relationships: {len(path.get('relationships', []))}\")\n",
+    "else:\n",
+    "    print(\"No path found between nodes\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 6: Graph Statistics\n",
+    "\n",
+    "Get comprehensive statistics about your graph."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Get graph statistics\n",
+    "stats = neptune_store.get_stats()\n",
+    "\n",
+    "print(\"Graph Statistics:\")\n",
+    "print(f\"  Total nodes: {stats.get('node_count', 'N/A')}\")\n",
+    "print(f\"  Total relationships: {stats.get('relationship_count', 'N/A')}\")\n",
+    "\n",
+    "print(\"\\nNode labels:\")\n",
+    "for label, count in stats.get('label_counts', {}).items():\n",
+    "    print(f\"  - {label}: {count}\")\n",
+    "\n",
+    "print(\"\\nRelationship types:\")\n",
+    "for rel_type, count in stats.get('relationship_type_counts', {}).items():\n",
+    "    print(f\"  - {rel_type}: {count}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Step 7: Connection Management\n",
+    "\n",
+    "Always close connections when done to free resources."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Check connection status\n",
+    "status = neptune_store.get_status()\n",
+    "print(f\"Connection status: {status}\")\n",
+    "\n",
+    "# Close the connection\n",
+    "neptune_store.close()\n",
+    "print(\"Connection closed\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Neptune-Specific Considerations\n",
+    "\n",
+    "### Native Element IDs\n",
+    "\n",
+    "Neptune uses native `~id` for element identification. Include `id` in properties to set a custom ID:\n",
+    "\n",
+    "```python\n",
+    "# Create a node with custom ID (include 'id' in properties)\n",
+    "node = neptune_store.create_node(\n",
+    "    labels=[\"Person\"],\n",
+    "    properties={\"id\": \"my-custom-id\", \"name\": \"Test\"}\n",
+    ")\n",
+    "\n",
+    "# Create a node with auto-generated UUID (omit 'id' from properties)\n",
+    "node = neptune_store.create_node(\n",
+    "    labels=[\"Person\"],\n",
+    "    properties={\"name\": \"Test\"}\n",
+    ")\n",
+    "\n",
+    "# The ID is used in id() function calls internally:\n",
+    "# MATCH (n) WHERE id(n) = 'my-custom-id' RETURN n\n",
+    "```\n",
+    "\n",
+    "### OpenCypher Considerations\n",
+    "\n",
+    "Amazon Neptune Database's OpenCypher implementation has some differences from Neo4j:\n",
+    "\n",
+    "1. **No `shortestPath()` function**: Use variable-length path patterns or `allShortestPaths()`\n",
+    "2. **Labels syntax**: Use `labels(n)` function to retrieve node labels\n",
+    "3. **Property updates**: Use `SET n += {props}` for merge behavior\n",
+    "\n",
+    "For the complete OpenCypher specification supported by Amazon Neptune Database, see the [AWS documentation](https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-opencypher.html).\n",
+    "\n",
+    "### Amazon Neptune Analytics\n",
+    "\n",
+    "For analytical (OLAP) workloads such as graph algorithms, aggregations, and large-scale traversals, consider [Amazon Neptune Analytics](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/what-is-neptune-analytics.html). Neptune Analytics complements Neptune Database by providing optimized performance for analytical queries while Neptune Database is optimized for transactional (OLTP) workloads.\n",
+    "\n",
+    "### Performance Tips\n",
+    "\n",
+    "1. **Use batch operations** for creating multiple nodes/relationships\n",
+    "2. **Use parameters** in queries to enable query caching\n",
+    "3. **Limit result sets** with `LIMIT` clause"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "\n",
+    "This notebook covered the Amazon Neptune Graph Store integration:\n",
+    "\n",
+    "- **IAM Authentication**: Secure AWS SigV4 signing\n",
+    "- **CRUD Operations**: Full node and relationship management\n",
+    "- **OpenCypher Queries**: Standard graph query language\n",
+    "- **Graph Analytics**: Neighbors and shortest path algorithms\n",
+    "- **Statistics & Monitoring**: Graph metrics and status\n",
+    "\n",
+    "### Key Takeaways\n",
+    "\n",
+    "- Neptune uses native `~id` for element identification\n",
+    "- IAM authentication is recommended for production\n",
+    "- Bolt protocol provides efficient binary query interface\n",
+    "- Semantica abstracts Neptune-specific syntax differences\n",
+    "\n",
+    "### Next Steps\n",
+    "\n",
+    "- [Graph Store (Neo4j/FalkorDB)](09_Graph_Store.ipynb) - Compare with other backends\n",
+    "- [Building Knowledge Graphs](07_Building_Knowledge_Graphs.ipynb) - Build production KGs\n",
+    "- [Graph Analytics](10_Graph_Analytics.ipynb) - Advanced analytics algorithms"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.9.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

cookbook/introduction/neptune-setup.yaml

@@ -0,0 +1,228 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: >
+  Amazon Neptune cluster with public endpoint, IAM authentication, and least-privilege
+  IAM user for Semantica cookbook. Uses db.t3.medium (most cost-effective Neptune instance type).
+
+Parameters:
+  EnvironmentName:
+    Type: String
+    Default: semantica-neptune
+    Description: Environment name prefix for resource naming
+
+Resources:
+  # =============================================================================
+  # VPC & NETWORKING
+  # =============================================================================
+
+  VPC:
+    Type: AWS::EC2::VPC
+    Properties:
+      CidrBlock: 10.0.0.0/16
+      EnableDnsHostnames: true
+      EnableDnsSupport: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-vpc
+
+  InternetGateway:
+    Type: AWS::EC2::InternetGateway
+    Properties:
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-igw
+
+  InternetGatewayAttachment:
+    Type: AWS::EC2::VPCGatewayAttachment
+    Properties:
+      InternetGatewayId: !Ref InternetGateway
+      VpcId: !Ref VPC
+
+  PublicSubnet1:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [0, !GetAZs '']
+      CidrBlock: 10.0.1.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-1
+
+  PublicSubnet2:
+    Type: AWS::EC2::Subnet
+    Properties:
+      VpcId: !Ref VPC
+      AvailabilityZone: !Select [1, !GetAZs '']
+      CidrBlock: 10.0.2.0/24
+      MapPublicIpOnLaunch: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-subnet-2
+
+  PublicRouteTable:
+    Type: AWS::EC2::RouteTable
+    Properties:
+      VpcId: !Ref VPC
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-public-rt
+
+  DefaultPublicRoute:
+    Type: AWS::EC2::Route
+    DependsOn: InternetGatewayAttachment
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      DestinationCidrBlock: 0.0.0.0/0
+      GatewayId: !Ref InternetGateway
+
+  PublicSubnet1RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet1
+
+  PublicSubnet2RouteTableAssociation:
+    Type: AWS::EC2::SubnetRouteTableAssociation
+    Properties:
+      RouteTableId: !Ref PublicRouteTable
+      SubnetId: !Ref PublicSubnet2
+
+  # =============================================================================
+  # SECURITY GROUP
+  # =============================================================================
+
+  NeptuneSecurityGroup:
+    Type: AWS::EC2::SecurityGroup
+    Properties:
+      GroupName: !Sub ${EnvironmentName}-neptune-sg
+      GroupDescription: Security group for Neptune cluster - allows Bolt protocol access
+      VpcId: !Ref VPC
+      SecurityGroupIngress:
+        - IpProtocol: tcp
+          FromPort: 8182
+          ToPort: 8182
+          CidrIp: 0.0.0.0/0
+          Description: Allow Bolt protocol access from anywhere
+      SecurityGroupEgress:
+        - IpProtocol: -1
+          CidrIp: 0.0.0.0/0
+          Description: Allow all outbound traffic
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-neptune-sg
+
+  # =============================================================================
+  # NEPTUNE CLUSTER
+  # =============================================================================
+
+  NeptuneSubnetGroup:
+    Type: AWS::Neptune::DBSubnetGroup
+    Properties:
+      DBSubnetGroupDescription: Subnet group for Neptune cluster
+      DBSubnetGroupName: !Sub ${EnvironmentName}-subnet-group
+      SubnetIds:
+        - !Ref PublicSubnet1
+        - !Ref PublicSubnet2
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-subnet-group
+
+  NeptuneCluster:
+    Type: AWS::Neptune::DBCluster
+    Properties:
+      DBClusterIdentifier: !Sub ${EnvironmentName}-cluster
+      DBSubnetGroupName: !Ref NeptuneSubnetGroup
+      VpcSecurityGroupIds:
+        - !Ref NeptuneSecurityGroup
+      EngineVersion: '1.4.6.3'
+      IamAuthEnabled: true
+      StorageEncrypted: true
+      DeletionProtection: false
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-cluster
+
+  NeptuneInstance:
+    Type: AWS::Neptune::DBInstance
+    Properties:
+      DBInstanceIdentifier: !Sub ${EnvironmentName}-instance
+      DBInstanceClass: db.t3.medium
+      DBClusterIdentifier: !Ref NeptuneCluster
+      PubliclyAccessible: true
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-instance
+
+  # =============================================================================
+  # IAM USER WITH LEAST PRIVILEGES
+  # =============================================================================
+
+  NeptuneUser:
+    Type: AWS::IAM::User
+    Properties:
+      UserName: !Sub ${EnvironmentName}-user
+      Tags:
+        - Key: Name
+          Value: !Sub ${EnvironmentName}-user
+
+  NeptuneUserPolicy:
+    Type: AWS::IAM::Policy
+    Properties:
+      PolicyName: !Sub ${EnvironmentName}-neptune-access
+      Users:
+        - !Ref NeptuneUser
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Sid: NeptuneDataAccess
+            Effect: Allow
+            Action:
+              - neptune-db:connect
+              - neptune-db:ReadDataViaQuery
+              - neptune-db:WriteDataViaQuery
+              - neptune-db:DeleteDataViaQuery
+            Resource: !Sub
+              - arn:aws:neptune-db:${AWS::Region}:${AWS::AccountId}:${ClusterResourceId}/*
+              - ClusterResourceId: !GetAtt NeptuneCluster.ClusterResourceId
+
+  NeptuneUserAccessKey:
+    Type: AWS::IAM::AccessKey
+    Properties:
+      UserName: !Ref NeptuneUser
+
+# =============================================================================
+# OUTPUTS
+# =============================================================================
+
+Outputs:
+  NeptuneEndpoint:
+    Description: Neptune cluster endpoint (hostname only) - use as NEPTUNE_ENDPOINT
+    Value: !GetAtt NeptuneCluster.Endpoint
+
+  NeptunePort:
+    Description: Neptune cluster port - use as NEPTUNE_PORT
+    Value: !GetAtt NeptuneCluster.Port
+
+  AwsAccessKeyId:
+    Description: Access key ID for the Neptune IAM user - use as AWS_ACCESS_KEY_ID
+    Value: !Ref NeptuneUserAccessKey
+
+  AwsSecretAccessKey:
+    Description: Secret access key for the Neptune IAM user - use as AWS_SECRET_ACCESS_KEY
+    Value: !GetAtt NeptuneUserAccessKey.SecretAccessKey
+
+  AwsRegion:
+    Description: AWS region where Neptune is deployed - use as AWS_REGION
+    Value: !Ref AWS::Region
+
+  NeptuneClusterResourceId:
+    Description: Neptune cluster resource ID (for IAM policy reference)
+    Value: !GetAtt NeptuneCluster.ClusterResourceId
+
+  VpcId:
+    Description: VPC ID
+    Value: !Ref VPC
+
+  SecurityGroupId:
+    Description: Neptune security group ID
+    Value: !Ref NeptuneSecurityGroup

cookbook/use_cases/advanced_rag/01_GraphRAG_Complete.ipynb

@@ -110,7 +110,7 @@
    "source": [
     "# Set up API keys\n",
     "# Note: In production, use environment variables: export GROQ_API_KEY=\"your-key\"\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"Your Groq API\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/advanced_rag/02_RAG_vs_GraphRAG_Comparison.ipynb

@@ -30,7 +30,7 @@
     "# Environment Setup\n",
     "import os\n",
     "\n",
-    "os.environ['GROQ_API_KEY'] = os.getenv('GROQ_API_KEY', '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')\n",
+    "os.environ['GROQ_API_KEY'] = os.getenv('GROQ_API_KEY', '')\n",
     "\n",
     "# Install Semantica and all required dependencies\n",
     "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers\n"
@@ -84,7 +84,7 @@
    "source": [
     "# Set up API keys\n",
     "# Note: In production, use environment variables: export GROQ_API_KEY=\"your-key\"\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"your-groq-api-key-here\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "print(\"API keys configured.\")\n"
    ]

cookbook/use_cases/biomedical/01_Drug_Discovery_Pipeline.ipynb

@@ -109,7 +109,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/biomedical/02_Genomic_Variant_Analysis.ipynb

@@ -85,7 +85,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n"
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n"
    ]
   },
   {

cookbook/use_cases/blockchain/01_DeFi_Protocol_Intelligence.ipynb

@@ -81,7 +81,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/finance/02_Fraud_Detection.ipynb

@@ -98,7 +98,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/intelligence/02_Intelligence_Analysis_Orchestrator_Worker.ipynb

@@ -83,7 +83,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb

@@ -80,7 +80,7 @@
    "source": [
     "import os\n",
     "\n",
-    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
+    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
     "\n",
     "# Configuration constants\n",
     "EMBEDDING_DIMENSION = 384\n",

docs/CodeExamples.md

@@ -138,6 +138,39 @@ async for item in feed_processor.stream_items():
     knowledge_graph.add_triplets(core.generate_triplets(semantics))
 ```
 
+### 🦆 Docling Clear Code Example
+
+High-accuracy document parsing with structural understanding:
+
+```python
+from semantica.parse import DoclingParser
+
+# 1. Initialize DoclingParser
+# Docling provides superior table extraction and structure understanding
+# Requires: pip install docling
+parser = DoclingParser(
+    enable_ocr=True,          # Enable OCR for scanned documents
+    export_format="markdown"  # Options: "markdown", "html", "json"
+)
+
+# 2. Parse a complex document
+# Supports PDF, DOCX, PPTX, XLSX, HTML, and images
+result = parser.parse("complex_invoice.pdf")
+
+# 3. Access structured content
+print(f"Content (Markdown):\n{result['full_text']}")
+
+# 4. Extract and iterate over tables with high precision
+for i, table in enumerate(result['tables']):
+    print(f"\nTable {i+1}:")
+    print(f"Headers: {table.get('headers', [])}")
+    print(f"Data rows: {len(table.get('rows', []))}")
+
+# 5. Get document metadata
+metadata = result['metadata']
+print(f"\nMetadata: {metadata.get('title')} ({result.get('total_pages')} pages)")
+```
+
 ### 📊 Structured Data Processing Module
 
 Handle structured and semi-structured data formats:

docs/DOCS_README.md

@@ -46,7 +46,7 @@ semantica/
 │   │   └── custom.css   # Custom styling
 │   └── assets/
 │       └── img/
-│           └── semantica_logo.png
+│           └── Semantica Updated Logo.png
 └── site/                # Generated site (created by mkdocs build)
 ```
 

docs/citation.md

@@ -12,22 +12,22 @@ How to cite Semantica in academic papers and research.
   author = {Hawksight AI},
   year = {2026},
   url = {https://github.com/Hawksight-AI/semantica},
-  version = {0.1.1},
+  version = {0.2.5},
   doi = {10.5281/zenodo.XXXXXXX}
 }
 ```
 
 ### APA
-Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.1.1) [Computer software]. https://github.com/Hawksight-AI/semantica
+Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.5) [Computer software]. https://github.com/Hawksight-AI/semantica
 
 ### MLA
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.1.1, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.5, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
 
 ### Chicago
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.1.1. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.5. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
 
 ### IEEE
-Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.1.1, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
+Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.5, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
 
 ---
 

docs/index.md

@@ -1,5 +1,5 @@
 <div align="center">
-  <img src="assets/img/semantica_logo.png" alt="Semantica Logo" width="450" height="auto">
+  <img src="assets/img/Semantica Updated Logo.png" alt="Semantica Logo" width="450" height="auto">
   
   <h1>🧠 Semantica</h1>
   
@@ -17,7 +17,7 @@
   
   <p><em>The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.</em></p>
   
-  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.1.1</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
+  <p>🆓 <strong>100% Open Source</strong> • 📜 <strong>MIT Licensed</strong> • 🚀 <strong>Latest Version: 0.2.3</strong> • 🚀 <strong>Production Ready</strong> • 🌍 <strong>Community Driven</strong></p>
   
   <p>
     <a href="getting-started/" class="md-button md-button--primary">Get Started</a>

docs/integrations/docling.md

@@ -0,0 +1,107 @@
+# Docling Integration
+
+Semantica features a native integration with **Docling**, the powerful document parsing library that excels at extracting structured data from complex documents like PDFs, DOCX, and PPTX.
+
+## Overview
+
+Docling is integrated into Semantica's `parse` module via the `DoclingParser`. This allows you to seamlessly convert unstructured documents into semantic structures that can be indexed, searched, and analyzed within the Semantica framework.
+
+- 📖 **Semantica Docling Integration Docs**: [Reference Guide](../reference/parse.md)
+- 💻 **Semantica Docling Integration GitHub**: [Source Code](https://github.com/Hawksight-AI/semantica/blob/main/semantica/parse/docling_parser.py)
+- 🧑🏽‍🍳 **Semantica Docling Integration Example**: [Docling Clear Code Example](../CodeExamples.md#docling-clear-code-example)
+- 📦 **Semantica Docling Integration PyPI**: [Installation Guide](../installation.md)
+
+---
+
+## 📖 Integration Documentation
+
+The `DoclingParser` provides a high-level interface for document processing. It supports:
+
+*   **Multi-format support**: PDF, DOCX, PPTX, HTML, and more.
+*   **Table Extraction**: High-fidelity table extraction with header detection.
+*   **OCR Support**: Built-in Optical Character Recognition for scanned documents.
+*   **Markdown Export**: Clean markdown output optimized for LLM consumption.
+
+### Basic Usage
+
+```python
+from semantica.parse import DoclingParser
+
+# Initialize with OCR enabled
+parser = DoclingParser(enable_ocr=True)
+
+# Parse a complex document
+result = parser.parse("financial_report.pdf")
+
+# Access the structured data
+print(f"Content: {result['full_text'][:200]}...")
+print(f"Found {len(result['tables'])} tables")
+```
+
+For more details, see the [Parse Reference](../reference/parse.md).
+
+---
+
+## 🧑🏽‍🍳 Integration Example
+
+We provide a detailed cookbook and clear code examples to help you get started quickly.
+
+### Docling Clear Code Example
+
+```python
+from semantica.parse import DoclingParser
+import json
+
+# 1. Initialize the Docling Parser with advanced config
+parser = DoclingParser(
+    enable_ocr=True, 
+    export_format="markdown"
+)
+
+# 2. Parse a complex document (PDF, DOCX, etc.)
+result = parser.parse("complex_invoice.pdf")
+
+# 3. Access the clean Markdown text
+print(f"--- Document Content ---\n{result['full_text']}")
+
+# 4. Iterate through extracted tables
+for i, table in enumerate(result['tables']):
+    print(f"\nTable {i+1} headers: {table.get('headers', [])}")
+    # Access table rows as a list of lists
+    for row in table.get('rows', [])[:3]:  # Print first 3 rows
+        print(f"  Row: {row}")
+
+# 5. Get document metadata
+metadata = result['metadata']
+print(f"\n--- Metadata ---\nTitle: {metadata.get('title')}")
+print(f"Total Pages: {result.get('total_pages')}")
+```
+
+See more in our [Code Examples](../CodeExamples.md).
+
+---
+
+## 💻 GitHub Source
+
+The integration is open-source and available on GitHub. You can explore the implementation, contribute improvements, or report issues.
+
+- [docling_parser.py](https://github.com/Hawksight-AI/semantica/blob/main/semantica/parse/docling_parser.py) - The core implementation of the Docling integration.
+
+---
+
+## 📦 PyPI & Installation
+
+Docling is an optional but highly recommended dependency for Semantica. You can install it along with Semantica or as a separate requirement.
+
+### Install via Semantica
+```bash
+pip install semantica
+```
+
+### Install Docling manually
+If you are working in a custom environment:
+```bash
+pip install docling
+```
+
+For full installation details, see the [Installation Guide](../installation.md).

docs/js/version-selector.js

@@ -9,7 +9,12 @@ document.addEventListener("DOMContentLoaded", function () {
 
         // Define versions
         var versions = [
-            { name: "0.1.1", url: "#", current: true },
+            { name: "0.2.4", url: "#", current: true },
+            { name: "0.2.3", url: "#", current: false },
+            { name: "0.2.2", url: "#", current: false },
+            { name: "0.2.1", url: "#", current: false },
+            { name: "0.2.0", url: "#", current: false },
+            { name: "0.1.1", url: "#", current: false },
             { name: "0.1.0", url: "#", current: false }
         ];
 

docs/reference/change_management.md

@@ -0,0 +1,938 @@
+# Change Management API Reference
+
+Comprehensive API documentation for the Enhanced Change Management module in Semantica.
+
+## Overview
+
+The `semantica.change_management` module provides enterprise-grade version control, audit trails, and compliance tracking for knowledge graphs and ontologies. It includes persistent storage backends, detailed change tracking, data integrity verification, and standardized metadata structures.
+
+## Module Structure
+
+```
+semantica.change_management/
+├── change_log.py              # Standardized metadata structures
+├── version_storage.py         # Storage abstraction and implementations
+├── managers.py                # Enhanced version managers
+├── ontology_version_manager.py # Ontology version management
+└── change_management_usage.md # Usage guide
+```
+
+## Quick Import
+
+```python
+from semantica.change_management import (
+    # Metadata
+    ChangeLogEntry,
+    
+    # Storage
+    VersionStorage,
+    InMemoryVersionStorage,
+    SQLiteVersionStorage,
+    
+    # Utilities
+    compute_checksum,
+    verify_checksum,
+    
+    # Version Managers
+    BaseVersionManager,
+    TemporalVersionManager,
+    OntologyVersionManager,
+    VersionManager,
+    OntologyVersion
+)
+```
+
+---
+
+## Core Classes
+
+### ChangeLogEntry
+
+Standardized metadata structure for version changes with validation.
+
+#### Class Definition
+
+```python
+@dataclass
+class ChangeLogEntry:
+    """
+    Standardized change log entry with validation.
+    
+    Attributes:
+        timestamp: ISO 8601 formatted timestamp
+        author: Email address of the change author
+        description: Change description (max 500 characters)
+        change_id: Optional ID linking to external systems
+    """
+    timestamp: str
+    author: str
+    description: str
+    change_id: Optional[str] = None
+```
+
+#### Methods
+
+##### `__post_init__()`
+
+Validates all fields after initialization.
+
+**Raises:**
+- `ValidationError`: If any field validation fails
+
+**Example:**
+```python
+entry = ChangeLogEntry(
+    timestamp="2024-01-30T12:00:00Z",
+    author="user@example.com",
+    description="Updated entity relationships",
+    change_id="TICKET-123"
+)
+```
+
+##### `create_now(author, description, change_id=None)` (classmethod)
+
+Creates a change log entry with the current timestamp.
+
+**Parameters:**
+- `author` (str): Email address of the change author
+- `description` (str): Change description (max 500 characters)
+- `change_id` (str, optional): ID linking to external systems
+
+**Returns:**
+- `ChangeLogEntry`: New instance with current timestamp
+
+**Example:**
+```python
+entry = ChangeLogEntry.create_now(
+    author="developer@company.com",
+    description="Fixed entity resolution bug",
+    change_id="JIRA-1234"
+)
+```
+
+#### Validation Rules
+
+- **Timestamp**: Must be valid ISO 8601 format with 'T' separator
+- **Author**: Must be valid email format (RFC 5322)
+- **Description**: Maximum 500 characters
+- **Change ID**: Optional, no validation
+
+---
+
+### VersionStorage
+
+Abstract base class for storage implementations.
+
+#### Class Definition
+
+```python
+class VersionStorage(ABC):
+    """
+    Abstract base class for version storage backends.
+    
+    Provides interface for saving, retrieving, and managing version snapshots.
+    """
+```
+
+#### Abstract Methods
+
+##### `save(snapshot)`
+
+Save a version snapshot.
+
+**Parameters:**
+- `snapshot` (Dict[str, Any]): Version snapshot dictionary with metadata
+
+**Raises:**
+- `ValidationError`: If snapshot data is invalid
+- `ProcessingError`: If save operation fails
+
+**Example:**
+```python
+snapshot = {
+    "label": "v1.0",
+    "timestamp": "2024-01-30T12:00:00Z",
+    "author": "user@example.com",
+    "description": "Initial version",
+    "data": {...}
+}
+storage.save(snapshot)
+```
+
+##### `get(label)`
+
+Retrieve a version snapshot by label.
+
+**Parameters:**
+- `label` (str): Version label to retrieve
+
+**Returns:**
+- `Optional[Dict[str, Any]]`: Snapshot dictionary or None if not found
+
+**Example:**
+```python
+snapshot = storage.get("v1.0")
+if snapshot:
+    print(f"Retrieved: {snapshot['label']}")
+```
+
+##### `list_all()`
+
+List all version snapshots.
+
+**Returns:**
+- `List[Dict[str, Any]]`: List of snapshot metadata dictionaries
+
+**Example:**
+```python
+versions = storage.list_all()
+for v in versions:
+    print(f"{v['label']}: {v['description']}")
+```
+
+##### `exists(label)`
+
+Check if a version exists.
+
+**Parameters:**
+- `label` (str): Version label to check
+
+**Returns:**
+- `bool`: True if version exists, False otherwise
+
+**Example:**
+```python
+if storage.exists("v1.0"):
+    print("Version exists")
+```
+
+##### `delete(label)`
+
+Delete a version snapshot.
+
+**Parameters:**
+- `label` (str): Version label to delete
+
+**Returns:**
+- `bool`: True if deleted, False if not found
+
+**Example:**
+```python
+if storage.delete("v1.0"):
+    print("Version deleted")
+```
+
+---
+
+### InMemoryVersionStorage
+
+In-memory version storage implementation.
+
+#### Class Definition
+
+```python
+class InMemoryVersionStorage(VersionStorage):
+    """
+    In-memory version storage implementation.
+    
+    Fast, volatile storage for development and testing.
+    Data is lost when the process ends.
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self):
+    """Initialize in-memory storage."""
+```
+
+**Example:**
+```python
+storage = InMemoryVersionStorage()
+```
+
+#### Performance Characteristics
+
+- **Save**: 0.37-16ms (10-1000 entities)
+- **Get**: 0.20-16ms (10-1000 entities)
+- **List**: <0.03ms
+- **Thread-safe**: Yes (uses RLock)
+
+#### Use Cases
+
+- Development and testing
+- Temporary version tracking
+- High-performance scenarios where persistence is not required
+
+---
+
+### SQLiteVersionStorage
+
+SQLite-based persistent version storage implementation.
+
+#### Class Definition
+
+```python
+class SQLiteVersionStorage(VersionStorage):
+    """
+    SQLite-based persistent version storage implementation.
+    
+    Provides persistence across process restarts with ACID guarantees.
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: str):
+    """
+    Initialize SQLite storage.
+    
+    Args:
+        storage_path: Path to SQLite database file
+    """
+```
+
+**Parameters:**
+- `storage_path` (str): Path to SQLite database file (created if doesn't exist)
+
+**Example:**
+```python
+storage = SQLiteVersionStorage("versions.db")
+```
+
+#### Database Schema
+
+```sql
+CREATE TABLE versions (
+    label TEXT PRIMARY KEY,
+    timestamp TEXT NOT NULL,
+    author TEXT NOT NULL,
+    description TEXT,
+    checksum TEXT,
+    snapshot_data TEXT NOT NULL,
+    created_at TEXT NOT NULL
+)
+```
+
+#### Performance Characteristics
+
+- **Save**: 7-25ms (10-1000 entities)
+- **Get**: 2-8ms (10-1000 entities)
+- **List**: 0.6-13ms
+- **Thread-safe**: Yes (uses RLock)
+- **ACID**: Full transaction support
+
+#### Use Cases
+
+- Production deployments
+- Long-term version storage
+- Compliance and audit requirements
+- Multi-process environments
+
+---
+
+### BaseVersionManager
+
+Abstract base class for version managers.
+
+#### Class Definition
+
+```python
+class BaseVersionManager(ABC):
+    """
+    Abstract base class for version managers.
+    
+    Provides common functionality for version management across
+    different data types (knowledge graphs, ontologies, etc.).
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: Optional[str] = None):
+    """
+    Initialize base version manager.
+    
+    Args:
+        storage_path: Path to SQLite database file for persistent storage.
+                     If None, uses in-memory storage.
+    """
+```
+
+**Parameters:**
+- `storage_path` (str, optional): Path to SQLite database file
+
+**Example:**
+```python
+# In-memory storage
+manager = BaseVersionManager()
+
+# Persistent storage
+manager = BaseVersionManager(storage_path="versions.db")
+```
+
+#### Abstract Methods
+
+##### `create_snapshot(data, version_label, author, description, **options)`
+
+Create a versioned snapshot of the data.
+
+**Parameters:**
+- `data` (Any): Data to snapshot
+- `version_label` (str): Version label
+- `author` (str): Email address of the author
+- `description` (str): Change description
+- `**options`: Additional options
+
+**Returns:**
+- `Dict[str, Any]`: Snapshot with metadata and checksum
+
+##### `compare_versions(version1, version2, **options)`
+
+Compare two versions and return detailed differences.
+
+**Parameters:**
+- `version1` (Any): First version (label or snapshot)
+- `version2` (Any): Second version (label or snapshot)
+- `**options`: Comparison options
+
+**Returns:**
+- `Dict[str, Any]`: Detailed differences
+
+#### Concrete Methods
+
+##### `list_versions()`
+
+List all version snapshots.
+
+**Returns:**
+- `List[Dict[str, Any]]`: List of version metadata
+
+**Example:**
+```python
+versions = manager.list_versions()
+for v in versions:
+    print(f"{v['label']}: {v['description']}")
+```
+
+##### `get_version(label)`
+
+Retrieve specific version by label.
+
+**Parameters:**
+- `label` (str): Version label
+
+**Returns:**
+- `Optional[Dict[str, Any]]`: Version snapshot or None
+
+**Example:**
+```python
+version = manager.get_version("v1.0")
+```
+
+##### `verify_checksum(snapshot)`
+
+Verify data integrity using checksum.
+
+**Parameters:**
+- `snapshot` (Dict[str, Any]): Snapshot to verify
+
+**Returns:**
+- `bool`: True if checksum is valid
+
+**Example:**
+```python
+is_valid = manager.verify_checksum(snapshot)
+```
+
+---
+
+### TemporalVersionManager
+
+Enhanced temporal version management engine for knowledge graphs.
+
+#### Class Definition
+
+```python
+class TemporalVersionManager(BaseVersionManager):
+    """
+    Enhanced temporal version management engine for knowledge graphs.
+    
+    Features:
+        - Persistent snapshot storage (SQLite or in-memory)
+        - Detailed change tracking with entity-level diffs
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+        - Version comparison with backward compatibility
+        - Input validation and security features
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: Optional[str] = None, **config):
+    """
+    Initialize enhanced temporal version manager.
+    
+    Args:
+        storage_path: Path to SQLite database file for persistent storage.
+                     If None, uses in-memory storage
+        **config: Additional configuration options
+    """
+```
+
+**Parameters:**
+- `storage_path` (str, optional): Path to SQLite database file
+- `**config`: Additional configuration options
+
+**Example:**
+```python
+# In-memory storage
+manager = TemporalVersionManager()
+
+# Persistent storage
+manager = TemporalVersionManager(storage_path="kg_versions.db")
+```
+
+#### Methods
+
+##### `create_snapshot(graph, version_label, author, description, **options)`
+
+Create and store snapshot with checksum and metadata.
+
+**Parameters:**
+- `graph` (Dict[str, Any]): Knowledge graph dict with "entities" and "relationships"
+- `version_label` (str): Version string (e.g., "v1.0")
+- `author` (str): Email address of the change author
+- `description` (str): Change description (max 500 chars)
+- `**options`: Additional options
+
+**Returns:**
+- `Dict[str, Any]`: Snapshot with metadata and checksum
+
+**Raises:**
+- `ValidationError`: If input validation fails
+- `ProcessingError`: If snapshot creation fails
+
+**Example:**
+```python
+graph = {
+    "entities": [
+        {"id": "e1", "name": "Entity 1", "type": "Person"},
+        {"id": "e2", "name": "Entity 2", "type": "Organization"}
+    ],
+    "relationships": [
+        {"source": "e1", "target": "e2", "type": "works_for"}
+    ]
+}
+
+snapshot = manager.create_snapshot(
+    graph,
+    version_label="v1.0",
+    author="user@example.com",
+    description="Initial knowledge graph"
+)
+
+print(f"Created: {snapshot['label']}")
+print(f"Checksum: {snapshot['checksum']}")
+```
+
+##### `compare_versions(version1, version2, **options)`
+
+Compare two versions with detailed entity and relationship diffs.
+
+**Parameters:**
+- `version1` (Union[str, Dict]): First version (label or snapshot dict)
+- `version2` (Union[str, Dict]): Second version (label or snapshot dict)
+- `**options`: Comparison options
+
+**Returns:**
+- `Dict[str, Any]`: Detailed differences including:
+  - `summary`: Aggregate statistics
+  - `entity_changes`: Entity-level changes
+  - `relationship_changes`: Relationship-level changes
+
+**Example:**
+```python
+diff = manager.compare_versions("v1.0", "v2.0")
+
+print(f"Entities added: {diff['summary']['entities_added']}")
+print(f"Entities modified: {diff['summary']['entities_modified']}")
+print(f"Relationships added: {diff['summary']['relationships_added']}")
+
+# Detailed entity changes
+for entity_id, changes in diff['entity_changes'].items():
+    print(f"Entity {entity_id}: {changes['status']}")
+    if changes['status'] == 'modified':
+        print(f"  Before: {changes['before']}")
+        print(f"  After: {changes['after']}")
+```
+
+#### Performance
+
+- **Snapshot Creation**: 1.40-54ms (50-2000 entities)
+- **Version Retrieval**: 0.65-26ms (50-2000 entities)
+- **Version Comparison**: 3.46-33ms (100-1000 entities)
+- **Concurrent Throughput**: 500+ operations/second
+
+---
+
+### OntologyVersionManager
+
+Enhanced version management for ontologies.
+
+#### Class Definition
+
+```python
+class OntologyVersionManager(BaseVersionManager):
+    """
+    Enhanced version management for ontologies.
+    
+    Features:
+        - Persistent ontology snapshot storage
+        - Structural comparison (classes, properties, axioms)
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+    """
+```
+
+#### Constructor
+
+```python
+def __init__(self, storage_path: Optional[str] = None, **config):
+    """
+    Initialize enhanced version manager for ontologies.
+    
+    Args:
+        storage_path: Path to SQLite database file for persistent storage.
+                     If None, uses in-memory storage
+        **config: Additional configuration options
+    """
+```
+
+**Example:**
+```python
+manager = OntologyVersionManager(storage_path="ontology_versions.db")
+```
+
+#### Methods
+
+##### `create_snapshot(ontology, version_label, author, description, **options)`
+
+Create ontology snapshot with metadata.
+
+**Parameters:**
+- `ontology` (Dict[str, Any]): Ontology dict with structure information
+- `version_label` (str): Version label
+- `author` (str): Email address of the author
+- `description` (str): Change description
+- `**options`: Additional options
+
+**Returns:**
+- `Dict[str, Any]`: Ontology snapshot with metadata
+
+**Example:**
+```python
+ontology = {
+    "uri": "https://example.com/ontology",
+    "version_info": {"version": "1.0", "date": "2024-01-30"},
+    "structure": {
+        "classes": ["Person", "Organization", "Location"],
+        "properties": ["name", "address", "email"],
+        "individuals": ["JohnDoe", "ACME_Corp"],
+        "axioms": ["Person hasAddress exactly 1 Location"]
+    }
+}
+
+snapshot = manager.create_snapshot(
+    ontology,
+    version_label="ont_v1.0",
+    author="architect@example.com",
+    description="Initial ontology design"
+)
+```
+
+##### `compare_versions(version1, version2, **options)`
+
+Compare ontology versions with structural analysis.
+
+**Parameters:**
+- `version1` (Union[str, Dict]): First version
+- `version2` (Union[str, Dict]): Second version
+- `**options`: Comparison options
+
+**Returns:**
+- `Dict[str, Any]`: Structural differences including:
+  - `classes_added`, `classes_removed`
+  - `properties_added`, `properties_removed`
+  - `individuals_added`, `individuals_removed`
+  - `axioms_added`, `axioms_removed`, `axioms_modified`
+
+**Example:**
+```python
+diff = manager.compare_versions("ont_v1.0", "ont_v2.0")
+
+print(f"Classes added: {diff['classes_added']}")
+print(f"Properties added: {diff['properties_added']}")
+print(f"Axioms modified: {diff['axioms_modified']}")
+```
+
+---
+
+## Utility Functions
+
+### compute_checksum
+
+Compute SHA-256 checksum for data integrity.
+
+#### Function Signature
+
+```python
+def compute_checksum(data: Dict[str, Any]) -> str:
+    """
+    Compute SHA-256 checksum for data.
+    
+    Args:
+        data: Dictionary to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+    """
+```
+
+**Parameters:**
+- `data` (Dict[str, Any]): Dictionary to compute checksum for
+
+**Returns:**
+- `str`: SHA-256 checksum as hexadecimal string
+
+**Example:**
+```python
+from semantica.change_management import compute_checksum
+
+data = {"entities": [...], "relationships": [...]}
+checksum = compute_checksum(data)
+print(f"Checksum: {checksum}")
+```
+
+**Performance:** 1.29-110ms (100-10,000 entities)
+
+---
+
+### verify_checksum
+
+Verify data integrity using stored checksum.
+
+#### Function Signature
+
+```python
+def verify_checksum(snapshot: Dict[str, Any]) -> bool:
+    """
+    Verify data integrity using checksum.
+    
+    Args:
+        snapshot: Snapshot dictionary with 'checksum' field
+        
+    Returns:
+        True if checksum is valid, False otherwise
+    """
+```
+
+**Parameters:**
+- `snapshot` (Dict[str, Any]): Snapshot dictionary with 'checksum' field
+
+**Returns:**
+- `bool`: True if checksum is valid, False otherwise
+
+**Example:**
+```python
+from semantica.change_management import verify_checksum
+
+snapshot = manager.get_version("v1.0")
+is_valid = verify_checksum(snapshot)
+
+if not is_valid:
+    print("WARNING: Data integrity compromised!")
+```
+
+**Performance:** 0.82-96ms (100-10,000 entities)
+
+---
+
+## Legacy Classes
+
+### VersionManager
+
+Original ontology version manager (moved from `semantica.ontology`).
+
+#### Import
+
+```python
+from semantica.change_management import VersionManager, OntologyVersion
+```
+
+**Note:** This class is maintained for backward compatibility. New projects should use `OntologyVersionManager`.
+
+---
+
+## Error Handling
+
+### ValidationError
+
+Raised when input validation fails.
+
+**Common Causes:**
+- Invalid email format
+- Description exceeds 500 characters
+- Invalid ISO 8601 timestamp
+- Missing required fields
+
+**Example:**
+```python
+from semantica.utils.exceptions import ValidationError
+
+try:
+    entry = ChangeLogEntry(
+        timestamp="invalid",
+        author="not-an-email",
+        description="x" * 501
+    )
+except ValidationError as e:
+    print(f"Validation failed: {e}")
+```
+
+### ProcessingError
+
+Raised when operations fail.
+
+**Common Causes:**
+- Database connection issues
+- File system errors
+- Concurrent modification conflicts
+
+**Example:**
+```python
+from semantica.utils.exceptions import ProcessingError
+
+try:
+    storage.save(snapshot)
+except ProcessingError as e:
+    print(f"Save failed: {e}")
+```
+
+---
+
+## Performance Considerations
+
+### Benchmarks
+
+Based on comprehensive performance testing:
+
+| Component | Small (100) | Medium (500) | Large (2000) |
+|-----------|-------------|--------------|--------------|
+| Snapshot Creation | 2.33ms | 10.70ms | 54.23ms |
+| Version Retrieval | 1.88ms | 7.33ms | 26.04ms |
+| Version Comparison | 3.46ms | 17.39ms | 32.83ms |
+| Checksum Compute | 1.29ms | 5.48ms | 22.15ms |
+| SQLite Save | 8.69ms | 13.37ms | 25.33ms |
+| InMemory Save | 1.18ms | 10.60ms | 14.11ms |
+
+### Optimization Tips
+
+1. **Use appropriate storage backend:**
+   - Development: `InMemoryVersionStorage`
+   - Production: `SQLiteVersionStorage`
+
+2. **Batch operations when possible:**
+   ```python
+   for data in batch:
+       manager.create_snapshot(data, ...)
+   ```
+
+3. **Implement retention policies:**
+   ```python
+   # Delete old versions periodically
+   for version in old_versions:
+       storage.delete(version['label'])
+   ```
+
+4. **Use concurrent operations:**
+   - Thread-safe: 500+ operations/second
+   - No performance degradation under load
+
+---
+
+## Compliance Features
+
+### HIPAA Compliance
+
+- Complete audit trails with author attribution
+- Timestamp tracking for all changes
+- Data integrity verification with checksums
+- Secure storage with access controls
+
+### SOX Compliance
+
+- Immutable change records
+- Detailed change descriptions
+- External system linking (change IDs)
+- Comprehensive audit reports
+
+### FDA 21 CFR Part 11
+
+- Electronic signatures (author email)
+- Data integrity verification
+- Audit trail generation
+- Tamper detection
+
+---
+
+## Examples
+
+### Complete Healthcare Example
+
+```python
+from semantica.change_management import TemporalVersionManager
+
+# Initialize with HIPAA-compliant storage
+manager = TemporalVersionManager(storage_path="hipaa_records.db")
+
+# Patient knowledge graph
+patient_kg = {
+    "entities": [
+        {"id": "patient_001", "type": "Patient", "name": "Jane Smith"},
+        {"id": "diagnosis_001", "type": "Diagnosis", "code": "I10"}
+    ],
+    "relationships": [
+        {"source": "patient_001", "target": "diagnosis_001", "type": "has_diagnosis"}
+    ]
+}
+
+# Create versioned record
+snapshot = manager.create_snapshot(
+    patient_kg,
+    "patient_001_v1.0",
+    "dr.williams@hospital.com",
+    "Initial diagnosis - Essential hypertension"
+)
+
+# Verify integrity
+assert manager.verify_checksum(snapshot), "Data integrity check failed"
+
+# Generate audit report
+for version in manager.list_versions():
+    print(f"{version['timestamp']}: {version['label']} by {version['author']}")
+```
+
+---
+
+## See Also
+
+- **Usage Guide**: `semantica/change_management/change_management_usage.md`
+- **Performance Tests**: `tests/change_management/test_performance.py`
+- **CHANGELOG**: `CHANGELOG.md`
+- **GitHub**: https://github.com/Hawksight-AI/semantica

docs/reference/ontology.md

@@ -63,6 +63,29 @@ The module uses several inference algorithms:
 
 ---
 
+## Ontology Ingestion
+
+Ingest existing ontology files directly into usable data structures using `OntologyIngestor`.
+
+**Function:** `ingest_ontology(source, method="file")`
+
+| Argument | Description |
+|----------|-------------|
+| `source` | File path, directory path, or list of paths |
+| `method` | Ingestion method (default: "file") |
+
+**Example:**
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest file
+data = ingest_ontology("ontology.ttl")
+
+# Ingest directory
+dataset = ingest_ontology("ontologies/")
+```
+
 ## Main Classes
 
 ### OntologyEngine
@@ -170,6 +193,17 @@ Manages external dependencies.
 | `import_external_ontology(uri, ontology)` | Load and merge external ontology |
 | `evaluate_alignment(uri, ontology)` | Assess alignment and compatibility |
 
+### OntologyIngestor
+
+Handles ingestion of existing ontologies from files and directories.
+
+**Methods:**
+
+| Method | Description |
+|--------|-------------|
+| `ingest_ontology(file_path)` | Ingest a single ontology file |
+| `ingest_directory(directory_path)` | Recursively ingest ontology files from a directory |
+
 ---
 
 ## Unified Engine Examples

docs/reference/parse.md

@@ -159,11 +159,11 @@ parser = DoclingParser()
 result = parser.parse("complex_table.pdf")
 
 # Access high-accuracy tables
-for table in result.tables:
-    print(table.headers)
+for table in result["tables"]:
+    print(table["headers"])
     
 # Get markdown representation
-print(result.markdown)
+print(result["full_text"])
 ```
 
 ### WebParser

docs/reference/provenance.md

@@ -0,0 +1,667 @@
+# Provenance Tracking Module
+
+**W3C PROV-O compliant provenance tracking for high-stakes domains requiring complete traceability**
+
+## Overview
+
+The Semantica provenance module provides W3C PROV-O compliant tracking for knowledge graphs, enabling complete end-to-end lineage from source documents to query responses. Designed for high-stakes domains where every decision must be explainable and auditable.
+
+### Key Features
+
+- ✅ **W3C PROV-O Compliant** — Implements PROV-O ontology (prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom)
+- ✅ **All 17 Modules Integrated** — Complete coverage across Semantica
+- ✅ **Source Tracking** — Document identifiers, page numbers, sections, and direct quotes supported
+- ✅ **Zero Breaking Changes** — 100% backward compatible, opt-in only
+- ✅ **Multiple Storage Backends** — InMemory (fast) and SQLite (persistent)
+- ✅ **Bridge Axiom Support** — Translation chain tracking for domain transformations (L1 → L2 → L3)
+- ✅ **Integrity Verification** — SHA-256 checksums for tamper detection
+- ✅ **Complete Lineage Tracing** — End-to-end from document to response
+
+---
+
+## Installation
+
+The provenance module is included with Semantica. No additional installation required.
+
+```python
+from semantica.provenance import ProvenanceManager
+```
+
+---
+
+## Core Components
+
+### ProvenanceManager
+
+Central manager for all provenance tracking operations.
+
+```python
+from semantica.provenance import ProvenanceManager
+
+# Initialize with in-memory storage (default)
+manager = ProvenanceManager()
+
+# Initialize with persistent SQLite storage
+manager = ProvenanceManager(storage_path="provenance.db")
+```
+
+**Methods:**
+- `track_entity(entity_id, source, entity_type, **metadata)` — Track entity provenance
+- `track_relationship(relationship_id, source, subject, predicate, obj, **metadata)` — Track relationship provenance
+- `track_chunk(chunk_id, source_document, chunk_text, start_char, end_char, **metadata)` — Track document chunk provenance
+- `track_property_source(entity_id, property_name, value, source, **metadata)` — Track property-level provenance
+- `get_lineage(entity_id)` — Retrieve complete lineage for an entity
+- `get_statistics()` — Get provenance statistics
+- `get_all_entries()` — Retrieve all provenance entries
+
+### Storage Backends
+
+#### InMemoryStorage
+
+Fast, non-persistent storage for development and testing.
+
+```python
+from semantica.provenance import ProvenanceManager, InMemoryStorage
+
+manager = ProvenanceManager(storage=InMemoryStorage())
+```
+
+#### SQLiteStorage
+
+Persistent storage for production use.
+
+```python
+from semantica.provenance import ProvenanceManager, SQLiteStorage
+
+storage = SQLiteStorage("provenance.db")
+manager = ProvenanceManager(storage=storage)
+```
+
+### Data Schemas
+
+#### ProvenanceEntry
+
+Core data structure for provenance tracking.
+
+```python
+from semantica.provenance import ProvenanceEntry
+from datetime import datetime
+
+entry = ProvenanceEntry(
+    entity_id="entity_1",
+    source="document.pdf",
+    timestamp=datetime.now(),
+    entity_type="named_entity",
+    metadata={"text": "Apple Inc.", "confidence": 0.95}
+)
+```
+
+#### SourceReference
+
+Structured source information with page and section details.
+
+```python
+from semantica.provenance import SourceReference
+
+source = SourceReference(
+    document="research_paper.pdf",
+    page=5,
+    section="Results",
+    confidence=0.98
+)
+```
+
+---
+
+## Module Integrations
+
+All Semantica modules have provenance-enabled versions. Enable tracking by setting `provenance=True`.
+
+### Semantic Extract
+
+```python
+from semantica.semantic_extract.semantic_extract_provenance import (
+    NERExtractorWithProvenance,
+    RelationExtractorWithProvenance,
+    EventDetectorWithProvenance,
+    CoreferenceResolverWithProvenance,
+    TripletExtractorWithProvenance
+)
+
+# Named Entity Recognition with provenance
+ner = NERExtractorWithProvenance(provenance=True)
+entities = ner.extract(
+    text="Apple Inc. was founded by Steve Jobs in Cupertino.",
+    source="company_history.pdf"
+)
+
+# Access provenance manager
+prov_manager = ner._prov_manager
+lineage = prov_manager.get_lineage("entity_id")
+```
+
+**Tracks:** Entity text, labels, confidence scores, source documents, character positions, extraction timestamps
+
+### LLM Providers
+
+```python
+from semantica.llms.llms_provenance import (
+    GroqLLMWithProvenance,
+    OpenAILLMWithProvenance,
+    HuggingFaceLLMWithProvenance,
+    LiteLLMWithProvenance
+)
+
+# Groq LLM with provenance
+llm = GroqLLMWithProvenance(
+    provenance=True,
+    model="llama-3.1-70b"
+)
+
+response = llm.generate("What is artificial intelligence?")
+
+# Access cost and performance data
+stats = llm._prov_manager.get_statistics()
+```
+
+**Tracks:** Model name, prompt/completion tokens, API costs, latency, generation parameters, prompts and responses
+
+### Pipeline Execution
+
+```python
+from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
+
+pipeline = PipelineWithProvenance(provenance=True)
+result = pipeline.run(data=input_data, source="input_file.json")
+```
+
+**Tracks:** Pipeline steps executed, duration, input/output data, execution status
+
+### Context Management
+
+```python
+from semantica.context.context_provenance import ContextManagerWithProvenance
+
+ctx = ContextManagerWithProvenance(provenance=True)
+ctx.add_context("Relevant background information", source="knowledge_base.txt")
+```
+
+**Tracks:** Context additions, sources, timestamps
+
+### Document Ingestion
+
+```python
+from semantica.ingest.ingest_provenance import PDFIngestorWithProvenance
+
+ingestor = PDFIngestorWithProvenance(provenance=True)
+documents = ingestor.ingest("research_paper.pdf")
+```
+
+**Tracks:** File paths, page counts, file metadata, ingestion timestamps
+
+### Embeddings Generation
+
+```python
+from semantica.embeddings.embeddings_provenance import EmbeddingGeneratorWithProvenance
+
+embedder = EmbeddingGeneratorWithProvenance(
+    provenance=True,
+    model="sentence-transformers/all-mpnet-base-v2"
+)
+embeddings = embedder.embed(["Text 1", "Text 2"], source="corpus.txt")
+```
+
+**Tracks:** Model name, embedding dimensions, generation timestamps
+
+### Graph Store
+
+```python
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+
+store = GraphStoreWithProvenance(provenance=True)
+store.add_node(entity_node, source="knowledge_graph.json")
+```
+
+**Tracks:** Nodes added, node properties, graph structure changes
+
+### Vector Store
+
+```python
+from semantica.vector_store.vector_store_provenance import VectorStoreWithProvenance
+
+store = VectorStoreWithProvenance(provenance=True)
+store.add_vectors(embedding_vectors, source="embeddings.npy")
+```
+
+**Tracks:** Vectors stored, dimensions, storage timestamps
+
+### Triplet Store
+
+```python
+from semantica.triplet_store.triplet_store_provenance import TripletStoreWithProvenance
+
+store = TripletStoreWithProvenance(provenance=True)
+store.add_triplet("Steve_Jobs", "founded", "Apple_Inc", source="knowledge_base.ttl")
+```
+
+**Tracks:** Subject, predicate, object, confidence scores, timestamps
+
+### Other Modules
+
+All remaining modules follow the same pattern:
+
+- **Reasoning** — `ReasoningEngineWithProvenance`
+- **Conflicts** — `SourceTrackerWithUnifiedBackend`
+- **Deduplication** — `DeduplicatorWithProvenance`
+- **Export** — `ExporterWithProvenance`
+- **Parse** — `ParserWithProvenance`
+- **Normalize** — `NormalizerWithProvenance`
+- **Ontology** — `OntologyManagerWithProvenance`
+- **Visualization** — `VisualizerWithProvenance`
+
+---
+
+## Usage Examples
+
+### Basic Entity Tracking
+
+```python
+from semantica.provenance import ProvenanceManager
+
+manager = ProvenanceManager()
+
+# Track entity
+manager.track_entity(
+    entity_id="entity_1",
+    source="document.pdf",
+    entity_type="organization",
+    metadata={
+        "name": "Apple Inc.",
+        "confidence": 0.95,
+        "extraction_method": "NER"
+    }
+)
+
+# Retrieve lineage
+lineage = manager.get_lineage("entity_1")
+print(f"Source: {lineage['source']}")
+print(f"Timestamp: {lineage['timestamp']}")
+print(f"Metadata: {lineage['metadata']}")
+```
+
+### Relationship Tracking
+
+```python
+# Track entities
+manager.track_entity("steve_jobs", "biography.pdf", "person")
+manager.track_entity("apple_inc", "biography.pdf", "organization")
+
+# Track relationship
+manager.track_relationship(
+    relationship_id="rel_1",
+    source="biography.pdf",
+    subject="steve_jobs",
+    predicate="founded",
+    obj="apple_inc",
+    metadata={"confidence": 0.92}
+)
+```
+
+### Lineage Chain Tracking
+
+```python
+# Create lineage chain: document → chunk → entity
+manager.track_entity("doc_1", "research_paper.pdf", "document")
+
+manager.track_chunk(
+    chunk_id="chunk_1",
+    source_document="doc_1",
+    chunk_text="Sample text content",
+    start_char=0,
+    end_char=100
+)
+
+manager.track_entity(
+    entity_id="entity_1",
+    source="chunk_1",
+    entity_type="named_entity",
+    metadata={"text": "Apple"}
+)
+
+# Retrieve complete lineage
+lineage = manager.get_lineage("entity_1")
+print(f"Lineage chain: {lineage['lineage_chain']}")
+```
+
+### Property-Level Provenance
+
+```python
+from semantica.provenance import SourceReference
+
+# Track entity
+manager.track_entity("company_1", "doc.pdf", "organization")
+
+# Track property sources
+manager.track_property_source(
+    entity_id="company_1",
+    property_name="revenue",
+    value="$394.3B",
+    source=SourceReference(
+        document="annual_report_2023.pdf",
+        page=5,
+        section="Financial Summary",
+        confidence=0.98
+    )
+)
+
+manager.track_property_source(
+    entity_id="company_1",
+    property_name="employees",
+    value="500",
+    source=SourceReference(
+        document="company_profile.pdf",
+        page=2,
+        confidence=0.90
+    )
+)
+```
+
+### End-to-End Workflow
+
+```python
+from semantica.provenance import ProvenanceManager
+from semantica.ingest.ingest_provenance import PDFIngestorWithProvenance
+from semantica.semantic_extract.semantic_extract_provenance import NERExtractorWithProvenance
+from semantica.llms.llms_provenance import GroqLLMWithProvenance
+from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+
+# Initialize
+manager = ProvenanceManager()
+
+# Step 1: Ingest
+ingestor = PDFIngestorWithProvenance(provenance=True)
+documents = ingestor.ingest("research_paper.pdf")
+
+# Step 2: Extract
+ner = NERExtractorWithProvenance(provenance=True)
+entities = ner.extract(documents[0].text, source="research_paper.pdf")
+
+# Step 3: LLM Analysis
+llm = GroqLLMWithProvenance(provenance=True)
+summary = llm.generate(f"Summarize: {documents[0].text[:500]}")
+
+# Step 4: Store
+graph = GraphStoreWithProvenance(provenance=True)
+for entity in entities:
+    graph.add_node(entity, source="research_paper.pdf")
+
+# Step 5: Retrieve provenance
+lineage = ner._prov_manager.get_lineage("entity_id")
+stats = ner._prov_manager.get_statistics()
+print(f"Total operations: {stats['total_entries']}")
+```
+
+---
+
+## Bridge Axioms
+
+Bridge axioms enable translation chain tracking across multiple abstraction layers.
+
+```python
+from semantica.provenance.bridge_axiom import BridgeAxiom, TranslationChain
+
+# Create bridge axiom
+axiom = BridgeAxiom(
+    source_layer="L1_ecological",
+    target_layer="L2_financial",
+    translation_rule="fish_biomass_to_revenue",
+    confidence=0.89
+)
+
+# Add provenance
+axiom.add_source_provenance(
+    document="DOI:10.1371/journal.pone.0023601",
+    location="Figure 2",
+    quote="Total fish biomass increased by 463%"
+)
+
+# Create translation chain
+chain = TranslationChain()
+chain.add_axiom(axiom)
+
+# Track complete chain
+provenance_data = chain.get_complete_provenance()
+```
+
+**Use Cases:**
+- Blue Finance: Ecological data → Financial metrics
+- Healthcare: Clinical data → Treatment recommendations
+- Legal: Evidence → Legal conclusions
+- Pharmaceutical: Research data → Drug efficacy claims
+
+---
+
+## Best Practices
+
+### 1. Always Provide Source Information
+
+```python
+# ✅ GOOD - Provides source
+entities = ner.extract(text, source="document.pdf")
+
+# ❌ BAD - No source information
+entities = ner.extract(text)
+```
+
+### 2. Use Descriptive Entity IDs
+
+```python
+# ✅ GOOD - Descriptive IDs
+manager.track_entity("company_apple_inc", source, "organization")
+
+# ❌ BAD - Generic IDs
+manager.track_entity("entity_1", source, "organization")
+```
+
+### 3. Include Rich Metadata
+
+```python
+# ✅ GOOD - Rich metadata
+manager.track_entity(
+    entity_id="person_steve_jobs",
+    source="biography.pdf",
+    entity_type="person",
+    metadata={
+        "full_name": "Steve Jobs",
+        "birth_year": 1955,
+        "confidence": 0.95,
+        "extraction_method": "NER_spacy"
+    }
+)
+```
+
+### 4. Enable Provenance for High-Stakes Operations
+
+```python
+# For high-stakes requirements
+llm = GroqLLMWithProvenance(provenance=True)  # Track all LLM calls
+ner = NERExtractorWithProvenance(provenance=True)  # Track all extractions
+```
+
+### 5. Use Persistent Storage for Production
+
+```python
+from semantica.provenance import ProvenanceManager, SQLiteStorage
+
+# Use SQLite for persistence
+storage = SQLiteStorage("provenance.db")
+manager = ProvenanceManager(storage=storage)
+```
+
+---
+
+## Performance
+
+### Benchmarks
+
+- **Entity tracking:** <5ms per operation
+- **Lineage retrieval:** <10ms for chains up to 100 levels
+- **Batch operations:** 1000+ entities/second
+- **Storage:** InMemory (fastest), SQLite (persistent)
+
+### Optimization Tips
+
+1. **Batch Operations:** Use batch methods for multiple entities
+2. **Selective Tracking:** Only track provenance for critical entities
+3. **Storage Choice:** Use InMemory for development, SQLite for production
+4. **Index Optimization:** SQLite automatically indexes entity_id and source_document
+
+---
+
+## Compliance Standards Support
+
+The provenance module provides **technical infrastructure** that supports compliance efforts:
+
+- **W3C PROV-O** — Implements PROV-O ontology data structures and relationships
+- **FDA 21 CFR Part 11** — Provides audit trails, checksums, and temporal tracking for electronic records
+- **SOX** — Enables financial data lineage tracking and integrity verification
+- **HIPAA** — Supports healthcare data integrity through checksums and source tracking
+- **TNFD** — Enables bridge axiom tracking for nature-to-financial translations
+
+**Important:** This module provides the *technical capabilities* for compliance. Organizations must implement additional policies, procedures, validation, and controls to meet specific regulatory requirements. Semantica does not provide regulatory certification or legal compliance guarantees.
+
+---
+
+## API Reference
+
+### ProvenanceManager
+
+#### `__init__(storage=None, storage_path=None)`
+
+Initialize provenance manager.
+
+**Parameters:**
+- `storage` (ProvenanceStorage, optional): Storage backend instance
+- `storage_path` (str, optional): Path for SQLite storage
+
+#### `track_entity(entity_id, source, entity_type, **metadata)`
+
+Track entity provenance.
+
+**Parameters:**
+- `entity_id` (str): Unique identifier for entity
+- `source` (str): Source document or identifier
+- `entity_type` (str): Type of entity
+- `**metadata`: Additional metadata
+
+**Returns:** ProvenanceEntry
+
+#### `track_relationship(relationship_id, source, subject, predicate, obj, **metadata)`
+
+Track relationship provenance.
+
+**Parameters:**
+- `relationship_id` (str): Unique identifier for relationship
+- `source` (str): Source document
+- `subject` (str): Subject entity ID
+- `predicate` (str): Relationship type
+- `obj` (str): Object entity ID
+- `**metadata`: Additional metadata
+
+**Returns:** ProvenanceEntry
+
+#### `track_chunk(chunk_id, source_document, chunk_text, start_char, end_char, **metadata)`
+
+Track document chunk provenance.
+
+**Parameters:**
+- `chunk_id` (str): Unique identifier for chunk
+- `source_document` (str): Source document ID
+- `chunk_text` (str): Text content of chunk
+- `start_char` (int): Start character position
+- `end_char` (int): End character position
+- `**metadata`: Additional metadata
+
+**Returns:** ProvenanceEntry
+
+#### `get_lineage(entity_id)`
+
+Retrieve complete lineage for an entity.
+
+**Parameters:**
+- `entity_id` (str): Entity identifier
+
+**Returns:** dict with lineage information
+
+#### `get_statistics()`
+
+Get provenance statistics.
+
+**Returns:** dict with statistics (total_entries, entities, relationships, chunks)
+
+---
+
+## Testing
+
+Run the provenance test suite:
+
+```bash
+# All provenance tests
+pytest tests/provenance/ -v
+
+# Specific test categories
+pytest tests/provenance/test_manager.py -v
+pytest tests/provenance/test_storage.py -v
+pytest tests/provenance/test_bridge_axiom.py -v
+pytest tests/provenance/test_integration.py -v
+
+# Module integration tests
+pytest tests/provenance/test_semantic_extract_provenance.py -v
+pytest tests/provenance/test_llms_provenance.py -v
+```
+
+---
+
+## Troubleshooting
+
+### Provenance Not Being Tracked
+
+```python
+# Check if provenance is enabled
+print(f"Provenance enabled: {obj.provenance}")
+print(f"Manager available: {obj._prov_manager is not None}")
+```
+
+### Performance Issues
+
+```python
+# Use batch operations
+entities = [{"id": f"entity_{i}"} for i in range(1000)]
+manager.track_entities_batch(entities, source="doc_1")
+```
+
+### Storage Growing Too Large
+
+```python
+# Use separate databases for different time periods
+manager_2026 = ProvenanceManager(storage_path="provenance_2026.db")
+```
+
+---
+
+## See Also
+
+- [Provenance Usage Guide](https://github.com/Hawksight-AI/semantica/blob/main/semantica/provenance/provenance_usage.md) — Comprehensive usage documentation
+- [Change Management](change_management.md) — Version control and audit trails
+- [Conflicts Module](conflicts.md) — Source tracking and conflict resolution
+- [Knowledge Graph](kg.md) — Entity and relationship tracking
+
+---
+
+## License
+
+MIT License - See [LICENSE](../../LICENSE) for details.
+
+## Support
+
+For issues or questions, please open an issue on GitHub or join our [Discord](https://discord.gg/RgaGTj9J).

docs/reference/semantic_extract.md

@@ -23,7 +23,7 @@ The **Semantic Extract Module** extracts structured information from unstructure
 - **High Accuracy**: LLM-based extraction for complex schemas
 - **Flexible Configuration**: Customize extraction for your domain
 - **Confidence Scores**: Get confidence scores for all extractions
-- **Batch Processing**: Efficient batch processing for large datasets
+- **Batch Processing**: Efficient parallel batch processing for large datasets
 - **Coreference Resolution**: Resolve pronouns to their entity references
 
 ### How It Works
@@ -184,16 +184,15 @@ Core entity extraction implementation used by notebooks and lower-level integrat
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `method` | str or list | `"ml"` | Method(s): "ml", "llm", "pattern", "regex", "huggingface" |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
-| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `provider`) |
+| `entity_types` | list | `None` | Filter for specific entity types |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `aggregation_strategy`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text)` | Alias for `extract_entities`. Get list of entities. |
-| `extract_entities(text)` | Get list of entities |
+| `extract(text, pipeline_id=None, **kwargs)` | Alias for `extract_entities`. Supports `max_workers`. |
+| `extract_entities(text, pipeline_id=None, **kwargs)` | Get list of entities. Supports `max_workers`. |
 
 **Example:**
 
@@ -204,11 +203,12 @@ from semantica.semantic_extract import NERExtractor
 extractor = NERExtractor(method="ml", model="en_core_web_trf")
 entities = extractor.extract("Elon Musk leads SpaceX.")
 
-# 2. LLM (OpenAI/Gemini/etc)
+# 2. LLM (OpenAI/Gemini/Groq/etc)
 extractor = NERExtractor(
     method="llm", 
-    provider="openai", 
-    model="gpt-4",
+    provider="groq", 
+    model="llama-3.3-70b-versatile",
+    max_tokens=2048, # Increased output limit
     temperature=0.0
 )
 
@@ -228,17 +228,19 @@ Extracts relationships between entities.
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"dependency"` | Method: "dependency", "pattern", "cooccurrence", "huggingface", "llm" |
 | `relation_types` | list | `None` | Specific relation types to extract |
 | `bidirectional` | bool | `False` | Extract bidirectional relations |
 | `confidence_threshold` | float | `0.6` | Minimum confidence score |
 | `max_distance` | int | `50` | Max token distance between entities |
+| `**config` | dict | `{}` | Method-specific config (e.g., `model`, `device` for HuggingFace) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract(text, entities)` | Alias for `extract_relations`. Find links. |
-| `extract_relations(text, entities)` | Find links |
+| `extract(text, entities, pipeline_id=None, **kwargs)` | Alias for `extract_relations`. Supports `max_workers`. |
+| `extract_relations(text, entities, pipeline_id=None, **kwargs)` | Find links. Supports `max_workers`. |
 
 **Example:**
 
@@ -253,7 +255,7 @@ entities = ner.extract_entities(text)
 # Basic relation extraction
 rel_extractor = RelationExtractor()
 relations = rel_extractor.extract(text, entities=entities)
-# [Relation(source="Elon Musk", target="SpaceX", type="founded")]
+# [Relation(subject="Elon Musk", predicate="founded", object="SpaceX")]
 
 # With configuration
 rel_extractor = RelationExtractor(
@@ -308,6 +310,7 @@ Identifies events with temporal information and participants.
 | `extract_participants` | bool | `True` | Extract event participants |
 | `extract_location` | bool | `True` | Extract event locations |
 | `extract_time` | bool | `True` | Extract temporal information |
+| `max_workers` | int | `1` | Threads for parallel batch processing |
 
 **Methods:**
 
@@ -336,17 +339,18 @@ Extracts RDF triplets (Subject-Predicate-Object).
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
+| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
+| `triplet_types` | list | `None` | Specific triplet types/predicates to extract |
 | `include_temporal` | bool | `False` | Include time information |
 | `include_provenance` | bool | `False` | Track source sentences |
-| `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
-| `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
+| `**kwargs` | dict | `{}` | Configuration options (e.g., `model`, `device`) |
 
 **Methods:**
 
 | Method | Description |
 |--------|-------------|
-| `extract_triplets(text)` | Get (S, P, O) tuples |
+| `extract(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Alias for `extract_triplets`. Supports `max_workers`. |
+| `extract_triplets(text, entities=None, relations=None, pipeline_id=None, **kwargs)` | Get (S, P, O) tuples. Supports `max_workers`. |
 
 **Example:**
 
@@ -371,6 +375,7 @@ Extracts structured semantic networks with nodes and edges.
 |-----------|------|---------|-------------|
 | `ner_method` | str | `None` | Method for node extraction |
 | `relation_method` | str | `None` | Method for edge extraction |
+| `max_workers` | int | `1` | Threads for parallel batch processing |
 | `**config` | dict | `{}` | Configuration for underlying extractors |
 
 **Methods:**
@@ -424,6 +429,44 @@ enhanced_entities = extractor.enhance_entities(text, entities)
 
 ---
 
+## Batch Processing & Provenance
+
+All extractors support batch processing for high-throughput extraction. You can pass a list of strings or a list of dictionaries (with `content` and `id` keys).
+
+**Features:**
+- **Progress Tracking**: Automatically shows a progress bar for large batches.
+- **Provenance Metadata**: Each extracted item includes `batch_index` and `document_id` in its `metadata`.
+
+```python
+from semantica.semantic_extract import NERExtractor
+
+documents = [
+    {"id": "doc_1", "content": "Apple Inc. was founded by Steve Jobs."},
+    {"id": "doc_2", "content": "Microsoft Corporation was founded by Bill Gates."}
+]
+
+extractor = NERExtractor()
+batch_results = extractor.extract(documents)
+
+for i, doc_entities in enumerate(batch_results):
+    print(f"Document {i} entities:")
+    for entity in doc_entities:
+        print(f"  - {entity.text} ({entity.label})")
+        print(f"    Provenance: Batch Index {entity.metadata['batch_index']}, Doc ID {entity.metadata.get('document_id')}")
+```
+
+## Robust Extraction Fallbacks
+
+The framework implements robust fallback chains to prevent empty results when primary methods fail (e.g., due to model unavailability or obscure text).
+
+- **NER**: `ML/LLM` -> `Pattern` -> `Last Resort` (Capitalized Words)
+- **Relation**: `Primary` -> `Pattern` -> `Last Resort` (Adjacency)
+- **Triplet**: `Primary` -> `Relation-to-Triplet` -> `Pattern`
+
+This ensures that you almost always get *some* structured data, even if it requires falling back to simpler heuristics.
+
+---
+
 ## Usage Examples
 
 ```python

docs/reference/vector_store.md

@@ -1,12 +1,12 @@
 # Vector Store
 
-> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, and Milvus with Hybrid Search.**
+> **Unified vector database interface supporting FAISS, Weaviate, Qdrant, Pinecone, and Milvus with Hybrid Search.**
 
 ---
 
 ## 🎯 Overview
 
-The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Milvus) and enables semantic search, RAG, and similarity matching.
+The **Vector Store Module** provides a unified interface for storing and searching vector embeddings. It supports multiple backends (FAISS, Weaviate, Qdrant, Pinecone, Milvus) and enables semantic search, RAG, and similarity matching.
 
 ### What is a Vector Store?
 
@@ -18,7 +18,7 @@ A **vector store** is a database optimized for storing and searching high-dimens
 
 ### Why Use the Vector Store Module?
 
-- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, and Milvus
+- **Multiple Backends**: Switch between FAISS (local), Weaviate, Qdrant, Pinecone, and Milvus
 - **Unified Interface**: Same API regardless of backend
 - **Hybrid Search**: Combine vector similarity with metadata filtering
 - **Performance**: Optimized for high-throughput search operations
@@ -38,8 +38,8 @@ A **vector store** is a database optimized for storing and searching high-dimens
 -   :material-database:{ .lg .middle } **Multi-Backend Support**
 
     ---
-
-    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, and Milvus
+    
+    Seamlessly switch between FAISS (Local), Weaviate, Qdrant, Pinecone, and Milvus
 
 -   :material-magnify-plus:{ .lg .middle } **Hybrid Search**
 
@@ -112,6 +112,8 @@ The main facade for all vector operations.
 | Method | Description |
 |--------|-------------|
 | `store_vectors(vectors, metadata)` | Store embeddings |
+| `add_documents(documents, metadata, batch_size, parallel)` | **(New)** Store documents with automatic embedding generation and parallelization |
+| `embed_batch(texts)` | **(New)** Generate embeddings for a batch of texts |
 | `search(query, k)` | Semantic search |
 | `delete(ids)` | Remove vectors |
 
@@ -120,15 +122,24 @@ The main facade for all vector operations.
 ```python
 from semantica.vector_store import VectorStore
 
-# Initialize (defaults to FAISS)
+# Initialize (defaults to FAISS, parallel enabled by default with 6 workers)
 store = VectorStore(backend="faiss", dimension=1536)
 
-# Store
+# 1. Store pre-computed vectors
 ids = store.store_vectors(
     vectors=[[0.1, 0.2, ...], ...],
     metadata=[{"text": "Hello"}, ...]
 )
 
+# 2. Store raw documents (High Performance)
+# Automatically handles embedding generation in parallel batches (uses default 6 workers)
+ids = store.add_documents(
+    documents=["Doc 1", "Doc 2", ...],
+    metadata=[{"id": 1}, {"id": 2}, ...],
+    batch_size=32,
+    parallel=True
+)
+
 # Search
 results = store.search(query_vector=[0.1, 0.2, ...], k=5)
 ```
@@ -771,6 +782,7 @@ print(f"Context: {context}")
 ---
 
 ## See Also
+- [High-Performance Usage Guide](../vector_store_usage.md) - **(New)** Parallel ingestion and batching guide
 - [Embeddings Module](embeddings.md) - Generates the vectors
 - [Context Module](context.md) - Uses vector store for memory
 - [Ingest Module](ingest.md) - Source of data

docs/vector_store_usage.md

@@ -0,0 +1,101 @@
+# High-Performance Vector Store Usage
+
+This guide demonstrates how to leverage the new high-performance features of the Semantica Vector Store, specifically designed for efficient batch processing and parallel ingestion of large document sets.
+
+## 🚀 Key Features
+
+- **Parallel Ingestion**: Utilize multi-threading to embed and store documents concurrently.
+- **Batch Processing**: Automatically group documents into batches to minimize overhead.
+- **Unified API**: A single `add_documents` method handles embedding generation and storage.
+
+---
+
+## ⚡ Quick Start: Parallel Ingestion
+
+The fastest way to ingest documents is using the `add_documents` method. Parallelization is enabled by default with optimized settings (6 workers).
+
+```python
+from semantica.vector_store import VectorStore
+import time
+
+store = VectorStore(
+    backend="faiss",
+    dimension=768,
+)
+
+documents = [f"This is document number {i} with some content." for i in range(1000)]
+metadata = [{"source": "generated", "id": i} for i in range(1000)]
+
+start_time = time.time()
+ids = store.add_documents(
+    documents=documents,
+    metadata=metadata,
+    batch_size=64,
+    parallel=True,
+)
+print(f"Ingested {len(ids)} documents in {time.time() - start_time:.2f}s")
+```
+
+---
+
+## 📊 Performance Comparison
+
+### Old Method (Sequential Loop)
+*Slower due to sequential processing and overhead per single item.*
+
+```python
+for doc in documents:
+    emb = embedder.generate(doc)
+    store.store_vectors([emb], [{"text": doc}])
+```
+
+### New Method (Parallel Batching)
+*Significantly faster (3x-10x) by utilizing thread pools and batch operations.*
+
+```python
+store.add_documents(documents, parallel=True)
+```
+
+---
+
+## 🛠 Configuration & Tuning
+
+### `max_workers`
+Controls the number of concurrent threads used for embedding generation.
+- **Default**: 6 (Optimized for most systems)
+- **Recommendation**: You generally don't need to change this. If you have very high core counts or specific throughput needs, you can override it.
+
+```python
+store = VectorStore(max_workers=16)
+```
+
+### `batch_size`
+Controls how many documents are processed in a single chunk.
+- **Default**: 32
+- **Recommendation**: 
+    - **Local Models**: 32-64 usually works well.
+    - **API Models (OpenAI, etc.)**: Larger batches (e.g., 100-200) can reduce network latency overhead.
+
+```python
+store.add_documents(documents, batch_size=100)
+```
+
+---
+
+## 🧩 Advanced: Manual Batch Embedding
+
+If you need the embeddings without storing them immediately, use `embed_batch`.
+
+```python
+vectors = store.embed_batch(
+    texts=documents[:100],
+)
+
+print(f"Generated {len(vectors)} vectors")
+```
+
+## ⚠️ Best Practices
+
+1.  **Metadata Consistency**: Ensure your `metadata` list has the same length as your `documents` list.
+2.  **Error Handling**: The `add_documents` method will propagate exceptions if embedding fails. Ensure your data is clean.
+3.  **Memory Usage**: Very large `batch_size` combined with high `max_workers` can increase memory usage. Monitor your system resources.

integrations/__init__.py

@@ -0,0 +1,7 @@
+"""
+Semantica Framework Integrations
+
+Optional integration packages for agentic frameworks (Google ADK, Claude Agent SDK, Agno, etc.).
+Each integration is self-contained, independently installable via extras_require, and maintains
+zero impact on core Semantica - keeping the semantic layer lean while maximizing ecosystem reach.
+"""

mkdocs.yml

@@ -107,6 +107,7 @@ nav:
     - installation.md
     - quickstart.md
   - Docs:
+    - Change Management: reference/change_management.md
     - Conflicts: reference/conflicts.md
     - Context: reference/context.md
     - Core: reference/core.md
@@ -122,6 +123,7 @@ nav:
     - Ontology: reference/ontology.md
     - Parse: reference/parse.md
     - Pipeline: reference/pipeline.md
+    - Provenance: reference/provenance.md
     - Reasoning: reference/reasoning.md
     - Seed: reference/seed.md
     - Semantic Extract: reference/semantic_extract.md
@@ -139,6 +141,8 @@ nav:
     - examples.md
     - Code Examples: CodeExamples.md
     - learning-more.md
+  - Integrations:
+    - Docling: integrations/docling.md
   - Cookbook: cookbook.md
   - Resources:
     - community-projects.md

pyproject.toml

@@ -4,309 +4,208 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantica"
-version = "0.1.1"
-description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering "
+version = "0.2.5"
+description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering"
 readme = "README.md"
-license = {text = "MIT"}
-authors = [
-    {name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com"}
-]
-maintainers = [
-    {name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com"}
-]
+license = { text = "MIT" }
+
+authors = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
+maintainers = [{ name = "Hawksight AI", email = "semantica-dev@users.noreply.github.com" }]
+
 requires-python = ">=3.8"
+
 classifiers = [
-    "Development Status :: 3 - Alpha",
-    "Intended Audience :: Developers",
-    "Intended Audience :: Science/Research",
-    "License :: OSI Approved :: MIT License",
-    "Operating System :: OS Independent",
-    "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
-    "Programming Language :: Python :: 3.11",
-    "Programming Language :: Python :: 3.12",
-    "Topic :: Scientific/Engineering :: Artificial Intelligence",
-    "Topic :: Software Development :: Libraries :: Python Modules",
-    "Topic :: Text Processing :: Linguistic",
-    "Topic :: Database :: Database Engines/Servers",
-    "Topic :: Internet :: WWW/HTTP :: Indexing/Search"
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Intended Audience :: Science/Research",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Software Development :: Libraries :: Python Modules"
 ]
+
 keywords = [
-    "semantic-layer", "knowledge-engineering", "nlp", "knowledge-graph",
-    "embeddings", "entity-extraction", "relationship-extraction", "rdf",
-    "ontology", "semantic-analysis", "ai", "machine-learning"
+  "semantic-layer", "knowledge-graph", "nlp", "embeddings",
+  "entity-extraction", "relationship-extraction", "rdf", "ontology"
 ]
+
+# ---------------- CORE DEPENDENCIES (SAFE DEFAULT) ----------------
 dependencies = [
-    "numpy>=1.21.0",
-    "pandas>=1.3.0",
-    "scikit-learn>=1.0.0",
-    "umap-learn>=0.5.0",
-    "spacy>=3.4.0",
-    "transformers>=4.20.0",
-    "torch>=1.12.0",
-    "sentence-transformers>=2.2.0",
-    "rdflib>=6.2.0",
-    "networkx>=2.8.0",
-    "matplotlib>=3.5.0",
-    "seaborn>=0.11.0",
-    "plotly>=5.10.0",
-    "ipywidgets>=8.0.0",
-    "requests>=2.28.0",
-    "GitPython>=3.1.30",
-    "chardet>=5.1.0",
-    "protobuf==4.25.8",
-    "grpcio==1.67.1",
-    "beautifulsoup4>=4.11.0",
-    "lxml>=4.9.0",
-    "pypdf2>=2.10.0",
-    "python-docx>=0.8.11",
-    "docling>=1.0.0",
-    "openpyxl>=3.0.10",
-    "pillow>=9.2.0",
-    "librosa>=0.9.0",
-    "opencv-python>=4.6.0",
-    "faiss-cpu>=1.7.0",
-    "fastembed>=0.2.0",
-    "onnxruntime>=1.17.0",
-    "tokenizers>=0.15.0",
-    "weaviate-client>=3.15.0",
-    "qdrant-client>=1.3.0",
-    "neo4j>=5.0.0",
-    "falkordb>=1.0.0",
-    "pymongo>=4.2.0",
-    "sqlalchemy>=1.4.0",
-    "psycopg2-binary>=2.9.0",
-    "pymysql>=1.0.0",
-    "redis>=4.3.0",
-    "celery>=5.2.0",
-    "kafka-python>=2.0.0",
-    "pulsar-client>=3.0.0",
-    "pika>=1.3.0",
-    "boto3>=1.24.0",
-    "azure-storage-blob>=12.12.0",
-    "google-cloud-storage>=2.5.0",
-    "pydantic>=2.0.0",
-    "fastmcp>=0.1.0",
-    "groq>=0.4.0",
-    "openai>=1.0.0",
-    "litellm>=1.0.0",
-    "click>=8.1.0",
-    "rich>=12.5.0",
-    "tqdm>=4.64.0",
-    "pyyaml>=6.0",
-    "toml>=0.10.0",
-    "python-dotenv>=0.20.0",
-    "loguru>=0.6.0",
-    "structlog>=22.1.0",
-    "prometheus-client>=0.14.0",
-    "opentelemetry-api>=1.12.0",
-    "opentelemetry-sdk>=1.12.0",
-    "opentelemetry-instrumentation",
-    "fastapi>=0.78.0",
-    "uvicorn>=0.18.0",
-    "pytest>=7.1.0",
-    "pytest-cov>=3.0.0",
-    "pytest-asyncio>=0.19.0",
-    "black>=22.6.0",
-    "isort>=5.10.0",
-    "flake8>=4.0.0",
-    "mypy>=0.971",
-    "pre-commit>=2.19.0"
+  "numpy>=1.21.0",
+  "pandas>=1.3.0",
+  "scikit-learn>=1.0.0",
+  "umap-learn>=0.5.0",
+  "spacy>=3.4.0",
+  "transformers>=4.20.0",
+  "torch>=1.12.0",
+  "sentence-transformers>=2.2.0",
+  "rdflib>=6.2.0",
+  "networkx>=2.8.0",
+  "matplotlib>=3.5.0",
+  "seaborn>=0.11.0",
+  "plotly>=5.10.0",
+  "ipywidgets>=8.0.0",
+  "requests>=2.28.0",
+  "GitPython>=3.1.30",
+  "chardet>=5.1.0",
+  "protobuf>=5.29.1,<7.0",
+  "grpcio>=1.71.2",
+  "beautifulsoup4>=4.11.0",
+  "lxml>=4.9.0",
+  "pypdf2>=2.10.0",
+  "python-docx>=0.8.11",
+  "openpyxl>=3.0.10",
+  "pillow>=9.2.0",
+  "librosa>=0.9.0",
+  "opencv-python>=4.6.0",
+  "faiss-cpu>=1.7.0",
+  "fastembed>=0.2.0",
+  "onnxruntime>=1.17.0",
+  "tokenizers>=0.15.0",
+  "pydantic>=2.0.0",
+  "click>=8.1.0",
+  "rich>=12.5.0",
+  "tqdm>=4.64.0",
+  "pyyaml>=6.0",
+  "toml>=0.10.0",
+  "python-dotenv>=0.20.0",
+  "loguru>=0.6.0",
+  "structlog>=22.1.0"
 ]
-[project.urls]
-Homepage = "https://github.com/Hawksight-AI/semantica"
-Repository = "https://github.com/Hawksight-AI/semantica"
-"Bug Tracker" = "https://github.com/Hawksight-AI/semantica/issues"
-Discussions = "https://github.com/Hawksight-AI/semantica/discussions"
 
+# ---------------- OPTIONAL DEPENDENCIES ----------------
 [project.optional-dependencies]
-dev = [
-    "pytest>=7.1.0",
-    "pytest-cov>=3.0.0",
-    "pytest-asyncio>=0.19.0",
-    "black>=22.6.0",
-    "isort>=5.10.0",
-    "flake8>=4.0.0",
-    "mypy>=0.971",
-    "pre-commit>=2.19.0",
-    "jupyter>=1.0.0",
-    "ipykernel>=6.15.0",
-    "notebook>=6.4.0"
-]
-viz = [
-    "pyvis>=0.3.0",
-    "graphviz>=0.20.0",
-    "umap-learn>=0.5.0",
-    "d3blocks>=1.0.0"
-]
-gpu = [
-    "torch>=1.12.0",
-    "faiss-gpu>=1.7.0",
-    "cupy>=10.0.0"
-]
-cloud = [
-    "boto3>=1.24.0",
-    "azure-storage-blob>=12.12.0",
-    "google-cloud-storage>=2.5.0",
-    "kubernetes>=24.0.0",
-    "helm>=3.10.0"
-]
-monitoring = [
-    "prometheus-client>=0.14.0",
-    "opentelemetry-api>=1.12.0",
-    "opentelemetry-sdk>=1.12.0",
-    "opentelemetry-instrumentation>=0.32.0",
-    "grafana-api>=1.0.0",
-    "elasticsearch>=8.5.0"
-]
-llm-openai = [
-    "openai>=1.0.0"
-]
-llm-gemini = [
-    "google-generativeai>=0.3.0"
-]
-llm-groq = [
-    "groq>=0.4.0"
-]
-llm-anthropic = [
-    "anthropic>=0.18.0"
-]
-llm-ollama = [
-    "ollama>=0.1.0"
-]
-llm-deepseek = [
-    "deepseek>=0.1.0"
-]
-llm-litellm = [
-    "litellm>=1.0.0"
-]
+
+# ---- LLM Providers ----
+llm-openai = ["openai>=1.0.0"]
+llm-groq = ["groq>=0.4.0"]
+llm-gemini = ["google-genai>=0.1.0"]
+llm-anthropic = ["anthropic>=0.18.0"]
+llm-ollama = ["ollama>=0.1.0"]
+llm-deepseek = ["deepseek>=0.1.0"]
+llm-litellm = ["litellm>=1.0.0"]
+llm-instructor = ["instructor>=1.0.0"]
+
 llm-all = [
-    "semantica[llm-openai,llm-gemini,llm-groq,llm-anthropic,llm-ollama,llm-deepseek,llm-litellm]"
-]
-models-huggingface = [
-    "transformers>=4.20.0",
-    "torch>=1.12.0"
-]
-split-tiktoken = [
-    "tiktoken>=0.5.0"
-]
-split-community = [
-    "python-louvain>=0.16"
-]
-split-topic = [
-    "bertopic>=0.15.0",
-    "gensim>=4.3.0"
-]
-split-all = [
-    "semantica[split-tiktoken,split-community,split-topic]"
-]
-graph-neo4j = [
-    "neo4j>=5.0.0"
-]
-graph-falkordb = [
-    "falkordb>=1.0.0",
-    "redis>=4.3.0"
-]
-graph-all = [
-    "semantica[graph-neo4j,graph-falkordb]"
-]
-parse-docling = [
-    "docling>=1.0.0"
-]
-all = [
-    "semantica[dev,viz,gpu,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,parse-docling]"
+  "semantica[llm-openai,llm-groq,llm-gemini,llm-anthropic,llm-ollama,llm-deepseek,llm-litellm,llm-instructor]"
 ]
 
+# ---- Document Parsing ----
+parse-docling = ["docling>=1.0.0"]
+
+# ---- Embedding / Models ----
+models-huggingface = [
+  "transformers>=4.20.0",
+  "torch>=1.12.0"
+]
+
+# ---- Graph Backends ----
+graph-neo4j = ["neo4j>=5.0.0"]
+graph-falkordb = ["falkordb>=1.0.0", "redis>=4.3.0"]
+graph-amazon-neptune = ["boto3>=1.24.0", "neo4j>=5.0.0"]
+
+graph-all = [
+  "semantica[graph-neo4j,graph-falkordb,graph-amazon-neptune]"
+]
+
+# ---- Vector Store Backends ----
+vectorstore-qdrant = ["qdrant-client>=1.0.0"]
+vectorstore-weaviate = ["weaviate-client>=4.0.0"]
+vectorstore-pinecone = ["pinecone-client>=3.0.0"]
+vectorstore-milvus = ["pymilvus>=2.0.0"]
+
+vectorstore-all = [
+  "semantica[vectorstore-qdrant,vectorstore-weaviate,vectorstore-pinecone,vectorstore-milvus]"
+]
+
+# ---- Infra / Queues / Workers ----
+infra = [
+  "redis>=4.3.0",
+  "celery>=5.2.0",
+  "kafka-python>=2.0.0",
+  "pulsar-client>=3.0.0",
+  "pika>=1.3.0"
+]
+
+# ---- Cloud Providers ----
+cloud = [
+  "boto3>=1.24.0",
+  "azure-storage-blob>=12.12.0",
+  "google-cloud-storage>=2.5.0"
+]
+
+# ---- Monitoring (FIXED) ----
+monitoring = [
+  "prometheus-client>=0.14.0",
+  "opentelemetry-api>=1.30.0,<2.0.0",
+  "opentelemetry-sdk>=1.30.0,<2.0.0",
+  "opentelemetry-semantic-conventions>=0.58b0,<0.61b0",
+  "opentelemetry-instrumentation>=0.58b0,<0.61b0"
+]
+
+# ---- Visualization ----
+viz = [
+  "pyvis>=0.3.0",
+  "graphviz>=0.20.0",
+  "d3blocks>=1.0.0"
+]
+
+# ---- GPU ----
+gpu = [
+  "faiss-gpu>=1.7.0",
+  "cupy>=10.0.0"
+]
+
+# ---- Splitting / Chunking ----
+split-tiktoken = ["tiktoken>=0.5.0"]
+split-community = ["python-louvain>=0.16"]
+split-topic = ["bertopic>=0.15.0", "gensim>=4.3.0"]
+
+split-all = [
+  "semantica[split-tiktoken,split-community,split-topic]"
+]
+
+# ---- Dev ----
+dev = [
+  "pytest>=7.1.0",
+  "pytest-cov>=3.0.0",
+  "pytest-asyncio>=0.19.0",
+  "black>=22.6.0",
+  "isort>=5.10.0",
+  "flake8>=4.0.0",
+  "mypy>=0.971",
+  "pre-commit>=2.19.0",
+  "jupyter>=1.0.0",
+  "ipykernel>=6.15.0"
+]
+
+# ---- Everything ----
+all = [
+  "semantica[dev,viz,gpu,infra,cloud,monitoring,llm-all,models-huggingface,split-all,graph-all,vectorstore-all,parse-docling]"
+]
+
+# ---------------- ENTRYPOINTS ----------------
 [project.scripts]
 semantica = "semantica.cli:main"
 semantica-server = "semantica.server:main"
 semantica-worker = "semantica.worker:main"
 
+# ---------------- TOOLING ----------------
 [tool.setuptools.packages.find]
 where = ["."]
 include = ["semantica*"]
-exclude = ["tests*", "docs*", "examples*"]
-
-[tool.setuptools.package-data]
-semantica = ["*.yaml", "*.yml", "*.json", "*.toml", "*.txt", "*.md"]
 
 [tool.black]
 line-length = 88
-target-version = ['py38', 'py39', 'py310', 'py311', 'py312']
-include = '\.pyi?$'
-extend-exclude = '''
-/(
-  # directories
-  \.eggs
-  | \.git
-  | \.hg
-  | \.mypy_cache
-  | \.tox
-  | \.venv
-  | build
-  | dist
-)/
-'''
 
 [tool.isort]
 profile = "black"
-multi_line_output = 3
-line_length = 88
-known_first_party = ["semantica"]
-known_third_party = ["numpy", "pandas", "scikit-learn", "spacy", "transformers", "torch"]
-
-[tool.mypy]
-python_version = "3.9"
-warn_return_any = true
-warn_unused_configs = true
-disallow_untyped_defs = true
-disallow_incomplete_defs = true
-check_untyped_defs = true
-disallow_untyped_decorators = true
-no_implicit_optional = true
-warn_redundant_casts = true
-warn_unused_ignores = true
-warn_no_return = true
-warn_unreachable = true
-strict_equality = true
-show_error_codes = true
 
 [tool.pytest.ini_options]
-minversion = "7.0"
-addopts = "-ra -q --strict-markers --strict-config"
 testpaths = ["tests"]
-python_files = ["test_*.py", "*_test.py"]
-python_classes = ["Test*"]
-python_functions = ["test_*"]
-markers = [
-    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
-    "integration: marks tests as integration tests",
-    "unit: marks tests as unit tests",
-    "gpu: marks tests that require GPU",
-    "cloud: marks tests that require cloud services"
-]
-
-[tool.coverage.run]
-source = ["semantica"]
-omit = [
-    "*/tests/*",
-    "*/test_*",
-    "*/__pycache__/*",
-    "*/migrations/*"
-]
-
-[tool.coverage.report]
-exclude_lines = [
-    "pragma: no cover",
-    "def __repr__",
-    "if self.debug:",
-    "if settings.DEBUG",
-    "raise AssertionError",
-    "raise NotImplementedError",
-    "if 0:",
-    "if __name__ == .__main__.:",
-    "class .*\\bProtocol\\):",
-    "@(abc\\.)?abstractmethod"
-]

pytest.ini

@@ -0,0 +1,4 @@
+[pytest]
+markers =
+    integration: marks tests as integration (deselect with '-m "not integration"')
+addopts = -ra

semantica/__init__.py

@@ -10,7 +10,7 @@ Main exports:
     - Config: Configuration management
 """
 
-__version__ = "0.1.1"
+__version__ = "0.2.5"
 __author__ = "Semantica Contributors"
 __license__ = "MIT"
 

semantica/change_management/__init__.py

@@ -0,0 +1,63 @@
+"""
+Enhanced Change Management Module for Semantica
+
+This module provides comprehensive change management capabilities including:
+- Persistent version storage (SQLite and in-memory)
+- Detailed change tracking and diff algorithms
+- Standardized metadata and audit trails
+- Data integrity verification with checksums
+- Enhanced version managers for KG and ontologies
+- Enterprise compliance support (HIPAA, SOX, FDA)
+
+Public API:
+    ChangeLogEntry: Standardized metadata for version changes
+    VersionStorage: Abstract storage interface
+    InMemoryVersionStorage: Fast in-memory storage backend
+    SQLiteVersionStorage: Persistent SQLite storage backend
+    compute_checksum: SHA-256 checksum computation
+    verify_checksum: Data integrity verification
+    EnhancedTemporalVersionManager: Advanced KG version management
+    EnhancedVersionManager: Advanced ontology version management
+"""
+
+from .change_log import ChangeLogEntry
+from .version_storage import (
+    VersionStorage,
+    InMemoryVersionStorage,
+    SQLiteVersionStorage,
+    compute_checksum,
+    verify_checksum
+)
+from .managers import (
+    BaseVersionManager,
+    TemporalVersionManager,
+    OntologyVersionManager
+)
+from .ontology_version_manager import VersionManager, OntologyVersion
+
+__all__ = [
+    # Change metadata
+    "ChangeLogEntry",
+    
+    # Storage backends
+    "VersionStorage",
+    "InMemoryVersionStorage", 
+    "SQLiteVersionStorage",
+    
+    # Integrity utilities
+    "compute_checksum",
+    "verify_checksum",
+    
+    # Version managers
+    "BaseVersionManager",
+    "TemporalVersionManager",
+    "OntologyVersionManager",
+    
+    # Ontology version management
+    "VersionManager",
+    "OntologyVersion"
+]
+
+__version__ = "1.0.0"
+__author__ = "Semantica Team"
+__description__ = "Enhanced Change Management for Semantica"

semantica/change_management/change_log.py

@@ -0,0 +1,108 @@
+"""
+Change Log Module
+
+This module provides standardized metadata structures for version changes
+across both ontology and knowledge graph versioning systems.
+
+Key Features:
+    - Standardized ChangeLogEntry dataclass
+    - Email validation for authors
+    - Timestamp handling in ISO 8601 format
+    - Optional change linking and tracking
+
+Main Classes:
+    - ChangeLogEntry: Standard metadata for version changes
+
+Example Usage:
+    >>> from semantica.common.change_log import ChangeLogEntry
+    >>> entry = ChangeLogEntry(
+    ...     timestamp="2024-01-15T10:30:00Z",
+    ...     author="alice@company.com",
+    ...     description="Added Customer entity"
+    ... )
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import re
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import List, Optional
+
+from ..utils.exceptions import ValidationError
+
+
+@dataclass
+class ChangeLogEntry:
+    """
+    Standard metadata for version changes.
+    
+    This dataclass provides a consistent structure for tracking changes
+    across both ontology and knowledge graph versioning systems.
+    
+    Attributes:
+        timestamp: ISO 8601 timestamp of the change
+        author: Email address of the change author
+        description: Description of the change (max 500 characters)
+        change_id: Optional unique identifier for the change
+        related_changes: Optional list of related change IDs
+    """
+    
+    timestamp: str
+    author: str
+    description: str
+    change_id: Optional[str] = None
+    related_changes: List[str] = field(default_factory=list)
+    
+    def __post_init__(self):
+        """Validate fields after initialization."""
+        self._validate_timestamp()
+        self._validate_author()
+        self._validate_description()
+    
+    def _validate_timestamp(self):
+        """Validate timestamp is in ISO 8601 format."""
+        try:
+            # More strict validation for ISO 8601 format
+            if 'T' not in self.timestamp:
+                raise ValueError("Missing 'T' separator")
+            datetime.fromisoformat(self.timestamp.replace('Z', '+00:00'))
+        except ValueError:
+            raise ValidationError(f"Invalid timestamp format: {self.timestamp}. Expected ISO 8601 format.")
+    
+    def _validate_author(self):
+        """Validate author is a valid email address."""
+        email_pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
+        if not re.match(email_pattern, self.author):
+            raise ValidationError(f"Invalid email format: {self.author}")
+    
+    def _validate_description(self):
+        """Validate description length."""
+        if len(self.description) > 500:
+            raise ValidationError(f"Description too long: {len(self.description)} characters (max 500)")
+        if not self.description.strip():
+            raise ValidationError("Description cannot be empty")
+    
+    @classmethod
+    def create_now(cls, author: str, description: str, change_id: Optional[str] = None, 
+                   related_changes: Optional[List[str]] = None) -> 'ChangeLogEntry':
+        """
+        Create a ChangeLogEntry with current timestamp.
+        
+        Args:
+            author: Email address of the change author
+            description: Description of the change
+            change_id: Optional unique identifier for the change
+            related_changes: Optional list of related change IDs
+            
+        Returns:
+            ChangeLogEntry with current timestamp
+        """
+        return cls(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description,
+            change_id=change_id,
+            related_changes=related_changes or []
+        )

semantica/change_management/managers.py

@@ -0,0 +1,497 @@
+"""
+Enhanced Version Managers Module
+
+This module provides enhanced version management capabilities for both knowledge graphs
+and ontologies, with comprehensive change tracking, persistent storage, and audit trails.
+
+Key Features:
+    - Enhanced TemporalVersionManager for knowledge graphs
+    - Enhanced VersionManager for ontologies  
+    - Detailed diff algorithms for entities and relationships
+    - Structural comparison for ontology elements
+    - Integration with storage backends and metadata
+
+Main Classes:
+    - EnhancedTemporalVersionManager: Advanced KG version management
+    - EnhancedVersionManager: Advanced ontology version management
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from abc import ABC, abstractmethod
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+from .change_log import ChangeLogEntry
+from .version_storage import VersionStorage, InMemoryVersionStorage, SQLiteVersionStorage, compute_checksum, verify_checksum
+from ..utils.exceptions import ValidationError, ProcessingError
+from ..utils.logging import get_logger
+
+
+class BaseVersionManager(ABC):
+    """
+    Abstract base class for enhanced version managers.
+    
+    Provides common functionality for version management across different data types.
+    """
+    
+    def __init__(self, storage_path: Optional[str] = None):
+        """
+        Initialize base version manager.
+        
+        Args:
+            storage_path: Path to SQLite database for persistent storage.
+                         If None, uses in-memory storage.
+        """
+        self.logger = get_logger(self.__class__.__name__.lower())
+        
+        # Initialize storage backend
+        if storage_path:
+            self.storage = SQLiteVersionStorage(storage_path)
+            self.logger.info(f"Initialized with SQLite storage: {storage_path}")
+        else:
+            self.storage = InMemoryVersionStorage()
+            self.logger.info("Initialized with in-memory storage")
+    
+    @abstractmethod
+    def create_snapshot(self, data: Any, version_label: str, author: str, description: str, **options) -> Dict[str, Any]:
+        """Create a versioned snapshot of the data."""
+        pass
+    
+    @abstractmethod
+    def compare_versions(self, version1: Any, version2: Any, **options) -> Dict[str, Any]:
+        """Compare two versions and return detailed differences."""
+        pass
+    
+    def list_versions(self) -> List[Dict[str, Any]]:
+        """List all version snapshots."""
+        return self.storage.list_all()
+    
+    def get_version(self, label: str) -> Optional[Dict[str, Any]]:
+        """Retrieve specific version by label."""
+        return self.storage.get(label)
+    
+    def verify_checksum(self, snapshot: Dict[str, Any]) -> bool:
+        """Verify the integrity of a snapshot using its checksum."""
+        return verify_checksum(snapshot)
+
+
+class TemporalVersionManager(BaseVersionManager):
+    """
+    Temporal version management engine for knowledge graphs.
+    
+    Provides comprehensive version/snapshot management capabilities including
+    persistent storage, detailed change tracking, and audit trails.
+    
+    Features:
+        - Persistent snapshot storage (SQLite or in-memory)
+        - Detailed change tracking with entity-level diffs
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+        - Version comparison with backward compatibility
+        - Input validation and security features
+    """
+    
+    def __init__(self, storage_path: Optional[str] = None, **config):
+        """
+        Initialize enhanced temporal version manager.
+        
+        Args:
+            storage_path: Path to SQLite database file for persistent storage.
+                         If None, uses in-memory storage
+            **config: Additional configuration options
+        """
+        super().__init__(storage_path)
+        self.config = config
+    
+    def create_snapshot(
+        self, 
+        graph: Dict[str, Any], 
+        version_label: str, 
+        author: str, 
+        description: str,
+        **options
+    ) -> Dict[str, Any]:
+        """
+        Create and store snapshot with checksum and metadata.
+        
+        Args:
+            graph: Knowledge graph dict with "entities" and "relationships"
+            version_label: Version string (e.g., "v1.0")
+            author: Email address of the change author
+            description: Change description (max 500 chars)
+            **options: Additional options
+            
+        Returns:
+            dict: Snapshot with metadata and checksum
+            
+        Raises:
+            ValidationError: If input validation fails
+            ProcessingError: If storage operation fails
+        """
+        # Validate inputs
+        change_entry = ChangeLogEntry(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description
+        )
+        
+        # Create snapshot
+        snapshot = {
+            "label": version_label,
+            "timestamp": change_entry.timestamp,
+            "author": change_entry.author,
+            "description": change_entry.description,
+            "entities": graph.get("entities", []).copy(),
+            "relationships": graph.get("relationships", []).copy(),
+            "metadata": options.get("metadata", {})
+        }
+        
+        # Compute and add checksum
+        snapshot["checksum"] = compute_checksum(snapshot)
+        
+        # Store snapshot
+        self.storage.save(snapshot)
+        
+        self.logger.info(f"Created snapshot '{version_label}' by {author}")
+        return snapshot
+    
+    def compare_versions(
+        self,
+        v1_label_or_dict,
+        v2_label_or_dict,
+        comparison_metrics: Optional[List[str]] = None,
+        **options,
+    ) -> Dict[str, Any]:
+        """
+        Compare two graph versions with detailed entity-level differences.
+        
+        Args:
+            v1_label_or_dict: First version (label string or snapshot dict)
+            v2_label_or_dict: Second version (label string or snapshot dict)
+            comparison_metrics: List of metrics to calculate (optional, unused)
+            **options: Additional comparison options (unused)
+            
+        Returns:
+            dict: Detailed version comparison results
+        """
+        # Handle both label strings and snapshot dictionaries
+        if isinstance(v1_label_or_dict, str):
+            version1 = self.storage.get(v1_label_or_dict)
+            if not version1:
+                raise ValidationError(f"Version not found: {v1_label_or_dict}")
+        else:
+            version1 = v1_label_or_dict
+            
+        if isinstance(v2_label_or_dict, str):
+            version2 = self.storage.get(v2_label_or_dict)
+            if not version2:
+                raise ValidationError(f"Version not found: {v2_label_or_dict}")
+        else:
+            version2 = v2_label_or_dict
+        
+        # Compute detailed diff
+        detailed_diff = self._compute_detailed_diff(version1, version2)
+        
+        # Maintain backward compatibility with summary
+        summary = {
+            "entities_added": len(detailed_diff["entities_added"]),
+            "entities_removed": len(detailed_diff["entities_removed"]),
+            "entities_modified": len(detailed_diff["entities_modified"]),
+            "relationships_added": len(detailed_diff["relationships_added"]),
+            "relationships_removed": len(detailed_diff["relationships_removed"]),
+            "relationships_modified": len(detailed_diff["relationships_modified"])
+        }
+        
+        return {
+            "version1": version1.get("label", "unknown"),
+            "version2": version2.get("label", "unknown"),
+            "summary": summary,
+            **detailed_diff
+        }
+    
+    def _compute_detailed_diff(self, version1: Dict[str, Any], version2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute detailed entity and relationship differences between versions.
+        
+        Args:
+            version1: First version snapshot
+            version2: Second version snapshot
+            
+        Returns:
+            Dict with detailed diff information
+        """
+        entities1 = {e.get("id", str(i)): e for i, e in enumerate(version1.get("entities", []))}
+        entities2 = {e.get("id", str(i)): e for i, e in enumerate(version2.get("entities", []))}
+        
+        relationships1 = {self._relationship_key(r): r for r in version1.get("relationships", [])}
+        relationships2 = {self._relationship_key(r): r for r in version2.get("relationships", [])}
+        
+        # Entity differences
+        entity_ids1 = set(entities1.keys())
+        entity_ids2 = set(entities2.keys())
+        
+        entities_added = [entities2[eid] for eid in entity_ids2 - entity_ids1]
+        entities_removed = [entities1[eid] for eid in entity_ids1 - entity_ids2]
+        
+        entities_modified = []
+        for eid in entity_ids1 & entity_ids2:
+            if entities1[eid] != entities2[eid]:
+                changes = self._compute_entity_changes(entities1[eid], entities2[eid])
+                entities_modified.append({
+                    "id": eid,
+                    "before": entities1[eid],
+                    "after": entities2[eid],
+                    "changes": changes
+                })
+        
+        # Relationship differences
+        rel_keys1 = set(relationships1.keys())
+        rel_keys2 = set(relationships2.keys())
+        
+        relationships_added = [relationships2[key] for key in rel_keys2 - rel_keys1]
+        relationships_removed = [relationships1[key] for key in rel_keys1 - rel_keys2]
+        
+        relationships_modified = []
+        for key in rel_keys1 & rel_keys2:
+            if relationships1[key] != relationships2[key]:
+                changes = self._compute_relationship_changes(relationships1[key], relationships2[key])
+                relationships_modified.append({
+                    "key": key,
+                    "before": relationships1[key],
+                    "after": relationships2[key],
+                    "changes": changes
+                })
+        
+        return {
+            "entities_added": entities_added,
+            "entities_removed": entities_removed,
+            "entities_modified": entities_modified,
+            "relationships_added": relationships_added,
+            "relationships_removed": relationships_removed,
+            "relationships_modified": relationships_modified
+        }
+    
+    def _relationship_key(self, relationship: Dict[str, Any]) -> str:
+        """Generate a unique key for a relationship."""
+        source = relationship.get("source", "")
+        target = relationship.get("target", "")
+        rel_type = relationship.get("type", relationship.get("relationship", ""))
+        return f"{source}|{rel_type}|{target}"
+    
+    def _compute_entity_changes(self, entity1: Dict[str, Any], entity2: Dict[str, Any]) -> Dict[str, Any]:
+        """Compute changes between two entity versions."""
+        changes = {}
+        all_keys = set(entity1.keys()) | set(entity2.keys())
+        
+        for key in all_keys:
+            val1 = entity1.get(key)
+            val2 = entity2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes
+    
+    def _compute_relationship_changes(self, rel1: Dict[str, Any], rel2: Dict[str, Any]) -> Dict[str, Any]:
+        """Compute changes between two relationship versions."""
+        changes = {}
+        all_keys = set(rel1.keys()) | set(rel2.keys())
+        
+        for key in all_keys:
+            val1 = rel1.get(key)
+            val2 = rel2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes
+
+
+class OntologyVersionManager(BaseVersionManager):
+    """
+    Version management for ontologies with structural comparison.
+    
+    Provides comprehensive version management for ontologies including
+    detailed structural analysis and change tracking.
+    
+    Features:
+        - Structural comparison of ontology elements
+        - Detailed diff for classes, properties, individuals, axioms
+        - Persistent storage with metadata
+        - Change tracking and audit trails
+    """
+    
+    def __init__(self, storage_path: Optional[str] = None, **config):
+        """
+        Initialize enhanced version manager.
+        
+        Args:
+            storage_path: Path to SQLite database file for persistent storage.
+                         If None, uses in-memory storage
+            **config: Additional configuration options
+        """
+        super().__init__(storage_path)
+        self.config = config
+        self.versions = {}  # In-memory version tracking for compatibility
+    
+    def create_snapshot(
+        self,
+        ontology_data: Dict[str, Any],
+        version_label: str,
+        author: str,
+        description: str,
+        **options
+    ) -> Dict[str, Any]:
+        """
+        Create ontology version snapshot.
+        
+        Args:
+            ontology_data: Ontology data dictionary
+            version_label: Version string (e.g., "v1.0")
+            author: Email address of the change author
+            description: Change description
+            **options: Additional options including metadata
+            
+        Returns:
+            dict: Ontology version snapshot
+        """
+        # Validate inputs
+        change_entry = ChangeLogEntry(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description
+        )
+        
+        # Create snapshot
+        snapshot = {
+            "label": version_label,
+            "timestamp": change_entry.timestamp,
+            "author": change_entry.author,
+            "description": change_entry.description,
+            "ontology_iri": ontology_data.get("uri", ""),
+            "version_info": ontology_data.get("version_info", {}),
+            "structure": ontology_data.get("structure", {}),
+            "metadata": options.get("metadata", {})
+        }
+        
+        # Compute and add checksum
+        snapshot["checksum"] = compute_checksum(snapshot)
+        
+        # Store snapshot
+        self.storage.save(snapshot)
+        
+        # Also store in memory for compatibility
+        self.versions[version_label] = snapshot
+        
+        self.logger.info(f"Created ontology snapshot '{version_label}' by {author}")
+        return snapshot
+    
+    def compare_versions(self, version1: str, version2: str, **options) -> Dict[str, Any]:
+        """
+        Compare two ontology versions with detailed structural analysis.
+        
+        Args:
+            version1: First version label
+            version2: Second version label
+            **options: Additional comparison options
+            
+        Returns:
+            Detailed comparison results including structural differences
+        """
+        # Get versions from storage
+        v1_snapshot = self.storage.get(version1)
+        v2_snapshot = self.storage.get(version2)
+        
+        if not v1_snapshot:
+            raise ValidationError(f"Version not found: {version1}")
+        if not v2_snapshot:
+            raise ValidationError(f"Version not found: {version2}")
+        
+        # Basic metadata comparison
+        metadata_changes = {}
+        if v1_snapshot.get("ontology_iri") != v2_snapshot.get("ontology_iri"):
+            metadata_changes["ontology_iri"] = {
+                "from": v1_snapshot.get("ontology_iri"),
+                "to": v2_snapshot.get("ontology_iri")
+            }
+        if v1_snapshot.get("version_info") != v2_snapshot.get("version_info"):
+            metadata_changes["version_info"] = {
+                "from": v1_snapshot.get("version_info"),
+                "to": v2_snapshot.get("version_info")
+            }
+        
+        # Structural comparison
+        structural_diff = self._compare_ontology_structures(v1_snapshot, v2_snapshot)
+        
+        return {
+            "version1": version1,
+            "version2": version2,
+            "metadata_changes": metadata_changes,
+            **structural_diff
+        }
+    
+    def _compare_ontology_structures(self, v1_snapshot: Dict[str, Any], v2_snapshot: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compare structural elements between two ontology versions.
+        
+        Args:
+            v1_snapshot: First ontology version snapshot
+            v2_snapshot: Second ontology version snapshot
+            
+        Returns:
+            Dictionary with structural differences
+        """
+        # Extract structural information
+        v1_structure = v1_snapshot.get("structure", {})
+        v2_structure = v2_snapshot.get("structure", {})
+        
+        # Compare classes
+        v1_classes = set(v1_structure.get("classes", []))
+        v2_classes = set(v2_structure.get("classes", []))
+        
+        classes_added = list(v2_classes - v1_classes)
+        classes_removed = list(v1_classes - v2_classes)
+        
+        # Compare properties
+        v1_properties = set(v1_structure.get("properties", []))
+        v2_properties = set(v2_structure.get("properties", []))
+        
+        properties_added = list(v2_properties - v1_properties)
+        properties_removed = list(v1_properties - v2_properties)
+        
+        # Compare individuals
+        v1_individuals = set(v1_structure.get("individuals", []))
+        v2_individuals = set(v2_structure.get("individuals", []))
+        
+        individuals_added = list(v2_individuals - v1_individuals)
+        individuals_removed = list(v1_individuals - v2_individuals)
+        
+        # Compare axioms/rules
+        v1_axioms = set(v1_structure.get("axioms", []))
+        v2_axioms = set(v2_structure.get("axioms", []))
+        
+        axioms_added = list(v2_axioms - v1_axioms)
+        axioms_removed = list(v1_axioms - v2_axioms)
+        
+        return {
+            "classes_added": classes_added,
+            "classes_removed": classes_removed,
+            "properties_added": properties_added,
+            "properties_removed": properties_removed,
+            "individuals_added": individuals_added,
+            "individuals_removed": individuals_removed,
+            "axioms_added": axioms_added,
+            "axioms_removed": axioms_removed,
+            "summary": {
+                "classes_added": len(classes_added),
+                "classes_removed": len(classes_removed),
+                "properties_added": len(properties_added),
+                "properties_removed": len(properties_removed),
+                "individuals_added": len(individuals_added),
+                "individuals_removed": len(individuals_removed),
+                "axioms_added": len(axioms_added),
+                "axioms_removed": len(axioms_removed)
+            }
+        }

semantica/change_management/ontology_version_manager.py

@@ -42,7 +42,7 @@ from typing import Any, Dict, List, Optional
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
-from .namespace_manager import NamespaceManager
+from ..ontology.namespace_manager import NamespaceManager
 
 
 @dataclass
@@ -198,14 +198,14 @@ class VersionManager:
 
     def compare_versions(self, version1: str, version2: str) -> Dict[str, Any]:
         """
-        Compare two ontology versions.
+        Compare two ontology versions with detailed structural analysis.
 
         Args:
             version1: First version
             version2: Second version
 
         Returns:
-            Comparison results
+            Detailed comparison results including structural differences
         """
         if version1 not in self.versions:
             raise ValidationError(f"Version not found: {version1}")
@@ -215,19 +215,85 @@ class VersionManager:
         v1 = self.versions[version1]
         v2 = self.versions[version2]
 
-        # Basic comparison
-        changes = []
+        # Basic metadata comparison
+        metadata_changes = {}
         if v1.ontology_iri != v2.ontology_iri:
-            changes.append("Ontology IRI changed")
+            metadata_changes["ontology_iri"] = {"from": v1.ontology_iri, "to": v2.ontology_iri}
         if v1.version_info != v2.version_info:
-            changes.append("Version info changed")
+            metadata_changes["version_info"] = {"from": v1.version_info, "to": v2.version_info}
+
+        # Structural comparison (if ontology data is available in metadata)
+        structural_diff = self._compare_ontology_structures(v1, v2)
 
         return {
             "version1": version1,
             "version2": version2,
-            "changes": changes,
-            "v1_iri": v1.ontology_iri,
-            "v2_iri": v2.ontology_iri,
+            "metadata_changes": metadata_changes,
+            **structural_diff
+        }
+
+    def _compare_ontology_structures(self, v1: OntologyVersion, v2: OntologyVersion) -> Dict[str, Any]:
+        """
+        Compare structural elements between two ontology versions.
+        
+        Args:
+            v1: First ontology version
+            v2: Second ontology version
+            
+        Returns:
+            Dictionary with structural differences
+        """
+        # Extract structural information from metadata if available
+        v1_structure = v1.metadata.get("structure", {})
+        v2_structure = v2.metadata.get("structure", {})
+        
+        # Compare classes
+        v1_classes = set(v1_structure.get("classes", []))
+        v2_classes = set(v2_structure.get("classes", []))
+        
+        classes_added = list(v2_classes - v1_classes)
+        classes_removed = list(v1_classes - v2_classes)
+        
+        # Compare properties
+        v1_properties = set(v1_structure.get("properties", []))
+        v2_properties = set(v2_structure.get("properties", []))
+        
+        properties_added = list(v2_properties - v1_properties)
+        properties_removed = list(v1_properties - v2_properties)
+        
+        # Compare individuals
+        v1_individuals = set(v1_structure.get("individuals", []))
+        v2_individuals = set(v2_structure.get("individuals", []))
+        
+        individuals_added = list(v2_individuals - v1_individuals)
+        individuals_removed = list(v1_individuals - v2_individuals)
+        
+        # Compare axioms/rules if available
+        v1_axioms = set(v1_structure.get("axioms", []))
+        v2_axioms = set(v2_structure.get("axioms", []))
+        
+        axioms_added = list(v2_axioms - v1_axioms)
+        axioms_removed = list(v1_axioms - v2_axioms)
+        
+        return {
+            "classes_added": classes_added,
+            "classes_removed": classes_removed,
+            "properties_added": properties_added,
+            "properties_removed": properties_removed,
+            "individuals_added": individuals_added,
+            "individuals_removed": individuals_removed,
+            "axioms_added": axioms_added,
+            "axioms_removed": axioms_removed,
+            "summary": {
+                "classes_added": len(classes_added),
+                "classes_removed": len(classes_removed),
+                "properties_added": len(properties_added),
+                "properties_removed": len(properties_removed),
+                "individuals_added": len(individuals_added),
+                "individuals_removed": len(individuals_removed),
+                "axioms_added": len(axioms_added),
+                "axioms_removed": len(axioms_removed)
+            }
         }
 
     def get_version(self, version: str) -> Optional[OntologyVersion]:

semantica/change_management/version_storage.py

@@ -0,0 +1,391 @@
+"""
+Version Storage Module
+
+This module provides abstract storage interfaces and concrete implementations
+for persistent version management in Semantica.
+
+Key Features:
+    - Abstract VersionStorage interface
+    - In-memory storage implementation
+    - SQLite-based persistent storage implementation
+    - Checksum computation and validation
+    - Thread-safe operations
+
+Main Classes:
+    - VersionStorage: Abstract base class for storage backends
+    - InMemoryVersionStorage: Dictionary-based in-memory storage
+    - SQLiteVersionStorage: SQLite-based persistent storage
+
+Example Usage:
+    >>> from semantica.common.version_storage import SQLiteVersionStorage
+    >>> storage = SQLiteVersionStorage("versions.db")
+    >>> storage.save(snapshot)
+    >>> versions = storage.list_all()
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import hashlib
+import json
+import sqlite3
+import threading
+from abc import ABC, abstractmethod
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+
+
+class VersionStorage(ABC):
+    """
+    Abstract base class for version storage backends.
+    
+    This interface defines the contract that all storage implementations
+    must follow for version management operations.
+    """
+    
+    @abstractmethod
+    def save(self, snapshot: Dict[str, Any]) -> None:
+        """
+        Save a version snapshot.
+        
+        Args:
+            snapshot: Version snapshot dictionary with metadata
+            
+        Raises:
+            ValidationError: If snapshot data is invalid
+            ProcessingError: If save operation fails
+        """
+        pass
+    
+    @abstractmethod
+    def get(self, label: str) -> Optional[Dict[str, Any]]:
+        """
+        Retrieve a version snapshot by label.
+        
+        Args:
+            label: Version label to retrieve
+            
+        Returns:
+            Snapshot dictionary or None if not found
+        """
+        pass
+    
+    @abstractmethod
+    def list_all(self) -> List[Dict[str, Any]]:
+        """
+        List all version snapshots.
+        
+        Returns:
+            List of snapshot metadata dictionaries
+        """
+        pass
+    
+    @abstractmethod
+    def exists(self, label: str) -> bool:
+        """
+        Check if a version exists.
+        
+        Args:
+            label: Version label to check
+            
+        Returns:
+            True if version exists, False otherwise
+        """
+        pass
+    
+    @abstractmethod
+    def delete(self, label: str) -> bool:
+        """
+        Delete a version snapshot.
+        
+        Args:
+            label: Version label to delete
+            
+        Returns:
+            True if deleted, False if not found
+        """
+        pass
+
+
+class InMemoryVersionStorage(VersionStorage):
+    """
+    In-memory version storage implementation.
+    
+    This implementation stores all version data in memory using a dictionary.
+    Data is lost when the process ends.
+    """
+    
+    def __init__(self):
+        """Initialize in-memory storage."""
+        self._storage: Dict[str, Dict[str, Any]] = {}
+        self._lock = threading.RLock()
+        self.logger = get_logger("in_memory_storage")
+    
+    def save(self, snapshot: Dict[str, Any]) -> None:
+        """Save snapshot to memory."""
+        label = snapshot.get("label")
+        if not label:
+            raise ValidationError("Snapshot must have a 'label' field")
+        
+        with self._lock:
+            if label in self._storage:
+                raise ValidationError(f"Version '{label}' already exists")
+            
+            # Deep copy to prevent external modifications
+            self._storage[label] = json.loads(json.dumps(snapshot))
+            self.logger.debug(f"Saved version '{label}' to memory")
+    
+    def get(self, label: str) -> Optional[Dict[str, Any]]:
+        """Retrieve snapshot from memory."""
+        with self._lock:
+            snapshot = self._storage.get(label)
+            if snapshot:
+                # Return deep copy to prevent external modifications
+                return json.loads(json.dumps(snapshot))
+            return None
+    
+    def list_all(self) -> List[Dict[str, Any]]:
+        """List all snapshots in memory."""
+        with self._lock:
+            # Return metadata only (without full graph data)
+            metadata_list = []
+            for label, snapshot in self._storage.items():
+                metadata = {
+                    "label": snapshot.get("label"),
+                    "timestamp": snapshot.get("timestamp"),
+                    "author": snapshot.get("author"),
+                    "description": snapshot.get("description"),
+                    "checksum": snapshot.get("checksum"),
+                    "entity_count": len(snapshot.get("entities", [])),
+                    "relationship_count": len(snapshot.get("relationships", []))
+                }
+                metadata_list.append(metadata)
+            return metadata_list
+    
+    def exists(self, label: str) -> bool:
+        """Check if version exists in memory."""
+        with self._lock:
+            return label in self._storage
+    
+    def delete(self, label: str) -> bool:
+        """Delete version from memory."""
+        with self._lock:
+            if label in self._storage:
+                del self._storage[label]
+                self.logger.debug(f"Deleted version '{label}' from memory")
+                return True
+            return False
+
+
+class SQLiteVersionStorage(VersionStorage):
+    """
+    SQLite-based persistent version storage implementation.
+    
+    This implementation stores version data in a SQLite database file,
+    providing persistence across process restarts.
+    """
+    
+    def __init__(self, storage_path: str):
+        """
+        Initialize SQLite storage.
+        
+        Args:
+            storage_path: Path to SQLite database file
+        """
+        self.storage_path = Path(storage_path)
+        self._lock = threading.RLock()
+        self.logger = get_logger("sqlite_storage")
+        
+        # Create directory if it doesn't exist
+        self.storage_path.parent.mkdir(parents=True, exist_ok=True)
+        
+        # Initialize database
+        self._init_database()
+    
+    def _init_database(self) -> None:
+        """Initialize SQLite database schema."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("""
+                    CREATE TABLE IF NOT EXISTS versions (
+                        label TEXT PRIMARY KEY,
+                        timestamp TEXT NOT NULL,
+                        author TEXT NOT NULL,
+                        description TEXT NOT NULL,
+                        checksum TEXT NOT NULL,
+                        snapshot_data TEXT NOT NULL,
+                        created_at TEXT NOT NULL
+                    )
+                """)
+                conn.commit()
+                self.logger.debug(f"Initialized SQLite database at {self.storage_path}")
+            finally:
+                conn.close()
+    
+    def save(self, snapshot: Dict[str, Any]) -> None:
+        """Save snapshot to SQLite database."""
+        label = snapshot.get("label")
+        if not label:
+            raise ValidationError("Snapshot must have a 'label' field")
+        
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                
+                # Check if version already exists
+                cursor.execute("SELECT label FROM versions WHERE label = ?", (label,))
+                if cursor.fetchone():
+                    raise ValidationError(f"Version '{label}' already exists")
+                
+                # Insert new version
+                cursor.execute("""
+                    INSERT INTO versions 
+                    (label, timestamp, author, description, checksum, snapshot_data, created_at)
+                    VALUES (?, ?, ?, ?, ?, ?, ?)
+                """, (
+                    label,
+                    snapshot.get("timestamp", ""),
+                    snapshot.get("author", ""),
+                    snapshot.get("description", ""),
+                    snapshot.get("checksum", ""),
+                    json.dumps(snapshot),
+                    datetime.now().isoformat()
+                ))
+                
+                conn.commit()
+                self.logger.debug(f"Saved version '{label}' to SQLite database")
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to save version to database: {e}")
+            finally:
+                conn.close()
+    
+    def get(self, label: str) -> Optional[Dict[str, Any]]:
+        """Retrieve snapshot from SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("""
+                    SELECT snapshot_data FROM versions WHERE label = ?
+                """, (label,))
+                
+                row = cursor.fetchone()
+                if not row:
+                    return None
+                
+                return json.loads(row[0])
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to retrieve version from database: {e}")
+            finally:
+                conn.close()
+    
+    def list_all(self) -> List[Dict[str, Any]]:
+        """List all snapshots in SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("""
+                    SELECT snapshot_data FROM versions ORDER BY timestamp DESC
+                """)
+                
+                metadata_list = []
+                for row in cursor.fetchall():
+                    snapshot = json.loads(row[0])
+                    
+                    metadata = {
+                        "label": snapshot.get("label"),
+                        "timestamp": snapshot.get("timestamp"),
+                        "author": snapshot.get("author"),
+                        "description": snapshot.get("description"),
+                        "checksum": snapshot.get("checksum"),
+                        "entity_count": len(snapshot.get("entities", [])),
+                        "relationship_count": len(snapshot.get("relationships", []))
+                    }
+                    metadata_list.append(metadata)
+                
+                return metadata_list
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to list versions from database: {e}")
+            finally:
+                conn.close()
+    
+    def exists(self, label: str) -> bool:
+        """Check if version exists in SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("SELECT 1 FROM versions WHERE label = ?", (label,))
+                return cursor.fetchone() is not None
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to check version existence: {e}")
+            finally:
+                conn.close()
+    
+    def delete(self, label: str) -> bool:
+        """Delete version from SQLite database."""
+        with self._lock:
+            conn = sqlite3.connect(str(self.storage_path))
+            try:
+                cursor = conn.cursor()
+                cursor.execute("DELETE FROM versions WHERE label = ?", (label,))
+                deleted = cursor.rowcount > 0
+                conn.commit()
+                
+                if deleted:
+                    self.logger.debug(f"Deleted version '{label}' from SQLite database")
+                
+                return deleted
+                
+            except sqlite3.Error as e:
+                raise ProcessingError(f"Failed to delete version from database: {e}")
+            finally:
+                conn.close()
+
+
+def compute_checksum(data: Dict[str, Any]) -> str:
+    """
+    Compute SHA-256 checksum for version data.
+    
+    Args:
+        data: Dictionary containing version data
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+    """
+    # Create a deterministic JSON representation
+    json_str = json.dumps(data, sort_keys=True, separators=(',', ':'))
+    return hashlib.sha256(json_str.encode('utf-8')).hexdigest()
+
+
+def verify_checksum(snapshot: Dict[str, Any]) -> bool:
+    """
+    Verify the integrity of a snapshot using its checksum.
+    
+    Args:
+        snapshot: Snapshot dictionary with checksum field
+        
+    Returns:
+        True if checksum is valid, False otherwise
+    """
+    stored_checksum = snapshot.get("checksum")
+    if not stored_checksum:
+        return False
+    
+    # Create copy without checksum for verification
+    data_copy = snapshot.copy()
+    data_copy.pop("checksum", None)
+    
+    computed_checksum = compute_checksum(data_copy)
+    return stored_checksum == computed_checksum

semantica/conflicts/conflicts_provenance.py

@@ -0,0 +1,64 @@
+"""
+Provenance-enabled wrapper for conflict detection with unified backend.
+
+This module provides unified provenance backend integration for SourceTracker
+while maintaining 100% backward compatibility.
+
+Usage:
+    from semantica.conflicts.conflicts_provenance import SourceTrackerWithUnifiedBackend
+    
+    tracker = SourceTrackerWithUnifiedBackend()
+    tracker.track_property_source(entity_id, property_name, value, source)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Dict, Any, List
+
+
+class SourceTrackerWithUnifiedBackend:
+    """SourceTracker using unified provenance backend."""
+    
+    def __init__(self, **config):
+        """Initialize with unified backend or fallback to legacy."""
+        from .source_tracker import SourceTracker
+        
+        try:
+            from semantica.provenance import ProvenanceManager
+            self._unified_manager = ProvenanceManager()
+            self._use_unified = True
+        except ImportError:
+            self._use_unified = False
+        
+        self._original_tracker = SourceTracker(**config)
+    
+    def track_property_source(self, entity_id: str, property_name: str, value: Any, source: Any, **metadata):
+        """Track property source with unified backend."""
+        if self._use_unified:
+            from semantica.provenance import SourceReference
+            
+            source_ref = SourceReference(
+                document=source.document if hasattr(source, 'document') else str(source),
+                page=getattr(source, 'page', None),
+                section=getattr(source, 'section', None),
+                confidence=getattr(source, 'confidence', 1.0)
+            )
+            
+            self._unified_manager.track_property_source(
+                entity_id=entity_id,
+                property_name=property_name,
+                value=value,
+                source=source_ref,
+                **metadata
+            )
+        else:
+            self._original_tracker.track_property_source(
+                entity_id, property_name, value, source, **metadata
+            )
+    
+    def __getattr__(self, name):
+        return getattr(self._original_tracker, name)
+
+
+__all__ = ['SourceTrackerWithUnifiedBackend']

semantica/context/context_provenance.py

@@ -0,0 +1,54 @@
+"""
+Provenance-enabled wrapper for context management.
+
+Usage:
+    from semantica.context.context_provenance import ContextManagerWithProvenance
+    
+    ctx = ContextManagerWithProvenance(provenance=True)
+    ctx.add_context("context data", source="doc1.pdf")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Any
+import uuid
+
+
+class ContextManagerWithProvenance:
+    """Context manager with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        """Initialize context manager with optional provenance."""
+        from .context_manager import ContextManager
+        
+        self.provenance = provenance
+        self._context_manager = ContextManager(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def add_context(self, context: Any, source: Optional[str] = None, **kwargs):
+        """Add context with provenance tracking."""
+        result = self._context_manager.add_context(context, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"context_{uuid.uuid4().hex[:8]}",
+                source=source or "context_manager",
+                entity_type="context",
+                metadata={"context_preview": str(context)[:100]}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._context_manager, name)
+
+
+__all__ = ['ContextManagerWithProvenance']

semantica/context/context_retriever.py

@@ -1411,7 +1411,7 @@ Instructions:
 Answer:"""
 
         try:
-            response = llm_provider.generate(prompt, temperature=0.3)
+            response = llm_provider.generate(prompt)
             return response
         except Exception as e:
             self.logger.warning(f"LLM generation failed: {e}")

semantica/deduplication/deduplication_provenance.py

@@ -0,0 +1,60 @@
+"""
+Provenance-enabled wrapper for deduplication.
+
+Tracks: duplicates found, merge operations, deduplication strategy
+
+Usage:
+    from semantica.deduplication.deduplication_provenance import DeduplicatorWithProvenance
+    
+    dedup = DeduplicatorWithProvenance(provenance=True)
+    unique_items = dedup.deduplicate(items)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import List, Any
+import uuid
+
+
+class DeduplicatorWithProvenance:
+    """Deduplicator with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .deduplicator import Deduplicator
+        
+        self.provenance = provenance
+        self._deduplicator = Deduplicator(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def deduplicate(self, items: List[Any], source: str = None, **kwargs):
+        """Deduplicate items with provenance tracking."""
+        unique_items = self._deduplicator.deduplicate(items, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            duplicates_found = len(items) - len(unique_items)
+            self._prov_manager.track_entity(
+                entity_id=f"dedup_{uuid.uuid4().hex[:8]}",
+                source=source or "deduplication",
+                entity_type="deduplication_operation",
+                metadata={
+                    "input_count": len(items),
+                    "output_count": len(unique_items),
+                    "duplicates_removed": duplicates_found
+                }
+            )
+        
+        return unique_items
+    
+    def __getattr__(self, name):
+        return getattr(self._deduplicator, name)
+
+
+__all__ = ['DeduplicatorWithProvenance']

semantica/embeddings/embeddings_provenance.py

@@ -0,0 +1,59 @@
+"""
+Provenance-enabled wrappers for embedding generation.
+
+Tracks: model, dimensions, input texts, embedding vectors
+
+Usage:
+    from semantica.embeddings.embeddings_provenance import EmbeddingGeneratorWithProvenance
+    
+    embedder = EmbeddingGeneratorWithProvenance(provenance=True)
+    embeddings = embedder.embed(["text1", "text2"])
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import List
+import uuid
+
+
+class EmbeddingGeneratorWithProvenance:
+    """Embedding generator with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .embedding_generator import EmbeddingGenerator
+        
+        self.provenance = provenance
+        self._generator = EmbeddingGenerator(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def embed(self, texts: List[str], source: str = None, **kwargs):
+        """Generate embeddings with provenance tracking."""
+        embeddings = self._generator.embed(texts, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"embed_{uuid.uuid4().hex[:8]}",
+                source=source or "embedding_generation",
+                entity_type="embeddings",
+                metadata={
+                    "model": getattr(self._generator, 'model', 'unknown'),
+                    "dimensions": len(embeddings[0]) if embeddings else 0,
+                    "count": len(embeddings)
+                }
+            )
+        
+        return embeddings
+    
+    def __getattr__(self, name):
+        return getattr(self._generator, name)
+
+
+__all__ = ['EmbeddingGeneratorWithProvenance']

semantica/export/export_provenance.py

@@ -0,0 +1,58 @@
+"""
+Provenance-enabled wrappers for export operations.
+
+Tracks: export format, destination, timestamp, data exported
+
+Usage:
+    from semantica.export.export_provenance import JSONExporterWithProvenance
+    
+    exporter = JSONExporterWithProvenance(provenance=True)
+    exporter.export(data, "output.json")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class ExporterWithProvenance:
+    """Base exporter with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .exporter import Exporter
+        
+        self.provenance = provenance
+        self._exporter = Exporter(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def export(self, data: Any, destination: str, **kwargs):
+        """Export data with provenance tracking."""
+        result = self._exporter.export(data, destination, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"export_{uuid.uuid4().hex[:8]}",
+                source="export_operation",
+                entity_type="export",
+                metadata={
+                    "destination": destination,
+                    "format": kwargs.get('format', 'unknown')
+                }
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._exporter, name)
+
+
+__all__ = ['ExporterWithProvenance']

semantica/graph_store/__init__.py

@@ -1,48 +1,71 @@
 """
 Graph Store Module
 
-This module provides comprehensive property graph database integration for the
-Semantica framework, supporting multiple graph database backends including Neo4j
-and FalkorDB for storing and querying knowledge graphs.
+This module provides comprehensive property graph database integration for
+the Semantica framework, supporting multiple graph database backends including
+Neo4j and FalkorDB for storing and querying knowledge graphs.
 
 Algorithms Used:
 
 Graph Store Management:
-    - Store Registration: Store type detection, store factory pattern, configuration management, default store selection
-    - Backend Pattern: Unified interface for multiple backends (Neo4j, FalkorDB), backend instantiation, backend-specific operation delegation
-    - Store Selection: Default store resolution, store ID lookup, store validation
+    - Store Registration: Store type detection, store factory pattern,
+      configuration management, default store selection
+    - Backend Pattern: Unified interface for multiple backends (Neo4j,
+      FalkorDB), backend instantiation, backend-specific operation delegation
+    - Store Selection: Default store resolution, store ID lookup,
+      store validation
 
 Node and Relationship Operations:
-    - Node Creation: Single node insertion, batch node insertion, property validation, label management, backend delegation
-    - Node Retrieval: Pattern matching (label/property filtering), Cypher query construction, result extraction, node reconstruction
-    - Node Update: Property update, label modification, atomic update operations, conflict detection
-    - Node Deletion: Node matching, cascade deletion (optional), deletion operation delegation, result verification
-    - Relationship Creation: Single relationship insertion, batch insertion, property validation, type management
+    - Node Creation: Single node insertion, batch node insertion,
+      property validation, label management, backend delegation
+    - Node Retrieval: Pattern matching (label/property filtering),
+      Cypher query construction, result extraction, node reconstruction
+    - Node Update: Property update, label modification, atomic update
+      operations, conflict detection
+    - Node Deletion: Node matching, cascade deletion (optional),
+      deletion operation delegation, result verification
+    - Relationship Creation: Single relationship insertion, batch insertion,
+      property validation, type management
     - Relationship Retrieval: Pattern matching, path queries, traversal queries
     - Relationship Update: Property update, type modification
-    - Relationship Deletion: Relationship matching, deletion operation delegation
+    - Relationship Deletion: Relationship matching, deletion operation
+      delegation
 
 Graph Query Execution:
-    - Cypher Query: Full Cypher query language support for Neo4j and FalkorDB (OpenCypher)
-    - Pattern Matching: Node and relationship pattern matching, variable binding, path matching
-    - Graph Traversal: BFS/DFS traversal, shortest path algorithms, path finding
+    - Cypher Query: Full Cypher query language support for Neo4j and
+      FalkorDB (OpenCypher)
+    - Pattern Matching: Node and relationship pattern matching, variable
+      binding, path matching
+    - Graph Traversal: BFS/DFS traversal, shortest path algorithms,
+      path finding
     - Aggregation: COUNT, SUM, AVG, MIN, MAX operations, GROUP BY support
-    - Query Optimization: Query caching, execution plan analysis, index utilization
+    - Query Optimization: Query caching, execution plan analysis,
+      index utilization
 
 Graph Analytics:
-    - Centrality Algorithms: Degree centrality, betweenness centrality, PageRank, closeness centrality
-    - Community Detection: Label propagation, Louvain modularity, connected components
-    - Path Algorithms: Shortest path, all shortest paths, Dijkstra, A* pathfinding
+    - Centrality Algorithms: Degree centrality, betweenness centrality,
+      PageRank, closeness centrality
+    - Community Detection: Label propagation, Louvain modularity,
+      connected components
+    - Path Algorithms: Shortest path, all shortest paths, Dijkstra,
+      A* pathfinding
     - Similarity: Node similarity, Jaccard similarity, cosine similarity
 
 Store Backends:
-    - Neo4j Store: Official Neo4j Python driver, Bolt protocol communication, transaction support, multi-database support, APOC procedures
-    - FalkorDB Store: Redis-based graph database, sparse matrix representation, linear algebra queries, OpenCypher support, ultra-fast performance
+    - Neo4j Store: Official Neo4j Python driver, Bolt protocol
+      communication, transaction support, multi-database support,
+      APOC procedures
+    - FalkorDB Store: Redis-based graph database, sparse matrix
+      representation, linear algebra queries, OpenCypher support,
+      ultra-fast performance
 
 Bulk Operations:
-    - Batch Processing: Chunking algorithm (fixed-size batch creation), batch size optimization, memory management for large datasets
-    - Transaction Management: ACID transaction support, batch commits, rollback on failure
-    - Progress Tracking: Load progress calculation, elapsed time tracking, throughput calculation
+    - Batch Processing: Chunking algorithm (fixed-size batch creation),
+      batch size optimization, memory management for large datasets
+    - Transaction Management: ACID transaction support, batch commits,
+      rollback on failure
+    - Progress Tracking: Load progress calculation, elapsed time tracking,
+      throughput calculation
 
 Key Features:
     - Multi-backend property graph support (Neo4j, FalkorDB)
@@ -79,33 +102,42 @@ Convenience Functions:
     - list_available_methods: List registered graph store methods
 
 Example Usage:
-    >>> from semantica.graph_store import GraphStore, create_node, create_relationship, execute_query
+    >>> from semantica.graph_store import GraphStore, create_node, \
+    ...     create_relationship, execute_query
     >>> # Using convenience functions
-    >>> node_id = create_node(labels=["Person"], properties={"name": "Alice", "age": 30})
-    >>> rel_id = create_relationship(start_id=node1_id, end_id=node2_id, rel_type="KNOWS", properties={"since": 2020})
-    >>> results = execute_query("MATCH (p:Person) WHERE p.age > 25 RETURN p.name")
+    >>> node_id = create_node(labels=["Person"],
+    ...                       properties={"name": "Alice", "age": 30})
+    >>> rel_id = create_relationship(start_id=node1_id, end_id=node2_id,
+    ...                              rel_type="KNOWS",
+    ...                              properties={"since": 2020})
+    >>> results = execute_query("MATCH (p:Person) WHERE p.age > 25 "
+    ...                         "RETURN p.name")
     >>> # Using classes directly
     >>> store = GraphStore(backend="neo4j", uri="bolt://localhost:7687")
-    >>> node_id = store.create_node(labels=["Person"], properties={"name": "Bob"})
+    >>> node_id = store.create_node(labels=["Person"],
+    ...                             properties={"name": "Bob"})
     >>> results = store.execute_query("MATCH (n) RETURN n LIMIT 10")
 
 Author: Semantica Contributors
 License: MIT
 """
 
-from .config import GraphStoreConfig, graph_store_config
-from .falkordb_store import (
-    FalkorDBStore,
-    FalkorDBClient,
-    FalkorDBGraph,
+from .amazon_neptune import (
+    AmazonNeptuneStore,
+    NeptuneAuthTokenManager,
+    NeptuneDriver,
+    NeptuneSession,
+    NeptuneTransaction,
 )
+from .config import GraphStoreConfig, graph_store_config
+from .falkordb_store import FalkorDBClient, FalkorDBGraph, FalkorDBStore
 from .graph_store import (
+    GraphAnalytics,
     GraphManager,
     GraphStore,
     NodeManager,
     QueryEngine,
     RelationshipManager,
-    GraphAnalytics,
 )
 from .methods import (
     create_node,
@@ -125,11 +157,7 @@ from .methods import (
     update_node,
     update_relationship,
 )
-from .neo4j_store import (
-    Neo4jStore,
-    Neo4jDriver,
-    Neo4jTransaction,
-)
+from .neo4j_store import Neo4jDriver, Neo4jStore, Neo4jTransaction
 from .registry import MethodRegistry, method_registry
 
 __all__ = [
@@ -144,6 +172,12 @@ __all__ = [
     "Neo4jStore",
     "Neo4jDriver",
     "Neo4jTransaction",
+    # Amazon Neptune
+    "AmazonNeptuneStore",
+    "NeptuneAuthTokenManager",
+    "NeptuneDriver",
+    "NeptuneSession",
+    "NeptuneTransaction",
     # FalkorDB
     "FalkorDBStore",
     "FalkorDBClient",
@@ -171,4 +205,3 @@ __all__ = [
     "MethodRegistry",
     "method_registry",
 ]
-

semantica/graph_store/config.py

@@ -6,7 +6,8 @@ supporting multiple configuration sources including environment variables, confi
 and programmatic configuration.
 
 Supported Configuration Sources:
-    - Environment variables: GRAPH_STORE_DEFAULT_BACKEND, GRAPH_STORE_NEO4J_URI, GRAPH_STORE_FALKORDB_HOST, etc.
+    - Environment variables: GRAPH_STORE_DEFAULT_BACKEND,
+      GRAPH_STORE_NEO4J_URI, GRAPH_STORE_FALKORDB_HOST, etc.
     - Config files: YAML, JSON, TOML formats
     - Programmatic: Python API for setting graph store configurations
 
@@ -44,7 +45,11 @@ from ..utils.logging import get_logger
 
 
 class GraphStoreConfig:
-    """Configuration manager for graph store module - supports .env files, environment variables, and programmatic config."""
+    """
+    Configuration manager for graph store module.
+
+    Supports .env files, environment variables, and programmatic config.
+    """
 
     def __init__(self, config_file: Optional[str] = None):
         """
@@ -124,6 +129,15 @@ class GraphStoreConfig:
             "GRAPH_STORE_FALKORDB_PORT": "falkordb_port",
             "GRAPH_STORE_FALKORDB_PASSWORD": "falkordb_password",
             "GRAPH_STORE_FALKORDB_GRAPH_NAME": "falkordb_graph_name",
+            # Amazon Neptune settings
+            "GRAPH_STORE_NEPTUNE_ENDPOINT": "neptune_endpoint",
+            "GRAPH_STORE_NEPTUNE_PORT": "neptune_port",
+            "GRAPH_STORE_NEPTUNE_REGION": "neptune_region",
+            "GRAPH_STORE_NEPTUNE_IAM_AUTH": "neptune_iam_auth",
+            "GRAPH_STORE_NEPTUNE_USE_SSL": "neptune_use_ssl",
+            "AWS_ACCESS_KEY_ID": "neptune_access_key",
+            "AWS_SECRET_ACCESS_KEY": "neptune_secret_key",
+            "AWS_SESSION_TOKEN": "neptune_session_token",
         }
 
         for env_var, config_key in env_mappings.items():
@@ -135,6 +149,7 @@ class GraphStoreConfig:
                     "timeout",
                     "max_retries",
                     "falkordb_port",
+                    "neptune_port",
                 ]:
                     try:
                         self._config[config_key] = int(value)
@@ -142,7 +157,11 @@ class GraphStoreConfig:
                         self.logger.warning(
                             f"Invalid integer value for {env_var}: {value}"
                         )
-                elif config_key in ["neo4j_encrypted"]:
+                elif config_key in [
+                    "neo4j_encrypted",
+                    "neptune_iam_auth",
+                    "neptune_use_ssl",
+                ]:
                     self._config[config_key] = value.lower() in [
                         "true",
                         "1",
@@ -171,6 +190,15 @@ class GraphStoreConfig:
             "falkordb_port": 6379,
             "falkordb_password": None,
             "falkordb_graph_name": "default",
+            # Amazon Neptune defaults
+            "neptune_endpoint": None,
+            "neptune_port": 8182,
+            "neptune_region": None,
+            "neptune_iam_auth": True,
+            "neptune_use_ssl": True,
+            "neptune_access_key": None,
+            "neptune_secret_key": None,
+            "neptune_session_token": None,
         }
 
         for key, default_value in defaults.items():
@@ -269,6 +297,24 @@ class GraphStoreConfig:
             "graph_name": self._config.get("falkordb_graph_name"),
         }
 
+    def get_neptune_config(self) -> Dict[str, Any]:
+        """
+        Get Amazon Neptune-specific configuration.
+
+        Returns:
+            Neptune configuration dictionary
+        """
+        return {
+            "endpoint": self._config.get("neptune_endpoint"),
+            "port": self._config.get("neptune_port"),
+            "region": self._config.get("neptune_region"),
+            "iam_auth": self._config.get("neptune_iam_auth"),
+            "use_ssl": self._config.get("neptune_use_ssl"),
+            "access_key": self._config.get("neptune_access_key"),
+            "secret_key": self._config.get("neptune_secret_key"),
+            "session_token": self._config.get("neptune_session_token"),
+        }
+
     def reset(self) -> None:
         """Reset configuration to defaults."""
         self._config.clear()
@@ -278,4 +324,3 @@ class GraphStoreConfig:
 
 # Global configuration instance
 graph_store_config = GraphStoreConfig()
-

semantica/graph_store/graph_store.py

@@ -34,7 +34,7 @@ License: MIT
 
 from typing import Any, Dict, List, Optional, Tuple, Union
 
-from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.exceptions import ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
 from .config import graph_store_config
@@ -214,7 +214,9 @@ class RelationshipManager:
         Returns:
             List of relationships
         """
-        return self.backend.get_relationships(node_id, rel_type, direction, limit, **options)
+        return self.backend.get_relationships(
+            node_id, rel_type, direction, limit, **options
+        )
 
     def delete(
         self,
@@ -290,6 +292,7 @@ class QueryEngine:
     ) -> str:
         """Generate cache key for query."""
         import hashlib
+
         key_str = f"{query}:{str(parameters)}"
         return hashlib.md5(key_str.encode()).hexdigest()
 
@@ -340,7 +343,9 @@ class GraphAnalytics:
         Returns:
             Path information or None
         """
-        return self.backend.shortest_path(start_node_id, end_node_id, rel_type, max_depth, **options)
+        return self.backend.shortest_path(
+            start_node_id, end_node_id, rel_type, max_depth, **options
+        )
 
     def get_neighbors(
         self,
@@ -363,7 +368,9 @@ class GraphAnalytics:
         Returns:
             List of neighboring nodes
         """
-        return self.backend.get_neighbors(node_id, rel_type, direction, depth, **options)
+        return self.backend.get_neighbors(
+            node_id, rel_type, direction, depth, **options
+        )
 
     def degree_centrality(
         self,
@@ -440,7 +447,7 @@ class GraphAnalytics:
             Component information
         """
         backend_type = type(self.backend).__name__
-        
+
         if "Neo4j" in backend_type:
             query = """
             CALL gds.wcc.stream({
@@ -452,16 +459,23 @@ class GraphAnalytics:
             """
             params = {"label": labels[0] if labels else "*"}
             result = self.backend.execute_query(query, params)
-            return [{"component": r["componentId"], "nodes": r["nodes"]} for r in result]
-            
+            return [
+                {"component": r["componentId"], "nodes": r["nodes"]} for r in result
+            ]
+
         elif "NetworkX" in backend_type:
             import networkx as nx
+
             G = self.backend.graph
             components = list(nx.connected_components(G))
-            return [{"component": i, "nodes": list(c)} for i, c in enumerate(components)]
-            
+            return [
+                {"component": i, "nodes": list(c)} for i, c in enumerate(components)
+            ]
+
         else:
-            raise NotImplementedError(f"connected_components not implemented for {backend_type}")
+            raise NotImplementedError(
+                f"connected_components not implemented for {backend_type}"
+            )
 
 
 class GraphManager:
@@ -534,7 +548,11 @@ class GraphStore:
             self.progress_tracker.enabled = True
 
         # Determine backend
-        self.backend = backend or config.get("backend") or graph_store_config.get("default_backend", "neo4j")
+        self.backend = (
+            backend
+            or config.get("backend")
+            or graph_store_config.get("default_backend", "neo4j")
+        )
         self.config = config
 
         # Initialize store backend
@@ -546,16 +564,25 @@ class GraphStore:
         """Initialize the appropriate store backend based on backend type."""
         if self.backend == "neo4j":
             from .neo4j_store import Neo4jStore
+
             neo4j_config = graph_store_config.get_neo4j_config()
             neo4j_config.update(self.config)
             self._store_backend = Neo4jStore(**neo4j_config)
 
         elif self.backend == "falkordb":
             from .falkordb_store import FalkorDBStore
+
             falkordb_config = graph_store_config.get_falkordb_config()
             falkordb_config.update(self.config)
             self._store_backend = FalkorDBStore(**falkordb_config)
 
+        elif self.backend == "neptune" or self.backend == "amazon_neptune":
+            from .amazon_neptune import AmazonNeptuneStore
+
+            neptune_config = graph_store_config.get_neptune_config()
+            neptune_config.update(self.config)
+            self._store_backend = AmazonNeptuneStore(**neptune_config)
+
         else:
             raise ValidationError(f"Unknown backend: {self.backend}")
 
@@ -621,7 +648,9 @@ class GraphStore:
         **options,
     ) -> List[Dict[str, Any]]:
         """Get nodes matching criteria."""
-        return self._manager.nodes.get(labels=labels, properties=properties, limit=limit, **options)
+        return self._manager.nodes.get(
+            labels=labels, properties=properties, limit=limit, **options
+        )
 
     def update_node(
         self,
@@ -665,7 +694,9 @@ class GraphStore:
         **options,
     ) -> List[Dict[str, Any]]:
         """Get relationships."""
-        return self._manager.relationships.get(node_id, rel_type, direction, limit, **options)
+        return self._manager.relationships.get(
+            node_id, rel_type, direction, limit, **options
+        )
 
     def delete_relationship(
         self,
@@ -723,7 +754,9 @@ class GraphStore:
             node_id, rel_type, direction, actual_depth, **options
         )
 
-    def query(self, query: str, parameters: Optional[Dict[str, Any]] = None, **options) -> List[Dict[str, Any]]:
+    def query(
+        self, query: str, parameters: Optional[Dict[str, Any]] = None, **options
+    ) -> List[Dict[str, Any]]:
         """
         Execute a query and return results (Compatibility method for ContextRetriever).
 
@@ -771,42 +804,46 @@ class GraphStore:
         # Convert to GraphStore format (labels, properties)
         graph_nodes = []
         for node in nodes:
-            # Extract label from type
-            labels = [node.get("type", "Entity")]
-            if isinstance(labels[0], str):
-                labels = [labels[0]] # Ensure list
+            # Extract labels - support both 'labels' array and 'type' string
+            labels = node.get("labels")
+            if not labels:
+                node_type = node.get("type", "Entity")
+                labels = [node_type] if isinstance(node_type, str) else node_type
+            if isinstance(labels, str):
+                labels = [labels]
 
             # Prepare properties
             props = node.get("properties", {}).copy()
-            
+
             # Ensure ID is preserved
             if "id" in node and "id" not in props:
                 props["id"] = node["id"]
-                
+
             # Ensure content/text is preserved
             if "content" in node and "content" not in props:
                 props["content"] = node["content"]
             if "text" in node and "text" not in props:
                 props["text"] = node["text"]
 
-            graph_nodes.append({
-                "labels": labels,
-                "properties": props
-            })
+            graph_nodes.append({"labels": labels, "properties": props})
 
         # Use batch creation
-        # Note: create_nodes expects dicts with 'labels' and 'properties' keys if passed directly?
+        # Note: create_nodes expects dicts with 'labels' and 'properties'
+        # keys if passed directly?
         # Let's check create_nodes signature implementation in manager.
-        # But here I'll assume create_nodes takes a list of such dicts or similar.
+        # But here I'll assume create_nodes takes a list of such dicts
+        # or similar.
         # Actually, let's look at create_nodes wrapper in this file:
         # def create_nodes(self, nodes: List[Dict[str, Any]], **options)
         # It passes to self._manager.nodes.create_batch(nodes)
-        
+
         # If create_batch expects specific format, I should match it.
         # Assuming create_batch is smart enough or expects standard format.
-        # To be safe, let's look at NodeManager.create_batch if possible, but I can't easily.
-        # Standard expectation: List of dicts where each dict has labels and properties.
-        
+        # To be safe, let's look at NodeManager.create_batch if possible,
+        # but I can't easily.
+        # Standard expectation: List of dicts where each dict has labels
+        # and properties.
+
         result = self.create_nodes(graph_nodes, **options)
         return len(result)
 
@@ -827,17 +864,21 @@ class GraphStore:
             target_id = edge.get("target_id")
             rel_type = edge.get("type", "RELATED_TO")
             properties = edge.get("properties", {}).copy()
-            
+
             # Preserve weight
             if "weight" in edge:
                 properties["weight"] = edge["weight"]
 
             if source_id and target_id:
                 try:
-                    self.create_relationship(source_id, target_id, rel_type, properties, **options)
+                    self.create_relationship(
+                        source_id, target_id, rel_type, properties, **options
+                    )
                     count += 1
                 except Exception as e:
-                    self.logger.warning(f"Failed to add edge {source_id}->{target_id}: {e}")
+                    self.logger.warning(
+                        f"Failed to add edge {source_id}->{target_id}: {e}"
+                    )
         return count
 
     def build_from_conversations(
@@ -872,27 +913,29 @@ class GraphStore:
             all_nodes = []
             all_edges = []
             seen_nodes = set()
-            
+
             for conv in conversations:
                 # Load conversation if string (file path)
                 conv_data = conv
                 if isinstance(conv, str):
                     from pathlib import Path
+
                     from ..utils.helpers import read_json_file
+
                     conv_data = read_json_file(Path(conv))
 
                 nodes, edges = self._process_conversation_to_elements(
-                    conv_data, 
+                    conv_data,
                     extract_intents=extract_intents,
-                    extract_sentiments=extract_sentiments
+                    extract_sentiments=extract_sentiments,
                 )
-                
+
                 # Add unique nodes
                 for node in nodes:
                     if node["id"] not in seen_nodes:
                         all_nodes.append(node)
                         seen_nodes.add(node["id"])
-                
+
                 all_edges.extend(edges)
 
             if link_entities:
@@ -904,13 +947,8 @@ class GraphStore:
             edge_count = self.add_edges(all_edges)
 
             self.progress_tracker.stop_tracking(tracking_id, status="completed")
-            
-            return {
-                "statistics": {
-                    "node_count": node_count,
-                    "edge_count": edge_count
-                }
-            }
+
+            return {"statistics": {"node_count": node_count, "edge_count": edge_count}}
 
         except Exception as e:
             self.progress_tracker.stop_tracking(
@@ -930,92 +968,112 @@ class GraphStore:
         """
         nodes = []
         edges = []
-        
+
         # Process entities
         for entity in entities:
             entity_id = entity.get("id") or entity.get("entity_id")
             if entity_id:
-                nodes.append({
-                    "id": entity_id,
-                    "type": entity.get("type", "entity"),
-                    "properties": {
-                        "content": entity.get("text") or entity.get("label") or entity_id,
-                        **entity
+                nodes.append(
+                    {
+                        "id": entity_id,
+                        "type": entity.get("type", "entity"),
+                        "properties": {
+                            "content": entity.get("text")
+                            or entity.get("label")
+                            or entity_id,
+                            **entity,
+                        },
                     }
-                })
+                )
 
         # Process relationships
         for rel in relationships:
             source = rel.get("source_id")
             target = rel.get("target_id")
             if source and target:
-                edges.append({
-                    "source_id": source,
-                    "target_id": target,
-                    "type": rel.get("type", "related_to"),
-                    "weight": rel.get("confidence", 1.0),
-                    "properties": rel
-                })
+                edges.append(
+                    {
+                        "source_id": source,
+                        "target_id": target,
+                        "type": rel.get("type", "related_to"),
+                        "weight": rel.get("confidence", 1.0),
+                        "properties": rel,
+                    }
+                )
 
         node_count = self.add_nodes(nodes)
         edge_count = self.add_edges(edges)
-        
+
         return {"statistics": {"node_count": node_count, "edge_count": edge_count}}
 
-    def _process_conversation_to_elements(self, conv_data: Dict[str, Any], **kwargs) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    def _process_conversation_to_elements(
+        self, conv_data: Dict[str, Any], **kwargs
+    ) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
         """Helper to process conversation into nodes and edges."""
         nodes = []
         edges = []
-        
+
         conv_id = conv_data.get("id") or f"conv_{hash(str(conv_data)) % 10000}"
 
         # Conversation node
-        nodes.append({
-            "id": conv_id,
-            "type": "conversation",
-            "properties": {
-                "content": conv_data.get("content", "") or conv_data.get("summary", ""),
-                "timestamp": conv_data.get("timestamp")
+        nodes.append(
+            {
+                "id": conv_id,
+                "type": "conversation",
+                "properties": {
+                    "content": conv_data.get("content", "")
+                    or conv_data.get("summary", ""),
+                    "timestamp": conv_data.get("timestamp"),
+                },
             }
-        })
+        )
 
         name_to_id = {}
-        extract_entities = kwargs.get("extract_entities", True) # Default true if not passed?
-        # Actually ContextGraph defaults to True in init, but here we are static.
+        # Note: extract_entities option is available but not used in this
+        # implementation. Default true if not passed. ContextGraph defaults
+        # to True in init, but here we are static.
         # Let's assume True unless told otherwise or check config.
-        
+
         # Extract entities
         for entity in conv_data.get("entities", []):
             entity_id = entity.get("id") or entity.get("entity_id")
-            entity_text = entity.get("text") or entity.get("label") or entity.get("name") or entity_id
+            entity_text = (
+                entity.get("text")
+                or entity.get("label")
+                or entity.get("name")
+                or entity_id
+            )
             entity_type = entity.get("type", "entity")
 
             # Generate ID if missing
             if not entity_id and entity_text:
                 import hashlib
-                entity_hash = hashlib.md5(f"{entity_text}_{entity_type}".encode()).hexdigest()[:12]
+
+                entity_hash = hashlib.md5(
+                    f"{entity_text}_{entity_type}".encode()
+                ).hexdigest()[:12]
                 entity_id = f"{entity_type.lower()}_{entity_hash}"
 
             if entity_id:
                 if entity_text:
                     name_to_id[entity_text] = entity_id
 
-                nodes.append({
-                    "id": entity_id,
-                    "type": "entity", # Normalize type?
-                    "properties": {
-                        "content": entity_text,
-                        "type": entity_type,
-                        **entity
+                nodes.append(
+                    {
+                        "id": entity_id,
+                        "type": "entity",  # Normalize type?
+                        "properties": {
+                            "content": entity_text,
+                            "type": entity_type,
+                            **entity,
+                        },
                     }
-                })
-                
+                )
+
                 # Edge: Conversation -> Entity
-                edges.append({
-                    "source_id": conv_id,
-                    "target_id": entity_id,
-                    "type": "mentions"
-                })
+                edges.append(
+                    {"source_id": conv_id, "target_id": entity_id, "type": "mentions"}
+                )
 
         # Extract relationships
         for rel in conv_data.get("relationships", []):
@@ -1029,43 +1087,54 @@ class GraphStore:
                 target = name_to_id[rel.get("target")]
 
             if source and target:
-                edges.append({
-                    "source_id": source,
-                    "target_id": target,
-                    "type": rel.get("type", "related_to"),
-                    "weight": rel.get("confidence", 1.0),
-                    "properties": rel
-                })
-                
+                edges.append(
+                    {
+                        "source_id": source,
+                        "target_id": target,
+                        "type": rel.get("type", "related_to"),
+                        "weight": rel.get("confidence", 1.0),
+                        "properties": rel,
+                    }
+                )
+
         return nodes, edges
 
-    def _link_entities_elements(self, nodes: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+    def _link_entities_elements(
+        self, nodes: List[Dict[str, Any]]
+    ) -> List[Dict[str, Any]]:
         """Link similar entities."""
         edges = []
         # Lazy import to avoid circular dependency
         try:
             from ..context.entity_linker import EntityLinker
-            linker = EntityLinker() # Use default config
+
+            linker = EntityLinker()  # Use default config
         except (ImportError, OSError):
             return []
 
         entity_nodes = [n for n in nodes if n.get("type") == "entity"]
         for i, node1 in enumerate(entity_nodes):
             content1 = node1["properties"].get("content", "")
-            if not content1: continue
-            
+            if not content1:
+                continue
+
             for node2 in entity_nodes[i + 1 :]:
                 content2 = node2["properties"].get("content", "")
-                if not content2: continue
-                
-                similarity = linker._calculate_text_similarity(content1.lower(), content2.lower())
+                if not content2:
+                    continue
+
+                similarity = linker._calculate_text_similarity(
+                    content1.lower(), content2.lower()
+                )
                 if similarity >= linker.similarity_threshold:
-                    edges.append({
-                        "source_id": node1["id"],
-                        "target_id": node2["id"],
-                        "type": "similar_to",
-                        "weight": similarity
-                    })
+                    edges.append(
+                        {
+                            "source_id": node1["id"],
+                            "target_id": node2["id"],
+                            "type": "similar_to",
+                            "weight": similarity,
+                        }
+                    )
         return edges
 
     @property
@@ -1087,4 +1156,3 @@ class GraphStore:
     def analytics(self) -> GraphAnalytics:
         """Get analytics engine."""
         return self._manager.analytics
-

semantica/graph_store/graph_store_provenance.py

@@ -0,0 +1,56 @@
+"""
+Provenance-enabled wrapper for graph storage.
+
+Tracks: nodes added, edges created
+
+Usage:
+    from semantica.graph_store.graph_store_provenance import GraphStoreWithProvenance
+    
+    store = GraphStoreWithProvenance(provenance=True)
+    store.add_node(node, source="doc1.pdf")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class GraphStoreWithProvenance:
+    """Graph store with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .graph_store import GraphStore
+        
+        self.provenance = provenance
+        self._store = GraphStore(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def add_node(self, node: Any, source: str = None, **kwargs):
+        """Add node with provenance tracking."""
+        result = self._store.add_node(node, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            node_id = getattr(node, 'id', f"node_{uuid.uuid4().hex[:8]}")
+            self._prov_manager.track_entity(
+                entity_id=node_id,
+                source=source or "graph_store",
+                entity_type="graph_node",
+                metadata={"properties": getattr(node, 'properties', {})}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._store, name)
+
+
+__all__ = ['GraphStoreWithProvenance']

semantica/ingest/__init__.py

@@ -86,6 +86,7 @@ Main Classes:
     - RepoIngestor: Git repository processing
     - EmailIngestor: Email protocol handling
     - DBIngestor: Database export handling
+    - OntologyIngestor: Ontology file processing
     - MethodRegistry: Registry for custom ingestion methods
     - IngestConfig: Configuration manager for ingest module
 
@@ -98,6 +99,7 @@ Convenience Functions:
     - ingest_repository: Repository ingestion wrapper
     - ingest_email: Email ingestion wrapper
     - ingest_database: Database ingestion wrapper
+    - ingest_ontology: Ontology ingestion wrapper
 
 
 Example Usage:
@@ -134,6 +136,7 @@ from .methods import (
     ingest_feed,
     ingest_file,
     ingest_mcp,
+    ingest_ontology,
     ingest_repository,
     ingest_stream,
     ingest_web,
@@ -166,6 +169,8 @@ from .web_ingestor import (
     WebIngestor,
 )
 
+from .ontology_ingestor import OntologyData, OntologyIngestor
+
 __all__ = [
     # File ingestion
     "FileIngestor",
@@ -216,6 +221,9 @@ __all__ = [
     "MCPClient",
     "MCPResource",
     "MCPTool",
+    # Ontology ingestion
+    "OntologyIngestor",
+    "OntologyData",
     # Registry and Methods
     "MethodRegistry",
     "method_registry",
@@ -227,6 +235,7 @@ __all__ = [
     "ingest_repository",
     "ingest_email",
     "ingest_database",
+    "ingest_ontology",
     "ingest_mcp",
     "get_ingest_method",
     "list_available_methods",

semantica/ingest/feed_ingestor.py

@@ -359,9 +359,14 @@ class FeedParser:
 
             return parser.parse(date_string)
         except (ImportError, OSError):
+            # If dateutil isn't available, fall through to raising ValueError
             pass
+        except Exception as e:
+            # If dateutil fails to parse, raise ValueError to signal invalid input
+            raise ValueError(f"Invalid date format: {date_string}") from e
 
-        return None
+        # No known formats matched and dateutil is unavailable; raise ValueError
+        raise ValueError(f"Invalid date format: {date_string}")
 
     def validate_feed(self, feed_data: FeedData) -> bool:
         """

semantica/ingest/ingest_provenance.py

@@ -0,0 +1,67 @@
+"""
+Provenance-enabled wrappers for document ingestion.
+
+Tracks: file paths, pages, metadata, ingestion timestamps
+
+Usage:
+    from semantica.ingest.ingest_provenance import PDFIngestorWithProvenance
+    
+    ingestor = PDFIngestorWithProvenance(provenance=True)
+    docs = ingestor.ingest("document.pdf")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, List
+import uuid
+
+
+class IngestProvenanceMixin:
+    """Mixin for ingest provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **kwargs):
+        self.provenance = provenance
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+
+
+class PDFIngestorWithProvenance(IngestProvenanceMixin):
+    """PDF ingestor with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .pdf_ingestor import PDFIngestor
+        
+        IngestProvenanceMixin.__init__(self, provenance=provenance)
+        self._ingestor = PDFIngestor(**config)
+    
+    def ingest(self, file_path: str, **kwargs):
+        """Ingest PDF with provenance tracking."""
+        docs = self._ingestor.ingest(file_path, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            for doc in docs:
+                doc_id = getattr(doc, 'id', f"doc_{uuid.uuid4().hex[:8]}")
+                self._prov_manager.track_entity(
+                    entity_id=doc_id,
+                    source=file_path,
+                    entity_type="document",
+                    metadata={
+                        "file_type": "pdf",
+                        "pages": getattr(doc, 'page_count', None)
+                    }
+                )
+        
+        return docs
+    
+    def __getattr__(self, name):
+        return getattr(self._ingestor, name)
+
+
+__all__ = ['PDFIngestorWithProvenance', 'IngestProvenanceMixin']

semantica/ingest/methods.py

@@ -150,6 +150,7 @@ from .email_ingestor import EmailData, EmailIngestor
 from .feed_ingestor import FeedData, FeedIngestor
 from .file_ingestor import FileIngestor, FileObject
 from .mcp_ingestor import MCPData, MCPIngestor
+from .ontology_ingestor import OntologyData, OntologyIngestor
 from .registry import method_registry
 from .repo_ingestor import RepoIngestor
 from .stream_ingestor import StreamIngestor, StreamProcessor
@@ -537,6 +538,66 @@ def ingest_email(
         raise
 
 
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], method: str = "file", **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source (convenience function).
+
+    This is a user-friendly wrapper that ingests ontologies using the specified method.
+
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+            - "file": Single file ingestion
+            - "directory": Directory ingestion with recursive scanning
+        **kwargs: Additional options passed to OntologyIngestor
+
+    Returns:
+        OntologyData, List[OntologyData] with ingestion results
+
+    Examples:
+        >>> from semantica.ingest.methods import ingest_ontology
+        >>> ontology = ingest_ontology("ontology.ttl")
+        >>> ontologies = ingest_ontology("./ontologies", method="directory")
+    """
+    # Check for custom method in registry
+    custom_method = method_registry.get("ontology", method)
+    if custom_method and custom_method != ingest_ontology:
+        try:
+            return custom_method(source, **kwargs)
+        except Exception as e:
+            logger.warning(
+                f"Custom method {method} failed: {e}, falling back to default"
+            )
+
+    try:
+        # Get config
+        config = ingest_config.get_method_config("ontology")
+        config.update(kwargs)
+
+        ingestor = OntologyIngestor(**config)
+
+        source_path = str(source) if isinstance(source, (str, Path)) else None
+        
+        if method == "file" and source_path:
+            if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+            return ingestor.ingest_ontology(source_path, **kwargs)
+        elif method == "directory" and source_path:
+             recursive = kwargs.get("recursive", ingest_config.get("recursive", True))
+             return ingestor.ingest_directory(source_path, recursive=recursive, **kwargs)
+        else:
+             # Default: try as file
+             if isinstance(source, list):
+                return [ingestor.ingest_ontology(str(s), **kwargs) for s in source]
+             return ingestor.ingest_ontology(str(source), **kwargs)
+
+    except Exception as e:
+        logger.error(f"Failed to ingest ontology: {e}")
+        raise
+
+
 def ingest_database(
     source: Union[str, Dict[str, Any]], method: Optional[str] = None, **kwargs
 ) -> Union[TableData, List[TableData], Dict[str, Any]]:
@@ -769,6 +830,7 @@ def ingest(
             - "repo": Repository ingestion
             - "email": Email ingestion
             - "db": Database ingestion
+            - "ontology": Ontology ingestion
         method: Optional specific ingestion method
         **kwargs: Additional options passed to ingestor
 
@@ -802,6 +864,8 @@ def ingest(
                 ("git@", "https://github.com", "https://gitlab.com")
             ):
                 source_type = "repo"
+            elif source_str.endswith((".ttl", ".owl", ".rdf", ".jsonld", ".n3", ".nt")):
+                source_type = "ontology"
             else:
                 source_type = "file"
         else:
@@ -830,6 +894,8 @@ def ingest(
             raise ProcessingError("Email ingestion requires configuration dictionary")
     elif source_type == "db":
         return {"data": ingest_database(sources, method=method, **kwargs)}
+    elif source_type == "ontology":
+        return {"ontology": ingest_ontology(sources, method=method or "file", **kwargs)}
     elif source_type == "mcp":
         return {"data": ingest_mcp(sources, method=method or "resources", **kwargs)}
     else:
@@ -909,5 +975,8 @@ method_registry.register("mcp", "default", ingest_mcp)
 method_registry.register("mcp", "resources", ingest_mcp)
 method_registry.register("mcp", "tools", ingest_mcp)
 method_registry.register("mcp", "all", ingest_mcp)
+method_registry.register("ontology", "default", ingest_ontology)
+method_registry.register("ontology", "file", ingest_ontology)
+method_registry.register("ontology", "directory", ingest_ontology)
 method_registry.register("ingest", "default", ingest)
 method_registry.register("ingest", "unified", ingest)

semantica/ingest/ontology_ingestor.py

@@ -0,0 +1,392 @@
+"""
+Ontology Ingestion Module
+
+This module provides capabilities to ingest external ontologies from files (OWL, RDF, TTL, etc.)
+and convert them into Semantica's internal ontology dictionary format.
+
+Supported Formats:
+    - Turtle (.ttl): Terse RDF Triple Language. A concise, human-readable
+      format for representing RDF graphs. Commonly used for writing
+      ontologies by hand.
+    - RDF/XML (.rdf, .owl): The XML serialization of RDF. The standard
+      format for OWL (Web Ontology Language) ontologies and often used
+      for data interchange.
+    - JSON-LD (.jsonld): JSON for Linked Data. A lightweight Linked Data
+      format that is easy for humans to read and for machines to parse
+      and generate. Ideal for web-based applications.
+    - N-Triples (.nt): A line-based, plain text format for encoding an
+      RDF graph. Each line represents a single triple. Very simple to
+      parse but verbose.
+    - Notation3 (.n3): A superset of Turtle that adds features like logic
+      and rules.
+
+Key Features:
+    - Support for multiple RDF formats (Turtle, RDF/XML, JSON-LD, N3, NT)
+    - Automatic parsing using rdflib
+    - Conversion to Semantica ontology structure
+    - Batch processing of ontology files
+    - Extraction of classes, properties, and metadata
+
+Example Usage:
+    >>> from semantica.ingest import OntologyIngestor
+    >>> ingestor = OntologyIngestor()
+    >>> ontology = ingestor.ingest_ontology("my_ontology.ttl")
+"""
+
+import os
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Union
+
+import rdflib
+from rdflib import RDF, RDFS, OWL, Graph
+
+from ..utils.exceptions import ProcessingError, ValidationError
+from ..utils.logging import get_logger
+from ..utils.progress_tracker import get_progress_tracker
+
+
+@dataclass
+class OntologyData:
+    """Ontology data representation."""
+
+    data: Dict[str, Any]
+    source_path: str
+    format: str
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    ingested_at: datetime = field(default_factory=datetime.now)
+
+
+class OntologyIngestor:
+    """
+    Ontology ingestion handler.
+
+    This class parses OWL/RDF files and converts them to Semantica's ontology dictionary format.
+    """
+
+    def __init__(self, config: Optional[Dict[str, Any]] = None, **kwargs):
+        """
+        Initialize ontology ingestor.
+
+        Args:
+            config: Optional configuration dictionary
+            **kwargs: Additional configuration parameters
+        """
+        self.logger = get_logger("ontology_ingestor")
+        self.progress = get_progress_tracker()
+        self.config = config or {}
+        self.config.update(kwargs)
+
+    def ingest_ontology(self, file_path: Union[str, Path], format: Optional[str] = None, **kwargs) -> OntologyData:
+        """
+        Ingest an ontology file.
+
+        Args:
+            file_path: Path to the ontology file (string or Path object)
+            format: Optional format hint (e.g., 'turtle', 'xml'). If None, rdflib guesses.
+            **kwargs: Additional arguments for rdflib parsing
+
+        Returns:
+            OntologyData object containing the parsed ontology and metadata
+        """
+        file_path = Path(file_path)
+
+        # Track file ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(file_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Ontology: {file_path.name}",
+        )
+
+        try:
+            # Validate file exists
+            if not file_path.exists():
+                raise ValidationError(f"File not found: {file_path}")
+
+            self.progress.update_tracking(tracking_id, message="Parsing RDF graph...")
+            g = Graph()
+            
+            # Use provided format or let rdflib guess based on extension
+            parse_kwargs = kwargs.copy()
+            if format:
+                parse_kwargs['format'] = format
+            
+            try:
+                g.parse(file_path, **parse_kwargs)
+            except Exception as e:
+                # Fallback: try to guess format from extension if not provided and initial parse failed
+                if not format:
+                    ext = os.path.splitext(file_path)[1].lower()
+                    fmt_map = {
+                        '.ttl': 'turtle',
+                        '.owl': 'xml', # OWL is often XML
+                        '.rdf': 'xml',
+                        '.jsonld': 'json-ld',
+                        '.n3': 'n3',
+                        '.nt': 'nt'
+                    }
+                    guessed_fmt = fmt_map.get(ext)
+                    if guessed_fmt:
+                        self.logger.info(f"Retrying with guessed format: {guessed_fmt}")
+                        g.parse(file_path, format=guessed_fmt, **kwargs)
+                    else:
+                        raise e
+                else:
+                    raise e
+
+            self.progress.update_tracking(tracking_id, message="Converting to internal format...")
+            
+            # Determine format for metadata
+            used_format = format
+            if not used_format:
+                ext = os.path.splitext(file_path)[1].lower()
+                fmt_map = {
+                    '.ttl': 'turtle',
+                    '.owl': 'xml',
+                    '.rdf': 'xml',
+                    '.jsonld': 'json-ld',
+                    '.n3': 'n3',
+                    '.nt': 'nt'
+                }
+                used_format = fmt_map.get(ext, 'unknown')
+
+            ontology_dict = self._convert_to_dict(g, source_path=str(file_path), format=used_format)
+
+            ontology_data = OntologyData(
+                data=ontology_dict,
+                source_path=str(file_path),
+                format=used_format,
+                metadata=ontology_dict.get("metadata", {}).copy()
+            )
+
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Successfully ingested ontology from {file_path}",
+            )
+            
+            return ontology_data
+
+        except Exception as e:
+            self.logger.error(f"Failed to ingest ontology: {str(e)}")
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise ProcessingError(f"Failed to ingest ontology: {str(e)}") from e
+
+    def ingest_directory(self, directory_path: Union[str, Path], recursive: bool = True, **kwargs) -> List[OntologyData]:
+        """
+        Ingest all ontology files in a directory.
+
+        Args:
+            directory_path: Path to the directory (string or Path object)
+            recursive: Whether to search recursively
+            **kwargs: Additional arguments
+
+        Returns:
+            List of OntologyData objects
+        """
+        directory_path = Path(directory_path)
+        ontologies = []
+        extensions = {'.ttl', '.owl', '.rdf', '.jsonld', '.n3', '.nt'}
+        
+        # Track directory ingestion
+        tracking_id = self.progress.start_tracking(
+            file=str(directory_path),
+            module="ingest",
+            submodule="OntologyIngestor",
+            message=f"Directory: {directory_path.name}",
+        )
+
+        try:
+            if not directory_path.exists():
+                raise ValidationError(f"Directory not found: {directory_path}")
+                
+            if not directory_path.is_dir():
+                raise ValidationError(f"Path is not a directory: {directory_path}")
+
+            files_to_process = []
+            for root, _, files in os.walk(directory_path):
+                for file in files:
+                    ext = os.path.splitext(file)[1].lower()
+                    if ext in extensions:
+                        files_to_process.append(os.path.join(root, file))
+                
+                if not recursive:
+                    break
+            
+            total_files = len(files_to_process)
+            self.progress.update_tracking(
+                tracking_id, message=f"Processing {total_files} ontology files"
+            )
+
+            for idx, file_path in enumerate(files_to_process, 1):
+                try:
+                    ont_data = self.ingest_ontology(file_path, **kwargs)
+                    ontologies.append(ont_data)
+                    
+                    self.progress.update_progress(
+                        tracking_id,
+                        processed=idx,
+                        total=total_files,
+                        message=f"Processing {idx}/{total_files}: {Path(file_path).name}"
+                    )
+                except Exception as e:
+                    self.logger.warning(f"Skipping {file_path}: {e}")
+            
+            self.progress.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Ingested {len(ontologies)} ontologies",
+            )
+            return ontologies
+
+        except Exception as e:
+            self.progress.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise
+
+    def _convert_to_dict(self, graph: Graph, source_path: str, format: str = "unknown") -> Dict[str, Any]:
+        """
+        Convert rdflib Graph to Semantica ontology dictionary.
+
+        Args:
+            graph: Parsed rdflib Graph
+            source_path: Source file path
+            format: Format of the ontology file
+
+        Returns:
+            Ontology dictionary
+        """
+        ontology = {
+            "uri": "",
+            "name": os.path.basename(source_path),
+            "version": "1.0",
+            "classes": [],
+            "properties": [],
+            "metadata": {
+                "source_path": source_path,
+                "ingested_at": datetime.now().isoformat(),
+                "format": format
+            }
+        }
+
+        # 1. Extract Ontology Metadata
+        for s, p, o in graph.triples((None, RDF.type, OWL.Ontology)):
+            ontology["uri"] = str(s)
+            
+            # Try to find label/comment/versionInfo
+            for _, _, label in graph.triples((s, RDFS.label, None)):
+                ontology["name"] = str(label)
+                
+            for _, _, comment in graph.triples((s, RDFS.comment, None)):
+                ontology["description"] = str(comment)
+                
+            for _, _, version in graph.triples((s, OWL.versionInfo, None)):
+                ontology["version"] = str(version)
+            
+            # Break after first ontology definition found (usually only one per file)
+            break
+            
+        # 2. Extract Classes
+        classes = {}
+        # Union of owl:Class and rdfs:Class
+        class_types = [OWL.Class, RDFS.Class]
+        for c_type in class_types:
+            for s, p, o in graph.triples((None, RDF.type, c_type)):
+                if isinstance(s, rdflib.BNode):
+                    continue # Skip blank nodes for now
+                    
+                uri = str(s)
+                if uri not in classes:
+                    cls_def = {
+                        "uri": uri,
+                        "name": self._get_local_name(uri),
+                        "type": "class"
+                    }
+                    
+                    # Add label/comment
+                    label = graph.value(s, RDFS.label)
+                    if label:
+                        cls_def["label"] = str(label)
+                        cls_def["name"] = str(label) # Prefer label as name if available? Or keep URI fragment? 
+                        # Keeping local name from URI is safer for internal IDs, label for display. 
+                        # But Semantica seems to use "name" for the identifier in some examples.
+                        # Let's keep name as local name or label if simple.
+                    
+                    comment = graph.value(s, RDFS.comment)
+                    if comment:
+                        cls_def["description"] = str(comment)
+                        
+                    # Superclasses
+                    parents = []
+                    for _, _, parent in graph.triples((s, RDFS.subClassOf, None)):
+                        if isinstance(parent, rdflib.URIRef):
+                            parents.append(str(parent))
+                    if parents:
+                        cls_def["parents"] = parents
+                        
+                    classes[uri] = cls_def
+        
+        ontology["classes"] = list(classes.values())
+
+        # 3. Extract Properties
+        properties = {}
+        # Object Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.ObjectProperty)):
+            self._add_property(graph, s, "object", properties)
+            
+        # Datatype Properties
+        for s, p, o in graph.triples((None, RDF.type, OWL.DatatypeProperty)):
+            self._add_property(graph, s, "data", properties)
+            
+        # RDF Properties (generic)
+        for s, p, o in graph.triples((None, RDF.type, RDF.Property)):
+            if str(s) not in properties: # Don't overwrite if already found as specific type
+                self._add_property(graph, s, "annotation", properties) # Default to annotation or generic
+        
+        ontology["properties"] = list(properties.values())
+        
+        return ontology
+
+    def _add_property(self, graph: Graph, subject: rdflib.term.Node, prop_type: str, properties_dict: Dict):
+        if isinstance(subject, rdflib.BNode):
+            return
+
+        uri = str(subject)
+        if uri in properties_dict:
+            return
+            
+        prop_def = {
+            "uri": uri,
+            "name": self._get_local_name(uri),
+            "type": prop_type
+        }
+        
+        label = graph.value(subject, RDFS.label)
+        if label:
+            prop_def["label"] = str(label)
+            
+        comment = graph.value(subject, RDFS.comment)
+        if comment:
+            prop_def["description"] = str(comment)
+            
+        # Domain and Range
+        domain = graph.value(subject, RDFS.domain)
+        if domain and isinstance(domain, rdflib.URIRef):
+            prop_def["domain"] = str(domain)
+            
+        range_val = graph.value(subject, RDFS.range)
+        if range_val and isinstance(range_val, rdflib.URIRef):
+            prop_def["range"] = str(range_val)
+            
+        properties_dict[uri] = prop_def
+
+    def _get_local_name(self, uri: str) -> str:
+        """Extract local name from URI."""
+        if '#' in uri:
+            return uri.split('#')[-1]
+        return uri.split('/')[-1]

semantica/ingest/pandas_ingestor.py

@@ -25,6 +25,8 @@ Example Usage:
 """
 
 import json
+import csv
+import chardet
 from dataclasses import dataclass, field
 from datetime import datetime
 from pathlib import Path
@@ -193,23 +195,9 @@ class PandasIngestor:
     def from_csv(
         self,
         file_path: Union[str, Path],
+        chunksize: Optional[int] = None,
         **pandas_options,
     ) -> PandasData:
-        """
-        Ingest data from CSV file.
-
-        This method reads a CSV file using pandas and ingests it as a DataFrame.
-
-        Args:
-            file_path: Path to CSV file
-            **pandas_options: Additional options passed to pd.read_csv()
-
-        Returns:
-            PandasData: Ingested data object
-
-        Raises:
-            ProcessingError: If CSV reading fails
-        """
         file_path = Path(file_path)
 
         if not file_path.exists():
@@ -223,15 +211,104 @@ class PandasIngestor:
         )
 
         try:
-            # Read CSV with pandas
-            dataframe = pd.read_csv(file_path, **pandas_options)
 
-            self.progress_tracker.update_tracking(
-                tracking_id, message="CSV read successfully, processing DataFrame..."
+            # ---------- Encoding Detection ----------
+            with open(file_path, "rb") as f:
+                raw = f.read(100_000)
+                encoding_info = chardet.detect(raw)
+            encoding = encoding_info.get("encoding") or "utf-8"
+
+        
+            # ---------- Delimiter & Header Detection ----------
+            with open(file_path, "r", encoding=encoding, errors="replace") as f:
+                sample = f.read(10000)
+                sniffer = csv.Sniffer()
+
+                try:
+                    dialect = sniffer.sniff(sample, delimiters=[",", ";", "\t", "|"])
+                    delimiter = dialect.delimiter
+                    quotechar = dialect.quotechar
+                except Exception:
+                    delimiter = ","
+                    quotechar = '"'
+
+                # Header handling: default to True (treat first row as header)
+                # unless user explicitly overrides via pandas_options['header'].
+                has_header = True
+                header_opt = pandas_options.get("header", None)
+                if header_opt is None:
+                    has_header = True
+                elif header_opt == 0 or header_opt == "infer":
+                    has_header = True
+                else:
+                    # Any explicit non-header setting (e.g., None or int>0) implies no header
+                    try:
+                        has_header = False if header_opt is None or int(header_opt) != 0 else True
+                    except Exception:
+                        has_header = False
+
+
+            skipped_rows = 0
+            dataframes = []
+
+            # ---------- CSV Reading (Chunked if needed) ----------
+            # Preserve explicit header setting (including None) if user provided it.
+            has_explicit_header = "header" in pandas_options
+            explicit_header = pandas_options.pop("header", None) if has_explicit_header else None
+            header_arg = explicit_header if has_explicit_header else (0 if has_header else None)
+
+            reader = pd.read_csv(
+                file_path,
+                sep=delimiter,
+                encoding=encoding,
+                encoding_errors="replace",
+                quoting=csv.QUOTE_MINIMAL,
+                header=header_arg,
+                quotechar=quotechar,
+                escapechar="\\",
+                engine="python",
+                on_bad_lines="warn",
+                chunksize=chunksize,
+                **pandas_options,
             )
 
-            # Ingest the DataFrame
-            return self.ingest_dataframe(dataframe, **pandas_options)
+            if chunksize:
+                for chunk in reader:
+                    dataframes.append(chunk)
+            else:
+                dataframes.append(reader)
+
+            dataframe = pd.concat(dataframes, ignore_index=True)
+
+            self.progress_tracker.update_tracking(
+                tracking_id,
+                message="CSV parsed successfully, ingesting DataFrame...",
+            )
+
+            # ---------- Ingest ----------
+            pandas_data = self.ingest_dataframe(dataframe)
+
+            # ---------- Metadata ----------
+            pandas_data.metadata.update(
+                {
+                    "source": "csv",
+                    "file": str(file_path),
+                    "detected_encoding": encoding,
+                    "encoding_confidence": encoding_info.get("confidence"),
+                    "detected_delimiter": delimiter,
+                    "header_detected": has_header,
+                    "chunksize": chunksize,
+                    "malformed_rows_skipped": skipped_rows,
+                }
+            )
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"CSV ingestion completed: {pandas_data.row_count} rows",
+            )
+
+            return pandas_data
 
         except Exception as e:
             self.progress_tracker.stop_tracking(

semantica/kg/graph_builder.py

@@ -89,6 +89,7 @@ class GraphBuilder:
         self.track_history = track_history
         self.version_snapshots = version_snapshots
         self.graph_store = graph_store
+        self.config = kwargs  # Store additional config for extractors
 
         # Initialize logging
         from ..utils.logging import get_logger
@@ -130,6 +131,11 @@ class GraphBuilder:
 
     def _process_item(self, item: Any, all_entities: List[Any], all_relationships: List[Any], **options):
         """Helper to process a single item and add to entities or relationships list."""
+        if isinstance(item, str):
+            # Treat string as text for extraction
+            self._extract_from_text(item, all_entities, all_relationships, **options)
+            return
+
         if hasattr(item, "text") and (hasattr(item, "label") or hasattr(item, "type")):
             # It's likely an Entity object
             entity_dict = {
@@ -155,7 +161,14 @@ class GraphBuilder:
             }
             all_relationships.append(rel_dict)
         elif isinstance(item, dict):
-            # Detect and normalize Entity objects inside dict
+            if "source_id" in item and "source" not in item:
+                item["source"] = item["source_id"]
+            if "target_id" in item and "target" not in item:
+                item["target"] = item["target_id"]
+            if "subject" in item and "source" not in item:
+                item["source"] = item["subject"]
+            if "object" in item and "target" not in item:
+                item["target"] = item["object"]
             if "source" in item and not isinstance(item["source"], str):
                 src = item["source"]
                 item["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -209,30 +222,68 @@ class GraphBuilder:
                 # If still nothing found and has 'text', try extraction
                 if not found_something and "text" in item:
                     text = item["text"]
-                    # Perform extraction if requested or if it's the only way
-                    if options.get("extract", True):
-                        from ..semantic_extract.ner_extractor import NERExtractor
-                        from ..semantic_extract.triplet_extractor import TripletExtractor
-                        
-                        ner_method = options.get("ner_method", "ml")
-                        triplet_method = options.get("triplet_method", "pattern")
-                        
-                        ner = NERExtractor(method=ner_method)
-                        entities = ner.extract_entities(text)
-                        for ent in entities:
-                            self._process_item(ent, all_entities, all_relationships, **options)
-                        
-                        # Only try triplets if specifically requested or if method provided
-                        if "triplet_method" in options or options.get("extract_relations", False):
-                            triplet = TripletExtractor(method=triplet_method)
-                            relations = triplet.extract_triplets(text)
-                            for rel in relations:
-                                self._process_item(rel, all_entities, all_relationships, **options)
-                        found_something = True
+                    self._extract_from_text(text, all_entities, all_relationships, **options)
+                    found_something = True
         else:
             # Unknown type
             pass
 
+    def _extract_from_text(self, text: str, all_entities: List[Any], all_relationships: List[Any], **options):
+        """Helper to extract knowledge from text using configured methods."""
+        if not options.get("extract", True):
+            return
+
+        from ..semantic_extract.ner_extractor import NERExtractor
+        from ..semantic_extract.relation_extractor import RelationExtractor
+        from ..semantic_extract.triplet_extractor import TripletExtractor
+        
+        # Default to LLM methods as per requirement
+        ner_method = options.get("ner_method", "llm")
+        relation_method = options.get("relation_method", "llm")
+        triplet_method = options.get("triplet_method", "llm")
+        
+        self.logger.info(f"Extracting knowledge from text ({len(text)} chars) using {ner_method}...")
+        
+        # 1. Extract Entities
+        ner = NERExtractor(method=ner_method, **self.config)
+        try:
+            entities = ner.extract_entities(text, **options)
+            extracted_count = len(entities)
+            self._extraction_stats["extracted_entities"] += extracted_count
+            self.logger.info(f"Extracted {extracted_count} entities")
+            for ent in entities:
+                self._process_item(ent, all_entities, all_relationships, **options)
+        except Exception as e:
+            self.logger.error(f"Entity extraction failed: {e}")
+            entities = []
+        
+        # 2. Extract Relations (if requested)
+        if options.get("extract_relations", True):
+            rel_extractor = RelationExtractor(method=relation_method, **self.config)
+            try:
+                # Pass entities if available to help relation extraction
+                relations = rel_extractor.extract_relations(text, entities=entities, **options)
+                extracted_count = len(relations)
+                self._extraction_stats["extracted_relations"] += extracted_count
+                self.logger.info(f"Extracted {extracted_count} relationships")
+                for rel in relations:
+                    self._process_item(rel, all_entities, all_relationships, **options)
+            except Exception as e:
+                self.logger.error(f"Relation extraction failed: {e}")
+
+        # 3. Extract Triplets (if requested)
+        if options.get("extract_triplets", True):
+            trip_extractor = TripletExtractor(method=triplet_method, **self.config)
+            try:
+                triplets = trip_extractor.extract_triplets(text, entities=entities, **options)
+                extracted_count = len(triplets)
+                self._extraction_stats["extracted_triplets"] += extracted_count
+                self.logger.info(f"Extracted {extracted_count} triplets")
+                for trip in triplets:
+                    self._process_item(trip, all_entities, all_relationships, **options)
+            except Exception as e:
+                self.logger.error(f"Triplet extraction failed: {e}")
+
     def build(
         self,
         sources: Union[List[Any], Any],
@@ -303,8 +354,31 @@ class GraphBuilder:
         elif not isinstance(sources, list):
             sources = [sources]
 
+        # Count input relationships for warning if all are dropped
+        input_relationships_count = 0
+        if isinstance(source_dict, dict):
+            rels = source_dict.get("relationships", [])
+            if isinstance(rels, list):
+                input_relationships_count += len(rels)
+            elif rels is not None:
+                input_relationships_count += 1
+        if explicit_relationships:
+            for rel_item in explicit_relationships:
+                if isinstance(rel_item, list):
+                    input_relationships_count += len(rel_item)
+                else:
+                    input_relationships_count += 1
+
         # Track graph building
         build_start_time = time.time()
+        
+        # Initialize extraction statistics for traceability
+        self._extraction_stats = {
+            "extracted_entities": 0,
+            "extracted_relations": 0,
+            "extracted_triplets": 0
+        }
+        
         tracking_id = self.progress_tracker.start_tracking(
             module="kg",
             submodule="GraphBuilder",
@@ -416,11 +490,12 @@ class GraphBuilder:
                         pipeline_id=pipeline_id,
                     )
                     
-                    # Check if relationships are already in dictionary format
                     sample_rel = relationships_list[0] if relationships_list else None
                     is_dict_format = isinstance(sample_rel, dict) and (
-                        "source" in sample_rel and "target" in sample_rel
-                    ) and not hasattr(sample_rel, "__dict__") # Ensure it's not a class instance
+                        ("source" in sample_rel and "target" in sample_rel)
+                        or ("source_id" in sample_rel and "target_id" in sample_rel)
+                        or ("subject" in sample_rel and "object" in sample_rel)
+                    ) and not hasattr(sample_rel, "__dict__")
                     
                     if is_dict_format:
                         # Fast path: directly append dictionaries after normalizing source/target
@@ -429,8 +504,15 @@ class GraphBuilder:
                             batch = relationships_list[i:i + batch_size]
                             for item in batch:
                                 if isinstance(item, dict):
-                                    # Normalize source/target if they are objects
                                     rel_dict = item.copy()
+                                    if "source_id" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["source_id"]
+                                    if "target_id" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["target_id"]
+                                    if "subject" in rel_dict and "source" not in rel_dict:
+                                        rel_dict["source"] = rel_dict["subject"]
+                                    if "object" in rel_dict and "target" not in rel_dict:
+                                        rel_dict["target"] = rel_dict["object"]
                                     if "source" in rel_dict and not isinstance(rel_dict["source"], str):
                                         src = rel_dict["source"]
                                         rel_dict["source"] = getattr(src, "id", getattr(src, "text", str(src)))
@@ -514,11 +596,19 @@ class GraphBuilder:
                 resolution_start = time.time()
                 resolved_entities = resolver_to_use.resolve_entities(all_entities)
                 resolution_time = time.time() - resolution_start
-                print(f"✅ Resolved to {len(resolved_entities)} unique entities ({resolution_time:.2f}s)")
+                print(f"[DONE] Resolved to {len(resolved_entities)} unique entities ({resolution_time:.2f}s)")
                 self.logger.info(
                     f"Entity resolution complete: {len(all_entities)} -> {len(resolved_entities)} unique entities"
                 )
 
+            if input_relationships_count > 0 and len(all_relationships) == 0:
+                warning_msg = (
+                    f"All relationships were dropped during graph building: "
+                    f"{input_relationships_count} input relationships, 0 in final graph"
+                )
+                self.logger.warning(warning_msg)
+                print(f"Warning: {warning_msg}")
+
             # Build graph structure
             print("Building graph structure...")
             structure_start = time.time()
@@ -534,7 +624,7 @@ class GraphBuilder:
                 },
             }
             structure_time = time.time() - structure_start
-            print(f"✅ Graph structure built ({structure_time:.2f}s)")
+            print(f"[DONE] Graph structure built ({structure_time:.2f}s)")
 
             # Persist to GraphStore if available
             if self.graph_store:
@@ -567,7 +657,7 @@ class GraphBuilder:
                 edge_time = time.time() - edge_start
                 total_store_time = time.time() - store_start
                 print(f"  Added {edge_count} edges ({edge_time:.2f}s)")
-                print(f"✅ GraphStore persistence complete ({total_store_time:.2f}s total)")
+                print(f"[DONE] GraphStore persistence complete ({total_store_time:.2f}s total)")
                 self.logger.info(f"Persisted {node_count} nodes and {edge_count} edges")
 
             # Detect and resolve conflicts if conflict detector is available
@@ -604,7 +694,14 @@ class GraphBuilder:
             
             # Print final summary with timing
             print(f"\n{'='*60}")
-            print(f"✅ Knowledge Graph Build Complete")
+            print(f"[INFO] Extraction Statistics")
+            print(f"   Extracted Entities: {self._extraction_stats['extracted_entities']}")
+            print(f"   Extracted Relationships: {self._extraction_stats['extracted_relations']}")
+            print(f"   Extracted Triplets: {self._extraction_stats['extracted_triplets']}")
+            print(f"{'='*60}")
+
+            print(f"\n{'='*60}")
+            print(f"[DONE] Knowledge Graph Build Complete")
             print(f"   Entities: {len(resolved_entities)}")
             print(f"   Relationships: {len(all_relationships)}")
             print(f"   Total time: {total_build_time:.2f}s")
@@ -623,6 +720,14 @@ class GraphBuilder:
             )
             raise
 
+    def build_single_source(
+        self,
+        kg_data: Dict[str, Any],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Dict[str, Any]:
+        return self.build(kg_data, pipeline_id=pipeline_id, **options)
+
     def add_temporal_edge(
         self,
         graph,

semantica/kg/provenance_tracker.py

@@ -1,19 +1,29 @@
 """
-Provenance Tracking Module
+Provenance Tracking Module (Enhanced with Unified Backend)
 
 This module provides comprehensive source tracking and lineage capabilities
 for the Semantica framework, enabling tracking of data origins and evolution
 for knowledge graph entities and relationships.
 
+IMPORTANT: This module now uses the unified semantica.provenance.ProvenanceManager
+backend for enhanced W3C PROV-O compliance and audit-grade tracking. All existing
+APIs remain 100% backward compatible.
+
+For new code, consider using the unified API:
+    >>> from semantica.provenance import ProvenanceManager
+    >>> prov_mgr = ProvenanceManager()
+
 Key Features:
     - Entity provenance tracking (source, timestamp, metadata)
     - Relationship provenance tracking
     - Lineage retrieval (complete provenance history)
     - Source aggregation (multiple sources per entity)
     - Temporal tracking (first seen, last updated)
+    - W3C PROV-O compliance (when using unified backend)
+    - Audit-grade integrity verification
 
 Main Classes:
-    - ProvenanceTracker: Main provenance tracking engine
+    - ProvenanceTracker: Main provenance tracking engine (backward compatible wrapper)
 
 Example Usage:
     >>> from semantica.kg import ProvenanceTracker
@@ -32,6 +42,13 @@ from typing import Any, Dict, List, Optional
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
 
+# Import unified provenance manager
+try:
+    from ..provenance import ProvenanceManager as UnifiedProvenanceManager
+    UNIFIED_AVAILABLE = True
+except ImportError:
+    UNIFIED_AVAILABLE = False
+
 
 class ProvenanceTracker:
     """
@@ -75,6 +92,10 @@ class ProvenanceTracker:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        self._use_unified = UNIFIED_AVAILABLE
+        if self._use_unified:
+            self._unified_manager = UnifiedProvenanceManager()
+
         self.logger.debug("Provenance tracker initialized")
 
     def track_entity(
@@ -89,10 +110,33 @@ class ProvenanceTracker:
 
         Args:
             entity_id: Entity identifier
-            source: Source identifier (e.g., "file_1", "api_endpoint_2")
+            source: Source identifier (e.g., "file_1", "api_endpoint_2", DOI)
             metadata: Optional metadata dictionary (e.g., confidence scores,
                      extraction methods, etc.)
         """
+        if self._use_unified:
+            # Delegate to unified manager
+            try:
+                self._unified_manager.track_entity(
+                    entity_id=entity_id,
+                    source=source,
+                    metadata=metadata
+                )
+            except Exception as e:
+                self.logger.warning(f"Unified tracking failed, using fallback: {e}")
+                self._track_entity_legacy(entity_id, source, metadata)
+        else:
+            # Use legacy implementation
+            self._track_entity_legacy(entity_id, source, metadata)
+        
+        self.logger.debug(
+            f"Tracked provenance for entity {entity_id} from source {source}"
+        )
+    
+    def _track_entity_legacy(
+        self, entity_id: str, source: str, metadata: Optional[Dict[str, Any]] = None
+    ) -> None:
+        """Legacy entity tracking implementation."""
         if entity_id not in self.provenance_data:
             self.provenance_data[entity_id] = {
                 "sources": [],
@@ -115,10 +159,6 @@ class ProvenanceTracker:
         if metadata:
             self.provenance_data[entity_id]["metadata"].update(metadata)
 
-        self.logger.debug(
-            f"Tracked provenance for entity {entity_id} from source {source}"
-        )
-
     def track_relationship(
         self,
         relationship_id: str,
@@ -175,9 +215,20 @@ class ProvenanceTracker:
                 - timestamp: ISO format timestamp
                 - metadata: Source metadata dictionary
         """
+        if self._use_unified:
+            # Get from unified manager
+            try:
+                return self._unified_manager.get_all_sources(entity_id)
+            except Exception as e:
+                self.logger.warning(f"Unified retrieval failed, using fallback: {e}")
+                return self._get_all_sources_legacy(entity_id)
+        else:
+            return self._get_all_sources_legacy(entity_id)
+    
+    def _get_all_sources_legacy(self, entity_id: str) -> List[Dict[str, Any]]:
+        """Legacy get all sources implementation."""
         if entity_id not in self.provenance_data:
             return []
-
         return self.provenance_data[entity_id].get("sources", [])
 
     def get_lineage(self, entity_id: str) -> Dict[str, Any]:
@@ -192,14 +243,38 @@ class ProvenanceTracker:
 
         Returns:
             dict: Complete lineage information containing:
-                - sources: List of all source entries
+                - sources: List of all source entries (legacy format)
                 - first_seen: ISO timestamp of first source
                 - last_updated: ISO timestamp of most recent source
                 - metadata: Aggregated metadata dictionary
+                - lineage_chain: Complete lineage chain (when using unified backend)
         """
+        if self._use_unified:
+            # Get from unified manager
+            try:
+                lineage = self._unified_manager.get_lineage(entity_id)
+                if not lineage:
+                    return {}
+                
+                # Convert to legacy format for backward compatibility
+                legacy_format = {
+                    "sources": self._unified_manager.get_all_sources(entity_id),
+                    "first_seen": lineage.get("first_seen"),
+                    "last_updated": lineage.get("last_updated"),
+                    "metadata": lineage.get("metadata", {}),  # Include metadata from lineage
+                    "lineage_chain": lineage.get("lineage_chain", [])
+                }
+                return legacy_format
+            except Exception as e:
+                self.logger.warning(f"Unified retrieval failed, using fallback: {e}")
+                return self._get_lineage_legacy(entity_id)
+        else:
+            return self._get_lineage_legacy(entity_id)
+    
+    def _get_lineage_legacy(self, entity_id: str) -> Dict[str, Any]:
+        """Legacy get lineage implementation."""
         if entity_id not in self.provenance_data:
             return {}
-
         return self.provenance_data[entity_id].copy()
 
     def get_provenance(self, entity_id: str) -> Optional[Dict[str, Any]]:
@@ -216,7 +291,18 @@ class ProvenanceTracker:
             dict: Complete provenance information (same as get_lineage()),
                   or None if entity is not tracked
         """
-        return self.provenance_data.get(entity_id)
+        if self._use_unified:
+            try:
+                prov = self._unified_manager.get_provenance(entity_id)
+                if not prov:
+                    return None
+                # Return in legacy format
+                return self.get_lineage(entity_id)
+            except Exception as e:
+                self.logger.warning(f"Unified retrieval failed, using fallback: {e}")
+                return self.provenance_data.get(entity_id)
+        else:
+            return self.provenance_data.get(entity_id)
 
     def track_entities_batch(
         self,

semantica/kg/temporal_query.py

@@ -631,39 +631,47 @@ class TemporalPatternDetector:
 
 class TemporalVersionManager:
     """
-    Temporal version management engine.
+    Enhanced temporal version management engine with persistent storage.
 
-    This class provides version/snapshot management capabilities for knowledge
-    graphs, enabling creation of temporal versions, version comparison, and
-    version history tracking.
+    This class provides comprehensive version/snapshot management capabilities for knowledge
+    graphs, including persistent storage, detailed change tracking, and audit trails.
 
     Features:
-        - Version snapshot creation
-        - Version comparison
-        - Version history tracking
-        - Automatic snapshotting (planned)
-        - Version rollback (planned)
+        - Persistent snapshot storage (SQLite or in-memory)
+        - Detailed change tracking with entity-level diffs
+        - SHA-256 checksums for data integrity
+        - Standardized metadata with author attribution
+        - Version comparison with backward compatibility
+        - Input validation and security features
 
     Example Usage:
+        >>> # In-memory storage
         >>> manager = TemporalVersionManager()
-        >>> version = manager.create_version(graph, version_label="v1.0")
-        >>> comparison = manager.compare_versions(version1, version2)
+        >>> # Persistent storage
+        >>> manager = TemporalVersionManager(storage_path="versions.db")
+        >>> snapshot = manager.create_snapshot(graph, "v1.0", 
+        ...     author="alice@company.com", description="Initial release")
+        >>> versions = manager.list_versions()
+        >>> diff = manager.compare_versions("v1.0", "v1.1")
     """
 
     def __init__(
         self,
+        storage_path: Optional[str] = None,
         snapshot_interval: Optional[int] = None,
         auto_snapshot: bool = False,
         version_strategy: str = "timestamp",
         **config,
     ):
         """
-        Initialize temporal version manager.
+        Initialize enhanced temporal version manager.
 
-        Sets up the version manager with snapshot configuration and versioning
-        strategy.
+        Sets up the version manager with storage backend, snapshot configuration,
+        and versioning strategy.
 
         Args:
+            storage_path: Path to SQLite database file for persistent storage.
+                         If None, uses in-memory storage (default: None)
             snapshot_interval: Interval for automatic snapshots in seconds
                              (optional, auto_snapshot must be True)
             auto_snapshot: Enable automatic snapshots (default: False)
@@ -671,11 +679,23 @@ class TemporalVersionManager:
                 - "timestamp": Use timestamps for version labels (default)
                 - "incremental": Use incremental version numbers (planned)
                 - "semantic": Use semantic versioning (planned)
-            **config: Additional configuration options (unused)
+            **config: Additional configuration options
         """
+        from semantica.change_management import ChangeLogEntry, VersionStorage, InMemoryVersionStorage, SQLiteVersionStorage
+        from ..utils.logging import get_logger
+        
         self.snapshot_interval = snapshot_interval
         self.auto_snapshot = auto_snapshot
         self.version_strategy = version_strategy
+        self.logger = get_logger("temporal_version_manager")
+        
+        # Initialize storage backend
+        if storage_path:
+            self.storage = SQLiteVersionStorage(storage_path)
+            self.logger.info(f"Initialized with SQLite storage: {storage_path}")
+        else:
+            self.storage = InMemoryVersionStorage()
+            self.logger.info("Initialized with in-memory storage")
 
     def create_version(
         self,
@@ -722,37 +742,282 @@ class TemporalVersionManager:
 
     def compare_versions(
         self,
-        version1: Dict[str, Any],
-        version2: Dict[str, Any],
+        v1_label_or_dict,
+        v2_label_or_dict,
         comparison_metrics: Optional[List[str]] = None,
         **options,
     ) -> Dict[str, Any]:
         """
-        Compare two graph versions.
+        Compare two graph versions with detailed entity-level differences.
 
-        This method compares two version snapshots and calculates differences
-        in entities and relationships.
+        This method compares two version snapshots and calculates detailed differences
+        in entities and relationships, maintaining backward compatibility.
 
         Args:
-            version1: First version snapshot dictionary
-            version2: Second version snapshot dictionary
+            v1_label_or_dict: First version (label string or snapshot dict)
+            v2_label_or_dict: Second version (label string or snapshot dict)
             comparison_metrics: List of metrics to calculate (optional, unused)
             **options: Additional comparison options (unused)
 
         Returns:
-            dict: Version comparison results containing:
-                - version1: Label of first version
-                - version2: Label of second version
-                - entities_added: Change in entity count (version2 - version1)
-                - relationships_added: Change in relationship count (version2 - version1)
+            dict: Detailed version comparison results containing:
+                - summary: Backward-compatible summary counts
+                - entities_added: List of added entities
+                - entities_removed: List of removed entities  
+                - entities_modified: List of modified entities with changes
+                - relationships_added: List of added relationships
+                - relationships_removed: List of removed relationships
+                - relationships_modified: List of modified relationships
         """
-        comparison = {
+        from ..utils.exceptions import ValidationError
+        
+        # Handle both label strings and snapshot dictionaries
+        if isinstance(v1_label_or_dict, str):
+            version1 = self.storage.get(v1_label_or_dict)
+            if not version1:
+                raise ValidationError(f"Version not found: {v1_label_or_dict}")
+        else:
+            version1 = v1_label_or_dict
+            
+        if isinstance(v2_label_or_dict, str):
+            version2 = self.storage.get(v2_label_or_dict)
+            if not version2:
+                raise ValidationError(f"Version not found: {v2_label_or_dict}")
+        else:
+            version2 = v2_label_or_dict
+        
+        # Compute detailed diff
+        detailed_diff = self._compute_detailed_diff(version1, version2)
+        
+        # Maintain backward compatibility with summary
+        summary = {
+            "entities_added": len(detailed_diff["entities_added"]),
+            "entities_removed": len(detailed_diff["entities_removed"]),
+            "entities_modified": len(detailed_diff["entities_modified"]),
+            "relationships_added": len(detailed_diff["relationships_added"]),
+            "relationships_removed": len(detailed_diff["relationships_removed"]),
+            "relationships_modified": len(detailed_diff["relationships_modified"])
+        }
+        
+        return {
             "version1": version1.get("label", "unknown"),
             "version2": version2.get("label", "unknown"),
-            "entities_added": len(version2.get("entities", []))
-            - len(version1.get("entities", [])),
-            "relationships_added": len(version2.get("relationships", []))
-            - len(version1.get("relationships", [])),
+            "summary": summary,
+            **detailed_diff
         }
 
-        return comparison
+    def create_snapshot(
+        self, 
+        graph: Dict[str, Any], 
+        version_label: str, 
+        author: str, 
+        description: str,
+        **options
+    ) -> Dict[str, Any]:
+        """
+        Create and store snapshot with checksum and metadata.
+        
+        Args:
+            graph: Knowledge graph dict with "entities" and "relationships"
+            version_label: Version string (e.g., "v1.0")
+            author: Email address of the change author
+            description: Change description (max 500 chars)
+            **options: Additional options
+            
+        Returns:
+            dict: Snapshot with metadata and checksum
+            
+        Raises:
+            ValidationError: If input validation fails
+            ProcessingError: If storage operation fails
+        """
+        from ..change_management import ChangeLogEntry, compute_checksum
+        from datetime import datetime
+        
+        # Validate inputs
+        change_entry = ChangeLogEntry(
+            timestamp=datetime.now().isoformat(),
+            author=author,
+            description=description
+        )
+        
+        # Create snapshot
+        snapshot = {
+            "label": version_label,
+            "timestamp": change_entry.timestamp,
+            "author": change_entry.author,
+            "description": change_entry.description,
+            "entities": graph.get("entities", []).copy(),
+            "relationships": graph.get("relationships", []).copy(),
+            "metadata": options.get("metadata", {})
+        }
+        
+        # Compute and add checksum
+        snapshot["checksum"] = compute_checksum(snapshot)
+        
+        # Store snapshot
+        self.storage.save(snapshot)
+        
+        self.logger.info(f"Created snapshot '{version_label}' by {author}")
+        return snapshot
+    
+    def list_versions(self) -> List[Dict[str, Any]]:
+        """
+        List all version snapshots.
+        
+        Returns:
+            List of version metadata dictionaries
+        """
+        return self.storage.list_all()
+    
+    def get_version(self, label: str) -> Optional[Dict[str, Any]]:
+        """
+        Retrieve specific version by label.
+        
+        Args:
+            label: Version label to retrieve
+            
+        Returns:
+            Snapshot dictionary or None if not found
+        """
+        return self.storage.get(label)
+    
+    def verify_checksum(self, snapshot: Dict[str, Any]) -> bool:
+        """
+        Verify the integrity of a snapshot using its checksum.
+        
+        Args:
+            snapshot: Snapshot dictionary with checksum field
+            
+        Returns:
+            True if checksum is valid, False otherwise
+        """
+        from ..change_management import verify_checksum
+        return verify_checksum(snapshot)
+    
+    def _compute_detailed_diff(self, version1: Dict[str, Any], version2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute detailed entity and relationship differences between versions.
+        
+        Args:
+            version1: First version snapshot
+            version2: Second version snapshot
+            
+        Returns:
+            Dict with detailed diff information
+        """
+        entities1 = {e.get("id", str(i)): e for i, e in enumerate(version1.get("entities", []))}
+        entities2 = {e.get("id", str(i)): e for i, e in enumerate(version2.get("entities", []))}
+        
+        relationships1 = {self._relationship_key(r): r for r in version1.get("relationships", [])}
+        relationships2 = {self._relationship_key(r): r for r in version2.get("relationships", [])}
+        
+        # Entity differences
+        entity_ids1 = set(entities1.keys())
+        entity_ids2 = set(entities2.keys())
+        
+        entities_added = [entities2[eid] for eid in entity_ids2 - entity_ids1]
+        entities_removed = [entities1[eid] for eid in entity_ids1 - entity_ids2]
+        
+        entities_modified = []
+        for eid in entity_ids1 & entity_ids2:
+            if entities1[eid] != entities2[eid]:
+                changes = self._compute_entity_changes(entities1[eid], entities2[eid])
+                entities_modified.append({
+                    "id": eid,
+                    "before": entities1[eid],
+                    "after": entities2[eid],
+                    "changes": changes
+                })
+        
+        # Relationship differences
+        rel_keys1 = set(relationships1.keys())
+        rel_keys2 = set(relationships2.keys())
+        
+        relationships_added = [relationships2[key] for key in rel_keys2 - rel_keys1]
+        relationships_removed = [relationships1[key] for key in rel_keys1 - rel_keys2]
+        
+        relationships_modified = []
+        for key in rel_keys1 & rel_keys2:
+            if relationships1[key] != relationships2[key]:
+                changes = self._compute_relationship_changes(relationships1[key], relationships2[key])
+                relationships_modified.append({
+                    "key": key,
+                    "before": relationships1[key],
+                    "after": relationships2[key],
+                    "changes": changes
+                })
+        
+        return {
+            "entities_added": entities_added,
+            "entities_removed": entities_removed,
+            "entities_modified": entities_modified,
+            "relationships_added": relationships_added,
+            "relationships_removed": relationships_removed,
+            "relationships_modified": relationships_modified
+        }
+    
+    def _relationship_key(self, relationship: Dict[str, Any]) -> str:
+        """
+        Generate a unique key for a relationship.
+        
+        Args:
+            relationship: Relationship dictionary
+            
+        Returns:
+            Unique string key for the relationship
+        """
+        source = relationship.get("source", "")
+        target = relationship.get("target", "")
+        rel_type = relationship.get("type", relationship.get("relationship", ""))
+        return f"{source}|{rel_type}|{target}"
+    
+    def _compute_entity_changes(self, entity1: Dict[str, Any], entity2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute changes between two entity versions.
+        
+        Args:
+            entity1: Original entity
+            entity2: Modified entity
+            
+        Returns:
+            Dictionary of changes
+        """
+        changes = {}
+        
+        # Check all keys from both entities
+        all_keys = set(entity1.keys()) | set(entity2.keys())
+        
+        for key in all_keys:
+            val1 = entity1.get(key)
+            val2 = entity2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes
+    
+    def _compute_relationship_changes(self, rel1: Dict[str, Any], rel2: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Compute changes between two relationship versions.
+        
+        Args:
+            rel1: Original relationship
+            rel2: Modified relationship
+            
+        Returns:
+            Dictionary of changes
+        """
+        changes = {}
+        
+        # Check all keys from both relationships
+        all_keys = set(rel1.keys()) | set(rel2.keys())
+        
+        for key in all_keys:
+            val1 = rel1.get(key)
+            val2 = rel2.get(key)
+            
+            if val1 != val2:
+                changes[key] = {"from": val1, "to": val2}
+        
+        return changes

semantica/llms/llms_provenance.py

@@ -0,0 +1,386 @@
+"""
+Provenance-enabled wrappers for LLM providers.
+
+This module provides provenance tracking for all LLM operations:
+- Groq LLM
+- OpenAI LLM
+- HuggingFace LLM
+- LiteLLM
+
+Tracks: model name, tokens (prompt/completion), cost, latency, prompts, responses
+
+All classes wrap the original LLM providers and add optional provenance tracking
+without modifying existing functionality.
+
+Usage:
+    from semantica.llms.llms_provenance import (
+        GroqLLMWithProvenance,
+        OpenAILLMWithProvenance
+    )
+    
+    # Enable provenance tracking
+    llm = GroqLLMWithProvenance(provenance=True)
+    response = llm.generate("What is artificial intelligence?")
+    
+    # Provenance automatically tracks:
+    # - Model used
+    # - Token counts
+    # - API costs
+    # - Latency
+    # - Prompt and response previews
+
+Features:
+    - Zero breaking changes - works exactly like original LLM classes
+    - Opt-in provenance via provenance=True parameter
+    - Tracks all API calls with complete metadata
+    - Cost tracking for budget monitoring
+    - Performance monitoring (latency)
+    - Graceful degradation if provenance module unavailable
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Dict, Any
+import time
+import uuid
+
+
+class LLMProvenanceMixin:
+    """
+    Mixin to add provenance tracking to any LLM provider.
+    
+    This mixin provides common provenance infrastructure for tracking
+    LLM API calls including tokens, costs, and performance metrics.
+    """
+    
+    def __init__(self, provenance: bool = False, **kwargs):
+        """
+        Initialize LLM provenance tracking.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **kwargs: Additional arguments passed to parent class
+        """
+        self.provenance = provenance
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                # Graceful degradation if provenance module not available
+                self.provenance = False
+    
+    def _track_llm_call(
+        self,
+        call_id: str,
+        prompt: str,
+        response: Any,
+        **metadata
+    ) -> None:
+        """
+        Track LLM API call with provenance.
+        
+        Args:
+            call_id: Unique identifier for this API call
+            prompt: Input prompt
+            response: LLM response
+            **metadata: Additional metadata (tokens, cost, latency, etc.)
+        """
+        if self.provenance and self._prov_manager:
+            # Extract response text
+            response_text = response
+            if hasattr(response, 'text'):
+                response_text = response.text
+            elif hasattr(response, 'content'):
+                response_text = response.content
+            elif not isinstance(response, str):
+                response_text = str(response)
+            
+            self._prov_manager.track_entity(
+                entity_id=call_id,
+                source=f"{self.__class__.__name__}_api",
+                entity_type="llm_generation",
+                metadata={
+                    "model": getattr(self, 'model', 'unknown'),
+                    "prompt_preview": prompt[:200] if len(prompt) > 200 else prompt,
+                    "response_preview": response_text[:200] if len(str(response_text)) > 200 else str(response_text),
+                    **metadata
+                }
+            )
+
+
+class GroqLLMWithProvenance(LLMProvenanceMixin):
+    """
+    Groq LLM with provenance tracking.
+    
+    Wraps the original GroqLLM and tracks all API calls with complete metadata.
+    
+    Example:
+        >>> llm = GroqLLMWithProvenance(provenance=True, model="llama-3.1-70b")
+        >>> response = llm.generate("Explain quantum computing")
+        >>> # API call is tracked with model, tokens, cost, latency
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize Groq LLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original GroqLLM
+        """
+        from .groq_llm import GroqLLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = GroqLLM(**config)
+        self.model = getattr(self._llm, 'model', 'groq')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response (same format as original GroqLLM)
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            # Extract token counts if available
+            prompt_tokens = None
+            completion_tokens = None
+            total_cost = None
+            
+            if hasattr(response, 'usage'):
+                prompt_tokens = getattr(response.usage, 'prompt_tokens', None)
+                completion_tokens = getattr(response.usage, 'completion_tokens', None)
+            
+            if hasattr(response, 'cost'):
+                total_cost = response.cost
+            
+            self._track_llm_call(
+                call_id=f"groq_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_tokens=(prompt_tokens + completion_tokens) if (prompt_tokens and completion_tokens) else None,
+                total_cost=total_cost,
+                latency_seconds=elapsed,
+                temperature=kwargs.get('temperature'),
+                max_tokens=kwargs.get('max_tokens'),
+                top_p=kwargs.get('top_p')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+class OpenAILLMWithProvenance(LLMProvenanceMixin):
+    """
+    OpenAI LLM with provenance tracking.
+    
+    Wraps the original OpenAILLM and tracks all API calls.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize OpenAI LLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original OpenAILLM
+        """
+        from .openai_llm import OpenAILLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = OpenAILLM(**config)
+        self.model = getattr(self._llm, 'model', 'openai')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            # Extract token counts if available
+            prompt_tokens = None
+            completion_tokens = None
+            total_cost = None
+            
+            if hasattr(response, 'usage'):
+                prompt_tokens = getattr(response.usage, 'prompt_tokens', None)
+                completion_tokens = getattr(response.usage, 'completion_tokens', None)
+            
+            if hasattr(response, 'cost'):
+                total_cost = response.cost
+            
+            self._track_llm_call(
+                call_id=f"openai_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_tokens=(prompt_tokens + completion_tokens) if (prompt_tokens and completion_tokens) else None,
+                total_cost=total_cost,
+                latency_seconds=elapsed,
+                temperature=kwargs.get('temperature'),
+                max_tokens=kwargs.get('max_tokens')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+class HuggingFaceLLMWithProvenance(LLMProvenanceMixin):
+    """
+    HuggingFace LLM with provenance tracking.
+    
+    Wraps the original HuggingFaceLLM and tracks all generations.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize HuggingFace LLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original HuggingFaceLLM
+        """
+        from .huggingface_llm import HuggingFaceLLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = HuggingFaceLLM(**config)
+        self.model = getattr(self._llm, 'model', 'huggingface')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            self._track_llm_call(
+                call_id=f"hf_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                latency_seconds=elapsed,
+                max_length=kwargs.get('max_length'),
+                temperature=kwargs.get('temperature')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+class LiteLLMWithProvenance(LLMProvenanceMixin):
+    """
+    LiteLLM with provenance tracking.
+    
+    Wraps the original LiteLLM and tracks all API calls across providers.
+    """
+    
+    def __init__(self, provenance: bool = False, **config):
+        """
+        Initialize LiteLLM with optional provenance.
+        
+        Args:
+            provenance: Enable provenance tracking (default: False)
+            **config: Configuration passed to original LiteLLM
+        """
+        from .lite_llm import LiteLLM
+        
+        LLMProvenanceMixin.__init__(self, provenance=provenance)
+        self._llm = LiteLLM(**config)
+        self.model = getattr(self._llm, 'model', 'litellm')
+    
+    def generate(self, prompt: str, **kwargs):
+        """
+        Generate response with provenance tracking.
+        
+        Args:
+            prompt: Input prompt
+            **kwargs: Additional generation parameters
+            
+        Returns:
+            LLM response
+        """
+        start_time = time.time()
+        response = self._llm.generate(prompt, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance:
+            # LiteLLM provides unified response format
+            prompt_tokens = None
+            completion_tokens = None
+            total_cost = None
+            
+            if hasattr(response, 'usage'):
+                prompt_tokens = getattr(response.usage, 'prompt_tokens', None)
+                completion_tokens = getattr(response.usage, 'completion_tokens', None)
+            
+            if hasattr(response, '_hidden_params') and 'response_cost' in response._hidden_params:
+                total_cost = response._hidden_params['response_cost']
+            
+            self._track_llm_call(
+                call_id=f"lite_call_{uuid.uuid4().hex[:8]}",
+                prompt=prompt,
+                response=response,
+                prompt_tokens=prompt_tokens,
+                completion_tokens=completion_tokens,
+                total_cost=total_cost,
+                latency_seconds=elapsed,
+                provider=kwargs.get('provider')
+            )
+        
+        return response
+    
+    def __getattr__(self, name):
+        """Delegate other methods to wrapped LLM."""
+        return getattr(self._llm, name)
+
+
+# Convenience exports
+__all__ = [
+    'GroqLLMWithProvenance',
+    'OpenAILLMWithProvenance',
+    'HuggingFaceLLMWithProvenance',
+    'LiteLLMWithProvenance',
+    'LLMProvenanceMixin',
+]

semantica/normalize/normalize_provenance.py

@@ -0,0 +1,53 @@
+"""
+Provenance-enabled wrapper for normalization.
+
+Usage:
+    from semantica.normalize.normalize_provenance import NormalizerWithProvenance
+    
+    normalizer = NormalizerWithProvenance(provenance=True)
+    normalized_data = normalizer.normalize(data)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class NormalizerWithProvenance:
+    """Normalizer with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .normalizer import Normalizer
+        
+        self.provenance = provenance
+        self._normalizer = Normalizer(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def normalize(self, data: Any, source: str = None, **kwargs):
+        """Normalize data with provenance tracking."""
+        result = self._normalizer.normalize(data, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"normalize_{uuid.uuid4().hex[:8]}",
+                source=source or "normalization",
+                entity_type="normalized_data",
+                metadata={"method": kwargs.get('method', 'default')}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._normalizer, name)
+
+
+__all__ = ['NormalizerWithProvenance']

semantica/normalize/text_normalizer.py

@@ -562,15 +562,19 @@ class SpecialCharacterProcessor:
         Returns:
             str: Text with normalized punctuation marks
         """
-        # Normalize quotes
-        text = re.sub(r'["""]', '"', text)
-        text = re.sub(r"[''']", "'", text)
+        # Replace common smart punctuation with ASCII equivalents
+        replacements = {
+            "\u2018": "'",   # Left single quotation mark
+            "\u2019": "'",   # Right single quotation mark
+            "\u201C": '"',   # Left double quotation mark
+            "\u201D": '"',   # Right double quotation mark
+            "\u2013": "-",   # En dash
+            "\u2014": "--",  # Em dash
+            "\u2026": "...", # Ellipsis
+        }
 
-        # Normalize dashes
-        text = re.sub(r"[–—]", "-", text)
-
-        # Normalize ellipsis
-        text = text.replace("…", "...")
+        for old, new in replacements.items():
+            text = text.replace(old, new)
 
         return text
 

semantica/ontology/__init__.py

@@ -109,12 +109,14 @@ Convenience Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory
 
 Example Usage:
-    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator
+    >>> from semantica.ontology import generate_ontology, infer_classes, OntologyGenerator, ingest_ontology
     >>> # Using convenience functions
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
     >>> # Using classes directly
     >>> from semantica.ontology import OntologyGenerator, ClassInferrer, PropertyGenerator
     >>> generator = OntologyGenerator(base_uri="https://example.org/ontology/")
@@ -154,7 +156,10 @@ from .property_generator import PropertyGenerator
 from .registry import MethodRegistry, method_registry
 from .requirements_spec import RequirementsSpec, RequirementsSpecManager
 from .reuse_manager import ReuseDecision, ReuseManager
-from .version_manager import OntologyVersion, VersionManager
+# VersionManager and OntologyVersion moved to change_management module
+# Import them directly from there: from semantica.change_management import VersionManager, OntologyVersion
+from semantica.ingest import OntologyData, OntologyIngestor
+from .methods import ingest_ontology
 
 __all__ = [
     # Main generators
@@ -180,8 +185,7 @@ __all__ = [
     # Management
     "ReuseManager",
     "ReuseDecision",
-    "VersionManager",
-    "OntologyVersion",
+    # VersionManager and OntologyVersion moved to change_management module
     "NamespaceManager",
     "NamingConventions",
     "ModuleManager",
@@ -200,4 +204,7 @@ __all__ = [
     # Configuration
     "OntologyConfig",
     "ontology_config",
+    "ingest_ontology",
+    "OntologyData",
+    "OntologyIngestor",
 ]

semantica/ontology/llm_generator.py

@@ -41,7 +41,7 @@ class LLMOntologyGenerator:
 
         try:
             result = self.provider.generate_structured(
-                prompt, model=self.model or options.get("model"), temperature=options.get("temperature", 0.2)
+                prompt, model=self.model or options.get("model"), temperature=options.get("temperature")
             )
         except Exception as e:
             self.progress.update_tracking(tracking_id, message="LLM generation failed")

semantica/ontology/methods.py

@@ -111,15 +111,19 @@ Main Functions:
     - create_associative_class: Associative class creation wrapper
     - get_ontology_method: Get ontology method by name
     - list_available_methods: List registered methods
+    - ingest_ontology: Ingest ontology from file or directory (via semantica.ingest)
 
 Example Usage:
-    >>> from semantica.ontology.methods import generate_ontology, infer_classes
+    >>> from semantica.ontology.methods import generate_ontology, infer_classes, ingest_ontology
     >>> ontology = generate_ontology({"entities": [...], "relationships": [...]}, method="default")
     >>> classes = infer_classes(entities, method="default")
+    >>> data = ingest_ontology("ontology.ttl")
 """
 
-from typing import Any, Callable, Dict, List, Optional
+from typing import Any, Callable, Dict, List, Optional, Union
+from pathlib import Path
 
+from semantica.ingest import ingest_ontology as _ingest_ontology, OntologyData
 from .registry import method_registry
 
 
@@ -172,4 +176,23 @@ def list_available_methods(task: Optional[str] = None) -> Dict[str, List[str]]:
     return method_registry.list_all(task)
 
 
-pass
+def ingest_ontology(
+    source: Union[str, Path, List[Union[str, Path]]], 
+    method: str = "file", 
+    **kwargs
+) -> Union[OntologyData, List[OntologyData]]:
+    """
+    Ingest ontology from source.
+    
+    This is a convenience wrapper around semantica.ingest.ingest_ontology.
+    
+    Args:
+        source: Ontology file path, directory path, or list of paths
+        method: Ingestion method (default: "file")
+        **kwargs: Additional options
+        
+    Returns:
+        OntologyData or List[OntologyData]
+    """
+    return _ingest_ontology(source, method=method, **kwargs)
+

semantica/ontology/ontology_provenance.py

@@ -0,0 +1,53 @@
+"""
+Provenance-enabled wrapper for ontology operations.
+
+Usage:
+    from semantica.ontology.ontology_provenance import OntologyManagerWithProvenance
+    
+    ontology = OntologyManagerWithProvenance(provenance=True)
+    ontology.add_concept(concept, source="ontology.owl")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class OntologyManagerWithProvenance:
+    """Ontology manager with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .ontology_manager import OntologyManager
+        
+        self.provenance = provenance
+        self._manager = OntologyManager(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def add_concept(self, concept: Any, source: str = None, **kwargs):
+        """Add concept with provenance tracking."""
+        result = self._manager.add_concept(concept, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"concept_{uuid.uuid4().hex[:8]}",
+                source=source or "ontology",
+                entity_type="ontology_concept",
+                metadata={"concept_name": str(concept)}
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._manager, name)
+
+
+__all__ = ['OntologyManagerWithProvenance']

semantica/ontology/ontology_usage.md

@@ -42,6 +42,18 @@ classes = inferrer.infer_classes(entities, build_hierarchy=True)
 properties = prop_gen.infer_properties(entities, relationships, classes)
 ```
 
+### Ingesting Ontologies
+
+```python
+from semantica.ingest import OntologyIngestor
+
+# Create ingestor
+ingestor = OntologyIngestor()
+
+# Ingest ontology
+ontology_data = ingestor.ingest_ontology("ontology.ttl")
+```
+
 ## Ontology Generation
 
 ### Basic Ontology Generation
@@ -115,6 +127,49 @@ ontology = engine.from_data(
 )
 ```
 
+## Ontology Ingestion
+
+### Basic Ingestion
+
+Ingest existing ontologies from files (Turtle, RDF/XML, JSON-LD, etc.) into `OntologyData` objects.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a single file
+ontology_data = ingest_ontology("path/to/ontology.ttl")
+
+print(f"Source: {ontology_data.source_path}")
+print(f"Format: {ontology_data.format}")
+print(f"Data keys: {ontology_data.data.keys()}")
+```
+
+### Ingesting Directories
+
+Ingest all ontology files in a directory recursively.
+
+```python
+from semantica.ontology import ingest_ontology
+
+# Ingest a directory
+ontologies = ingest_ontology("path/to/ontologies_dir/")
+
+for ont in ontologies:
+    print(f"Ingested: {ont.source_path} ({ont.format})")
+```
+
+### Unified Ingestion Interface
+
+You can also use the unified `semantica.ingest` interface.
+
+```python
+from semantica.ingest import ingest
+
+# Ingest as "ontology" source type
+result = ingest("path/to/ontology.ttl", source_type="ontology")
+ontology_data = result["ontology"]
+```
+
 ## Class Inference
 
 ### Basic Class Inference

semantica/ontology/reuse_manager.py

@@ -363,3 +363,90 @@ class ReuseManager:
     def list_known_ontologies(self) -> List[str]:
         """List known ontology URIs."""
         return list(self.known_ontologies.keys())
+
+    def merge_ontology_data(
+        self, target: Dict[str, Any], source: Dict[str, Any], **options
+    ) -> Dict[str, Any]:
+        """
+        Merge source ontology data into target ontology.
+
+        Merges classes, properties, and metadata from source to target.
+        Handles deduplication based on URI and name.
+
+        Args:
+            target: Target ontology dictionary (modified in-place)
+            source: Source ontology dictionary
+            **options: Merge options:
+                - overwrite: Whether to overwrite existing elements (default: False)
+                - merge_metadata: Whether to merge metadata (default: True)
+
+        Returns:
+            Merged target ontology
+        """
+        tracking_id = self.progress_tracker.start_tracking(
+            module="ontology",
+            submodule="ReuseManager",
+            message=f"Merging ontology {source.get('name', 'unknown')} into {target.get('name', 'unknown')}",
+        )
+
+        try:
+            overwrite = options.get("overwrite", False)
+            
+            # Helper to merge lists of dicts (classes/properties)
+            def merge_lists(target_list, source_list, key_field="uri"):
+                existing_keys = {item.get(key_field): i for i, item in enumerate(target_list) if item.get(key_field)}
+                
+                for item in source_list:
+                    key = item.get(key_field)
+                    if not key:
+                        # Fallback to name if URI missing
+                        key = item.get("name")
+                        
+                    if key in existing_keys:
+                        if overwrite:
+                            target_list[existing_keys[key]] = item
+                    else:
+                        target_list.append(item)
+                        if key:
+                            existing_keys[key] = len(target_list) - 1
+
+            # Merge Classes
+            if "classes" in source:
+                if "classes" not in target:
+                    target["classes"] = []
+                merge_lists(target["classes"], source["classes"])
+
+            # Merge Properties
+            if "properties" in source:
+                if "properties" not in target:
+                    target["properties"] = []
+                merge_lists(target["properties"], source["properties"])
+
+            # Merge Metadata
+            if options.get("merge_metadata", True) and "metadata" in source:
+                if "metadata" not in target:
+                    target["metadata"] = {}
+                # Update with source metadata, preserving target's specific fields if needed
+                # Here we just update
+                target["metadata"].update(source["metadata"])
+                
+            # Merge Imports
+            if "imports" in source:
+                if "imports" not in target:
+                    target["imports"] = []
+                for imp in source["imports"]:
+                    if imp not in target["imports"]:
+                        target["imports"].append(imp)
+
+            self.progress_tracker.stop_tracking(
+                tracking_id,
+                status="completed",
+                message=f"Merged ontology data successfully",
+            )
+            return target
+
+        except Exception as e:
+            self.progress_tracker.stop_tracking(
+                tracking_id, status="failed", message=str(e)
+            )
+            raise

semantica/parse/parse_provenance.py

@@ -0,0 +1,58 @@
+"""
+Provenance-enabled wrappers for parsing operations.
+
+Tracks: file parsed, format, structure, parsing method
+
+Usage:
+    from semantica.parse.parse_provenance import JSONParserWithProvenance
+    
+    parser = JSONParserWithProvenance(provenance=True)
+    data = parser.parse("data.json")
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class ParserWithProvenance:
+    """Base parser with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .parser import Parser
+        
+        self.provenance = provenance
+        self._parser = Parser(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def parse(self, file_path: str, **kwargs):
+        """Parse file with provenance tracking."""
+        data = self._parser.parse(file_path, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"parse_{uuid.uuid4().hex[:8]}",
+                source=file_path,
+                entity_type="parsed_data",
+                metadata={
+                    "file_path": file_path,
+                    "format": kwargs.get('format', 'unknown')
+                }
+            )
+        
+        return data
+    
+    def __getattr__(self, name):
+        return getattr(self._parser, name)
+
+
+__all__ = ['ParserWithProvenance']

semantica/parse/parse_usage.md

@@ -170,17 +170,17 @@ result = parser.parse("complex_invoice.pdf")
 
 # 2. Extract structured content
 # result contains the full Docling document object if available
-print(f"Extracted Text (Markdown): {result.markdown}")
+print(f"Extracted Text (Markdown): {result['full_text']}")
 
 # 3. Access extracted tables with high accuracy
-for i, table in enumerate(result.tables):
-    print(f"Table {i+1} headers: {table.headers}")
-    print(f"Table {i+1} row count: {len(table.rows)}")
+for i, table in enumerate(result['tables']):
+    print(f"Table {i+1} headers: {table.get('headers', [])}")
+    print(f"Table {i+1} row count: {len(table.get('rows', []))}")
 
 # 4. Extract metadata
-metadata = result.metadata
-print(f"Title: {metadata.title}")
-print(f"Page Count: {metadata.page_count}")
+metadata = result['metadata']
+print(f"Title: {metadata.get('title')}")
+print(f"Page Count: {metadata.get('page_count')}")
 ```
 
 #### Advanced Configuration
@@ -198,7 +198,7 @@ parser = DoclingParser(
 
 # Parse with specific export format
 result = parser.parse("scanned_document.pdf")
-print(f"HTML Content: {result.html}")
+print(f"HTML Content: {result['full_text']}")
 
 # Batch processing
 results = parser.parse_batch(["doc1.pdf", "doc2.docx"])
@@ -504,23 +504,22 @@ pdf_parser = PDFParser()
 pdf_data = pdf_parser.parse("document.pdf", extract_text=True, extract_tables=True)
 
 # Access pages
-for page_dict in pdf_data.get("pages", []):
-    page = PDFPage(**page_dict)
-    print(f"Page {page.page_number}: {len(page.text)} characters")
-    print(f"  Tables: {len(page.tables)}")
-    print(f"  Images: {len(page.images)}")
+for page in pdf_data.get("pages", []):
+    print(f"Page {page['page_number']}: {len(page['text'])} characters")
+    print(f"  Tables: {len(page['tables'])}")
+    print(f"  Images: {len(page['images'])}")
 
 # Access metadata
-metadata = PDFMetadata(**pdf_data.get("metadata", {}))
-print(f"Title: {metadata.title}")
-print(f"Author: {metadata.author}")
-print(f"Page Count: {metadata.page_count}")
+metadata = pdf_data.get("metadata", {})
+print(f"Title: {metadata.get('title')}")
+print(f"Author: {metadata.get('author')}")
+print(f"Page Count: {metadata.get('page_count')}")
 ```
 
 ### DOCX Parser
 
 ```python
-from semantica.parse import DOCXParser, DocxSection, DocxMetadata
+from semantica.parse import DOCXParser
 
 docx_parser = DOCXParser()
 
@@ -528,15 +527,14 @@ docx_parser = DOCXParser()
 docx_data = docx_parser.parse("document.docx", extract_tables=True)
 
 # Access sections
-for section_dict in docx_data.get("sections", []):
-    section = DocxSection(**section_dict)
-    print(f"Section: {section.heading} (Level {section.level})")
-    print(f"  Content: {section.content[:100]}...")
+for section in docx_data.get("sections", []):
+    print(f"Section: {section['heading']} (Level {section['level']})")
+    print(f"  Content: {section['content'][:100]}...")
 
 # Access metadata
-metadata = DocxMetadata(**docx_data.get("metadata", {}))
-print(f"Title: {metadata.title}")
-print(f"Author: {metadata.author}")
+metadata = docx_data.get("metadata", {})
+print(f"Title: {metadata.get('title')}")
+print(f"Author: {metadata.get('author')}")
 ```
 
 ### JSON Parser

semantica/pipeline/pipeline_builder.py

@@ -32,12 +32,14 @@ License: MIT
 
 from dataclasses import dataclass, field
 from enum import Enum
-from typing import Any, Callable, Dict, List, Optional, Union
+from typing import Any, Callable, Dict, List, Optional, Union, TYPE_CHECKING
 
 from ..utils.exceptions import ProcessingError, ValidationError
 from ..utils.logging import get_logger
 from ..utils.progress_tracker import get_progress_tracker
-from .pipeline_validator import PipelineValidator
+
+if TYPE_CHECKING:
+    from .pipeline_validator import PipelineValidator
 
 
 class StepStatus(Enum):
@@ -104,6 +106,8 @@ class PipelineBuilder:
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
+        from .pipeline_validator import PipelineValidator
+
         self.validator = PipelineValidator(**self.config)
         self.steps: List[PipelineStep] = []
         self.step_registry: Dict[str, Callable] = {}

semantica/pipeline/pipeline_provenance.py

@@ -0,0 +1,67 @@
+"""
+Provenance-enabled wrapper for pipeline execution.
+
+This module provides provenance tracking for end-to-end pipeline workflows,
+capturing all steps, inputs, outputs, and transformations.
+
+Usage:
+    from semantica.pipeline.pipeline_provenance import PipelineWithProvenance
+    
+    pipeline = PipelineWithProvenance(provenance=True)
+    result = pipeline.run(data)
+    # Tracks all pipeline steps with complete lineage
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, Any, Dict, List
+import uuid
+import time
+
+
+class PipelineWithProvenance:
+    """Pipeline executor with complete provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        """Initialize pipeline with optional provenance."""
+        from .pipeline import Pipeline
+        
+        self.provenance = provenance
+        self._pipeline = Pipeline(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def run(self, data: Any, source: Optional[str] = None, **kwargs):
+        """Run pipeline with provenance tracking."""
+        pipeline_id = f"pipeline_{uuid.uuid4().hex[:8]}"
+        start_time = time.time()
+        
+        result = self._pipeline.run(data, **kwargs)
+        elapsed = time.time() - start_time
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=pipeline_id,
+                source=source or "pipeline_execution",
+                entity_type="pipeline_run",
+                metadata={
+                    "steps": len(self._pipeline.steps) if hasattr(self._pipeline, 'steps') else 0,
+                    "duration_seconds": elapsed,
+                    "status": "completed"
+                }
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._pipeline, name)
+
+
+__all__ = ['PipelineWithProvenance']

semantica/provenance/__init__.py

@@ -0,0 +1,69 @@
+"""
+Provenance Tracking Module for Semantica
+
+Provides audit-grade provenance tracking for high-stakes domains requiring
+complete traceability (blue finance, healthcare, legal, pharma).
+
+This module consolidates and enhances provenance tracking from:
+- kg.ProvenanceTracker (entity/relationship tracking)
+- split.ProvenanceTracker (chunk tracking)
+- conflicts.SourceTracker (source tracking)
+
+Key Features:
+    - W3C PROV-O compliant tracking
+    - End-to-end lineage (doc → chunk → entity → KG → query → response)
+    - Bridge axiom translation chains (L1 → L2 → L3)
+    - Audit-grade source tracking (DOI + page + quote)
+    - Zero breaking changes (opt-in only)
+    - No new dependencies (stdlib only)
+
+Example Usage:
+    >>> # Enable provenance tracking
+    >>> from semantica.semantic_extract import NERExtractor
+    >>> ner = NERExtractor(provenance=True)
+    >>> entities = ner.extract("Steve Jobs founded Apple.")
+    >>> 
+    >>> # Trace lineage
+    >>> from semantica.provenance import ProvenanceManager
+    >>> prov_mgr = ProvenanceManager()
+    >>> lineage = prov_mgr.get_lineage(entities[0].id)
+    >>> 
+    >>> # Track with source details
+    >>> prov_mgr.track_entity(
+    ...     entity_id="entity_1",
+    ...     source="DOI:10.1371/journal.pone.0023601",
+    ...     source_location="Figure 2",
+    ...     source_quote="Total fish biomass increased by 463%...",
+    ...     confidence=0.92
+    ... )
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from .schemas import ProvenanceEntry, SourceReference
+from .storage import ProvenanceStorage, InMemoryStorage, SQLiteStorage
+from .manager import ProvenanceManager
+from .integrity import compute_checksum, verify_checksum
+
+__all__ = [
+    # Core schemas
+    "ProvenanceEntry",
+    "SourceReference",
+    
+    # Storage backends
+    "ProvenanceStorage",
+    "InMemoryStorage",
+    "SQLiteStorage",
+    
+    # Manager
+    "ProvenanceManager",
+    
+    # Utilities
+    "compute_checksum",
+    "verify_checksum",
+]
+
+__version__ = "1.0.0"
+__author__ = "Semantica Team"
+__description__ = "Audit-Grade Provenance Tracking for Semantica"

semantica/provenance/bridge_axiom.py

@@ -0,0 +1,435 @@
+"""
+Bridge Axiom Translation Chain Tracking
+
+This module provides bridge axiom tracking for multi-layer provenance chains,
+enabling translation from one domain to another across all high-stakes domains.
+
+Supported Domain Translations:
+    - Ecological → Financial (Blue Finance, Natural Capital)
+    - Clinical → Diagnostic (Healthcare, Medical Research)
+    - Evidence → Legal Conclusion (Legal, Forensic)
+    - Research Data → Drug Efficacy (Pharmaceutical, Clinical Trials)
+    - Raw Data → Financial Metrics (Finance, Risk Assessment)
+    - Sensor Data → Security Threat (Intelligence, Cybersecurity)
+    - Biological Data → Biomedical Insights (Biomedical Research)
+    - Asset Data → Portfolio Risk (Asset Management)
+
+Features:
+    - Bridge axiom definition and tracking
+    - Translation chain provenance
+    - Coefficient source tracking (DOI + page + quote)
+    - Multi-layer lineage tracing
+    - Confidence propagation
+
+Examples:
+    >>> # Blue Finance: Ecological → Financial
+    >>> ba_finance = BridgeAxiom(
+    ...     axiom_id="BA-FINANCE-001",
+    ...     name="biomass_tourism_elasticity",
+    ...     rule="1% biomass increase → 0.346% tourism revenue increase",
+    ...     coefficient=0.346,
+    ...     source_doi="10.1038/s41586-021-03371-z",
+    ...     input_domain="ecological",
+    ...     output_domain="financial"
+    ... )
+    >>> 
+    >>> # Healthcare: Clinical Observation → Diagnosis Probability
+    >>> ba_health = BridgeAxiom(
+    ...     axiom_id="BA-HEALTH-001",
+    ...     name="fever_influenza_correlation",
+    ...     rule="Fever >38°C increases influenza probability by 0.65",
+    ...     coefficient=0.65,
+    ...     source_doi="10.1001/jama.2020.12345",
+    ...     input_domain="clinical_observation",
+    ...     output_domain="diagnostic_probability"
+    ... )
+    >>> 
+    >>> # Legal: Evidence Strength → Conviction Probability
+    >>> ba_legal = BridgeAxiom(
+    ...     axiom_id="BA-LEGAL-001",
+    ...     name="dna_match_conviction",
+    ...     rule="DNA match increases conviction probability by 0.95",
+    ...     coefficient=0.95,
+    ...     source_doi="10.1016/j.forsciint.2019.12345",
+    ...     input_domain="forensic_evidence",
+    ...     output_domain="legal_conclusion"
+    ... )
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from dataclasses import dataclass, field
+from typing import Optional, Dict, Any, List
+from datetime import datetime
+import uuid
+
+
+@dataclass
+class BridgeAxiom:
+    """
+    Bridge axiom for domain translation with provenance.
+    
+    Represents a rule that translates data from one domain to another,
+    with complete source tracking for audit-grade provenance.
+    
+    Attributes:
+        axiom_id: Unique axiom identifier (e.g., "BA-001")
+        name: Human-readable axiom name
+        rule: Rule description in natural language
+        coefficient: Numeric coefficient for translation
+        source_doi: DOI of source paper/document
+        source_page: Page/table/figure in source
+        source_quote: Direct quote supporting the coefficient
+        confidence: Confidence score (0.0-1.0)
+        input_domain: Input domain (e.g., "ecological")
+        output_domain: Output domain (e.g., "financial")
+        metadata: Additional metadata
+    
+    Examples:
+        >>> # Blue Finance
+        >>> ba_finance = BridgeAxiom(
+        ...     axiom_id="BA-FINANCE-001",
+        ...     name="biomass_tourism_elasticity",
+        ...     rule="1% biomass increase → 0.346% tourism revenue increase",
+        ...     coefficient=0.346,
+        ...     source_doi="10.1038/s41586-021-03371-z",
+        ...     input_domain="ecological",
+        ...     output_domain="financial"
+        ... )
+        >>> 
+        >>> # Healthcare
+        >>> ba_health = BridgeAxiom(
+        ...     axiom_id="BA-HEALTH-001",
+        ...     name="symptom_diagnosis_correlation",
+        ...     rule="Symptom X increases diagnosis Y probability by 0.75",
+        ...     coefficient=0.75,
+        ...     source_doi="10.1001/jama.2020.12345",
+        ...     input_domain="clinical_symptom",
+        ...     output_domain="diagnosis"
+        ... )
+        >>> 
+        >>> # Pharmaceutical
+        >>> ba_pharma = BridgeAxiom(
+        ...     axiom_id="BA-PHARMA-001",
+        ...     name="dosage_efficacy_relationship",
+        ...     rule="10mg increase → 0.15 efficacy improvement",
+        ...     coefficient=0.15,
+        ...     source_doi="10.1056/NEJMoa2020123",
+        ...     input_domain="drug_dosage",
+        ...     output_domain="clinical_efficacy"
+        ... )
+    """
+    
+    axiom_id: str
+    name: str
+    rule: str
+    coefficient: float
+    source_doi: str
+    source_page: str
+    source_quote: Optional[str] = None
+    confidence: float = 1.0
+    input_domain: str = "unknown"
+    output_domain: str = "unknown"
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def apply(
+        self,
+        input_entity: str,
+        input_value: float,
+        prov_manager: Optional[Any] = None,
+        **kwargs
+    ) -> Dict[str, Any]:
+        """
+        Apply bridge axiom to input value with provenance tracking.
+        
+        Args:
+            input_entity: Input entity identifier
+            input_value: Input value to transform
+            prov_manager: ProvenanceManager instance (optional)
+            **kwargs: Additional parameters
+            
+        Returns:
+            Dictionary with result and provenance information
+            
+        Example:
+            >>> result = ba.apply(
+            ...     input_entity="cabo_pulmo_biomass",
+            ...     input_value=463,
+            ...     prov_manager=prov_mgr
+            ... )
+            >>> print(result["output_value"])
+            160.098
+        """
+        # Calculate output value
+        output_value = input_value * self.coefficient
+        
+        # Generate output entity ID
+        output_entity = f"{input_entity}_transformed_{self.axiom_id}"
+        
+        # Track provenance if manager provided
+        if prov_manager:
+            try:
+                # Track the bridge axiom application
+                prov_manager.track_entity(
+                    entity_id=output_entity,
+                    source=self.source_doi,
+                    entity_type="bridge_axiom_result",
+                    activity_id=f"bridge_axiom_application_{self.axiom_id}",
+                    source_location=self.source_page,
+                    source_quote=self.source_quote,
+                    confidence=self.confidence,
+                    metadata={
+                        "axiom_id": self.axiom_id,
+                        "axiom_name": self.name,
+                        "rule": self.rule,
+                        "coefficient": self.coefficient,
+                        "input_entity": input_entity,
+                        "input_value": input_value,
+                        "output_value": output_value,
+                        "input_domain": self.input_domain,
+                        "output_domain": self.output_domain,
+                        **kwargs
+                    }
+                )
+            except Exception:
+                pass  # Graceful failure
+        
+        return {
+            "axiom_id": self.axiom_id,
+            "axiom_name": self.name,
+            "input_entity": input_entity,
+            "input_value": input_value,
+            "output_entity": output_entity,
+            "output_value": output_value,
+            "coefficient": self.coefficient,
+            "confidence": self.confidence,
+            "source_doi": self.source_doi,
+            "source_page": self.source_page,
+            "input_domain": self.input_domain,
+            "output_domain": self.output_domain
+        }
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert bridge axiom to dictionary."""
+        return {
+            "axiom_id": self.axiom_id,
+            "name": self.name,
+            "rule": self.rule,
+            "coefficient": self.coefficient,
+            "source_doi": self.source_doi,
+            "source_page": self.source_page,
+            "source_quote": self.source_quote,
+            "confidence": self.confidence,
+            "input_domain": self.input_domain,
+            "output_domain": self.output_domain,
+            "metadata": self.metadata
+        }
+
+
+@dataclass
+class TranslationChain:
+    """
+    Multi-layer translation chain with complete provenance.
+    
+    Tracks a complete translation from source data through multiple
+    bridge axioms to final output (e.g., L1 → L2 → L3).
+    
+    Attributes:
+        chain_id: Unique chain identifier
+        layers: List of layer dictionaries
+        confidence: Overall confidence score
+        metadata: Additional metadata
+    
+    Example:
+        >>> chain = TranslationChain(
+        ...     chain_id="chain_001",
+        ...     layers=[
+        ...         {"layer": "L1", "type": "ecological", "value": 463},
+        ...         {"layer": "L2", "type": "bridge_axiom", "axiom": "BA-001"},
+        ...         {"layer": "L3", "type": "financial", "value": 29.27}
+        ...     ]
+        ... )
+    """
+    
+    chain_id: str
+    layers: List[Dict[str, Any]] = field(default_factory=list)
+    confidence: float = 1.0
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def add_layer(
+        self,
+        layer_name: str,
+        layer_type: str,
+        value: Any,
+        source: Optional[str] = None,
+        **kwargs
+    ) -> None:
+        """
+        Add a layer to the translation chain.
+        
+        Args:
+            layer_name: Layer name (e.g., "L1", "L2", "L3")
+            layer_type: Layer type (e.g., "ecological", "bridge_axiom", "financial")
+            value: Layer value
+            source: Source document/DOI
+            **kwargs: Additional layer metadata
+        """
+        layer = {
+            "layer": layer_name,
+            "type": layer_type,
+            "value": value,
+            "source": source,
+            "timestamp": datetime.utcnow().isoformat(),
+            **kwargs
+        }
+        self.layers.append(layer)
+    
+    def get_layer(self, layer_name: str) -> Optional[Dict[str, Any]]:
+        """
+        Get a specific layer by name.
+        
+        Args:
+            layer_name: Layer name to retrieve
+            
+        Returns:
+            Layer dictionary or None
+        """
+        for layer in self.layers:
+            if layer.get("layer") == layer_name:
+                return layer
+        return None
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert translation chain to dictionary."""
+        return {
+            "chain_id": self.chain_id,
+            "layers": self.layers,
+            "confidence": self.confidence,
+            "metadata": self.metadata
+        }
+
+
+def create_translation_chain(
+    input_data: Dict[str, Any],
+    bridge_axioms: List[BridgeAxiom],
+    prov_manager: Optional[Any] = None
+) -> TranslationChain:
+    """
+    Create a complete translation chain through multiple bridge axioms.
+    
+    Args:
+        input_data: Input data dictionary with 'entity_id' and 'value'
+        bridge_axioms: List of BridgeAxiom objects to apply in sequence
+        prov_manager: ProvenanceManager instance (optional)
+        
+    Returns:
+        TranslationChain object with complete provenance
+        
+    Example:
+        >>> input_data = {
+        ...     "entity_id": "cabo_pulmo_biomass",
+        ...     "value": 463,
+        ...     "source": "DOI:10.1371/journal.pone.0023601"
+        ... }
+        >>> axioms = [ba_001, ba_002]
+        >>> chain = create_translation_chain(input_data, axioms, prov_mgr)
+    """
+    chain_id = str(uuid.uuid4())
+    chain = TranslationChain(chain_id=chain_id)
+    
+    # Add L1 (input layer)
+    chain.add_layer(
+        layer_name="L1",
+        layer_type="input",
+        value=input_data.get("value"),
+        source=input_data.get("source"),
+        entity_id=input_data.get("entity_id")
+    )
+    
+    # Apply bridge axioms sequentially
+    current_value = input_data.get("value")
+    current_entity = input_data.get("entity_id")
+    
+    for i, axiom in enumerate(bridge_axioms):
+        # Apply axiom
+        result = axiom.apply(
+            input_entity=current_entity,
+            input_value=current_value,
+            prov_manager=prov_manager
+        )
+        
+        # Add bridge axiom layer
+        chain.add_layer(
+            layer_name=f"L{i+2}_axiom",
+            layer_type="bridge_axiom",
+            value=axiom.coefficient,
+            source=axiom.source_doi,
+            axiom_id=axiom.axiom_id,
+            axiom_name=axiom.name,
+            rule=axiom.rule
+        )
+        
+        # Update for next iteration
+        current_value = result["output_value"]
+        current_entity = result["output_entity"]
+        
+        # Update chain confidence (minimum of all confidences)
+        chain.confidence = min(chain.confidence, axiom.confidence)
+    
+    # Add final output layer
+    chain.add_layer(
+        layer_name=f"L{len(bridge_axioms)+2}",
+        layer_type="output",
+        value=current_value,
+        entity_id=current_entity
+    )
+    
+    return chain
+
+
+def trace_translation_chain(
+    chain: TranslationChain,
+    prov_manager: Any
+) -> Dict[str, Any]:
+    """
+    Trace complete provenance for a translation chain.
+    
+    Args:
+        chain: TranslationChain object
+        prov_manager: ProvenanceManager instance
+        
+    Returns:
+        Dictionary with complete provenance trace
+        
+    Example:
+        >>> trace = trace_translation_chain(chain, prov_mgr)
+        >>> print(trace["layers"])
+    """
+    trace = {
+        "chain_id": chain.chain_id,
+        "layers": [],
+        "confidence": chain.confidence,
+        "provenance": []
+    }
+    
+    for layer in chain.layers:
+        layer_trace = {
+            "layer": layer.get("layer"),
+            "type": layer.get("type"),
+            "value": layer.get("value"),
+            "source": layer.get("source")
+        }
+        
+        # Get provenance for entities in this layer
+        entity_id = layer.get("entity_id")
+        if entity_id:
+            try:
+                lineage = prov_manager.get_lineage(entity_id)
+                layer_trace["provenance"] = lineage
+            except Exception:
+                pass
+        
+        trace["layers"].append(layer_trace)
+    
+    return trace

semantica/provenance/integrity.py

@@ -0,0 +1,178 @@
+"""
+Integrity Verification Utilities
+
+This module provides utilities for data integrity verification using
+SHA-256 checksums, ensuring provenance data has not been tampered with.
+
+Features:
+    - SHA-256 checksum computation
+    - Checksum verification
+    - Data integrity validation
+    - Tamper detection
+
+Compliance:
+    - FDA 21 CFR Part 11 (electronic records)
+    - SOX (Sarbanes-Oxley)
+    - HIPAA (healthcare data integrity)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import hashlib
+from typing import Any, Dict, Optional
+from .schemas import ProvenanceEntry
+
+
+def compute_checksum(entry: ProvenanceEntry) -> str:
+    """
+    Compute SHA-256 checksum for a provenance entry.
+    
+    Creates a deterministic checksum based on critical provenance fields
+    to detect any tampering or corruption of provenance data.
+    
+    Args:
+        entry: ProvenanceEntry to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+        
+    Example:
+        >>> entry = ProvenanceEntry(
+        ...     entity_id="entity_123",
+        ...     entity_type="entity",
+        ...     activity_id="extraction",
+        ...     source_document="DOI:10.1371/..."
+        ... )
+        >>> checksum = compute_checksum(entry)
+        >>> print(checksum)
+        'a3b2c1d4e5f6...'
+    """
+    # Concatenate critical fields for checksum
+    data = (
+        f"{entry.entity_id}"
+        f"{entry.entity_type}"
+        f"{entry.activity_id}"
+        f"{entry.source_document}"
+        f"{entry.timestamp}"
+        f"{entry.confidence}"
+    )
+    
+    return hashlib.sha256(data.encode('utf-8')).hexdigest()
+
+
+def verify_checksum(entry: ProvenanceEntry, expected_checksum: Optional[str] = None) -> bool:
+    """
+    Verify checksum for a provenance entry.
+    
+    Computes the current checksum and compares it with the expected checksum
+    to detect tampering or corruption.
+    
+    Args:
+        entry: ProvenanceEntry to verify
+        expected_checksum: Expected checksum (uses entry.checksum if None)
+        
+    Returns:
+        True if checksum matches, False otherwise
+        
+    Example:
+        >>> entry = ProvenanceEntry(...)
+        >>> entry.checksum = compute_checksum(entry)
+        >>> is_valid = verify_checksum(entry)
+        >>> print(is_valid)
+        True
+    """
+    if expected_checksum is None:
+        expected_checksum = entry.checksum
+    
+    if expected_checksum is None:
+        return False
+    
+    current_checksum = compute_checksum(entry)
+    return current_checksum == expected_checksum
+
+
+def compute_data_checksum(data: str) -> str:
+    """
+    Compute SHA-256 checksum for arbitrary data.
+    
+    Args:
+        data: String data to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+        
+    Example:
+        >>> checksum = compute_data_checksum("some data")
+        >>> print(checksum)
+        'a3b2c1d4e5f6...'
+    """
+    return hashlib.sha256(data.encode('utf-8')).hexdigest()
+
+
+def verify_data_checksum(data: str, expected_checksum: str) -> bool:
+    """
+    Verify checksum for arbitrary data.
+    
+    Args:
+        data: String data to verify
+        expected_checksum: Expected checksum
+        
+    Returns:
+        True if checksum matches, False otherwise
+        
+    Example:
+        >>> data = "some data"
+        >>> checksum = compute_data_checksum(data)
+        >>> is_valid = verify_data_checksum(data, checksum)
+        >>> print(is_valid)
+        True
+    """
+    current_checksum = compute_data_checksum(data)
+    return current_checksum == expected_checksum
+
+
+def compute_dict_checksum(data: Dict[str, Any]) -> str:
+    """
+    Compute SHA-256 checksum for dictionary data.
+    
+    Sorts keys to ensure deterministic checksum computation.
+    
+    Args:
+        data: Dictionary data to compute checksum for
+        
+    Returns:
+        SHA-256 checksum as hexadecimal string
+        
+    Example:
+        >>> data = {"key1": "value1", "key2": "value2"}
+        >>> checksum = compute_dict_checksum(data)
+        >>> print(checksum)
+        'a3b2c1d4e5f6...'
+    """
+    # Sort keys for deterministic checksum
+    sorted_items = sorted(data.items())
+    data_str = str(sorted_items)
+    return hashlib.sha256(data_str.encode('utf-8')).hexdigest()
+
+
+def verify_dict_checksum(data: Dict[str, Any], expected_checksum: str) -> bool:
+    """
+    Verify checksum for dictionary data.
+    
+    Args:
+        data: Dictionary data to verify
+        expected_checksum: Expected checksum
+        
+    Returns:
+        True if checksum matches, False otherwise
+        
+    Example:
+        >>> data = {"key1": "value1", "key2": "value2"}
+        >>> checksum = compute_dict_checksum(data)
+        >>> is_valid = verify_dict_checksum(data, checksum)
+        >>> print(is_valid)
+        True
+    """
+    current_checksum = compute_dict_checksum(data)
+    return current_checksum == expected_checksum

semantica/provenance/manager.py

@@ -0,0 +1,574 @@
+"""
+Unified Provenance Manager
+
+This module provides the central ProvenanceManager class that consolidates
+provenance tracking from multiple Semantica modules:
+    - kg.ProvenanceTracker (entity/relationship tracking)
+    - split.ProvenanceTracker (chunk tracking)
+    - conflicts.SourceTracker (source tracking)
+
+The ProvenanceManager provides a unified API for all provenance operations
+while maintaining backward compatibility with existing tracker interfaces.
+
+Features:
+    - W3C PROV-O compliant tracking
+    - Entity and relationship tracking
+    - Chunk provenance tracking
+    - Source and property tracking
+    - Complete lineage tracing
+    - Multiple storage backends
+    - Integrity verification
+    - Batch operations
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Optional, List, Dict, Any
+from datetime import datetime
+
+from .schemas import ProvenanceEntry, SourceReference, PropertySource
+from .storage import ProvenanceStorage, InMemoryStorage, SQLiteStorage
+from .integrity import compute_checksum
+
+
+class ProvenanceManager:
+    """
+    Unified provenance tracking manager.
+    
+    Consolidates and enhances provenance tracking from:
+    - kg.ProvenanceTracker: Entity/relationship tracking with temporal info
+    - split.ProvenanceTracker: Chunk tracking with parent-child relationships
+    - conflicts.SourceTracker: Source tracking with credibility scores
+    
+    Example:
+        >>> # Basic usage
+        >>> prov_mgr = ProvenanceManager()
+        >>> prov_mgr.track_entity("entity_1", source="doc_1")
+        >>> 
+        >>> # With persistent storage
+        >>> prov_mgr = ProvenanceManager(storage_path="provenance.db")
+        >>> 
+        >>> # Trace lineage
+        >>> lineage = prov_mgr.get_lineage("entity_1")
+    """
+    
+    def __init__(
+        self,
+        storage: Optional[ProvenanceStorage] = None,
+        storage_path: Optional[str] = None
+    ):
+        """
+        Initialize provenance manager.
+        
+        Args:
+            storage: Custom storage backend (optional)
+            storage_path: Path to SQLite database (optional, uses in-memory if None)
+        """
+        if storage:
+            self.storage = storage
+        elif storage_path:
+            self.storage = SQLiteStorage(storage_path)
+        else:
+            self.storage = InMemoryStorage()
+    
+    # === Entity Tracking (from kg.ProvenanceTracker) ===
+    
+    def track_entity(
+        self,
+        entity_id: str,
+        source: str,
+        metadata: Optional[Dict[str, Any]] = None,
+        **kwargs
+    ) -> ProvenanceEntry:
+        """
+        Track entity provenance (kg.ProvenanceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            source: Source identifier (document ID, DOI, file path)
+            metadata: Optional metadata dictionary
+            **kwargs: Additional fields (confidence, source_location, etc.)
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> prov_mgr.track_entity(
+            ...     entity_id="entity_1",
+            ...     source="DOI:10.1371/journal.pone.0023601",
+            ...     metadata={"confidence": 0.92}
+            ... )
+        """
+        # Validate entity_id
+        if entity_id is None:
+            raise ValueError("entity_id cannot be None")
+        if not isinstance(entity_id, str):
+            raise TypeError(f"entity_id must be a string, got {type(entity_id).__name__}")
+        
+        if not isinstance(entity_id, str):
+            raise TypeError(f"entity_id must be a string, got {type(entity_id).__name__}")
+        
+        if not isinstance(entity_id, str):
+            raise TypeError(f"entity_id must be a string, got {type(entity_id).__name__}")
+        
+        # Check if entity already exists
+        existing = self.storage.retrieve(entity_id)
+        parent_id = kwargs.get("parent_entity_id")
+        
+        # If source is a known entity, link it as parent (unless parent already set)
+        if not parent_id and source and isinstance(source, str):
+            try:
+                # Check if source exists in storage
+                # trace_lineage is cheaper than retrieve for just checking existence? Or retrieve?
+                # retrieve returns the *latest* entry for that ID
+                source_entity = self.storage.retrieve(source)
+                if source_entity:
+                    parent_id = source
+            except Exception:
+                pass
+
+        # If entity exists, preserve history by archiving the old state
+        if existing:
+            # Create a history entry for the previous state
+            # Use timestamp or counter for uniqueness
+            import copy
+            history_entry = copy.deepcopy(existing)
+            history_id = f"{entity_id}:v:{existing.last_updated}"
+            
+            # Ensure unique ID if update happens same second
+            if self.storage.retrieve(history_id):
+                 history_id = f"{history_id}:{datetime.utcnow().microsecond}"
+            
+            history_entry.entity_id = history_id
+            
+            # Store the history entry
+            try:
+                self.storage.store(history_entry)
+                # Link new entry to this history entry
+                parent_id = history_id
+            except Exception:
+                pass # If history archiving fails, proceed with update but lose history (graceful degradation)
+
+        entry = ProvenanceEntry(
+            entity_id=entity_id,
+            entity_type=kwargs.get("entity_type", "entity"),
+            activity_id=kwargs.get("activity_id", "entity_tracking"),
+            source_document=source,
+            source_location=kwargs.get("source_location"),
+            source_quote=kwargs.get("source_quote"),
+            confidence=kwargs.get("confidence", 1.0),
+            metadata=metadata or {},
+            first_seen=existing.first_seen if existing else datetime.utcnow().isoformat(),
+            last_updated=datetime.utcnow().isoformat(),
+            parent_entity_id=parent_id  # Link to history or explicit parent
+        )
+        
+        # Compute checksum for integrity
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass  # Graceful failure - don't break main functionality
+        
+        return entry
+    
+    def track_relationship(
+        self,
+        relationship_id: str,
+        source: str,
+        metadata: Optional[Dict[str, Any]] = None,
+        **kwargs
+    ) -> ProvenanceEntry:
+        """
+        Track relationship provenance (kg.ProvenanceTracker compatible).
+        
+        Args:
+            relationship_id: Relationship identifier
+            source: Source identifier
+            metadata: Optional metadata dictionary
+            **kwargs: Additional fields
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> prov_mgr.track_relationship(
+            ...     relationship_id="rel_1",
+            ...     source="doc_1",
+            ...     metadata={"type": "founded"}
+            ... )
+        """
+        entry = ProvenanceEntry(
+            entity_id=relationship_id,
+            entity_type="relationship",
+            activity_id=kwargs.get("activity_id", "relationship_tracking"),
+            source_document=source,
+            source_location=kwargs.get("source_location"),
+            confidence=kwargs.get("confidence", 1.0),
+            metadata=metadata or {},
+            first_seen=datetime.utcnow().isoformat(),
+            last_updated=datetime.utcnow().isoformat()
+        )
+        
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass
+        
+        return entry
+    
+    # === Chunk Tracking (from split.ProvenanceTracker) ===
+    
+    def track_chunk(
+        self,
+        chunk_id: str,
+        source_document: str,
+        source_path: Optional[str] = None,
+        start_index: int = 0,
+        end_index: int = 0,
+        parent_chunk_id: Optional[str] = None,
+        **metadata
+    ) -> ProvenanceEntry:
+        """
+        Track chunk provenance (split.ProvenanceTracker compatible).
+        
+        Args:
+            chunk_id: Chunk identifier
+            source_document: Source document identifier
+            source_path: Path to source document
+            start_index: Start character index
+            end_index: End character index
+            parent_chunk_id: Parent chunk ID (if chunk was split)
+            **metadata: Additional metadata
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> prov_mgr.track_chunk(
+            ...     chunk_id="chunk_1",
+            ...     source_document="doc_1",
+            ...     source_path="/path/to/doc.pdf",
+            ...     start_index=0,
+            ...     end_index=500
+            ... )
+        """
+        entry = ProvenanceEntry(
+            entity_id=chunk_id,
+            entity_type="chunk",
+            activity_id="chunking",
+            source_document=source_document,
+            source_location=source_path,
+            start_index=start_index,
+            end_index=end_index,
+            parent_entity_id=parent_chunk_id,
+            metadata=metadata,
+            timestamp=datetime.utcnow().isoformat()
+        )
+        
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass
+        
+        return entry
+    
+    # === Source Tracking (from conflicts.SourceTracker) ===
+    
+    def track_property_source(
+        self,
+        entity_id: str,
+        property_name: str,
+        value: Any,
+        source: SourceReference,
+        **metadata
+    ) -> ProvenanceEntry:
+        """
+        Track property source (conflicts.SourceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            property_name: Property name
+            value: Property value
+            source: SourceReference object
+            **metadata: Additional metadata
+            
+        Returns:
+            ProvenanceEntry object
+            
+        Example:
+            >>> source = SourceReference(
+            ...     document="DOI:10.1038/...",
+            ...     page=4,
+            ...     confidence=0.92
+            ... )
+            >>> prov_mgr.track_property_source(
+            ...     entity_id="entity_1",
+            ...     property_name="biomass_increase",
+            ...     value="463%",
+            ...     source=source
+            ... )
+        """
+        entry = ProvenanceEntry(
+            entity_id=f"{entity_id}_{property_name}",
+            entity_type="property",
+            activity_id="property_tracking",
+            source_document=source.document,
+            source_location=f"page_{source.page}" if source.page else source.section,
+            confidence=source.confidence,
+            credibility=source.metadata.get("credibility"),
+            metadata={
+                "entity_id": entity_id,
+                "property_name": property_name,
+                "value": value,
+                **metadata,
+                **source.metadata
+            },
+            timestamp=datetime.utcnow().isoformat()
+        )
+        
+        entry.checksum = compute_checksum(entry)
+        
+        try:
+            self.storage.store(entry)
+        except Exception:
+            pass
+        
+        return entry
+    
+    # === Batch Operations ===
+    
+    def track_entities_batch(
+        self,
+        entities: List[Dict[str, Any]],
+        source: str,
+        **metadata
+    ) -> int:
+        """
+        Track multiple entities in batch.
+        
+        Args:
+            entities: List of entity dictionaries with 'id' key
+            source: Source identifier
+            **metadata: Metadata to apply to all entities
+            
+        Returns:
+            Number of entities tracked
+            
+        Example:
+            >>> entities = [
+            ...     {"id": "entity_1", "confidence": 0.9},
+            ...     {"id": "entity_2", "confidence": 0.85}
+            ... ]
+            >>> count = prov_mgr.track_entities_batch(entities, "doc_1")
+        """
+        tracked_count = 0
+        
+        for entity in entities:
+            entity_id = entity.get("id") or entity.get("entity_id")
+            if not entity_id:
+                continue
+            
+            entity_metadata = {**metadata, **entity.get("metadata", {})}
+            
+            try:
+                self.track_entity(entity_id, source, entity_metadata)
+                tracked_count += 1
+            except Exception:
+                pass  # Continue with other entities
+        
+        return tracked_count
+    
+    def track_chunks_batch(
+        self,
+        chunks: List[Dict[str, Any]],
+        source_document: str,
+        source_path: Optional[str] = None,
+        **metadata
+    ) -> int:
+        """
+        Track multiple chunks in batch.
+        
+        Args:
+            chunks: List of chunk dictionaries
+            source_document: Source document identifier
+            source_path: Path to source document
+            **metadata: Metadata to apply to all chunks
+            
+        Returns:
+            Number of chunks tracked
+        """
+        tracked_count = 0
+        
+        for chunk in chunks:
+            chunk_id = chunk.get("id") or chunk.get("chunk_id")
+            if not chunk_id:
+                continue
+            
+            try:
+                self.track_chunk(
+                    chunk_id=chunk_id,
+                    source_document=source_document,
+                    source_path=source_path,
+                    start_index=chunk.get("start_index", 0),
+                    end_index=chunk.get("end_index", 0),
+                    parent_chunk_id=chunk.get("parent_chunk_id"),
+                    **{**metadata, **chunk.get("metadata", {})}
+                )
+                tracked_count += 1
+            except Exception:
+                pass
+        
+        return tracked_count
+    
+    # === Lineage Retrieval ===
+    
+    def get_lineage(self, entity_id: str) -> Dict[str, Any]:
+        """
+        Get complete lineage for an entity.
+        
+        Compatible with all existing tracker interfaces.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            Dictionary containing lineage information including metadata
+            
+        Example:
+            >>> lineage = prov_mgr.get_lineage("entity_1")
+            >>> print(lineage["source_documents"])
+            ['DOI:10.1371/...', 'doc_2']
+            >>> print(lineage["metadata"])
+            {'text': 'Apple Inc.', 'label': 'ORG'}
+        """
+        lineage_entries = self.storage.trace_lineage(entity_id)
+        
+        if not lineage_entries:
+            return {}
+        
+        # Aggregate metadata from all lineage entries
+        # Most recent entry's metadata takes precedence
+        aggregated_metadata = {}
+        for entry in lineage_entries:
+            if entry.metadata:
+                meta = entry.metadata
+                if isinstance(meta, str):
+                    try:
+                        import json
+                        meta = json.loads(meta)
+                    except (json.JSONDecodeError, TypeError):
+                        pass
+                
+                if isinstance(meta, dict):
+                    aggregated_metadata.update(meta)
+        
+        return {
+            "entity_id": entity_id,
+            "lineage_chain": [entry.to_dict() for entry in lineage_entries],
+            "source_documents": list(set(
+                e.source_document for e in lineage_entries 
+                if e.source_document
+            )),
+            "first_seen": min(
+                (e.first_seen for e in lineage_entries if e.first_seen),
+                default=None
+            ),
+            "last_updated": max(
+                (e.last_updated for e in lineage_entries if e.last_updated),
+                default=None
+            ),
+            "entity_count": len(lineage_entries),
+            "metadata": aggregated_metadata  # Add metadata key
+        }
+    
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage and return raw entries.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        return self.storage.trace_lineage(entity_id)
+    
+    def get_all_sources(self, entity_id: str) -> List[Dict[str, Any]]:
+        """
+        Get all sources for an entity (kg.ProvenanceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of source dictionaries
+        """
+        lineage_entries = self.storage.trace_lineage(entity_id)
+        
+        sources = []
+        for entry in lineage_entries:
+            if entry.source_document:
+                sources.append({
+                    "source": entry.source_document,
+                    "location": entry.source_location,
+                    "timestamp": entry.timestamp,
+                    "confidence": entry.confidence,
+                    "metadata": entry.metadata
+                })
+        
+        return sources
+    
+    def get_provenance(self, entity_id: str) -> Optional[Dict[str, Any]]:
+        """
+        Get provenance for entity (kg.ProvenanceTracker compatible).
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            Provenance dictionary or None
+        """
+        entry = self.storage.retrieve(entity_id)
+        if entry:
+            return entry.to_dict()
+        return None
+    
+    # === Utility Methods ===
+    
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        return self.storage.clear()
+    
+    def get_statistics(self) -> Dict[str, Any]:
+        """
+        Get provenance statistics.
+        
+        Returns:
+            Dictionary with statistics
+        """
+        all_entries = self.storage.retrieve_all()
+        
+        entity_types = {}
+        for entry in all_entries:
+            entity_types[entry.entity_type] = entity_types.get(entry.entity_type, 0) + 1
+        
+        return {
+            "total_entries": len(all_entries),
+            "entity_types": entity_types,
+            "unique_sources": len(set(
+                e.source_document for e in all_entries 
+                if e.source_document
+            ))
+        }

semantica/provenance/schemas.py

@@ -0,0 +1,277 @@
+"""
+W3C PROV-O Compliant Provenance Schemas
+
+This module provides dataclasses for provenance tracking that comply with
+W3C PROV-O (Provenance Ontology) standards while consolidating functionality
+from existing Semantica provenance trackers.
+
+Consolidates:
+    - kg.ProvenanceTracker: Entity/relationship tracking with temporal info
+    - split.ProvenanceInfo: Chunk tracking with parent-child relationships
+    - conflicts.SourceReference: Source tracking with credibility scores
+
+W3C PROV-O Mapping:
+    - ProvenanceEntry.entity_id → prov:Entity
+    - ProvenanceEntry.activity_id → prov:Activity
+    - ProvenanceEntry.agent_id → prov:Agent
+    - ProvenanceEntry.parent_entity_id → prov:wasDerivedFrom
+    - ProvenanceEntry.used_entities → prov:used
+    - ProvenanceEntry.timestamp → prov:generatedAtTime
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from dataclasses import dataclass, field
+from typing import Optional, List, Dict, Any
+from datetime import datetime
+
+
+@dataclass
+class ProvenanceEntry:
+    """
+    W3C PROV-O compliant provenance entry.
+    
+    This unified schema consolidates provenance tracking from:
+    - kg.ProvenanceTracker (entity/relationship tracking)
+    - split.ProvenanceInfo (chunk tracking)
+    - conflicts.SourceReference (source tracking)
+    
+    Attributes:
+        entity_id: Unique identifier for the entity (prov:Entity)
+        entity_type: Type of entity (entity, chunk, relationship, property, etc.)
+        activity_id: Activity that generated this entity (prov:Activity)
+        agent_id: Agent responsible for the activity (prov:Agent)
+        source_document: Source document identifier (DOI, file path, URL)
+        source_location: Location within source (page, figure, char range)
+        source_quote: Direct quote from source (for audit trail)
+        timestamp: When this provenance entry was created (prov:generatedAtTime)
+        first_seen: When entity was first tracked (from kg.ProvenanceTracker)
+        last_updated: When entity was last updated (from kg.ProvenanceTracker)
+        confidence: Confidence score for this provenance entry (0.0-1.0)
+        checksum: SHA-256 checksum for integrity verification
+        parent_entity_id: Parent entity ID (prov:wasDerivedFrom)
+        used_entities: List of entities used to create this entity (prov:used)
+        start_index: Start character index (from split.ProvenanceInfo)
+        end_index: End character index (from split.ProvenanceInfo)
+        credibility: Source credibility score (from conflicts.SourceTracker)
+        metadata: Additional metadata dictionary
+        version: Provenance schema version
+    
+    Example:
+        >>> entry = ProvenanceEntry(
+        ...     entity_id="entity_123",
+        ...     entity_type="named_entity",
+        ...     activity_id="ner_extraction",
+        ...     source_document="DOI:10.1371/journal.pone.0023601",
+        ...     source_location="Figure 2",
+        ...     source_quote="Total fish biomass increased by 463%",
+        ...     confidence=0.92
+        ... )
+    """
+    
+    # W3C PROV-O core entities
+    entity_id: str
+    entity_type: str
+    activity_id: str
+    agent_id: str = "semantica"
+    
+    # Audit-grade source tracking
+    source_document: str = ""
+    source_location: Optional[str] = None
+    source_quote: Optional[str] = None
+    
+    # Temporal tracking (from kg.ProvenanceTracker)
+    timestamp: str = field(default_factory=lambda: datetime.utcnow().isoformat())
+    first_seen: Optional[str] = None
+    last_updated: Optional[str] = None
+    
+    # Quality metrics
+    confidence: float = 1.0
+    checksum: Optional[str] = None
+    
+    # Chain of custody (W3C PROV-O)
+    parent_entity_id: Optional[str] = None
+    used_entities: List[str] = field(default_factory=list)
+    
+    # Chunk-specific fields (from split.ProvenanceInfo)
+    start_index: Optional[int] = None
+    end_index: Optional[int] = None
+    
+    # Source credibility (from conflicts.SourceTracker)
+    credibility: Optional[float] = None
+    
+    # Metadata
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    version: str = "1.0"
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Convert provenance entry to dictionary.
+        
+        Returns:
+            Dictionary representation of provenance entry
+        """
+        return {
+            "entity_id": self.entity_id,
+            "entity_type": self.entity_type,
+            "activity_id": self.activity_id,
+            "agent_id": self.agent_id,
+            "source_document": self.source_document,
+            "source_location": self.source_location,
+            "source_quote": self.source_quote,
+            "timestamp": self.timestamp,
+            "first_seen": self.first_seen,
+            "last_updated": self.last_updated,
+            "confidence": self.confidence,
+            "checksum": self.checksum,
+            "parent_entity_id": self.parent_entity_id,
+            "used_entities": self.used_entities,
+            "start_index": self.start_index,
+            "end_index": self.end_index,
+            "credibility": self.credibility,
+            "metadata": self.metadata,
+            "version": self.version,
+        }
+    
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "ProvenanceEntry":
+        """
+        Create provenance entry from dictionary.
+        
+        Args:
+            data: Dictionary containing provenance data
+            
+        Returns:
+            ProvenanceEntry instance
+        """
+        return cls(**data)
+
+
+@dataclass
+class SourceReference:
+    """
+    Source reference for provenance tracking.
+    
+    Compatible with conflicts.SourceReference for backward compatibility.
+    
+    Attributes:
+        document: Document identifier (DOI, file path, URL)
+        page: Page number within document
+        section: Section identifier within document
+        line: Line number within document
+        timestamp: When this source was accessed/created
+        confidence: Confidence score for this source (0.0-1.0)
+        metadata: Additional metadata dictionary
+    
+    Example:
+        >>> source = SourceReference(
+        ...     document="DOI:10.1038/s41586-021-03371-z",
+        ...     page=4,
+        ...     section="Table S4",
+        ...     confidence=0.92
+        ... )
+    """
+    
+    document: str
+    page: Optional[int] = None
+    section: Optional[str] = None
+    line: Optional[int] = None
+    timestamp: Optional[datetime] = None
+    confidence: float = 1.0
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Convert source reference to dictionary.
+        
+        Returns:
+            Dictionary representation of source reference
+        """
+        return {
+            "document": self.document,
+            "page": self.page,
+            "section": self.section,
+            "line": self.line,
+            "timestamp": self.timestamp.isoformat() if self.timestamp else None,
+            "confidence": self.confidence,
+            "metadata": self.metadata,
+        }
+    
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "SourceReference":
+        """
+        Create source reference from dictionary.
+        
+        Args:
+            data: Dictionary containing source reference data
+            
+        Returns:
+            SourceReference instance
+        """
+        if "timestamp" in data and isinstance(data["timestamp"], str):
+            data["timestamp"] = datetime.fromisoformat(data["timestamp"])
+        return cls(**data)
+
+
+@dataclass
+class PropertySource:
+    """
+    Property source information for conflict tracking.
+    
+    Compatible with conflicts.PropertySource for backward compatibility.
+    
+    Attributes:
+        property_name: Name of the property
+        value: Value of the property
+        sources: List of source references
+        entity_id: Entity this property belongs to
+        metadata: Additional metadata dictionary
+    
+    Example:
+        >>> prop_source = PropertySource(
+        ...     property_name="biomass_increase",
+        ...     value="463%",
+        ...     sources=[source_ref],
+        ...     entity_id="cabo_pulmo_mpa"
+        ... )
+    """
+    
+    property_name: str
+    value: Any
+    sources: List[SourceReference] = field(default_factory=list)
+    entity_id: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Convert property source to dictionary.
+        
+        Returns:
+            Dictionary representation of property source
+        """
+        return {
+            "property_name": self.property_name,
+            "value": self.value,
+            "sources": [s.to_dict() for s in self.sources],
+            "entity_id": self.entity_id,
+            "metadata": self.metadata,
+        }
+    
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "PropertySource":
+        """
+        Create property source from dictionary.
+        
+        Args:
+            data: Dictionary containing property source data
+            
+        Returns:
+            PropertySource instance
+        """
+        if "sources" in data:
+            data["sources"] = [
+                SourceReference.from_dict(s) if isinstance(s, dict) else s
+                for s in data["sources"]
+            ]
+        return cls(**data)

semantica/provenance/storage.py

@@ -0,0 +1,489 @@
+"""
+Provenance Storage Backends
+
+This module provides storage backends for provenance tracking, including
+in-memory and persistent SQLite storage with W3C PROV-O compliance.
+
+Storage Backends:
+    - InMemoryStorage: Fast in-memory storage for development/testing
+    - SQLiteStorage: Persistent SQLite storage for production use
+
+Features:
+    - W3C PROV-O compliant schema
+    - Lineage tracing with BFS traversal
+    - Efficient entity retrieval
+    - Type-based filtering
+    - Integrity verification support
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from abc import ABC, abstractmethod
+from typing import List, Optional, Dict, Any
+import sqlite3
+import json
+from collections import deque
+
+from .schemas import ProvenanceEntry
+
+
+class ProvenanceStorage(ABC):
+    """
+    Abstract storage interface for provenance tracking.
+    
+    All storage backends must implement these methods to ensure
+    consistent provenance tracking across different storage types.
+    """
+    
+    @abstractmethod
+    def store(self, entry: ProvenanceEntry) -> None:
+        """
+        Store a provenance entry.
+        
+        Args:
+            entry: ProvenanceEntry to store
+        """
+        pass
+    
+    @abstractmethod
+    def retrieve(self, entity_id: str) -> Optional[ProvenanceEntry]:
+        """
+        Retrieve a provenance entry by entity ID.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            ProvenanceEntry if found, None otherwise
+        """
+        pass
+    
+    @abstractmethod
+    def retrieve_all(self, entity_type: Optional[str] = None) -> List[ProvenanceEntry]:
+        """
+        Retrieve all provenance entries, optionally filtered by type.
+        
+        Args:
+            entity_type: Optional entity type filter
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        pass
+    
+    @abstractmethod
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage for an entity.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects in lineage chain
+        """
+        pass
+    
+    @abstractmethod
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        pass
+
+
+class InMemoryStorage(ProvenanceStorage):
+    """
+    Fast in-memory storage for provenance tracking.
+    
+    Suitable for:
+        - Development and testing
+        - Short-lived processes
+        - Small to medium datasets
+        - When persistence is not required
+    
+    Features:
+        - O(1) entity retrieval
+        - BFS lineage tracing
+        - Type-based filtering
+        - No external dependencies
+    
+    Example:
+        >>> storage = InMemoryStorage()
+        >>> storage.store(entry)
+        >>> lineage = storage.trace_lineage("entity_123")
+    """
+    
+    def __init__(self):
+        """Initialize in-memory storage."""
+        self._entries: Dict[str, ProvenanceEntry] = {}
+    
+    def store(self, entry: ProvenanceEntry) -> None:
+        """
+        Store a provenance entry in memory.
+        
+        Args:
+            entry: ProvenanceEntry to store
+        """
+        self._entries[entry.entity_id] = entry
+    
+    def retrieve(self, entity_id: str) -> Optional[ProvenanceEntry]:
+        """
+        Retrieve a provenance entry by entity ID.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            ProvenanceEntry if found, None otherwise
+        """
+        return self._entries.get(entity_id)
+    
+    def retrieve_all(self, entity_type: Optional[str] = None) -> List[ProvenanceEntry]:
+        """
+        Retrieve all provenance entries, optionally filtered by type.
+        
+        Args:
+            entity_type: Optional entity type filter
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        if entity_type:
+            return [
+                entry for entry in self._entries.values()
+                if entry.entity_type == entity_type
+            ]
+        return list(self._entries.values())
+    
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage using BFS traversal.
+        
+        Traces both parent entities (wasDerivedFrom) and used entities
+        to build complete provenance chain.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects in lineage chain
+        """
+        lineage = []
+        visited = set()
+        queue = deque([entity_id])
+        
+        while queue:
+            current_id = queue.popleft()
+            
+            if current_id in visited:
+                continue
+            
+            visited.add(current_id)
+            entry = self.retrieve(current_id)
+            
+            if entry:
+                lineage.append(entry)
+                
+                # Add parent entity to queue
+                if entry.parent_entity_id:
+                    queue.append(entry.parent_entity_id)
+                
+                # Add used entities to queue
+                for used_id in entry.used_entities:
+                    if used_id not in visited:
+                        queue.append(used_id)
+        
+        return lineage
+    
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        count = len(self._entries)
+        self._entries.clear()
+        return count
+
+
+class SQLiteStorage(ProvenanceStorage):
+    """
+    Persistent SQLite storage for provenance tracking.
+    
+    Suitable for:
+        - Production use
+        - Long-term provenance tracking
+        - Large datasets
+        - Audit trail requirements
+        - Regulatory compliance
+    
+    Features:
+        - W3C PROV-O compliant schema
+        - Persistent storage
+        - Efficient indexing
+        - Transaction support
+        - Integrity verification
+    
+    Example:
+        >>> storage = SQLiteStorage("provenance.db")
+        >>> storage.store(entry)
+        >>> lineage = storage.trace_lineage("entity_123")
+    """
+    
+    def __init__(self, db_path: str = "provenance.db"):
+        """
+        Initialize SQLite storage.
+        
+        Args:
+            db_path: Path to SQLite database file
+        """
+        self.db_path = db_path
+        self._init_db()
+    
+    def _init_db(self) -> None:
+        """Create tables with W3C PROV-O compliant schema."""
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        cursor.execute("""
+            CREATE TABLE IF NOT EXISTS provenance (
+                entity_id TEXT PRIMARY KEY,
+                entity_type TEXT NOT NULL,
+                activity_id TEXT NOT NULL,
+                agent_id TEXT DEFAULT 'semantica',
+                source_document TEXT,
+                source_location TEXT,
+                source_quote TEXT,
+                timestamp TEXT NOT NULL,
+                first_seen TEXT,
+                last_updated TEXT,
+                confidence REAL DEFAULT 1.0,
+                checksum TEXT,
+                parent_entity_id TEXT,
+                used_entities TEXT,
+                start_index INTEGER,
+                end_index INTEGER,
+                credibility REAL,
+                metadata TEXT,
+                version TEXT DEFAULT '1.0'
+            )
+        """)
+        
+        # Create indexes for efficient querying
+        cursor.execute("""
+            CREATE INDEX IF NOT EXISTS idx_entity_type 
+            ON provenance(entity_type)
+        """)
+        
+        cursor.execute("""
+            CREATE INDEX IF NOT EXISTS idx_source_document 
+            ON provenance(source_document)
+        """)
+        
+        cursor.execute("""
+            CREATE INDEX IF NOT EXISTS idx_parent_entity 
+            ON provenance(parent_entity_id)
+        """)
+        
+        conn.commit()
+        conn.close()
+    
+    def store(self, entry: ProvenanceEntry) -> None:
+        """
+        Store a provenance entry in SQLite database.
+        
+        Args:
+            entry: ProvenanceEntry to store
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            cursor.execute("""
+                INSERT OR REPLACE INTO provenance VALUES (
+                    ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
+                )
+            """, (
+                entry.entity_id,
+                entry.entity_type,
+                entry.activity_id,
+                entry.agent_id,
+                entry.source_document,
+                entry.source_location,
+                entry.source_quote,
+                entry.timestamp,
+                entry.first_seen,
+                entry.last_updated,
+                entry.confidence,
+                entry.checksum,
+                entry.parent_entity_id,
+                json.dumps(entry.used_entities),
+                entry.start_index,
+                entry.end_index,
+                entry.credibility,
+                json.dumps(entry.metadata),
+                entry.version
+            ))
+            
+            conn.commit()
+        finally:
+            conn.close()
+    
+    def retrieve(self, entity_id: str) -> Optional[ProvenanceEntry]:
+        """
+        Retrieve a provenance entry by entity ID.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            ProvenanceEntry if found, None otherwise
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            cursor.execute("""
+                SELECT * FROM provenance WHERE entity_id = ?
+            """, (entity_id,))
+            
+            row = cursor.fetchone()
+            if not row:
+                return None
+            
+            return self._row_to_entry(row)
+        finally:
+            conn.close()
+    
+    def retrieve_all(self, entity_type: Optional[str] = None) -> List[ProvenanceEntry]:
+        """
+        Retrieve all provenance entries, optionally filtered by type.
+        
+        Args:
+            entity_type: Optional entity type filter
+            
+        Returns:
+            List of ProvenanceEntry objects
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            if entity_type:
+                cursor.execute("""
+                    SELECT * FROM provenance WHERE entity_type = ?
+                """, (entity_type,))
+            else:
+                cursor.execute("SELECT * FROM provenance")
+            
+            rows = cursor.fetchall()
+            return [self._row_to_entry(row) for row in rows]
+        finally:
+            conn.close()
+    
+    def trace_lineage(self, entity_id: str) -> List[ProvenanceEntry]:
+        """
+        Trace complete lineage using BFS traversal.
+        
+        Args:
+            entity_id: Entity identifier
+            
+        Returns:
+            List of ProvenanceEntry objects in lineage chain
+        """
+        lineage = []
+        visited = set()
+        queue = deque([entity_id])
+        
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            while queue:
+                current_id = queue.popleft()
+                
+                if current_id in visited:
+                    continue
+                
+                visited.add(current_id)
+                
+                cursor.execute("""
+                    SELECT * FROM provenance WHERE entity_id = ?
+                """, (current_id,))
+                
+                row = cursor.fetchone()
+                if row:
+                    entry = self._row_to_entry(row)
+                    lineage.append(entry)
+                    
+                    # Add parent entity to queue
+                    if entry.parent_entity_id:
+                        queue.append(entry.parent_entity_id)
+                    
+                    # Add used entities to queue
+                    for used_id in entry.used_entities:
+                        if used_id not in visited:
+                            queue.append(used_id)
+            
+            return lineage
+        finally:
+            conn.close()
+    
+    def clear(self) -> int:
+        """
+        Clear all provenance data.
+        
+        Returns:
+            Number of entries cleared
+        """
+        conn = sqlite3.connect(self.db_path)
+        cursor = conn.cursor()
+        
+        try:
+            cursor.execute("SELECT COUNT(*) FROM provenance")
+            count = cursor.fetchone()[0]
+            
+            cursor.execute("DELETE FROM provenance")
+            conn.commit()
+            
+            return count
+        finally:
+            conn.close()
+    
+    def _row_to_entry(self, row: tuple) -> ProvenanceEntry:
+        """
+        Convert database row to ProvenanceEntry.
+        
+        Args:
+            row: Database row tuple
+            
+        Returns:
+            ProvenanceEntry object
+        """
+        return ProvenanceEntry(
+            entity_id=row[0],
+            entity_type=row[1],
+            activity_id=row[2],
+            agent_id=row[3],
+            source_document=row[4] or "",
+            source_location=row[5],
+            source_quote=row[6],
+            timestamp=row[7],
+            first_seen=row[8],
+            last_updated=row[9],
+            confidence=row[10],
+            checksum=row[11],
+            parent_entity_id=row[12],
+            used_entities=json.loads(row[13]) if row[13] else [],
+            start_index=row[14],
+            end_index=row[15],
+            credibility=row[16],
+            metadata=json.loads(row[17]) if row[17] else {},
+            version=row[18]
+        )

semantica/reasoning/reasoning_provenance.py

@@ -0,0 +1,58 @@
+"""
+Provenance-enabled wrappers for reasoning operations.
+
+Tracks: premises, conclusions, inference rules, confidence scores
+
+Usage:
+    from semantica.reasoning.reasoning_provenance import ReasoningEngineWithProvenance
+    
+    reasoner = ReasoningEngineWithProvenance(provenance=True)
+    result = reasoner.infer(premises)
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+from typing import Any
+import uuid
+
+
+class ReasoningEngineWithProvenance:
+    """Reasoning engine with provenance tracking."""
+    
+    def __init__(self, provenance: bool = False, **config):
+        from .reasoning_engine import ReasoningEngine
+        
+        self.provenance = provenance
+        self._engine = ReasoningEngine(**config)
+        self._prov_manager = None
+        
+        if provenance:
+            try:
+                from semantica.provenance import ProvenanceManager
+                self._prov_manager = ProvenanceManager()
+            except ImportError:
+                self.provenance = False
+    
+    def infer(self, premises: Any, source: str = None, **kwargs):
+        """Perform inference with provenance tracking."""
+        result = self._engine.infer(premises, **kwargs)
+        
+        if self.provenance and self._prov_manager:
+            self._prov_manager.track_entity(
+                entity_id=f"inference_{uuid.uuid4().hex[:8]}",
+                source=source or "reasoning_engine",
+                entity_type="inference",
+                metadata={
+                    "premises_count": len(premises) if hasattr(premises, '__len__') else 1,
+                    "confidence": getattr(result, 'confidence', None)
+                }
+            )
+        
+        return result
+    
+    def __getattr__(self, name):
+        return getattr(self._engine, name)
+
+
+__all__ = ['ReasoningEngineWithProvenance']

semantica/semantic_extract/__init__.py

@@ -15,6 +15,8 @@ Key Features:
     - Semantic network construction
     - LLM-based extraction enhancement
     - Extraction validation and quality assessment
+    - Batch processing with provenance tracking (batch_index, document_id)
+    - Robust fallback mechanisms (ML -> Pattern -> Last Resort)
 
 Main Classes:
     - NamedEntityRecognizer: Main NER coordinator (confidence_threshold, merge_overlapping)

semantica/semantic_extract/cache.py

@@ -0,0 +1,184 @@
+"""
+Result Caching Module
+
+This module provides caching mechanisms for extraction results to avoid redundant
+computations and API calls. It implements an LRU (Least Recently Used) cache
+with Time-To-Live (TTL) support.
+
+Key Features:
+    - LRU Caching: Evicts least recently used items when cache is full
+    - TTL Support: Expires items after a configurable duration
+    - Namespaced Caching: Separate caches for entities, relations, and triplets
+    - Hash-based Keys: Uses stable hashing for text and parameters
+
+Classes:
+    - ExtractionCache: Main cache manager
+    - CacheItem: Container for cached data with metadata
+
+Author: Semantica Contributors
+License: MIT
+"""
+
+import time
+import hashlib
+import json
+from collections import OrderedDict
+from typing import Any, Dict, Optional, Union, List
+from threading import Lock
+
+from ..utils.logging import get_logger
+
+class CacheItem:
+    """Container for cached data."""
+    def __init__(self, value: Any, ttl: Optional[int] = None):
+        self.value = value
+        self.timestamp = time.time()
+        self.ttl = ttl
+
+    def is_expired(self) -> bool:
+        """Check if item has expired."""
+        if self.ttl is None:
+            return False
+        return time.time() - self.timestamp > self.ttl
+
+class ExtractionCache:
+    """
+    LRU Cache for extraction results.
+    Thread-safe implementation.
+    """
+    def __init__(self, max_size: int = 1000, ttl: int = 3600):
+        """
+        Initialize the cache.
+        
+        Args:
+            max_size: Maximum number of items to store per namespace
+            ttl: Time to live in seconds (default 1 hour)
+        """
+        self.max_size = max_size
+        self.ttl = ttl
+        self._caches: Dict[str, OrderedDict] = {
+            "entities": OrderedDict(),
+            "relations": OrderedDict(),
+            "triplets": OrderedDict()
+        }
+        self._locks: Dict[str, Lock] = {
+            "entities": Lock(),
+            "relations": Lock(),
+            "triplets": Lock()
+        }
+        self.logger = get_logger("extraction_cache")
+        self.enabled = True
+
+    def _generate_key(self, text: str, **params) -> str:
+        """
+        Generate a stable cache key based on text and parameters.
+        
+        Note: Sensitive parameters like 'api_key' are excluded from the cache key
+        to prevent security risks and ensure cache sharing where appropriate.
+        """
+        # Filter out sensitive keys
+        sensitive_keys = {'api_key', 'token', 'password', 'secret', 'auth', 'authorization'}
+        filtered_params = {k: v for k, v in params.items() if k.lower() not in sensitive_keys}
+
+        # Create a stable string representation of params
+        # Sort keys to ensure consistent ordering
+        param_str = json.dumps(filtered_params, sort_keys=True, default=str)
+        
+        # Combine text and params
+        content = f"{text}|{param_str}"
+        
+        # Return hash (SHA-256 for better security than MD5)
+        return hashlib.sha256(content.encode('utf-8')).hexdigest()
+
+    def get(self, namespace: str, text: str, **params) -> Optional[Any]:
+        """
+        Retrieve item from cache.
+        
+        Args:
+            namespace: Cache namespace ("entities", "relations", "triplets")
+            text: Input text used for extraction
+            **params: Extraction parameters used
+            
+        Returns:
+            Cached result or None if not found/expired
+        """
+        if not self.enabled:
+            return None
+
+        if namespace not in self._caches:
+            return None
+
+        key = self._generate_key(text, **params)
+        
+        with self._locks[namespace]:
+            cache = self._caches[namespace]
+            if key in cache:
+                item = cache[key]
+                
+                # Check expiration
+                if item.is_expired():
+                    del cache[key]
+                    return None
+                
+                # Move to end (mark as recently used)
+                cache.move_to_end(key)
+                return item.value
+        
+        return None
+
+    def set(self, namespace: str, text: str, value: Any, **params) -> None:
+        """
+        Add item to cache.
+        
+        Args:
+            namespace: Cache namespace
+            text: Input text
+            value: Result to cache
+            **params: Extraction parameters
+        """
+        if not self.enabled:
+            return
+
+        if namespace not in self._caches:
+            self.logger.warning(f"Unknown cache namespace: {namespace}")
+            return
+
+        key = self._generate_key(text, **params)
+        item = CacheItem(value, self.ttl)
+        
+        with self._locks[namespace]:
+            cache = self._caches[namespace]
+            
+            # If key exists, update and move to end
+            if key in cache:
+                cache.move_to_end(key)
+            
+            cache[key] = item
+            
+            # Evict if full
+            if len(cache) > self.max_size:
+                cache.popitem(last=False)  # Remove first (least recently used)
+
+    def clear(self, namespace: Optional[str] = None):
+        """Clear cache(s)."""
+        if namespace:
+            if namespace in self._caches:
+                with self._locks[namespace]:
+                    self._caches[namespace].clear()
+        else:
+            for ns in self._caches:
+                with self._locks[ns]:
+                    self._caches[ns].clear()
+
+    def get_stats(self) -> Dict[str, Dict[str, int]]:
+        """Get cache statistics."""
+        stats = {}
+        for ns, cache in self._caches.items():
+            stats[ns] = {
+                "size": len(cache),
+                "max_size": self.max_size
+            }
+        return stats
+
+# Global cache instance
+extraction_cache = ExtractionCache()

semantica/semantic_extract/config.py

@@ -40,8 +40,9 @@ License: MIT
 """
 
 import os
+import multiprocessing
 from pathlib import Path
-from typing import Dict, Optional
+from typing import Dict, Optional, Any
 
 from ..utils.logging import get_logger
 
@@ -53,9 +54,23 @@ class Config:
         """Initialize configuration manager."""
         self.logger = get_logger("config")
         self._configs: Dict[str, Dict] = {}
+        # Default optimization settings
+        self._configs["optimization"] = {
+            "enable_cache": True,
+            "cache_size": 1000,
+            "max_workers": 8,
+            "enable_batching": True,
+            "batch_size": 10,
+            "max_tokens_per_batch": 2000
+        }
         self._load_config_file(config_file)
         self._load_env_vars()
 
+    def get_optimization_config(self) -> Dict:
+        """Get optimization configuration."""
+        return self._configs.get("optimization", {})
+
+
     def _load_config_file(self, config_file: Optional[str]):
         """Load configuration from file."""
         if config_file and Path(config_file).exists():
@@ -114,6 +129,66 @@ class Config:
             return self._configs[provider].get("api_key")
         return os.getenv(f"{provider.upper()}_API_KEY")
 
+    def get(self, key: str, default: Any = None) -> Any:
+        """
+        Get configuration value by key.
+        Searches in top-level configs and optimization settings.
+        """
+        # 1. Check top-level keys
+        if key in self._configs:
+            return self._configs[key]
+            
+        # 2. Check optimization settings (common keys)
+        if "optimization" in self._configs and key in self._configs["optimization"]:
+            return self._configs["optimization"][key]
+            
+        # 3. Handle specific mapping for optimization keys
+        # Map cache_enabled -> enable_cache if needed
+        if key == "cache_enabled":
+            return self._configs.get("optimization", {}).get("enable_cache", default)
+            
+        return default
+
 
 # Global config instance
 config = Config()
+
+
+def resolve_max_workers(
+    explicit: Optional[int] = None,
+    local_config: Optional[Dict[str, Any]] = None,
+    methods: Optional[Any] = None,
+) -> int:
+    def to_int(val: Any, default: int) -> int:
+        try:
+            return int(val)
+        except Exception:
+            return default
+
+    if isinstance(methods, str):
+        normalized_methods = [methods]
+    elif isinstance(methods, (list, tuple, set)):
+        normalized_methods = [m for m in methods if isinstance(m, str)]
+    else:
+        normalized_methods = []
+
+    if explicit is not None:
+        value = to_int(explicit, 1)
+    elif local_config and "max_workers" in local_config:
+        value = to_int(local_config.get("max_workers", 1), 1)
+    else:
+        value = to_int(config.get("max_workers", 5), 5)
+
+    if "ml" in normalized_methods and explicit is None and not (local_config and "max_workers" in local_config):
+        value = 1
+
+    if value < 1:
+        value = 1
+
+    cpu_count = multiprocessing.cpu_count() or 1
+    if value > cpu_count:
+        value = cpu_count
+    if value > 32:
+        value = 32
+
+    return value

semantica/semantic_extract/coreference_resolver.py

@@ -86,6 +86,7 @@ class CoreferenceChain:
     mentions: List[Mention]
     representative: Mention
     entity_type: Optional[str] = None
+    metadata: Dict[str, Any] = field(default_factory=dict)
 
 
 class CoreferenceResolver:
@@ -121,12 +122,18 @@ class CoreferenceResolver:
         )
         self.chain_builder = CoreferenceChainBuilder(**self.config.get("chain", {}))
 
-    def resolve_coreferences(self, text: str, **options) -> List[CoreferenceChain]:
+    def resolve_coreferences(
+        self,
+        text: str,
+        entities: Optional[List[Entity]] = None,
+        **options
+    ) -> List[CoreferenceChain]:
         """
         Resolve coreferences in text.
 
         Args:
             text: Input text
+            entities: List of entities (optional)
             **options: Resolution options
 
         Returns:
@@ -139,6 +146,8 @@ class CoreferenceResolver:
         )
 
         try:
+            from .ner_extractor import NERExtractor
+
             total_steps = 4  # Extract mentions, resolve pronouns, detect coreferences, build chains
             current_step = 0
             
@@ -151,8 +160,38 @@ class CoreferenceResolver:
                 total=total_steps,
                 message=f"Extracting mentions... ({current_step}/{total_steps}, remaining: {remaining_steps} steps)"
             )
+            
+            # Extract pronouns
             mentions = self._extract_mentions(text)
 
+            # Add entities as mentions
+            if entities is None:
+                # Extract entities if not provided
+                ner_config = self.config.get("ner", {})
+                if "ner_method" in self.config:
+                    ner_config["method"] = self.config["ner_method"]
+                ner = NERExtractor(
+                    **ner_config,
+                    **{
+                        k: v
+                        for k, v in self.config.items()
+                        if k not in ["ner", "relation", "chain", "entity", "pronoun"]
+                    },
+                )
+                entities = ner.extract_entities(text, **options)
+
+            if entities:
+                for entity in entities:
+                    mentions.append(
+                        Mention(
+                            text=entity.text,
+                            start_char=entity.start_char,
+                            end_char=entity.end_char,
+                            mention_type="entity",
+                            metadata={"entity_label": entity.label, "confidence": entity.confidence},
+                        )
+                    )
+            
             # Step 2: Resolve pronouns
             current_step += 1
             remaining_steps = total_steps - current_step
@@ -201,20 +240,128 @@ class CoreferenceResolver:
             self.progress_tracker.stop_tracking(
                 tracking_id, status="failed", message=str(e)
             )
+            verbose_mode = options.get("verbose", False) or self.config.get("verbose", False)
+            if verbose_mode:
+                import sys
+                print(f"    [CoreferenceResolver] ERROR: Resolution failed: {e}", flush=True, file=sys.stderr)
+                import traceback
+                traceback.print_exc(file=sys.stderr)
             raise
 
-    def resolve(self, text: str, **options) -> List[CoreferenceChain]:
+    def resolve(
+        self,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        entities: Optional[Union[List[Entity], List[List[Entity]]]] = None,
+        pipeline_id: Optional[str] = None,
+        **kwargs
+    ) -> Union[List[CoreferenceChain], List[List[CoreferenceChain]]]:
         """
-        Resolve coreferences in text (alias for resolve_coreferences).
+        Resolve coreferences in text or list of documents.
+        Handles batch processing with progress tracking.
 
         Args:
-            text: Input text
-            **options: Resolution options
+            text: Input text or list of documents
+            entities: List of entities or list of list of entities (optional)
+            pipeline_id: Optional pipeline ID for progress tracking
+            **kwargs: Resolution options
 
         Returns:
-            list: List of coreference chains
+            Union[List[CoreferenceChain], List[List[CoreferenceChain]]]: Resolved coreference chains
         """
-        return self.resolve_coreferences(text, **options)
+        if isinstance(text, list):
+            # Handle batch resolution with progress tracking
+            tracking_id = self.progress_tracker.start_tracking(
+                module="semantic_extract",
+                submodule="CoreferenceResolver",
+                message=f"Batch resolving coreferences from {len(text)} documents",
+                pipeline_id=pipeline_id,
+            )
+
+            try:
+                results = []
+                total_items = len(text)
+                total_chains_count = 0
+                
+                # Determine update interval
+                if total_items <= 10:
+                    update_interval = 1
+                else:
+                    update_interval = max(1, min(10, total_items // 100))
+                
+                # Initial progress update
+                self.progress_tracker.update_progress(
+                    tracking_id,
+                    processed=0,
+                    total=total_items,
+                    message=f"Starting batch resolution... 0/{total_items} (remaining: {total_items})"
+                )
+
+                for idx, item in enumerate(text):
+                    # Prepare arguments for single item
+                    doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                    
+                    doc_entities = None
+                    if entities and idx < len(entities):
+                        doc_entities = entities[idx]
+
+                    # Resolve
+                    chains = self.resolve_coreferences(doc_text, entities=doc_entities, **kwargs)
+
+                    # Add provenance metadata
+                    for chain in chains:
+                        # Update chain metadata
+                        if chain.metadata is None:
+                            chain.metadata = {}
+                        chain.metadata["batch_index"] = idx
+                        if isinstance(item, dict) and "id" in item:
+                            chain.metadata["document_id"] = item["id"]
+
+                        # Update mentions metadata
+                        for mention in chain.mentions:
+                            if mention.metadata is None:
+                                mention.metadata = {}
+                            mention.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                mention.metadata["document_id"] = item["id"]
+                        
+                        # Update representative metadata
+                        if chain.representative:
+                            if chain.representative.metadata is None:
+                                chain.representative.metadata = {}
+                            chain.representative.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                chain.representative.metadata["document_id"] = item["id"]
+
+                    results.append(chains)
+                    total_chains_count += len(chains)
+
+                    # Update progress
+                    if (idx + 1) % update_interval == 0 or (idx + 1) == total_items:
+                        remaining = total_items - (idx + 1)
+                        self.progress_tracker.update_progress(
+                            tracking_id,
+                            processed=idx + 1,
+                            total=total_items,
+                            message=f"Processing... {idx + 1}/{total_items} (remaining: {remaining}) - Resolved {total_chains_count} chains"
+                        )
+
+                self.progress_tracker.stop_tracking(
+                    tracking_id,
+                    status="completed",
+                    message=f"Batch resolution completed. Processed {len(results)} documents, resolved {total_chains_count} chains.",
+                )
+                return results
+
+            except Exception as e:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message=str(e)
+                )
+                raise
+
+        else:
+            # Single item
+            return self.resolve_coreferences(text, entities=entities, **kwargs)
+
     def _extract_mentions(self, text: str) -> List[Mention]:
         """Extract all mentions from text."""
         mentions = []
@@ -346,15 +493,49 @@ class PronounResolver:
             if m.mention_type == "entity" or m.mention_type == "nominal"
         ]
 
-        # Simple resolution: find closest preceding entity
+        # Simple resolution: find closest preceding entity with compatible type
+        pronoun_types = {
+            "he": ["PERSON"],
+            "him": ["PERSON"],
+            "his": ["PERSON"],
+            "she": ["PERSON"],
+            "her": ["PERSON"],
+            "it": ["ORG", "GPE", "LOC", "PRODUCT", "EVENT", "FAC", "WORK_OF_ART", "LAW", "LANGUAGE", "DATE", "TIME", "PERCENT", "MONEY", "QUANTITY", "ORDINAL", "CARDINAL"],
+            "its": ["ORG", "GPE", "LOC", "PRODUCT", "EVENT", "FAC", "WORK_OF_ART", "LAW", "LANGUAGE", "DATE", "TIME", "PERCENT", "MONEY", "QUANTITY", "ORDINAL", "CARDINAL"],
+            "they": ["ORG", "GPE", "PERSON", "NORP"], # Can be groups of people or organizations
+            "them": ["ORG", "GPE", "PERSON", "NORP"],
+            "their": ["ORG", "GPE", "PERSON", "NORP"],
+        }
+
         for pronoun in pronouns:
             # Find preceding entities
             preceding = [e for e in entities if e.end_char < pronoun.start_char]
 
             if preceding:
-                # Take closest
-                antecedent = max(preceding, key=lambda e: e.end_char)
+                pronoun_lower = pronoun.text.lower()
+                compatible_types = pronoun_types.get(pronoun_lower)
+                
+                antecedent = None
+                
+                if compatible_types:
+                    # Filter by type
+                    compatible = [
+                        e for e in preceding 
+                        if e.metadata and e.metadata.get("entity_label") in compatible_types
+                    ]
+                    if compatible:
+                        # Take closest compatible
+                        antecedent = max(compatible, key=lambda e: e.end_char)
+                
+                # Fallback to closest if no compatible found or pronoun type unknown
+                if antecedent is None:
+                    antecedent = max(preceding, key=lambda e: e.end_char)
+
                 resolutions.append((pronoun.text, antecedent.text))
+                
+                # Update pronoun metadata and link to antecedent
+                pronoun.entity_id = antecedent.text
+                pronoun.metadata["antecedent_text"] = antecedent.text
 
         return resolutions
 
@@ -431,32 +612,59 @@ class CoreferenceChainBuilder:
             list: List of coreference chains
         """
         chains = []
+        processed_indices = set()
 
-        # Simple implementation: group by text similarity
-        processed = set()
-
-        for mention in mentions:
-            if mention.text.lower() in processed:
+        for i, mention in enumerate(mentions):
+            if i in processed_indices:
                 continue
 
-            # Find similar mentions
-            similar = [
-                m
-                for m in mentions
-                if m.text.lower() == mention.text.lower()
-                or self._similar_mentions(mention.text, m.text)
-            ]
+            # Start a new group
+            group = [mention]
+            processed_indices.add(i)
 
-            if len(similar) > 1:
-                processed.add(mention.text.lower())
+            # Find related mentions
+            for j, other in enumerate(mentions):
+                if j in processed_indices:
+                    continue
 
-                # Representative is first (leftmost) mention
-                representative = min(similar, key=lambda m: m.start_char)
+                is_related = False
+
+                # 1. Text similarity
+                if (
+                    other.text.lower() == mention.text.lower()
+                    or self._similar_mentions(mention.text, other.text)
+                ):
+                    is_related = True
+                
+                # 2. Pronoun resolution (entity_id matches text or entity_id matches entity_id)
+                elif mention.entity_id and (
+                    mention.entity_id == other.text 
+                    or mention.entity_id == other.entity_id
+                ):
+                    is_related = True
+                elif other.entity_id and (
+                    other.entity_id == mention.text
+                    or other.entity_id == mention.entity_id
+                ):
+                    is_related = True
+
+                if is_related:
+                    group.append(other)
+                    processed_indices.add(j)
+
+            if len(group) > 1:
+                # Representative is first (leftmost) mention, or prefer entity over pronoun
+                # Prefer entity mention as representative
+                entities = [m for m in group if m.mention_type != "pronoun"]
+                if entities:
+                    representative = min(entities, key=lambda m: m.start_char)
+                else:
+                    representative = min(group, key=lambda m: m.start_char)
 
                 chain = CoreferenceChain(
-                    mentions=similar,
+                    mentions=group,
                     representative=representative,
-                    entity_type=similar[0].metadata.get("entity_label"),
+                    entity_type=representative.metadata.get("entity_label"),
                 )
                 chains.append(chain)
 

semantica/semantic_extract/event_detector.py

@@ -85,79 +85,215 @@ class Event:
 class EventDetector:
     """Event detection and extraction handler."""
 
-    def __init__(
-        self,
-        event_types: Optional[List[str]] = None,
-        extract_participants: bool = True,
-        extract_location: bool = True,
-        extract_time: bool = True,
-        method: Union[str, List[str]] = None,
-        config=None,
-        **kwargs
-    ):
+    def __init__(self, method: str = "llm", **config):
         """
         Initialize event detector.
 
         Args:
-            event_types: Specific event types to detect (e.g., ["launch", "acquisition"])
-            extract_participants: Whether to extract event participants
-            extract_location: Whether to extract event locations
-            extract_time: Whether to extract temporal information
-            method: Extraction method(s) for underlying NER/relation extractors.
-                   Can be passed to ner_method and relation_method in config.
-            config: Legacy config dict (deprecated, use kwargs)
-            **kwargs: Configuration options:
-                - ner_method: Method for NER extraction (if entities need to be extracted)
-                - relation_method: Method for relation extraction (if relations need to be extracted)
-                - Other options passed to sub-components
+            method: Extraction method ("llm", "pattern")
+            **config: Configuration options
         """
         self.logger = get_logger("event_detector")
-        self.config = config or {}
-        self.config.update(kwargs)
+        self.config = config
+        self.method = method
         self.progress_tracker = get_progress_tracker()
+        
         # Ensure progress tracker is enabled
         if not self.progress_tracker.enabled:
             self.progress_tracker.enabled = True
 
-        # Store parameters
-        self.event_types_filter = event_types
-        self.extract_participants = extract_participants
-        self.extract_location = extract_location
-        self.extract_time = extract_time
+        # Initialize components
+        self.event_classifier = EventClassifier(**config)
+        self.temporal_processor = TemporalEventProcessor(**config)
+
+        # Configure extraction options
+        self.extract_participants = config.get("extract_participants", True)
+        self.extract_location = config.get("extract_location", True)
+        self.extract_time = config.get("extract_time", True)
+        self.event_types_filter = config.get("event_types", [])
+
+        # Define event patterns
+        self.event_patterns = {
+            "acquisition": r"\b(acquired|acquisition|buying|bought|merger|merged)\b",
+            "partnership": r"\b(partnered|partnership|collaborate|collaboration)\b",
+            "launch": r"\b(launch|launched|releasing|released|unveil|unveiled)\b",
+            "investment": r"\b(invest|invested|investment|funding|raised)\b",
+            "legal": r"\b(sue|sued|lawsuit|litigation|legal action)\b",
+        }
+        
+        # Pre-compile location patterns
+        self.location_patterns = [
+            re.compile(r"in\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)"),
+            re.compile(r"at\s+([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*)"),
+        ]
+        
+        # Pre-compile time patterns
+        self.time_patterns = [
+            re.compile(r"on\s+([A-Z][a-z]+\s+\d{1,2},?\s+\d{4})"),
+            re.compile(r"in\s+(\d{4})"),
+            re.compile(r"(\d{1,2}[/-]\d{1,2}[/-]\d{2,4})"),
+        ]
 
-        # Store method for passing to extractors if needed
         if method is not None:
             self.config["ner_method"] = method
             self.config["relation_method"] = method
 
-        self.event_classifier = EventClassifier(**self.config.get("classifier", {}))
-        self.temporal_processor = TemporalEventProcessor(
-            **self.config.get("temporal", {})
-        )
-        self.relationship_extractor = EventRelationshipExtractor(
-            **self.config.get("relationship", {})
-        )
+    def extract(
+        self,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        pipeline_id: Optional[str] = None,
+        **kwargs
+    ) -> Union[List[Event], List[List[Event]]]:
+        """
+        Detect events in text or list of documents.
+        Handles batch processing with progress tracking.
 
-        # Event patterns
-        self.event_patterns = {
-            "founded": r"founded|created|established",
-            "acquired": r"acquired|bought|purchased",
-            "launched": r"launched|released|introduced",
-            "announced": r"announced|declared|stated",
-            "meeting": r"met|meeting|conference|summit",
-        }
+        Args:
+            text: Input text or list of documents
+            pipeline_id: Optional pipeline ID for progress tracking
+            **kwargs: Detection options
 
-    def detect_events(self, text: str, **options) -> List[Event]:
+        Returns:
+            Union[List[Event], List[List[Event]]]: Detected events
+        """
+        if isinstance(text, list):
+            # Handle batch detection with progress tracking
+            tracking_id = self.progress_tracker.start_tracking(
+                module="semantic_extract",
+                submodule="EventDetector",
+                message=f"Batch detecting events from {len(text)} documents",
+                pipeline_id=pipeline_id,
+            )
+
+            try:
+                results = [None] * len(text)  # Pre-allocate to maintain order
+                total_items = len(text)
+                total_events_count = 0
+                processed_count = 0
+                
+                # Determine update interval
+                if total_items <= 10:
+                    update_interval = 1
+                else:
+                    update_interval = max(1, min(10, total_items // 100))
+                
+                # Initial progress update
+                self.progress_tracker.update_progress(
+                    tracking_id,
+                    processed=0,
+                    total=total_items,
+                    message=f"Starting batch detection... 0/{total_items} (remaining: {total_items})"
+                )
+
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=[self.config.get("ner_method"), self.config.get("relation_method"), self.config.get("method")],
+                )
+
+                def process_item(idx, item):
+                    try:
+                        # Prepare arguments for single item
+                        doc_text = item["content"] if isinstance(item, dict) and "content" in item else str(item)
+                        
+                        # Detect
+                        events = self.detect_events(doc_text, **kwargs)
+
+                        # Add provenance metadata
+                        for event in events:
+                            if event.metadata is None:
+                                event.metadata = {}
+                            event.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                event.metadata["document_id"] = item["id"]
+                        
+                        return idx, events
+                    except Exception as e:
+                        self.logger.error(f"Error processing item {idx}: {e}")
+                        # Return empty list on failure to continue processing
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
+                    
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit tasks
+                        future_to_idx = {}
+                        for idx, item in enumerate(text):
+                            future = executor.submit(process_item, idx, item)
+                            future_to_idx[future] = idx
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, events = future.result()
+                            results[idx] = events
+                            total_events_count += len(events)
+                            processed_count += 1
+                            
+                            # Update progress
+                            if processed_count % update_interval == 0 or processed_count == total_items:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, events = process_item(idx, item)
+                        results[idx] = events
+                        total_events_count += len(events)
+                        processed_count += 1
+
+                        # Update progress
+                        if processed_count % update_interval == 0 or processed_count == total_items:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing... {processed_count}/{total_items} (remaining: {remaining}) - Detected {total_events_count} events"
+                            )
+
+                self.progress_tracker.stop_tracking(
+                    tracking_id,
+                    status="completed",
+                    message=f"Batch detection completed. Processed {len(results)} documents, detected {total_events_count} events.",
+                )
+                return results
+
+            except Exception as e:
+                self.progress_tracker.stop_tracking(
+                    tracking_id, status="failed", message=str(e)
+                )
+                raise
+
+        else:
+            # Single item
+            return self.detect_events(text, **kwargs)
+
+    def detect_events(
+        self,
+        text: Union[str, List[str], List[Dict[str, Any]]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Event], List[List[Event]]]:
         """
         Detect events in text content.
 
         Args:
             text: Input text
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Detection options
 
         Returns:
             list: List of detected events
         """
+        if isinstance(text, list):
+            return self.extract(text, pipeline_id=pipeline_id, **options)
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="EventDetector",

semantica/semantic_extract/extraction_validator.py

@@ -12,8 +12,8 @@ Supported Methods (for future extensibility):
 
 Algorithms Used:
     - Confidence Thresholding: Statistical threshold-based filtering
-    - Duplicate Detection: Set-based and similarity-based deduplication
-    - Consistency Checking: Rule-based and graph-based consistency validation
+    - Duplicate Detection: (Removed - handled by external module)
+    - Consistency Checking: (Removed - handled by external module)
     - Quality Scoring: Weighted scoring algorithms for extraction quality
     - Validation Metrics: Precision, recall, F1-score calculations
     - Boundary Validation: Character position and text boundary checking
@@ -22,7 +22,6 @@ Key Features:
     - Entity validation with confidence checking
     - Relation validation and consistency checking
     - Quality scoring and metrics calculation
-    - Duplicate detection
     - Confidence-based filtering
     - Validation result reporting
     - Method parameter support for future method-specific validation
@@ -47,8 +46,10 @@ Author: Semantica Contributors
 License: MIT
 """
 
+from typing import List, Dict, Any, Optional, Set, Tuple, Union
 from dataclasses import dataclass, field
-from typing import Any, Dict, List, Optional
+from datetime import datetime
+import re
 
 from ..utils.exceptions import ProcessingError
 from ..utils.logging import get_logger
@@ -66,6 +67,7 @@ class ValidationResult:
     errors: List[str] = field(default_factory=list)
     warnings: List[str] = field(default_factory=list)
     metrics: Dict[str, Any] = field(default_factory=dict)
+    metadata: Dict[str, Any] = field(default_factory=dict)
 
 
 class ExtractionValidator:
@@ -79,7 +81,6 @@ class ExtractionValidator:
             method: Validation method (for future extensibility, currently unused)
             **config: Configuration options:
                 - min_confidence: Minimum confidence threshold (default: 0.5)
-                - validate_consistency: Check consistency (default: True)
         """
         self.logger = get_logger("extraction_validator")
         self.config = config
@@ -90,19 +91,30 @@ class ExtractionValidator:
 
         self.method = method  # Reserved for future method-based validation
         self.min_confidence = config.get("min_confidence", 0.5)
-        self.validate_consistency = config.get("validate_consistency", True)
 
-    def validate_entities(self, entities: List[Entity], **options) -> ValidationResult:
+    def validate_entities(self, entities: Union[List[Entity], List[List[Entity]]], **options) -> Union[ValidationResult, List[ValidationResult]]:
         """
         Validate extracted entities.
+        Handles both single list and batch list of entities.
 
         Args:
-            entities: List of entities
+            entities: List of entities or list of list of entities
             **options: Validation options
 
         Returns:
-            ValidationResult: Validation result
+            ValidationResult or List[ValidationResult]: Validation result(s)
         """
+        # Handle batch validation
+        if entities and isinstance(entities, list) and len(entities) > 0 and isinstance(entities[0], list):
+            results = []
+            for idx, batch_entities in enumerate(entities):
+                res = self.validate_entities(batch_entities, **options)
+                # Ensure metadata has batch index
+                if "batch_index" not in res.metadata:
+                    res.metadata["batch_index"] = idx
+                results.append(res)
+            return results
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="ExtractionValidator",
@@ -126,14 +138,6 @@ class ExtractionValidator:
                     f"{len(low_confidence)} entities below confidence threshold"
                 )
 
-            # Check for duplicates
-            self.progress_tracker.update_tracking(
-                tracking_id, message="Checking for duplicates..."
-            )
-            entity_texts = [e.text.lower() for e in entities]
-            duplicates = len(entity_texts) - len(set(entity_texts))
-            if duplicates > 0:
-                warnings.append(f"{duplicates} duplicate entities found")
 
             # Check for empty entities
             empty_entities = [e for e in entities if not e.text.strip()]
@@ -148,8 +152,7 @@ class ExtractionValidator:
                     [e for e in entities if min_confidence <= e.confidence < 0.8]
                 ),
                 "low_confidence": len(low_confidence),
-                "unique_entities": len(set(entity_texts)),
-                "duplicates": duplicates,
+                "unique_entities": len(set(e.text for e in entities)),
                 "entity_types": len(set(e.label for e in entities)),
                 "average_confidence": sum(e.confidence for e in entities)
                 / len(entities)
@@ -162,12 +165,23 @@ class ExtractionValidator:
 
             valid = len(errors) == 0
 
+            # Collect metadata from entities
+            metadata = {}
+            if entities:
+                first = entities[0]
+                if hasattr(first, "metadata") and first.metadata:
+                    if "batch_index" in first.metadata:
+                        metadata["batch_index"] = first.metadata["batch_index"]
+                    if "document_id" in first.metadata:
+                        metadata["document_id"] = first.metadata["document_id"]
+
             result = ValidationResult(
                 valid=valid,
                 score=score,
                 errors=errors,
                 warnings=warnings,
                 metrics=metrics,
+                metadata=metadata,
             )
 
             self.progress_tracker.stop_tracking(
@@ -184,18 +198,30 @@ class ExtractionValidator:
             raise
 
     def validate_relations(
-        self, relations: List[Relation], **options
-    ) -> ValidationResult:
+        self, relations: Union[List[Relation], List[List[Relation]]], **options
+    ) -> Union[ValidationResult, List[ValidationResult]]:
         """
         Validate extracted relations.
+        Handles both single list and batch list of relations.
 
         Args:
-            relations: List of relations
+            relations: List of relations or list of list of relations
             **options: Validation options
 
         Returns:
-            ValidationResult: Validation result
+            ValidationResult or List[ValidationResult]: Validation result(s)
         """
+        # Handle batch validation
+        if relations and isinstance(relations, list) and len(relations) > 0 and isinstance(relations[0], list):
+            results = []
+            for idx, batch_relations in enumerate(relations):
+                res = self.validate_relations(batch_relations, **options)
+                # Ensure metadata has batch index
+                if "batch_index" not in res.metadata:
+                    res.metadata["batch_index"] = idx
+                results.append(res)
+            return results
+
         errors = []
         warnings = []
         metrics = {}
@@ -218,12 +244,6 @@ class ExtractionValidator:
         if invalid_relations:
             errors.append(f"{len(invalid_relations)} invalid relations found")
 
-        # Check consistency
-        if self.validate_consistency:
-            consistency_issues = self._check_consistency(relations)
-            if consistency_issues:
-                warnings.append(f"{len(consistency_issues)} consistency issues found")
-
         # Calculate metrics
         metrics = {
             "total_relations": len(relations),
@@ -244,31 +264,25 @@ class ExtractionValidator:
 
         valid = len(errors) == 0
 
+        # Collect metadata from relations
+        metadata = {}
+        if relations:
+            first = relations[0]
+            if hasattr(first, "metadata") and first.metadata:
+                if "batch_index" in first.metadata:
+                    metadata["batch_index"] = first.metadata["batch_index"]
+                if "document_id" in first.metadata:
+                    metadata["document_id"] = first.metadata["document_id"]
+
         return ValidationResult(
-            valid=valid, score=score, errors=errors, warnings=warnings, metrics=metrics
+            valid=valid, 
+            score=score, 
+            errors=errors, 
+            warnings=warnings, 
+            metrics=metrics,
+            metadata=metadata
         )
 
-    def _check_consistency(self, relations: List[Relation]) -> List[str]:
-        """Check consistency of relations."""
-        issues = []
-
-        # Check for contradictory relations
-        relation_pairs = {}
-        for relation in relations:
-            key = (relation.subject.text, relation.object.text)
-            if key not in relation_pairs:
-                relation_pairs[key] = []
-            relation_pairs[key].append(relation.predicate)
-
-        # Find contradictions (e.g., "founded_by" and "founded" for same pair)
-        for key, predicates in relation_pairs.items():
-            if len(set(predicates)) > 1:
-                # Check for obvious contradictions
-                if "founded_by" in predicates and "founded" in predicates:
-                    issues.append(f"Contradictory relations for {key}")
-
-        return issues
-
     def _calculate_entity_score(
         self, entities: List[Entity], metrics: Dict[str, Any]
     ) -> float:

semantica/semantic_extract/llm_extraction.py

@@ -93,7 +93,7 @@ class LLMExtraction:
             **config: Configuration options:
                 - model: Model name (default depends on provider)
                 - api_key: API key (from environment if not provided)
-                - temperature: Temperature for generation
+                - temperature: Temperature for generation (None = use model's default)
         """
         self.logger = get_logger("llm_extraction")
         self.config = config
@@ -104,11 +104,16 @@ class LLMExtraction:
 
         self.provider_name = provider
         self.model = config.get("model")
-        self.temperature = config.get("temperature", 0.3)
+        self.temperature = config.get("temperature")  # None = use model default
 
         # Initialize provider using new system
         try:
-            self.provider = create_provider(provider, **config)
+            # Sanitize config: remove api_key if it's None/empty to allow fallback
+            provider_config = config.copy()
+            if "api_key" in provider_config and not provider_config["api_key"]:
+                del provider_config["api_key"]
+                
+            self.provider = create_provider(provider, **provider_config)
         except Exception as e:
             self.logger.warning(f"Failed to initialize {provider} provider: {e}")
             self.provider = None
@@ -289,7 +294,14 @@ Return the enhanced relation list in JSON format."""
     ) -> List[Entity]:
         """Parse LLM response for entities."""
         # Simplified parsing - in practice would parse JSON
-        # For now, return original entities
+        # For now, return original entities with updated metadata
+        for entity in original_entities:
+            if entity.metadata is None:
+                entity.metadata = {}
+            entity.metadata.update({
+                "enhanced_by": self.provider_name,
+                "model": self.model
+            })
         return original_entities
 
     def _parse_relation_response(
@@ -297,7 +309,14 @@ Return the enhanced relation list in JSON format."""
     ) -> List[Relation]:
         """Parse LLM response for relations."""
         # Simplified parsing - in practice would parse JSON
-        # For now, return original relations
+        # For now, return original relations with updated metadata
+        for relation in original_relations:
+            if relation.metadata is None:
+                relation.metadata = {}
+            relation.metadata.update({
+                "enhanced_by": self.provider_name,
+                "model": self.model
+            })
         return original_relations
 
 

semantica/semantic_extract/ner_extractor.py

@@ -20,7 +20,12 @@ Algorithms Used:
     - Transformer Models: BERT, RoBERTa, DistilBERT for token classification
     - Large Language Models: GPT, Claude, Gemini for zero-shot/few-shot extraction
     - Ensemble Voting: Majority voting and confidence-weighted aggregation
-    - Deduplication: Set-based and similarity-based entity deduplication
+    - Weighted Confidence Scoring:
+        * Formula: Score = (0.5 * Method_Confidence) + (0.5 * Type_Similarity_Score)
+        * Method_Confidence: Confidence score from the extraction algorithm
+        * Type_Similarity_Score: Semantic match with user-provided entity types (Exact=1.0, Synonym=0.95, Embedding=Cosine_Sim)
+    - Hybrid Similarity Matching: Exact -> Synonym -> Substring -> Semantic Embedding (Batch Optimized)
+    - Last Resort Fallback: Capitalized word heuristic when all other methods fail
 
 Key Features:
     - Multiple extraction methods:
@@ -31,8 +36,9 @@ Key Features:
         * HuggingFace: Custom HuggingFace NER models
         * LLM-based: Large language model extraction
     - Fallback chain support: Try methods in order until one succeeds
+    - Robust Fallbacks: Prevents empty results via ML -> Pattern -> Last Resort chain
     - Ensemble voting: Combine results from multiple methods
-    - Post-processing: Entity boundary validation and deduplication
+    - Post-processing: Entity boundary validation
     - Multiple entity type support (PERSON, ORG, GPE, DATE, etc.)
     - Confidence scoring and filtering
     - Batch processing capabilities
@@ -90,7 +96,12 @@ class Entity:
 class NERExtractor:
     """Named Entity Recognition extractor."""
 
-    def __init__(self, method: Union[str, List[str]] = "ml", **config):
+    def __init__(
+        self, 
+        method: Union[str, List[str]] = "ml", 
+        entity_types: Optional[List[str]] = None,
+        **config
+    ):
         """
         Initialize NER extractor.
 
@@ -103,6 +114,8 @@ class NERExtractor:
                 - "huggingface": HuggingFace model
                 - "llm": LLM-based extraction
                 - List of methods for fallback chain
+            entity_types: List of entity types to extract (e.g., ["PERSON", "ORG"]).
+                          If provided, extraction methods will try to limit/focus on these types.
             **config: Configuration options:
                 - model: Model name (for ML/HuggingFace methods)
                 - huggingface_model: HuggingFace model name
@@ -115,6 +128,7 @@ class NERExtractor:
         """
         self.logger = get_logger("ner_extractor")
         self.config = config
+        self.entity_types = entity_types
 
         # Method configuration
         self.method = method if isinstance(method, list) else [method]
@@ -164,8 +178,11 @@ class NERExtractor:
             )
             
             try:
-                results = []
+                results = [None] * len(text)
                 total_items = len(text)
+                total_entities_count = 0
+                processed_count = 0
+                
                 # Update more frequently: every 1% or at least every 10 items, but always update for small datasets
                 if total_items <= 10:
                     update_interval = 1  # Update every item for small datasets
@@ -173,49 +190,107 @@ class NERExtractor:
                     update_interval = max(1, min(10, total_items // 100))
                 
                 # Initial progress update - ALWAYS show this
-                remaining = total_items
                 self.progress_tracker.update_progress(
                     tracking_id,
                     processed=0,
                     total=total_items,
-                    message=f"Starting batch extraction... 0/{total_items} (remaining: {remaining})"
+                    message=f"Starting batch extraction... 0/{total_items}"
                 )
                 
-                for idx, item in enumerate(text, 1):
+                from .config import resolve_max_workers
+                max_workers = resolve_max_workers(
+                    explicit=kwargs.get("max_workers"),
+                    local_config=self.config,
+                    methods=self.method,
+                )
+                
+                # Helper function for single item processing
+                def process_item(idx, item):
                     try:
+                        current_entities = []
                         if isinstance(item, dict) and "content" in item:
-                            results.append(self.extract_entities(item["content"], **kwargs))
+                            current_entities = self.extract_entities(item["content"], **kwargs)
                         elif isinstance(item, str):
-                            results.append(self.extract_entities(item, **kwargs))
+                            current_entities = self.extract_entities(item, **kwargs)
                         else:
                             # Try converting to string
                             try:
-                                results.append(self.extract_entities(str(item), **kwargs))
+                                current_entities = self.extract_entities(str(item), **kwargs)
                             except Exception:
-                                results.append([])
-                    except Exception:
-                        results.append([])
+                                current_entities = []
+                        
+                        # Add provenance metadata
+                        for ent in current_entities:
+                            if ent.metadata is None:
+                                ent.metadata = {}
+                            ent.metadata["batch_index"] = idx
+                            if isinstance(item, dict) and "id" in item:
+                                ent.metadata["document_id"] = item["id"]
+                        
+                        return idx, current_entities
+                    except Exception as e:
+                        self.logger.warning(f"Failed to process item {idx}: {e}")
+                        return idx, []
+
+                if max_workers > 1:
+                    import concurrent.futures
                     
-                    remaining = total_items - idx
-                    # Update progress: always update for small datasets, or at intervals for large ones
-                    should_update = (
-                        idx % update_interval == 0 or 
-                        idx == total_items or 
-                        idx == 1 or
-                        total_items <= 10  # Always update for small datasets
-                    )
-                    if should_update:
-                        self.progress_tracker.update_progress(
-                            tracking_id,
-                            processed=idx,
-                            total=total_items,
-                            message=f"Processing documents... {idx}/{total_items} (remaining: {remaining})"
+                    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+                        # Submit all tasks
+                        future_to_idx = {
+                            executor.submit(process_item, idx, item): idx 
+                            for idx, item in enumerate(text)
+                        }
+                        
+                        for future in concurrent.futures.as_completed(future_to_idx):
+                            idx, entities = future.result()
+                            results[idx] = entities
+                            total_entities_count += len(entities)
+                            processed_count += 1
+                            
+                            # Update progress
+                            should_update = (
+                                processed_count % update_interval == 0 or 
+                                processed_count == total_items or 
+                                processed_count == 1 or
+                                total_items <= 10
+                            )
+                            if should_update:
+                                remaining = total_items - processed_count
+                                self.progress_tracker.update_progress(
+                                    tracking_id,
+                                    processed=processed_count,
+                                    total=total_items,
+                                    message=f"Processing documents... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                                )
+                else:
+                    # Sequential processing
+                    for idx, item in enumerate(text):
+                        _, entities = process_item(idx, item)
+                        results[idx] = entities
+                        total_entities_count += len(entities)
+                        processed_count += 1
+                        
+                        # Update progress
+                        should_update = (
+                            processed_count % update_interval == 0 or 
+                            processed_count == total_items or 
+                            processed_count == 1 or
+                            total_items <= 10
                         )
+                        if should_update:
+                            remaining = total_items - processed_count
+                            self.progress_tracker.update_progress(
+                                tracking_id,
+                                processed=processed_count,
+                                total=total_items,
+                                message=f"Processing documents... {processed_count}/{total_items} (remaining: {remaining}) - Extracted {total_entities_count} entities so far"
+                            )
                 
                 self.progress_tracker.stop_tracking(
                     tracking_id,
                     status="completed",
-                    message=f"Extracted entities from {len(results)} documents",
+                    message=f"Batch extraction completed. Processed {len(results)} documents, extracted {total_entities_count} entities.",
                 )
                 return results
             except Exception as e:
@@ -226,12 +301,18 @@ class NERExtractor:
         else:
             return self.extract_entities(text, **kwargs)
 
-    def extract_entities(self, text: str, **options) -> List[Entity]:
+    def extract_entities(
+        self,
+        text: Union[str, List[Dict[str, Any]], List[str]],
+        pipeline_id: Optional[str] = None,
+        **options,
+    ) -> Union[List[Entity], List[List[Entity]]]:
         """
         Extract named entities from text.
 
         Args:
             text: Input text
+            pipeline_id: Optional pipeline ID for progress tracking (batch mode)
             **options: Extraction options:
                 - entity_types: Filter by entity types (list)
                 - min_confidence: Minimum confidence threshold
@@ -240,6 +321,9 @@ class NERExtractor:
         Returns:
             list: List of extracted entities
         """
+        if isinstance(text, list):
+            return self.extract(text, pipeline_id=pipeline_id, **options)
+
         tracking_id = self.progress_tracker.start_tracking(
             module="semantic_extract",
             submodule="NERExtractor",
@@ -260,10 +344,12 @@ class NERExtractor:
                 methods = [methods]
 
             min_confidence = options.get("min_confidence", self.min_confidence)
-            entity_types = options.get("entity_types")
+            entity_types = options.get("entity_types", self.entity_types)
 
             # Merge config with options
             all_options = {**self.config, **options}
+            if entity_types:
+                all_options["entity_types"] = entity_types
 
             # Try each method in order (fallback chain)
             all_entities = []
@@ -278,8 +364,11 @@ class NERExtractor:
                     # Prepare method-specific options
                     method_options = all_options.copy()
                     if method_name == "huggingface":
-                        method_options["model"] = all_options.get(
-                            "huggingface_model", self.huggingface_model
+                        # Prioritize runtime options over config/defaults
+                        method_options["model"] = (
+                            options.get("huggingface_model") 
+                            or options.get("model") 
+                            or self.huggingface_model
                         )
                         method_options["device"] = all_options.get("device")
                     elif method_name == "llm":
@@ -289,40 +378,46 @@ class NERExtractor:
                         method_options["model"] = all_options.get(
                             "llm_model", all_options.get("model")
                         )
-                        # Pass api_key if provided (needed for all providers)
-                        if "api_key" in all_options:
-                            method_options["api_key"] = all_options["api_key"]
-                        elif "api_key" not in method_options:
-                            # Try to get from environment as fallback
+                        # Ensure api_key is populated: check explicitly provided or fallback to env
+                        current_key = method_options.get("api_key")
+                        if not current_key:
+                            # Not found or empty/None, try environment
                             import os
-                            provider = method_options.get("provider", "openai")
-                            env_key = f"{provider.upper()}_API_KEY"
+                            provider_name = method_options.get("provider", "openai")
+                            env_key = f"{provider_name.upper()}_API_KEY"
                             api_key = os.getenv(env_key)
                             if api_key:
                                 method_options["api_key"] = api_key
-                        # Pass entity_types to LLM method so it can use them in the prompt
-                        if entity_types:
-                            method_options["entity_types"] = entity_types
 
                     entities = method_func(text, **method_options)
 
-                    # Filter by confidence and entity types
-                    filtered = [e for e in entities if e.confidence >= min_confidence]
+                    # Apply weighted scoring if entity_types are provided
                     if entity_types:
-                        # Case-insensitive and flexible matching for entity types
-                        entity_types_lower = {et.lower() for et in entity_types}
-                        filtered = [
-                            e for e in filtered 
-                            if e.label.lower() in entity_types_lower 
-                            or any(et.lower() in e.label.lower() or e.label.lower() in et.lower() 
-                                   for et in entity_types)
-                        ]
+                        try:
+                            from .methods import calculate_weighted_confidence
+                            for e in entities:
+                                e.confidence = calculate_weighted_confidence(
+                                    item_type=e.label,
+                                    original_confidence=e.confidence,
+                                    valid_types=entity_types,
+                                    item_text=e.text
+                                )
+                        except ImportError:
+                            pass
 
+                    # Filter by confidence
+                    filtered = [e for e in entities if e.confidence >= min_confidence]
+                    
                     if filtered:
                         all_entities.append((method_name, filtered))
 
                         # If not using ensemble, return first successful result
                         if not self.ensemble_voting:
+                            # Ensure default metadata
+                            for e in filtered:
+                                if e.metadata is None: e.metadata = {}
+                                if "batch_index" not in e.metadata: e.metadata["batch_index"] = 0
+
                             self.progress_tracker.stop_tracking(
                                 tracking_id,
                                 status="completed",
@@ -342,7 +437,8 @@ class NERExtractor:
             elif all_entities:
                 entities = all_entities[0][1]  # Use first successful method
             else:
-                entities = []
+                # Fallback to pattern-based extraction if all models fail
+                entities = self._extract_fallback(text)
 
             # Post-processing if enabled
             if self.post_process and entities:
@@ -390,19 +486,12 @@ class NERExtractor:
     def _post_process_entities(self, entities: List[Entity], text: str) -> List[Entity]:
         """Post-process entities for refinement."""
         processed = []
-        seen = set()
 
         for entity in entities:
             # Check boundaries
             if entity.start_char < 0 or entity.end_char > len(text):
                 continue
 
-            # Check for duplicates
-            key = (entity.text.lower(), entity.label, entity.start_char)
-            if key in seen:
-                continue
-            seen.add(key)
-
             # Validate entity text matches
             actual_text = text[entity.start_char : entity.end_char]
             if actual_text.lower() != entity.text.lower():
@@ -457,6 +546,7 @@ class NERExtractor:
     def _extract_fallback(self, text: str) -> List[Entity]:
         """Fallback entity extraction using simple patterns."""
         entities = []
+        import re
 
         # Simple patterns for common entity types
         patterns = {
@@ -466,20 +556,49 @@ class NERExtractor:
             "DATE": r"\b(\d{1,2}[/-]\d{1,2}[/-]\d{2,4}|\d{4})\b",
         }
 
-        import re
+        # Track covered ranges to avoid overlaps
+        covered_ranges = set()
 
         for label, pattern in patterns.items():
             for match in re.finditer(pattern, text):
-                entities.append(
-                    Entity(
-                        text=match.group(1),
-                        label=label,
-                        start_char=match.start(),
-                        end_char=match.end(),
-                        confidence=0.7,  # Lower confidence for pattern-based
-                        metadata={"extraction_method": "pattern"},
+                start, end = match.start(), match.end()
+                # Check overlap
+                is_overlap = any(r_start < end and r_end > start for r_start, r_end in covered_ranges)
+                if not is_overlap:
+                    # Use group 1 if available, else group 0
+                    text_val = match.group(1) if match.lastindex and match.lastindex >= 1 else match.group(0)
+                    
+                    entities.append(
+                        Entity(
+                            text=text_val,
+                            label=label,
+                            start_char=start,
+                            end_char=end,
+                            confidence=0.7,  # Lower confidence for pattern-based
+                            metadata={"extraction_method": "pattern"},
+                        )
                     )
-                )
+                    covered_ranges.add((start, end))
+
+        # Last Resort: If no entities found, try single capitalized words as generic entities
+        if not entities:
+            # Match any capitalized word of length > 2
+            cap_pattern = r"\b[A-Z][a-z]{2,}\b"
+            for match in re.finditer(cap_pattern, text):
+                start, end = match.start(), match.end()
+                is_overlap = any(r_start < end and r_end > start for r_start, r_end in covered_ranges)
+                if not is_overlap:
+                    entities.append(
+                        Entity(
+                            text=match.group(0),
+                            label="UNKNOWN",
+                            start_char=start,
+                            end_char=end,
+                            confidence=0.5,
+                            metadata={"extraction_method": "last_resort_pattern"},
+                        )
+                    )
+                    covered_ranges.add((start, end))
 
         return entities
 
@@ -494,7 +613,7 @@ class NERExtractor:
         Returns:
             list: List of entity lists for each text
         """
-        return [self.extract_entities(text, **options) for text in texts]
+        return self.extract(texts, **options)
 
     def classify_entities(self, entities: List[Entity]) -> Dict[str, List[Entity]]:
         """