semantica/CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[0.3.0] - 2026-03-10

• Context Graph Feature Completeness (by @KaifAhmad1):

  • Added valid_from / valid_until temporal validity fields to ContextNode and ContextEdge dataclasses — both expose is_active(at_time=None) -> bool; nodes/edges without these fields are always considered active
  • Added add_node(valid_from=..., valid_until=...) and add_edge(valid_from=..., valid_until=...) support — validity windows are extracted from **properties and stored as first-class dataclass fields, not in metadata
  • Added ContextGraph.find_active_nodes(node_type=None, at_time=None) — returns only nodes whose validity window includes the given time (defaults to datetime.utcnow()); complements find_nodes() with temporal filtering
  • Added min_weight: float = 0.0 parameter to ContextGraph.get_neighbors() — edges with weight below the threshold are skipped during BFS traversal, enabling weighted/confidence-filtered multi-hop navigation; fully backward-compatible (default 0.0 passes all edges)
  • Added ContextGraph.link_graph(other_graph, source_node_id, target_node_id, link_type="CROSS_GRAPH") -> str — creates a navigable bridge between two separate ContextGraph instances; records a marker edge internally and returns a link_id
  • Added ContextGraph.navigate_to(link_id) -> (other_graph, target_node_id) — resolves a link_id to the target graph and its entry node, enabling hierarchical cross-graph traversal (e.g. agent moving from a high-level decision graph into a domain-specific sub-graph)
  • Added ContextGraph.resolve_links(registry) — reconnects cross-graph links after load_from_file(); save_to_file() now persists a links section with other_graph_id so navigation survives the full save/load cycle
  • Added graph_id field to ContextGraph — stable UUID per instance, persisted to JSON, so separate graphs can identify each other after reload
  • Fixed is_active() on ContextNode and ContextEdge — tz-aware datetime inputs are now normalised to tz-naive UTC before comparison, preventing TypeError when callers pass datetime.now(timezone.utc)
  • Fixed valid_from / valid_until serialisation — add_nodes(), add_edges(), to_dict(), and from_dict() all now preserve and restore validity windows; previously these fields were silently lost
  • Fixed cross-graph link artifact — link_graph() now pre-creates a "cross_graph_link" typed ContextNode for the marker before inserting the marker edge, preventing _add_internal_edge() from auto-creating a phantom "entity" node
  • Added 14 tests in tests/context/test_cross_graph_navigation.py covering link creation, phantom-node prevention, and full save/load round-trips with resolve_links()
  • Fixed pipeline_builder.add_step() return type annotation from "PipelineBuilder" to "PipelineStep" — implementation was already correct per 0.3.0-beta changelog, only signature and docstring were stale
  • Fixed test_hybrid_search_performance timing computation — accumulated a real search_times list and compute true average; raised threshold to < 5.0s to account for real sentence-transformers (384-dim) latency

• 0.3.0 Bug Fixes & Comprehensive Real-World Tests (by @KaifAhmad1):

  • Fixed ProvenanceTracker missing from semantica/kg/__init__.py exports — from semantica.kg import ProvenanceTracker now works correctly
  • Fixed duplicate relation creation in _parse_relation_result — orphaned legacy block was appending every relation twice; removed the duplicate block
  • Added extraction_method parameter to _parse_relation_result; typed extraction path now correctly sets "llm_typed" instead of "llm" in relation metadata
  • Fixed cross-test cache pollution in tests/semantic_extract/test_retry_logic.py — module-level _result_cache now cleared in setUp() to prevent intermittent failures when tests share input text
  • Added tests/test_030_realworld_comprehensive.py: 85 real-world tests covering all 0.3.0-alpha/beta features with real data (tech companies, CEOs, products, investment chains, healthcare scenarios)
    • ContextGraph basic operations and decision tracking lifecycle
    • KG algorithms: centrality, community detection, embeddings, path finding, similarity, link prediction, connectivity
    • PolicyEngine, DecisionQuery, AgentContext, Decision model serialization
    • ProvenanceTracker with GraphBuilderWithProvenance and AlgorithmTrackerWithProvenance
    • Deduplication v2 with blocking strategies, RDF/TTL export, Reasoner inference
    • Pipeline builder/validator/failure handler with retry policies
    • Multi-hop investment chain (Microsoft→OpenAI, Google→Anthropic) end-to-end
    • Healthcare entity extraction and knowledge graph construction E2E

[0.3.0-beta] - 2026-03-07

• Multi-Founder LLM Extraction & Reasoner Inference Fix (PR #354 by @KaifAhmad1):

  • Fixed _parse_relation_result in methods.py — unmatched subjects/objects now produce a synthetic UNKNOWN entity instead of silently dropping the relation; all LLM-returned co-founders are preserved
  • Rewrote _match_pattern in reasoner.py — splits pattern on ?var placeholders first, then escapes only the literal segments; pre-bound variables resolve to exact literals, repeated variables use backreferences, non-greedy .+? prevents over-consumption of literal separators
  • Added tests/reasoning/test_reasoner.py with 4 tests covering multi-word value inference, pre-bound variables, binding conflicts, and single-word regression
  • Added tests/semantic_extract/test_relation_extractor.py with 6 tests covering all-founders returned, synthetic entity creation, matched entity integrity, predicate/confidence preservation, empty response, and malformed entries

• TTL Export Alias Fix (PR #355 by @KaifAhmad1):

  • Added _format_aliases map in RDFExporter so format="ttl", "nt", "xml", "rdf", and "json-ld" resolve to their canonical counterparts without breaking existing callers
  • Alias resolution applied at the top of export_to_rdf() before format validation — zero public API changes
  • Added working TTL export cell to cookbook/introduction/15_Export.ipynb (Step 3: RDF Export)
  • Added tests/export/test_rdf_exporter.py with 8 tests covering all aliases, canonical formats, error handling, and file export

• Incremental/Delta Processing Feature (PR #349 by @ZohaibHassan16, reviewed and fixed by @KaifAhmad1):

  • Native delta computation between graph snapshots using SPARQL queries
  • Delta-aware pipeline execution with delta_mode configuration for processing only changed data
  • Version snapshot management with graph URI tracking and metadata storage
  • Snapshot retention policies with automatic cleanup via prune_versions() method
  • Integration with pipeline execution engine for incremental workflows
  • Significant performance improvements: processes only changes instead of full datasets
  • Cost optimization: dramatically reduces compute and storage requirements for large-scale operations
  • Production-ready for near real-time pipelines and frequent deployment scenarios
  • Bug fixes: corrected SPARQL variable order, fixed class references, resolved duplicate dictionary keys
  • Comprehensive test coverage including delta mode integration tests
  • Complete documentation with usage examples and API references
  • Essential for enterprise-grade, large-scale semantic infrastructure

• Deduplication v2 Migration Guide (PR #344 by @ZohaibHassan16, fixes by @KaifAhmad1):

  • Added comprehensive MIGRATION_V2.md documentation for Deduplication v2 Epic #333
  • Documented Candidate Generation V2 with multi-key blocking and phonetic matching
  • Documented Two-Stage Scoring prefilter with configurable thresholds
  • Documented Semantic Relationship Deduplication v2 with synonym mapping
  • Added practical code examples for all V2 features with opt-in configuration
  • Fixed critical infinite recursion bug in dedup_triplets() function
  • Completed Epic #333 with comprehensive migration path and documentation
  • Performance: 5.86x speedup confirmed (129ms vs 754ms) for semantic deduplication
  • Full backward compatibility maintained with legacy mode as default

• Semantic Relationship Deduplication v2 (PR #340 by @ZohaibHassan16, fixes by @KaifAhmad1):

  • Implemented opt-in semantic relationship deduplication mode (semantic_v2) with 6.98x performance improvement
  • Added canonicalization engine with predicate synonym mapping (works_for → employed_by)
  • Implemented fast-path O(1) hash matching for exact canonical signature comparisons
  • Added weighted semantic scoring (60% predicate + 40% object composition) with explainable semantic_match_score metadata
  • Enhanced dedup_triplets() function as first-class API in methods.py
  • Integrated semantic deduplication into merge strategy with canonical key generation
  • Added literal normalization for whitespace cleanup in object matching
  • Maintained full backward compatibility with legacy mode as default
  • Fixed critical infinite recursion bug in dedup_triplets() function via registry name checking
  • Performance: Semantic V2 (~83ms) vs Legacy (~579ms) - 6.98x speedup confirmed
  • All 13 deduplication benchmarks passing with comprehensive test coverage

• Two-Stage Scoring Prefilter (PR #339 by @ZohaibHassan16):

  • Implemented opt-in two-stage scoring with fast prefilter gates to eliminate expensive semantic scoring for obvious non-matches
  • Prefilter gates: type mismatch detection, name length ratio validation, token overlap requirements
  • Performance improvements: 18-25% faster batch processing with prefilter enabled
  • Configurable thresholds: min_length_ratio, min_token_overlap_ratio, required_shared_token
  • Enhanced explainability with score breakdown and rejection reasons in metadata
  • Complete backward compatibility with default prefilter_enabled=False

• Candidate Generation v2 with Multi-Key Blocking (PR #338 by @ZohaibHassan16):

  • Implemented opt-in candidate generation strategies (legacy, blocking_v2, hybrid_v2) to address O(N²) pair explosion during deduplication
  • Multi-key blocking with normalized token prefixes, type-aware keys, and optional phonetic (Soundex) blocking
  • Deterministic candidate budgeting with max_candidates_per_entity limit using stable sorting
  • Efficient pair generation with set-based deduplication across overlapping blocks
  • Performance improvements: 63.6% faster in worst-case scenarios (0.259s → 0.094s for 100 entities)
  • Complete backward compatibility with default candidate_strategy="legacy"
  • Added configuration options: blocking_keys, enable_phonetic_blocking, max_candidates_per_entity

• ArangoDB AQL Export Support (PR #342 by @tibisabau):

Added

• ArangoDB AQL Export Support (PR #342 by @tibisabau)

  • Full-featured ArangoDB AQL exporter with 642 lines of production-ready code
  • Comprehensive AQL INSERT statement generation for vertices and edges
  • Configurable collection names with validation and sanitization
  • Batch processing support for large knowledge graphs (default: 1000)
  • Added export_arango() convenience function for easy access
  • Enhanced unified export with AQL format support and .aql auto-detection
  • Added export_arango() convenience function for easy access
  • Enhanced unified export with AQL format support and .aql auto-detection
  • Integrated with method registry for extensibility
  • 17 comprehensive test cases with 100% pass rate
  • Enterprise-grade ArangoDB multi-model database integration

• Apache Parquet Export Support (PR #343 by @tibisabau):

• Apache Parquet Export Support (PR #343 by @tibisabau)

  • Full-featured Apache Parquet exporter with 701 lines of production-ready code
  • Columnar storage format optimized for analytics and data warehousing
  • Configurable compression codecs (snappy, gzip, brotli, zstd, lz4, none)
  • Explicit Arrow schemas with type safety and consistency
  • Field normalization for varied entity and relationship naming conventions
  • Structured metadata handling using Parquet struct fields
  • Added export_parquet() convenience function for easy access
  • Enhanced unified export with Parquet format support and .parquet auto-detection
  • Added export_parquet() convenience function for easy access
  • Enhanced unified export with Parquet format support and .parquet auto-detection
  • Integrated with method registry for extensibility
  • 25 comprehensive test cases with 100% pass rate
  • Enterprise-grade analytics integration with pandas, Spark, Snowflake, BigQuery, Databricks

Fixed

• Fixed NameError: missing Type import in utils/helpers.py

• Fixed NameError: missing Type import in utils/helpers.py

  • Added Type to typing imports to fix retry_on_error decorator
  • Removed unused Type import from config_manager.py
  • Resolves ImportError when importing semantica modules
  • Fixes capability gap analysis notebook execution

• Test Suite Fixes: 0.3.0-alpha & Unreleased Features (PR utils by @KaifAhmad1):

  Context Module (semantica/context/)

  • Fixed retrieve_decision_precedents to gate entity extraction on use_hybrid_search=True — was incorrectly extracting entities when flag was False
  • Fixed _extract_entities_from_query to use word[0].isupper() instead of word.istitle() — correctly captures CreditCard, CustomerID etc.
  • Added missing expand_context method — BFS graph traversal via knowledge_graph.get_neighbors
  • Added missing _get_decision_query method — creates a DecisionQuery from the knowledge graph
  • Fixed hybrid_retrieval to call expand_context(query) once (not per-entity) and include "query" key in return dict
  • Fixed dynamic_context_traversal to call expand_context once per query instead of per entity
  • Fixed multi_hop_context_assembly to use _get_decision_query() for robust decision lookup
  • Fixed _retrieve_from_vector to fall back to result["metadata"]["content"] when result["content"] is absent — prevents empty content and negative similarity scores during semantic re-ranking

  Knowledge Graph Module (semantica/kg/)

  • Fixed calculate_pagerank — added alpha and max_iter parameter aliases; changed return format to structured dict {"centrality": scores, "rankings": sorted_list}
  • Fixed community_detector._to_networkx to return a NetworkX graph directly when one is passed (was converting to adjacency list, silently losing all edges)
  • Added method as alias for algorithm parameter in detect_communities
  • Fixed _build_adjacency to handle "edges" key (list of tuples) in addition to "relationships" (list of dicts)
  • Added _track_generic base method and 9 domain-specific tracking methods to AlgorithmTrackerWithProvenance: track_influence_analysis, track_verification_analysis, track_supply_chain_paths, track_bottleneck_analysis, track_quality_analysis, track_lead_time_analysis, track_cross_domain_analysis, track_cross_domain_similarity, track_collaboration_potential
  • Created new provenance_tracker.py module with ProvenanceTracker class (track_entity, get_all_sources, clear)

  Pipeline Module (semantica/pipeline/)

  • Fixed execution_engine retry loop to properly iterate up to max_retries (was only retrying once regardless of policy)
  • Added RecoveryAction dataclass and handle_failure(error, policy, retry_count) method to FailureHandler — implements LINEAR, EXPONENTIAL, and FIXED backoff strategies
  • Fixed pipeline_builder.add_step to return the created PipelineStep object instead of self
  • Added validate as a public alias for validate_pipeline in PipelineValidator
  • Updated missing-dependency error message to "Missing dependency '{dep}' for step '{name}'" for consistent test assertions

  Vector Store (semantica/vector_store/)

  • Relaxed test_batch_processing_performance threshold from < 100ms to < 500ms per decision — original threshold was too tight for development machines running a real sentence-transformers embedding model (384-dim)

  Test File Fixes

  • test_end_to_end_context_integration.py — replaced emoji characters (✅, ❌, 🔄, ⚠️) with ASCII equivalents ([OK], [FAIL], [...], [WARN]) to fix Windows cp1252 encoding error
  • test_context_retriever_precedents.py — moved assert_called_once_with inside with patch.object block; fixed assertion to use decision.scenario not decision.decision_id; removed "iPhone" (lowercase-first) from entity extraction assertion
  • test_real_world_scenarios.py — fixed duplicate source= keyword argument (renamed to label=); fixed cross-domain analysis loop to iterate over all social network users instead of only academic_users
  • test_pipeline_comprehensive.py — changed test_pipeline_validator_missing_deps to call validator.validate(builder) directly instead of builder.build() which raises ValidationError before validation can complete

  Results: ~840 tests passing, 36 skipped (external services), 0 failed

[0.3.0-alpha] - 2026-02-19

Added / Changed

• Decision Tracking System: Complete decision lifecycle management with audit trails and provenance tracking
• Advanced KG Algorithms: Node2Vec embeddings, centrality analysis, community detection for decision insights
• Enhanced Context Module: Unified AgentContext with granular feature flags and decision tracking integration
• Vector Store Features: Hybrid search combining semantic, structural, and category similarity
• Policy Management: Versioning, compliance checking, and exception handling
• Production Ready Architecture: Scalable design with comprehensive error handling and validation

Fixed

• Fixed import issues in test suite (ProvenanceTracker location fixes)
• Fixed causal analyzer validation (max_depth bounds checking)
• Fixed test compatibility with updated method signatures
• Fixed mock object setup in test suites
• Comprehensive test suite fixes for decision tracking features

Testing

• 113+ tests passing across context and core modules
• Comprehensive decision tracking test coverage
• Enhanced error handling and edge case testing
• Fixed all critical test failures for release readiness

Documentation

• Enhanced context module documentation

• Updated API references for decision tracking features

• Comprehensive usage guides and examples

• Fixed: Context Graphs decision tracking bugs and added comprehensive test coverage (PR #315 by @KaifAhmad1)

  • Fixed empty/None decision ID handling in ContextGraph.add_decision()
  • Fixed None metadata handling to prevent TypeError
  • Fixed causal chain depth logic and node exclusion
  • Fixed nonexistent node handling in add_causal_relationship()
  • Added missing properties field in to_dict serialization
  • Added missing from_dict method for graph deserialization
  • Fixed precedent search direction in find_precedents()
  • Fixed UUID generation logic in all decision models
  • Added comprehensive test suite with 9 tests covering all features
  • All 71 context tests now passing (100% success rate)

• Fixed: PolicyEngine latest version selection on ContextGraph; AgentContext fallback robustness and secure logging (PR #TBD by @KaifAhmad1)

• Tests: Added ContextGraph fallback and AgentContext smoke tests; full suite passing

  • Apache AGE Backend Security Fixes (PR #311 by @Sameer6305, fixes by @KaifAhmad1):
  • Added AgeStore class with GraphStore API compatibility
  • Fixed SQL injection vulnerabilities with input validation
  • Added psycopg2-binary dependency and migration guide
  • Fixed parameter replacement and test mock leakage
  • Enhanced error handling and Unicode display issues

• Context Engineering Enhancement (PR #307 by @KaifAhmad1):

  • Comprehensive decision tracking system with full lifecycle management (record → analyze → query → precedent → influence)
  • Advanced KG algorithm integration: centrality analysis, community detection, node embeddings with ContextGraph
  • Enhanced AgentContext with granular feature flags for decision tracking, KG algorithms, and vector store features
  • PolicyException model replacing conflicting Exception name for meaningful business domain modeling
  • GraphStore validation preventing runtime failures with explicit capability checking
  • Hybrid search combining semantic, structural, and category similarity with configurable weights
  • Decision influence analysis with centrality measures and causal chain tracking
  • Policy management with versioning, compliance checking, and exception handling
  • Production-ready architecture with audit trails, security, and scalability features
  • 9 critical bug fixes: logging, security, audit trails, API compatibility, Cypher queries, centrality access, validation, naming
  • Comprehensive documentation with usage guides, production examples, and API references
  • 100% test coverage with all validation tests passing (9/9 tests)
  • Enterprise-grade features for financial services, healthcare, legal, and business domains
  • Complete backward compatibility with existing semantica components
  • Performance optimizations: caching, indexing, and efficient graph operations

• Added PgVector Store Support (PR #303 by @Sameer6305, @KaifAhmad1):

  • Native PostgreSQL vector storage using pgvector extension with full integration
  • Multiple distance metrics: cosine, L2/Euclidean, inner product with automatic score normalization
  • Advanced indexing: HNSW and IVFFlat for approximate nearest neighbor search with tunable parameters
  • JSONB metadata storage with flexible filtering capabilities and batch operations
  • Connection pooling support with psycopg3/psycopg2 fallback and efficient resource management
  • Comprehensive VectorStore integration with backend delegation and unified API
  • Idempotent index creation and table management with safe migration support
  • Production-ready security: SQL injection protection with psycopg_sql.SQL() and input validation
  • Performance optimizations: UUID4-based IDs, batch executemany operations, connection pooling
  • Full backward compatibility with existing vector store implementations
  • 36+ comprehensive test cases with Docker integration and dependency skipping
  • Complete documentation with setup guides, examples, and performance tuning
  • CI/CD integration: resolved benchmark compatibility and fixed documentation links

• Improved Vector Store for Decision Tracking (PR #293 by @KaifAhmad1):

  • Comprehensive decision tracking capabilities with hybrid search combining semantic and structural embeddings
  • New DecisionEmbeddingPipeline for generating semantic and structural embeddings with KG algorithm integration
  • HybridSimilarityCalculator with configurable weights (semantic: 0.7, structural: 0.3)
  • DecisionContext high-level interface for decision management with explainable AI features
  • ContextRetriever with hybrid precedent search and multi-hop reasoning
  • User-friendly convenience API: quick_decision(), find_precedents(), explain(), similar_to(), batch_decisions(), filter_decisions()
  • Knowledge Graph algorithm integration: Node2Vec, PathFinder, CommunityDetector, CentralityCalculator, SimilarityCalculator, ConnectivityAnalyzer
  • Explainable AI with path tracing, confidence scoring, and comprehensive decision explanations
  • Performance optimizations: 0.028s per decision processing, 0.031s search performance, ~0.8KB per decision memory usage
  • 100% backward compatibility maintained with existing VectorStore functionality
  • 34+ comprehensive tests covering all functionality including end-to-end scenarios and performance benchmarks
  • Real-world validation examples for banking and insurance domains
  • Documentation with clear imports, examples, and API references

• Improved Graph Algorithms in KG Module (PR #292 by @KaifAhmad1):

  • Complete algorithm suite with 30+ graph algorithms across 7 categories
  • Node Embeddings: Node2Vec, DeepWalk, Word2Vec for structural similarity analysis
  • Similarity Analysis: Cosine, Euclidean, Manhattan, Correlation metrics with batch processing
  • Path Finding: Dijkstra, A*, BFS, K-shortest paths for route and network analysis
  • Link Prediction: Preferential attachment, Jaccard, Adamic-Adar for network completion
  • Centrality Analysis: Degree, Betweenness, Closeness, PageRank for importance ranking
  • Community Detection: Louvain, Leiden, Label propagation for clustering analysis
  • Connectivity Analysis: Components, bridges, density for network robustness
  • Unified provenance tracking system with GraphBuilderWithProvenance and AlgorithmTrackerWithProvenance
  • Complete execution tracking with metadata, timestamps, and reproducibility IDs
  • Comprehensive test coverage with 5 test suites and 40+ test methods
  • Professional documentation overhaul for all modules and reference documentation
  • Enterprise-ready functionality with error handling and NetworkX compatibility
  • Performance optimizations with sparse matrix operations and batch processing
  • Full backward compatibility maintained with gradual migration support

• Improved Security Configuration with Dependabot:

  • Configured bi-weekly security updates with manual review by @KaifAhmad1
  • Implemented automated security scans (Monday & Thursday at 7 AM IST) with Bandit, Safety, Semgrep
  • Added security-critical package grouping (cryptography, requests, urllib3, certifi, pyopenssl)
  • Enterprise-grade security with audit trail, compliance features, and zero auto-merge
  • Optimized IST timezone scheduling (Security scans: 7 AM IST, PRs: 9 AM IST)
  • Aligned with new Dependabot features: open-source proxy support, smart dependency grouping for Snowflake/Arrow/benchmark features, private registry support, semantic commit prefixes, and latest GitHub security best practices

• ResourceScheduler Deadlock Fix and Performance Improvements (PR #299, #301 by @d4ndr4d3, @KaifAhmad1):

  • Fixed critical deadlock in ResourceScheduler by replacing threading.Lock() with threading.RLock()
  • Resolved nested lock acquisition issue in allocate_resources() → allocate_cpu/memory/gpu() calls
  • Added allocation validation with ValidationError when no resources can be allocated
  • Improved performance by moving progress tracking updates outside lock scope
  • Implemented comprehensive resource cleanup on allocation failures to prevent leaks
  • Added complete regression test suite (6 tests) for deadlock prevention and edge cases
  • Improved error handling and documentation for better operator visibility
  • Zero breaking changes, maintains thread safety and backward compatibility

[0.2.7] - 2026-02-09

Added / Changed

• Snowflake Connector for Data Ingestion (PR #276 by @Sameer6305):

  • Native Snowflake connector with multi-authentication (password, OAuth, key-pair, SSO)
  • Table and query ingestion with pagination, schema introspection, batch processing
  • SQL injection prevention via identifier escaping, OAuth token validation
  • Progress tracking integration, context manager support, document export
  • 24 comprehensive unit tests with mocking, complete documentation and examples
  • Added as optional dependency db-snowflake with snowflake-connector-python>=3.0.0

• Apache Arrow Export Support (PR #273 by @Sameer6305):

  • Added Apache Arrow exporter with explicit schemas, entity/relationship export, compression support
  • Integrated with export module and method registry, Pandas/DuckDB compatible
  • 20 unit tests + 1 integration test, complete documentation with examples

• Comprehensive Benchmark Suite with Regression CLI (PR #289 by @ZohaibHassan16, @KaifAhmad1):

  • 137+ benchmarks across all 10 Semantica modules (Input, Core, Storage, Context, QA, Ontology, etc.)
  • Environment-agnostic design with robust mocking system for CI/CD compatibility
  • Statistical regression detection using Z-score analysis with configurable thresholds
  • Automated performance auditing via GitHub Actions workflow
  • Comprehensive documentation suite (benchmarks.md, architecture guides, usage examples)
  • Zero breaking changes, production-ready with ultra-fast text processing (>10,000 ops/s)
  • Added benchmark runner CLI: python benchmarks/benchmark_runner.py

[0.2.6] - 2026-02-03

Added / Changed

• W3C PROV-O Compliant Provenance Tracking (#254, #246):

  • Comprehensive provenance tracking system with W3C PROV-O compliance across all 17 Semantica modules
  • Core Module: ProvenanceManager, W3C PROV-O schemas, storage backends (InMemory, SQLite), SHA-256 integrity verification
  • Module Integrations: Semantic Extract, LLMs (Groq, OpenAI, HuggingFace, LiteLLM), Pipeline, Context, Ingest, Embeddings, Graph/Vector/Triplet stores, Reasoning, Conflicts, Deduplication, Export, Parse, Normalize, Ontology, Visualization
  • Features: Complete lineage tracking (Document → Chunk → Entity → Relationship → Graph), LLM tracking (tokens, costs, latency), source tracking, bridge axioms for domain transformations
  • Compliance Infrastructure: W3C PROV-O, FDA 21 CFR Part 11, SOX, HIPAA, TNFD
  • Testing: 237 tests covering core functionality, all 17 module integrations, edge cases, backward compatibility
  • Design: Opt-in with provenance=False by default, zero breaking changes, no new dependencies
  • Contributed by @KaifAhmad1

• Enhanced Change Management Module (#248, #243):

  • Enterprise-grade version control for knowledge graphs and ontologies with persistent storage and audit trails
  • Core Classes: TemporalVersionManager (KG versioning), OntologyVersionManager (ontology versioning), ChangeLogEntry (metadata)
  • Storage: SQLite (persistent) and in-memory backends with thread-safe operations
  • Features: SHA-256 checksums, detailed entity/relationship diffs, structural ontology comparison, email validation
  • Compliance Infrastructure: HIPAA, SOX, FDA 21 CFR Part 11 with immutable audit trails
  • Testing: 104 tests (100% pass) - unit, integration, compliance, performance, edge cases
  • Performance: 17.6ms for 10k entities, 510+ ops/sec concurrent, handles 5k+ entity graphs
  • Migration: Backward compatible, simplified class names, zero external dependencies
  • Contributed by @KaifAhmad1

• CSV Ingestion Enhancements (PR #244 by @saloni0318)

  • Auto-detect CSV encoding (chardet) and delimiter (csv.Sniffer)
  • Tolerant decoding and malformed-row handling (on_bad_lines='warn')
  • Optional chunked reading for large files; metadata tracks detected values
  • Expanded unit tests covering delimiters, quoted/multiline fields, header overrides, chunks, and NaN preservation

• Tests: Comprehensive units for TextNormalizer (PR #242 by @ZohaibHassan16)

  • Added focused test coverage for TextNormalizer behavior across inputs

• Tests: Register integration mark and tidy ingest test warnings (PR #241 by @KaifAhmad1)

  • Introduced integration test marker and reduced noisy warnings in ingest tests

• Ingest Unit Tests (#239, #232):

  • Comprehensive unit tests for ingestion modules (file, web, and feed ingestors)
  • Coverage: File scanning (local/cloud S3/GCS/Azure), web ingestion (URL/sitemap/robots.txt), RSS/Atom feed parsing
  • Testing: 998 lines of test code with mocked external dependencies for fast, isolated execution
  • Results: file_ingestor (86%), web_ingestor (86%), feed_ingestor (80%) coverage
  • Covers happy paths, edge cases, and error handling
  • Contributed by @Mohammed2372

Fixed

• Temperature Compatibility Fix (#256, #252):

  • Fixed hardcoded temperature=0.3 that broke compatibility with models requiring specific temperature values (e.g., gpt-5-mini)
  • Added _add_if_set helper method to BaseProvider that only passes parameters when explicitly set
  • When temperature=None, parameter is omitted allowing APIs to use model defaults
  • Updated all 5 providers: OpenAI, Groq, Gemini, Ollama, DeepSeek
  • Reduced code by ~85 lines with cleaner parameter handling
  • Comprehensive test coverage added (10 temperature tests, all passing)
  • Backward compatible - no breaking changes
  • Contributed by @F0rt1s and @IGES-Institut

• JenaStore Empty Graph Bug (#257, #258):

  • Fixed ProcessingError: Graph not initialized when operating on empty (but initialized) graphs
  • Replaced implicit if not self.graph: checks with explicit if self.graph is None: validation in 5 methods (add_triplets, get_triplets, delete_triplet, execute_sparql, serialize)
  • Properly distinguishes None (uninitialized) from empty graphs (initialized with 0 triplets)
  • Unblocks benchmarking suite, fresh deployments, and testing workflows
  • Contributed by @ZohaibHassan16

[0.2.5] - 2026-01-27

Added

• Pinecone Vector Store Support:
  • Implemented native Pinecone support (PineconeStore) with full CRUD capabilities.
  • Added support for serverless and pod-based indexes, namespaces, and metadata filtering.
  • Integrated with VectorStore unified interface and registry.
  • (Closes #219, Resolves #220)
• Configurable LLM Retry Logic:
  • Exposed max_retries parameter in NERExtractor, RelationExtractor, TripletExtractor and low-level extraction methods (extract_entities_llm, extract_relations_llm, extract_triplets_llm).
  • Defaults to 3 retries to prevent infinite loops during JSON validation failures or API timeouts.
  • Propagated retry configuration through chunked processing helpers to ensure consistent behavior for long documents.
  • Updated 03_Earnings_Call_Analysis.ipynb to use max_retries=3 by default.

Added

• Bring Your Own Model (BYOM) Support:
  • Enabled full support for custom Hugging Face models in NERExtractor, RelationExtractor, and TripletExtractor.
  • Added support for custom tokenizers in HuggingFaceModelLoader to handle models with non-standard tokenization requirements.
  • Implemented robust fallback logic for model selection: runtime options (extract(model=...)) now correctly override configuration defaults.
• Enhanced NER Implementation:
  • Added configurable aggregation strategies (simple, first, average, max) to extract_entities_huggingface for better sub-word token handling.
  • Implemented robust IOB/BILOU parsing to reconstruct entities from raw model outputs when structured output is unavailable.
  • Added confidence scoring for aggregated entities.
• Relation Extraction Improvements:
  • Implemented standard entity marker technique (wrapping subject/object with <subj>, <obj> tags) in extract_relations_huggingface for compatibility with sequence classification models.
  • Added structured output parsing to convert raw model predictions into validated Relation objects.
• Triplet Extraction Completion:
  • Added specialized parsing for Seq2Seq models (e.g., REBEL) in extract_triplets_huggingface to generate structured triplets directly from text.
  • Implemented post-processing logic to clean and validate generated triplets.

Fixed

• LLM Extraction Stability:
  • Fixed infinite retry loops in BaseProvider by strictly enforcing max_retries limit during structured output generation.
  • Resolved stuck execution in earnings call analysis notebooks when using smaller models (e.g., Llama 3 8B) that frequently produce invalid JSON.
• Model Parameter Precedence:
  • Fixed issue where configuration defaults took precedence over runtime arguments in Hugging Face extractors. Runtime options now correctly override config values.
• Import Handling:
  • Fixed circular import issues in test suites by implementing robust mocking strategies.

[0.2.4] - 2026-01-22

Added

• Ontology Ingestion Module:
  • Implemented OntologyIngestor in semantica.ingest for parsing RDF/OWL files (Turtle, RDF/XML, JSON-LD, N3) into standardized OntologyData objects.
  • Added ingest_ontology convenience function and integrated it into the unified ingest(source_type="ontology") interface.
  • Added recursive directory scanning support for batch ontology ingestion.
  • Exposed ingestion tools in semantica.ontology for better discoverability.
  • Added OntologyData dataclass for consistent metadata handling (source path, format, timestamps).
• Documentation:
  • Ontology Usage Guide: Updated ontology_usage.md with comprehensive examples for single-file and directory ingestion.
  • API Reference: Updated ontology.md with OntologyIngestor class documentation and method details.
• Tests:
  • Comprehensive Test Suite: Added tests/ingest/test_ontology_ingestor.py covering all supported formats, error handling, and unified interface integration.
  • Demo Script: Added examples/demo_ontology_ingest.py for end-to-end usage demonstration.

[0.2.3] - 2026-01-20

Fixed

• LLM Relation Extraction Parsing:
  • Fixed relation extraction returning zero relations despite successful API calls to Groq and other providers
  • Normalized typed responses from instructor/OpenAI/Groq to consistent dict format before parsing
  • Added structured JSON fallback when typed generation yields zero relations to avoid silent empty outputs
  • Removed acceptance of extra kwargs (max_tokens, max_entities_prompt) from relation extraction internals
  • Filtered kwargs passed to provider LLM calls to only temperature and verbose
• API Parameter Handling:
  • Limited kwargs forwarded in chunked extraction helper to prevent parameter leakage
  • Ensured minimal, safe parameters are passed to provider calls
• Pipeline Circular Import (Issues #192, #193):
  • Fixed circular import between pipeline_builder and pipeline_validator triggered during semantica.pipeline import
  • Lazy-loaded PipelineValidator inside PipelineBuilder.__init__ and guarded type hints with TYPE_CHECKING
  • Ensured from semantica.deduplication import DuplicateDetector no longer fails even when pipeline module is imported
• JupyterLab Progress Output (Issue #181):
  • Added SEMANTICA_DISABLE_JUPYTER_PROGRESS environment variable to disable rich Jupyter/Colab progress tables
  • When enabled, progress falls back to console-style output, preventing infinite scrolling and JupyterLab out-of-memory errors

Added

• Comprehensive Test Suite:
• • Added unit tests (tests/test_relations_llm.py) with mocked LLM provider covering both typed and structured response paths
• • Added integration tests (tests/integration/test_relations_groq.py) for real Groq API calls with environment variable API key
• • Tests validate relation extraction completion and result parsing across different response formats
• Amazon Neptune Dev Environment:
• • Added CloudFormation template (cookbook/introduction/neptune-setup.yaml) to provision a dev Neptune cluster with public endpoint and IAM auth enabled
• • Documented deployment, cost estimates, and IAM User vs IAM Role best practices in cookbook/introduction/21_Amazon_Neptune_Store.ipynb
• • Added cfn-lint to .pre-commit-config.yaml for validating CloudFormation templates while excluding neptune-setup.yaml from generic YAML linters
• Vector Store High-Performance Ingestion:
• • Added VectorStore.add_documents for high-throughput ingestion with automatic embedding generation, batching, and parallel processing
• • Added VectorStore.embed_batch helper for generating embeddings for lists of texts without immediately storing them
• • Enabled default parallel ingestion in VectorStore with max_workers=6 for common workloads
• • Added dedicated documentation page docs/vector_store_usage.md describing high-performance vector store usage and configuration
• • Added tests/vector_store/test_vector_store_parallel.py covering parallel vs sequential performance, error handling, and edge cases for add_documents and embed_batch

Changed

• Relation Extraction API:
• • Simplified parameter interface by removing unused kwargs that were previously ignored
• • Improved error handling and verbose logging for debugging relation extraction issues
• • Enhanced robustness of post-response parsing across different LLM providers
• Vector Store Defaults and Examples:
• • Standardized VectorStore default concurrency to max_workers=6 for parallel ingestion
• • Updated vector store reference documentation and usage guides to rely on implicit defaults instead of requiring manual max_workers configuration in examples

[0.2.2] - 2026-01-15

Added

• Parallel Extraction Engine:
  • Implemented high-throughput parallel batch processing across all core extractors (NERExtractor, RelationExtractor, TripletExtractor, EventDetector, SemanticNetworkExtractor) using concurrent.futures.ThreadPoolExecutor.
  • Added max_workers configuration parameter (default: 1) to all extractor extract() methods, allowing users to tune concurrency based on available CPU cores or API rate limits.
  • Parallel Chunking: Implemented parallel processing for large document chunking in _extract_entities_chunked and _extract_relations_chunked, significantly reducing latency for long-form text analysis.
  • Thread-Safe Progress Tracking: Enhanced ProgressTracker to handle concurrent updates from multiple threads without race conditions during batch processing.
• Semantic Extract Performance & Regression:
  • Added edge-case regression suite covering max worker defaults, LLM prompt entity filtering, and extractor reuse.
  • Added a runnable real-use-case benchmark script for batch latency across NERExtractor, RelationExtractor, TripletExtractor, EventDetector, SemanticAnalyzer, and SemanticNetworkExtractor.
  • Added Groq LLM smoke tests that exercise LLM-based entities/relations/triplets when GROQ_API_KEY is available via environment configuration.

Security

• Credential Sanitization:
  • Removed hardcoded API keys from 8 cookbook notebooks to prevent secret leakage.
  • Enforced environment variable usage for GROQ_API_KEY across all examples.
• Secure Caching:
  • Updated ExtractionCache to exclude sensitive parameters (e.g., api_key, token, password) from cache key generation, preventing secret leakage and enabling safe cache sharing.
  • Upgraded cache key hashing algorithm from MD5 to SHA-256 for enhanced collision resistance and security.

Changed

• Gemini SDK Migration:
  • Migrated GeminiProvider to use the new google-genai SDK (v0.1.0+) to address deprecation warnings.
  • Implemented graceful fallback to google.generativeai for backward compatibility.
• Dependency Resolution:
  • Pinned opentelemetry-api and opentelemetry-sdk to 1.37.0 to resolve pip conflicts.
  • Updated protobuf and grpcio constraints for better stability.
• Entity Filtering Scope:
  • Removed entity filtering from non-LLM extraction flows to avoid accuracy regressions.
  • Applied entity downselection only to LLM relation prompt construction, while matching returned entities against the full original entity list.
• Batch Concurrency Defaults:
  • Standardized max_workers defaulting across semantic_extract and tuned for low-latency: ML-backed methods default to single-worker, while pattern/regex/rules/LLM/huggingface methods use a higher parallelism default capped by CPU.
  • Raised the global optimization.max_workers default to 8 for better throughput on batch workloads.

Performance

• Bottleneck Optimization (GitHub Issue #186):
  • Resolved Bottleneck #1 (Sequential Processing): Replaced sequential for loops with parallel execution for both document-level batches and intra-document chunks.
  • Performance Gains: Achieved ~1.89x speedup in real-world extraction scenarios (tested with Groq llama-3.3-70b-versatile on standard datasets).
  • Initialization Optimization: Refactored test suite to use class-level setUpClass for LLM provider initialization, eliminating redundant API client creation overhead.
• Low-Latency Entity Matching:
  • Avoided heavyweight embedding stack imports on common matches by improving fast matching heuristics and short-circuiting before embedding similarity.
  • Optimized entity matching to prioritize exact/substring/word-boundary matches and only fall back to embedding similarity when needed, reducing CPU overhead in LLM relation/triplet mapping.

[0.2.1] - 2026-01-12

Fixed

• LLM Output Stability (Bug #176):
  • Fixed incomplete JSON output issues by correctly propagating max_tokens parameter in extract_relations_llm.
  • Implemented automatic error handling that halves chunk sizes and retries when LLM context or output limits are exceeded.
  • Fixed AttributeError in provider integration by ensuring consistent parameter passing via **kwargs.
• Constraint Relaxations:
  • Removed hardcoded max_length constraints from Entity, Relation, and Triplet classes to support long-form semantic extraction (e.g., long descriptions or names).
• Fixed orchestrator lazy property initialization and configuration normalization logic in Orchestrator.
• Resolved AssertionError in orchestrator tests by aligning test mocks with production component usage.
• Fixed dependency compatibility issues by pinning protobuf>=5.29.1,<7.0 and grpcio>=1.71.2.
• Added missing dependencies GitPython and chardet to pyproject.toml.
• Verified and aligned FileObject.text property usage in GraphRAG notebooks for consistent content decoding.

Changed

• Chunking Defaults:
  • Increased default max_text_length for auto-chunking to 64,000 characters (from 32k/16k) for OpenAI, Anthropic, Gemini, Groq, and DeepSeek providers.
  • Unified chunking logic across extract_entities_llm, extract_relations_llm, and extract_triplets_llm.
• Groq Support:
  • Standardized Groq provider defaults to use llama-3.3-70b-versatile with a 64k context window.
  • Added native support for max_tokens and max_completion_tokens to prevent output truncation.

Added

• Testing:
  • Added tests/reproduce_issue_176.py to validate max_tokens propagation and chunking behavior across all extractors.

[0.2.0] - 2026-01-10

Added

• Amazon Neptune Support:
  • Added AmazonNeptuneStore providing Amazon Neptune graph database integration via Bolt protocol and OpenCypher.
  • Implemented NeptuneAuthTokenManager extending Neo4j AuthManager for AWS IAM SigV4 signing with automatic token refresh.
  • Added robust connection handling: retry logic with backoff for transient errors (signature expired, connection closed) and driver recreation.
  • Added graph-amazon-neptune optional dependency group (boto3, neo4j).
  • Comprehensive test suite covering all GraphStore interface methods.
• Docling Integration:
  • Added DoclingParser in semantica.parse for high-fidelity document parsing using the Docling library.
  • Supports multi-format parsing (PDF, DOCX, PPTX, XLSX, HTML, images) with superior table extraction and structure understanding.
  • Implemented as a standalone parser supporting local execution, OCR, and multiple export formats (Markdown, HTML, JSON).
• Robust Extraction Fallbacks:
  • Implemented comprehensive fallback chains ("ML/LLM" -> "Pattern" -> "Last Resort") across NERExtractor, RelationExtractor, and TripletExtractor to prevent empty result lists.
  • Added "Last Resort" pattern matching in NERExtractor to identify capitalized words as generic entities when all other methods fail.
  • Added "Last Resort" adjacency-based relation extraction in RelationExtractor to create weak connections between adjacent entities if no relations are found.
  • Added fallback logic in TripletExtractor to convert relations to triplets or use rule-based extraction if standard methods fail.
• Provenance & Tracking:
  • Added count tracking to batch processing logs in NERExtractor, RelationExtractor, and TripletExtractor.
  • Added batch_index and document_id to the metadata of all extracted entities, relations, triplets, semantic roles, and clusters for better traceability.
• Semantic Extract Improvements:
  • Introduced auto-chunking for long text processing in LLM extraction methods (extract_entities_llm, extract_relations_llm, extract_triplets_llm).
  • Added silent_fail parameter to LLM extraction methods for configurable error handling.
  • Implemented robust JSON parsing and automatic retry logic (3 attempts with exponential backoff) in BaseProvider for all LLM providers.
  • Enhanced GroqProvider with better diagnostics and connectivity testing.
  • Added comprehensive entity, relation, and triplet deduplication for chunked extraction.
  • Added semantica/semantic_extract/schemas.py with canonical Pydantic models for consistent structured output.
• Testing:
  • Added comprehensive robustness test suite tests/semantic_extract/test_robustness_fallback.py for validating extraction fallbacks and metadata propagation.
  • Added comprehensive unit test suite tests/embeddings/test_model_switching.py for verifying dynamic model transitions and dimension updates.
  • Added end-to-end integration test suite for Knowledge Graph pipeline validation (GraphBuilder -> EntityResolver -> GraphAnalyzer).
• Other:
  • Added missing dependencies GitPython and chardet to pyproject.toml.
  • Robustified ID extraction across CentralityCalculator, CommunityDetector, and ConnectivityAnalyzer to handle various entity formats.
  • Improved Entity class hashability and equality logic in utils/types.py.

Changed

• Deduplication & Conflict Logic:
  • Removed internal deduplication logic from NERExtractor, RelationExtractor, and TripletExtractor.
  • Removed consistency/conflict checking from ExtractionValidator to defer to dedicated semantica/conflicts module.
  • Removed _deduplicate_* methods from semantica/semantic_extract/methods.py.
• Batch Processing & Consistency:
  • Standardized batch processing across all extractors (NERExtractor, RelationExtractor, TripletExtractor, SemanticNetworkExtractor, EventDetector, SemanticAnalyzer, CoreferenceResolver) using a unified extract/analyze/resolve method pattern with progress tracking.
  • Added provenance metadata (batch_index, document_id) to SemanticNetwork nodes/edges, Event objects, SemanticRole results, CoreferenceChain mentions, and SemanticCluster (tracking source document_ids).
  • Updated SemanticClusterer.cluster and SemanticAnalyzer.cluster_semantically to accept list of dictionaries (with content and id keys) for better document tracking during clustering.
  • Removed legacy check_triplet_consistency from TripletExtractor.
  • Removed validate_consistency and _check_consistency from ExtractionValidator.
• Weighted Scoring:
  • Clarified weighted confidence scoring (50% Method Confidence + 50% Type Similarity) in comments.
  • Explicitly labeled "Type Similarity" as "user-provided" in code comments to remove ambiguity.
• Refactoring:
  • Fixed orchestrator lazy property initialization and configuration normalization logic in Orchestrator.
  • Verified and aligned FileObject.text property usage in GraphRAG notebooks for consistent content decoding.

Fixed

• Critical Fixes:
  • Resolved NameError in extraction_validator.py by adding missing Union import.
  • Resolved issues where extractors would return empty lists for valid input text when primary extraction methods failed.
  • Fixed metadata initialization issue in batch processing where batch_index and document_id were occasionally missing from extracted items.
  • Ensured LLMExtraction methods (enhance_entities, enhance_relations) return original input instead of failing or returning empty results when LLM providers are unavailable.
• Component Fixes:
  • Fixed model switching bug in TextEmbedder where internal state was not cleared, preventing dynamic updates between fastembed and sentence_transformers (#160).
  • Implemented model-intrinsic embedding dimension detection in TextEmbedder to ensure consistency between models and vector databases.
  • Updated set_model to properly refresh configuration and dimensions during model switches.
  • Fixed TypeError: unhashable type: 'Entity' in GraphAnalyzer when processing graphs with raw Entity objects or dictionaries in relationships (#159).
  • Resolved AssertionError in orchestrator tests by aligning test mocks with production component usage.
  • Fixed dependency compatibility issues by pinning protobuf==4.25.3 and grpcio==1.67.1.
  • Fixed a bug in TripletExtractor where the validate_triplets method was shadowed by an internal attribute.
  • Fixed incorrect TextSplitter import path in the semantic_extract.methods module.

[0.1.1] - 2026-01-05

Added

• Exported DoclingParser and DoclingMetadata from semantica.parse for easier access.
• Added comprehensive DoclingParser usage examples to README and documentation.
• Added Windows-specific troubleshooting note for PyTorch DLL issues.

Fixed

• Fixed DoclingParser import/export issues across platforms (Windows, Linux, Google Colab).
• Improved error messaging when optional docling dependency is missing.
• Fixed versioning inconsistencies across the framework.

[0.1.0] - 2025-12-31

Added

• New command-line interface (semantica CLI) with support for knowledge base building and info commands.
• Integrated FastAPI-based REST API server for remote access to framework functionality.
• Dedicated background worker component for scalable task processing and pipeline execution.
• Framework-level versioning configuration for PyPI distribution.
• Automated release workflow with Trusted Publishing support.

Changed

• Updated versioning across the framework to 0.1.0.
• Refined entry point configurations in pyproject.toml.
• Improved lazy module loading for core framework components.

[0.0.5] - 2025-11-26

Changed

• Configured Trusted Publishing for secure automated PyPI deployments

[0.0.4] - 2025-11-26

Changed

• Fixed PyPI deployment issues from v0.0.3

[0.0.3] - 2025-11-25

Changed

• Simplified CI/CD workflows - removed failing tests and strict linting
• Combined release and PyPI publishing into single workflow
• Simplified security scanning to weekly pip-audit only
• Streamlined GitHub Actions configuration

Added

• Comprehensive issue templates (Bug, Feature, Documentation, Support, Grant/Partnership)
• Updated pull request template with clear guidelines
• Community support documentation (SUPPORT.md)
• Funding and sponsorship configuration (FUNDING.yml)
• GitHub configuration README for maintainers
• 10+ new domain-specific cookbook examples (Finance, Healthcare, Cybersecurity, etc.)

Removed

• Redundant scripts folder (8 shell/PowerShell scripts)
• Unnecessary automation workflows (label-issues, mark-answered)
• Excessive issue templates

[0.0.2] - 2025-11-25

Changed

• Updated README with streamlined content and better examples
• Added more notebooks to cookbook
• Improved documentation structure

[0.0.1] - 2024-01-XX

Added

• Core framework architecture
• Universal data ingestion (multiple file formats)
• Semantic intelligence engine (NER, relation extraction, event detection)
• Knowledge graph construction with entity resolution
• 6-stage ontology generation pipeline
• GraphRAG engine for hybrid retrieval
• Multi-agent system infrastructure
• Production-ready quality assurance modules
• Comprehensive documentation with MkDocs
• Cookbook with interactive tutorials
• Support for multiple vector stores (Weaviate, Qdrant, FAISS)
• Support for multiple graph databases (Neo4j, NetworkX, RDFLib)
• Temporal knowledge graph support
• Conflict detection and resolution
• Deduplication and entity merging
• Schema template enforcement
• Seed data management
• Multi-format export (RDF, JSON-LD, CSV, GraphML)
• Visualization tools
• Pipeline orchestration
• Streaming support (Kafka, RabbitMQ, Kinesis)
• Context engineering for AI agents
• Reasoning and inference engine

Documentation

• Getting started guide
• API reference for all modules
• Concepts and architecture documentation
• Use case examples
• Cookbook tutorials
• Community projects showcase

---

Types of Changes

• Added for new features
• Changed for changes in existing functionality
• Deprecated for soon-to-be removed features
• Removed for now removed features
• Fixed for any bug fixes
• Security for vulnerability fixes

Migration Guides

When breaking changes are introduced, migration guides will be provided in the release notes and documentation.

---

For detailed release notes, see GitHub Releases.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

• Fixed: PolicyEngine latest version selection on ContextGraph; AgentContext fallback robustness and secure logging (PR #TBD by @KaifAhmad1)

• Tests: Added ContextGraph fallback and AgentContext smoke tests; full suite passing

• Context Engineering Enhancement (PR #307 by @KaifAhmad1):

  • Comprehensive decision tracking system with full lifecycle management (record → analyze → query → precedent → influence)
  • Advanced KG algorithm integration: centrality analysis, community detection, node embeddings with ContextGraph
  • Enhanced AgentContext with granular feature flags for decision tracking, KG algorithms, and vector store features
  • PolicyException model replacing conflicting Exception name for meaningful business domain modeling
  • GraphStore validation preventing runtime failures with explicit capability checking
  • Hybrid search combining semantic, structural, and category similarity with configurable weights
  • Decision influence analysis with centrality measures and causal chain tracking
  • Policy management with versioning, compliance checking, and exception handling
  • Production-ready architecture with audit trails, security, and scalability features
  • 9 critical bug fixes: logging, security, audit trails, API compatibility, Cypher queries, centrality access, validation, naming
  • Comprehensive documentation with usage guides, production examples, and API references
  • 100% test coverage with all validation tests passing (9/9 tests)
  • Enterprise-grade features for financial services, healthcare, legal, and business domains
  • Complete backward compatibility with existing semantica components
  • Performance optimizations: caching, indexing, and efficient graph operations

• Added PgVector Store Support (PR #303 by @Sameer6305, @KaifAhmad1):

  • Native PostgreSQL vector storage using pgvector extension with full integration
  • Multiple distance metrics: cosine, L2/Euclidean, inner product with automatic score normalization
  • Advanced indexing: HNSW and IVFFlat for approximate nearest neighbor search with tunable parameters
  • JSONB metadata storage with flexible filtering capabilities and batch operations
  • Connection pooling support with psycopg3/psycopg2 fallback and efficient resource management
  • Comprehensive VectorStore integration with backend delegation and unified API
  • Idempotent index creation and table management with safe migration support
  • Production-ready security: SQL injection protection with psycopg_sql.SQL() and input validation
  • Performance optimizations: UUID4-based IDs, batch executemany operations, connection pooling
  • Full backward compatibility with existing vector store implementations
  • 36+ comprehensive test cases with Docker integration and dependency skipping
  • Complete documentation with setup guides, examples, and performance tuning
  • CI/CD integration: resolved benchmark compatibility and fixed documentation links

• Improved Vector Store for Decision Tracking (PR #293 by @KaifAhmad1):

  • Comprehensive decision tracking capabilities with hybrid search combining semantic and structural embeddings
  • New DecisionEmbeddingPipeline for generating semantic and structural embeddings with KG algorithm integration
  • HybridSimilarityCalculator with configurable weights (semantic: 0.7, structural: 0.3)
  • DecisionContext high-level interface for decision management with explainable AI features
  • ContextRetriever with hybrid precedent search and multi-hop reasoning
  • User-friendly convenience API: quick_decision(), find_precedents(), explain(), similar_to(), batch_decisions(), filter_decisions()
  • Knowledge Graph algorithm integration: Node2Vec, PathFinder, CommunityDetector, CentralityCalculator, SimilarityCalculator, ConnectivityAnalyzer
  • Explainable AI with path tracing, confidence scoring, and comprehensive decision explanations
  • Performance optimizations: 0.028s per decision processing, 0.031s search performance, ~0.8KB per decision memory usage
  • 100% backward compatibility maintained with existing VectorStore functionality
  • 34+ comprehensive tests covering all functionality including end-to-end scenarios and performance benchmarks
  • Real-world validation examples for banking and insurance domains
  • Documentation with clear imports, examples, and API references

• Improved Graph Algorithms in KG Module (PR #292 by @KaifAhmad1):

  • Complete algorithm suite with 30+ graph algorithms across 7 categories
  • Node Embeddings: Node2Vec, DeepWalk, Word2Vec for structural similarity analysis
  • Similarity Analysis: Cosine, Euclidean, Manhattan, Correlation metrics with batch processing
  • Path Finding: Dijkstra, A*, BFS, K-shortest paths for route and network analysis
  • Link Prediction: Preferential attachment, Jaccard, Adamic-Adar for network completion
  • Centrality Analysis: Degree, Betweenness, Closeness, PageRank for importance ranking
  • Community Detection: Louvain, Leiden, Label propagation for clustering analysis
  • Connectivity Analysis: Components, bridges, density for network robustness
  • Unified provenance tracking system with GraphBuilderWithProvenance and AlgorithmTrackerWithProvenance
  • Complete execution tracking with metadata, timestamps, and reproducibility IDs
  • Comprehensive test coverage with 5 test suites and 40+ test methods
  • Professional documentation overhaul for all modules and reference documentation
  • Enterprise-ready functionality with error handling and NetworkX compatibility
  • Performance optimizations with sparse matrix operations and batch processing
  • Full backward compatibility maintained with gradual migration support

• Improved Security Configuration with Dependabot:

  • Configured bi-weekly security updates with manual review by @KaifAhmad1
  • Implemented automated security scans (Monday & Thursday at 7 AM IST) with Bandit, Safety, Semgrep
  • Added security-critical package grouping (cryptography, requests, urllib3, certifi, pyopenssl)
  • Enterprise-grade security with audit trail, compliance features, and zero auto-merge
  • Optimized IST timezone scheduling (Security scans: 7 AM IST, PRs: 9 AM IST)
  • Aligned with new Dependabot features: open-source proxy support, smart dependency grouping for Snowflake/Arrow/benchmark features, private registry support, semantic commit prefixes, and latest GitHub security best practices

• ResourceScheduler Deadlock Fix and Performance Improvements (PR #299, #301 by @d4ndr4d3, @KaifAhmad1):

  • Fixed critical deadlock in ResourceScheduler by replacing threading.Lock() with threading.RLock()
  • Resolved nested lock acquisition issue in allocate_resources() → allocate_cpu/memory/gpu() calls
  • Added allocation validation with ValidationError when no resources can be allocated
  • Improved performance by moving progress tracking updates outside lock scope
  • Implemented comprehensive resource cleanup on allocation failures to prevent leaks
  • Added complete regression test suite (6 tests) for deadlock prevention and edge cases
  • Improved error handling and documentation for better operator visibility
  • Zero breaking changes, maintains thread safety and backward compatibility

[0.2.7] - 2026-02-09

Added / Changed

• Snowflake Connector for Data Ingestion (PR #276 by @Sameer6305):

  • Native Snowflake connector with multi-authentication (password, OAuth, key-pair, SSO)
  • Table and query ingestion with pagination, schema introspection, batch processing
  • SQL injection prevention via identifier escaping, OAuth token validation
  • Progress tracking integration, context manager support, document export
  • 24 comprehensive unit tests with mocking, complete documentation and examples
  • Added as optional dependency db-snowflake with snowflake-connector-python>=3.0.0

• Apache Arrow Export Support (PR #273 by @Sameer6305):

  • Added Apache Arrow exporter with explicit schemas, entity/relationship export, compression support
  • Integrated with export module and method registry, Pandas/DuckDB compatible
  • 20 unit tests + 1 integration test, complete documentation with examples

• Comprehensive Benchmark Suite with Regression CLI (PR #289 by @ZohaibHassan16, @KaifAhmad1):

  • 137+ benchmarks across all 10 Semantica modules (Input, Core, Storage, Context, QA, Ontology, etc.)
  • Environment-agnostic design with robust mocking system for CI/CD compatibility
  • Statistical regression detection using Z-score analysis with configurable thresholds
  • Automated performance auditing via GitHub Actions workflow
  • Comprehensive documentation suite (benchmarks.md, architecture guides, usage examples)
  • Zero breaking changes, production-ready with ultra-fast text processing (>10,000 ops/s)
  • Added benchmark runner CLI: python benchmarks/benchmark_runner.py

[0.2.6] - 2026-02-03

Added / Changed

• W3C PROV-O Compliant Provenance Tracking (#254, #246):

  • Comprehensive provenance tracking system with W3C PROV-O compliance across all 17 Semantica modules
  • Core Module: ProvenanceManager, W3C PROV-O schemas, storage backends (InMemory, SQLite), SHA-256 integrity verification
  • Module Integrations: Semantic Extract, LLMs (Groq, OpenAI, HuggingFace, LiteLLM), Pipeline, Context, Ingest, Embeddings, Graph/Vector/Triplet stores, Reasoning, Conflicts, Deduplication, Export, Parse, Normalize, Ontology, Visualization
  • Features: Complete lineage tracking (Document → Chunk → Entity → Relationship → Graph), LLM tracking (tokens, costs, latency), source tracking, bridge axioms for domain transformations
  • Compliance Infrastructure: W3C PROV-O, FDA 21 CFR Part 11, SOX, HIPAA, TNFD
  • Testing: 237 tests covering core functionality, all 17 module integrations, edge cases, backward compatibility
  • Design: Opt-in with provenance=False by default, zero breaking changes, no new dependencies
  • Contributed by @KaifAhmad1

• Enhanced Change Management Module (#248, #243):

  • Enterprise-grade version control for knowledge graphs and ontologies with persistent storage and audit trails
  • Core Classes: TemporalVersionManager (KG versioning), OntologyVersionManager (ontology versioning), ChangeLogEntry (metadata)
  • Storage: SQLite (persistent) and in-memory backends with thread-safe operations
  • Features: SHA-256 checksums, detailed entity/relationship diffs, structural ontology comparison, email validation
  • Compliance Infrastructure: HIPAA, SOX, FDA 21 CFR Part 11 with immutable audit trails
  • Testing: 104 tests (100% pass) - unit, integration, compliance, performance, edge cases
  • Performance: 17.6ms for 10k entities, 510+ ops/sec concurrent, handles 5k+ entity graphs
  • Migration: Backward compatible, simplified class names, zero external dependencies
  • Contributed by @KaifAhmad1

• CSV Ingestion Enhancements (PR #244 by @saloni0318)

  • Auto-detect CSV encoding (chardet) and delimiter (csv.Sniffer)
  • Tolerant decoding and malformed-row handling (on_bad_lines='warn')
  • Optional chunked reading for large files; metadata tracks detected values
  • Expanded unit tests covering delimiters, quoted/multiline fields, header overrides, chunks, and NaN preservation

• Tests: Comprehensive units for TextNormalizer (PR #242 by @ZohaibHassan16)

  • Added focused test coverage for TextNormalizer behavior across inputs

• Tests: Register integration mark and tidy ingest test warnings (PR #241 by @KaifAhmad1)

  • Introduced integration test marker and reduced noisy warnings in ingest tests

• Ingest Unit Tests (#239, #232):

  • Comprehensive unit tests for ingestion modules (file, web, and feed ingestors)
  • Coverage: File scanning (local/cloud S3/GCS/Azure), web ingestion (URL/sitemap/robots.txt), RSS/Atom feed parsing
  • Testing: 998 lines of test code with mocked external dependencies for fast, isolated execution
  • Results: file_ingestor (86%), web_ingestor (86%), feed_ingestor (80%) coverage
  • Covers happy paths, edge cases, and error handling
  • Contributed by @Mohammed2372

Fixed

• Temperature Compatibility Fix (#256, #252):

  • Fixed hardcoded temperature=0.3 that broke compatibility with models requiring specific temperature values (e.g., gpt-5-mini)
  • Added _add_if_set helper method to BaseProvider that only passes parameters when explicitly set
  • When temperature=None, parameter is omitted allowing APIs to use model defaults
  • Updated all 5 providers: OpenAI, Groq, Gemini, Ollama, DeepSeek
  • Reduced code by ~85 lines with cleaner parameter handling
  • Comprehensive test coverage added (10 temperature tests, all passing)
  • Backward compatible - no breaking changes
  • Contributed by @F0rt1s and @IGES-Institut

• JenaStore Empty Graph Bug (#257, #258):

  • Fixed ProcessingError: Graph not initialized when operating on empty (but initialized) graphs
  • Replaced implicit if not self.graph: checks with explicit if self.graph is None: validation in 5 methods (add_triplets, get_triplets, delete_triplet, execute_sparql, serialize)
  • Properly distinguishes None (uninitialized) from empty graphs (initialized with 0 triplets)
  • Unblocks benchmarking suite, fresh deployments, and testing workflows
  • Contributed by @ZohaibHassan16

[0.2.5] - 2026-01-27

Added

• Pinecone Vector Store Support:
  • Implemented native Pinecone support (PineconeStore) with full CRUD capabilities.
  • Added support for serverless and pod-based indexes, namespaces, and metadata filtering.
  • Integrated with VectorStore unified interface and registry.
  • (Closes #219, Resolves #220)
• Configurable LLM Retry Logic:
  • Exposed max_retries parameter in NERExtractor, RelationExtractor, TripletExtractor and low-level extraction methods (extract_entities_llm, extract_relations_llm, extract_triplets_llm).
  • Defaults to 3 retries to prevent infinite loops during JSON validation failures or API timeouts.
  • Propagated retry configuration through chunked processing helpers to ensure consistent behavior for long documents.
  • Updated 03_Earnings_Call_Analysis.ipynb to use max_retries=3 by default.

Added

• Bring Your Own Model (BYOM) Support:
  • Enabled full support for custom Hugging Face models in NERExtractor, RelationExtractor, and TripletExtractor.
  • Added support for custom tokenizers in HuggingFaceModelLoader to handle models with non-standard tokenization requirements.
  • Implemented robust fallback logic for model selection: runtime options (extract(model=...)) now correctly override configuration defaults.
• Enhanced NER Implementation:
  • Added configurable aggregation strategies (simple, first, average, max) to extract_entities_huggingface for better sub-word token handling.
  • Implemented robust IOB/BILOU parsing to reconstruct entities from raw model outputs when structured output is unavailable.
  • Added confidence scoring for aggregated entities.
• Relation Extraction Improvements:
  • Implemented standard entity marker technique (wrapping subject/object with <subj>, <obj> tags) in extract_relations_huggingface for compatibility with sequence classification models.
  • Added structured output parsing to convert raw model predictions into validated Relation objects.
• Triplet Extraction Completion:
  • Added specialized parsing for Seq2Seq models (e.g., REBEL) in extract_triplets_huggingface to generate structured triplets directly from text.
  • Implemented post-processing logic to clean and validate generated triplets.

Fixed

• LLM Extraction Stability:
  • Fixed infinite retry loops in BaseProvider by strictly enforcing max_retries limit during structured output generation.
  • Resolved stuck execution in earnings call analysis notebooks when using smaller models (e.g., Llama 3 8B) that frequently produce invalid JSON.
• Model Parameter Precedence:
  • Fixed issue where configuration defaults took precedence over runtime arguments in Hugging Face extractors. Runtime options now correctly override config values.
• Import Handling:
  • Fixed circular import issues in test suites by implementing robust mocking strategies.

[0.2.4] - 2026-01-22

Added

• Ontology Ingestion Module:
  • Implemented OntologyIngestor in semantica.ingest for parsing RDF/OWL files (Turtle, RDF/XML, JSON-LD, N3) into standardized OntologyData objects.
  • Added ingest_ontology convenience function and integrated it into the unified ingest(source_type="ontology") interface.
  • Added recursive directory scanning support for batch ontology ingestion.
  • Exposed ingestion tools in semantica.ontology for better discoverability.
  • Added OntologyData dataclass for consistent metadata handling (source path, format, timestamps).
• Documentation:
  • Ontology Usage Guide: Updated ontology_usage.md with comprehensive examples for single-file and directory ingestion.
  • API Reference: Updated ontology.md with OntologyIngestor class documentation and method details.
• Tests:
  • Comprehensive Test Suite: Added tests/ingest/test_ontology_ingestor.py covering all supported formats, error handling, and unified interface integration.
  • Demo Script: Added examples/demo_ontology_ingest.py for end-to-end usage demonstration.

[0.2.3] - 2026-01-20

Fixed

• LLM Relation Extraction Parsing:
  • Fixed relation extraction returning zero relations despite successful API calls to Groq and other providers
  • Normalized typed responses from instructor/OpenAI/Groq to consistent dict format before parsing
  • Added structured JSON fallback when typed generation yields zero relations to avoid silent empty outputs
  • Removed acceptance of extra kwargs (max_tokens, max_entities_prompt) from relation extraction internals
  • Filtered kwargs passed to provider LLM calls to only temperature and verbose
• API Parameter Handling:
  • Limited kwargs forwarded in chunked extraction helper to prevent parameter leakage
  • Ensured minimal, safe parameters are passed to provider calls
• Pipeline Circular Import (Issues #192, #193):
  • Fixed circular import between pipeline_builder and pipeline_validator triggered during semantica.pipeline import
  • Lazy-loaded PipelineValidator inside PipelineBuilder.__init__ and guarded type hints with TYPE_CHECKING
  • Ensured from semantica.deduplication import DuplicateDetector no longer fails even when pipeline module is imported
• JupyterLab Progress Output (Issue #181):
  • Added SEMANTICA_DISABLE_JUPYTER_PROGRESS environment variable to disable rich Jupyter/Colab progress tables
  • When enabled, progress falls back to console-style output, preventing infinite scrolling and JupyterLab out-of-memory errors

Added

• Comprehensive Test Suite:
• • Added unit tests (tests/test_relations_llm.py) with mocked LLM provider covering both typed and structured response paths
• • Added integration tests (tests/integration/test_relations_groq.py) for real Groq API calls with environment variable API key
• • Tests validate relation extraction completion and result parsing across different response formats
• Amazon Neptune Dev Environment:
• • Added CloudFormation template (cookbook/introduction/neptune-setup.yaml) to provision a dev Neptune cluster with public endpoint and IAM auth enabled
• • Documented deployment, cost estimates, and IAM User vs IAM Role best practices in cookbook/introduction/21_Amazon_Neptune_Store.ipynb
• • Added cfn-lint to .pre-commit-config.yaml for validating CloudFormation templates while excluding neptune-setup.yaml from generic YAML linters
• Vector Store High-Performance Ingestion:
• • Added VectorStore.add_documents for high-throughput ingestion with automatic embedding generation, batching, and parallel processing
• • Added VectorStore.embed_batch helper for generating embeddings for lists of texts without immediately storing them
• • Enabled default parallel ingestion in VectorStore with max_workers=6 for common workloads
• • Added dedicated documentation page docs/vector_store_usage.md describing high-performance vector store usage and configuration
• • Added tests/vector_store/test_vector_store_parallel.py covering parallel vs sequential performance, error handling, and edge cases for add_documents and embed_batch

Changed

• Relation Extraction API:
• • Simplified parameter interface by removing unused kwargs that were previously ignored
• • Improved error handling and verbose logging for debugging relation extraction issues
• • Enhanced robustness of post-response parsing across different LLM providers
• Vector Store Defaults and Examples:
• • Standardized VectorStore default concurrency to max_workers=6 for parallel ingestion
• • Updated vector store reference documentation and usage guides to rely on implicit defaults instead of requiring manual max_workers configuration in examples

[0.2.2] - 2026-01-15

Added

• Parallel Extraction Engine:
  • Implemented high-throughput parallel batch processing across all core extractors (NERExtractor, RelationExtractor, TripletExtractor, EventDetector, SemanticNetworkExtractor) using concurrent.futures.ThreadPoolExecutor.
  • Added max_workers configuration parameter (default: 1) to all extractor extract() methods, allowing users to tune concurrency based on available CPU cores or API rate limits.
  • Parallel Chunking: Implemented parallel processing for large document chunking in _extract_entities_chunked and _extract_relations_chunked, significantly reducing latency for long-form text analysis.
  • Thread-Safe Progress Tracking: Enhanced ProgressTracker to handle concurrent updates from multiple threads without race conditions during batch processing.
• Semantic Extract Performance & Regression:
  • Added edge-case regression suite covering max worker defaults, LLM prompt entity filtering, and extractor reuse.
  • Added a runnable real-use-case benchmark script for batch latency across NERExtractor, RelationExtractor, TripletExtractor, EventDetector, SemanticAnalyzer, and SemanticNetworkExtractor.
  • Added Groq LLM smoke tests that exercise LLM-based entities/relations/triplets when GROQ_API_KEY is available via environment configuration.

Security

• Credential Sanitization:
  • Removed hardcoded API keys from 8 cookbook notebooks to prevent secret leakage.
  • Enforced environment variable usage for GROQ_API_KEY across all examples.
• Secure Caching:
  • Updated ExtractionCache to exclude sensitive parameters (e.g., api_key, token, password) from cache key generation, preventing secret leakage and enabling safe cache sharing.
  • Upgraded cache key hashing algorithm from MD5 to SHA-256 for enhanced collision resistance and security.

Changed

• Gemini SDK Migration:
  • Migrated GeminiProvider to use the new google-genai SDK (v0.1.0+) to address deprecation warnings.
  • Implemented graceful fallback to google.generativeai for backward compatibility.
• Dependency Resolution:
  • Pinned opentelemetry-api and opentelemetry-sdk to 1.37.0 to resolve pip conflicts.
  • Updated protobuf and grpcio constraints for better stability.
• Entity Filtering Scope:
  • Removed entity filtering from non-LLM extraction flows to avoid accuracy regressions.
  • Applied entity downselection only to LLM relation prompt construction, while matching returned entities against the full original entity list.
• Batch Concurrency Defaults:
  • Standardized max_workers defaulting across semantic_extract and tuned for low-latency: ML-backed methods default to single-worker, while pattern/regex/rules/LLM/huggingface methods use a higher parallelism default capped by CPU.
  • Raised the global optimization.max_workers default to 8 for better throughput on batch workloads.

Performance

• Bottleneck Optimization (GitHub Issue #186):
  • Resolved Bottleneck #1 (Sequential Processing): Replaced sequential for loops with parallel execution for both document-level batches and intra-document chunks.
  • Performance Gains: Achieved ~1.89x speedup in real-world extraction scenarios (tested with Groq llama-3.3-70b-versatile on standard datasets).
  • Initialization Optimization: Refactored test suite to use class-level setUpClass for LLM provider initialization, eliminating redundant API client creation overhead.
• Low-Latency Entity Matching:
  • Avoided heavyweight embedding stack imports on common matches by improving fast matching heuristics and short-circuiting before embedding similarity.
  • Optimized entity matching to prioritize exact/substring/word-boundary matches and only fall back to embedding similarity when needed, reducing CPU overhead in LLM relation/triplet mapping.

[0.2.1] - 2026-01-12

Fixed

• LLM Output Stability (Bug #176):
  • Fixed incomplete JSON output issues by correctly propagating max_tokens parameter in extract_relations_llm.
  • Implemented automatic error handling that halves chunk sizes and retries when LLM context or output limits are exceeded.
  • Fixed AttributeError in provider integration by ensuring consistent parameter passing via **kwargs.
• Constraint Relaxations:
  • Removed hardcoded max_length constraints from Entity, Relation, and Triplet classes to support long-form semantic extraction (e.g., long descriptions or names).
• Fixed orchestrator lazy property initialization and configuration normalization logic in Orchestrator.
• Resolved AssertionError in orchestrator tests by aligning test mocks with production component usage.
• Fixed dependency compatibility issues by pinning protobuf>=5.29.1,<7.0 and grpcio>=1.71.2.
• Added missing dependencies GitPython and chardet to pyproject.toml.
• Verified and aligned FileObject.text property usage in GraphRAG notebooks for consistent content decoding.

Changed

• Chunking Defaults:
  • Increased default max_text_length for auto-chunking to 64,000 characters (from 32k/16k) for OpenAI, Anthropic, Gemini, Groq, and DeepSeek providers.
  • Unified chunking logic across extract_entities_llm, extract_relations_llm, and extract_triplets_llm.
• Groq Support:
  • Standardized Groq provider defaults to use llama-3.3-70b-versatile with a 64k context window.
  • Added native support for max_tokens and max_completion_tokens to prevent output truncation.

Added

• Testing:
  • Added tests/reproduce_issue_176.py to validate max_tokens propagation and chunking behavior across all extractors.

[0.2.0] - 2026-01-10

Added

• Amazon Neptune Support:
  • Added AmazonNeptuneStore providing Amazon Neptune graph database integration via Bolt protocol and OpenCypher.
  • Implemented NeptuneAuthTokenManager extending Neo4j AuthManager for AWS IAM SigV4 signing with automatic token refresh.
  • Added robust connection handling: retry logic with backoff for transient errors (signature expired, connection closed) and driver recreation.
  • Added graph-amazon-neptune optional dependency group (boto3, neo4j).
  • Comprehensive test suite covering all GraphStore interface methods.
• Docling Integration:
  • Added DoclingParser in semantica.parse for high-fidelity document parsing using the Docling library.
  • Supports multi-format parsing (PDF, DOCX, PPTX, XLSX, HTML, images) with superior table extraction and structure understanding.
  • Implemented as a standalone parser supporting local execution, OCR, and multiple export formats (Markdown, HTML, JSON).
• Robust Extraction Fallbacks:
  • Implemented comprehensive fallback chains ("ML/LLM" -> "Pattern" -> "Last Resort") across NERExtractor, RelationExtractor, and TripletExtractor to prevent empty result lists.
  • Added "Last Resort" pattern matching in NERExtractor to identify capitalized words as generic entities when all other methods fail.
  • Added "Last Resort" adjacency-based relation extraction in RelationExtractor to create weak connections between adjacent entities if no relations are found.
  • Added fallback logic in TripletExtractor to convert relations to triplets or use rule-based extraction if standard methods fail.
• Provenance & Tracking:
  • Added count tracking to batch processing logs in NERExtractor, RelationExtractor, and TripletExtractor.
  • Added batch_index and document_id to the metadata of all extracted entities, relations, triplets, semantic roles, and clusters for better traceability.
• Semantic Extract Improvements:
  • Introduced auto-chunking for long text processing in LLM extraction methods (extract_entities_llm, extract_relations_llm, extract_triplets_llm).
  • Added silent_fail parameter to LLM extraction methods for configurable error handling.
  • Implemented robust JSON parsing and automatic retry logic (3 attempts with exponential backoff) in BaseProvider for all LLM providers.
  • Enhanced GroqProvider with better diagnostics and connectivity testing.
  • Added comprehensive entity, relation, and triplet deduplication for chunked extraction.
  • Added semantica/semantic_extract/schemas.py with canonical Pydantic models for consistent structured output.
• Testing:
  • Added comprehensive robustness test suite tests/semantic_extract/test_robustness_fallback.py for validating extraction fallbacks and metadata propagation.
  • Added comprehensive unit test suite tests/embeddings/test_model_switching.py for verifying dynamic model transitions and dimension updates.
  • Added end-to-end integration test suite for Knowledge Graph pipeline validation (GraphBuilder -> EntityResolver -> GraphAnalyzer).
• Other:
  • Added missing dependencies GitPython and chardet to pyproject.toml.
  • Robustified ID extraction across CentralityCalculator, CommunityDetector, and ConnectivityAnalyzer to handle various entity formats.
  • Improved Entity class hashability and equality logic in utils/types.py.

Changed

• Deduplication & Conflict Logic:
  • Removed internal deduplication logic from NERExtractor, RelationExtractor, and TripletExtractor.
  • Removed consistency/conflict checking from ExtractionValidator to defer to dedicated semantica/conflicts module.
  • Removed _deduplicate_* methods from semantica/semantic_extract/methods.py.
• Batch Processing & Consistency:
  • Standardized batch processing across all extractors (NERExtractor, RelationExtractor, TripletExtractor, SemanticNetworkExtractor, EventDetector, SemanticAnalyzer, CoreferenceResolver) using a unified extract/analyze/resolve method pattern with progress tracking.
  • Added provenance metadata (batch_index, document_id) to SemanticNetwork nodes/edges, Event objects, SemanticRole results, CoreferenceChain mentions, and SemanticCluster (tracking source document_ids).
  • Updated SemanticClusterer.cluster and SemanticAnalyzer.cluster_semantically to accept list of dictionaries (with content and id keys) for better document tracking during clustering.
  • Removed legacy check_triplet_consistency from TripletExtractor.
  • Removed validate_consistency and _check_consistency from ExtractionValidator.
• Weighted Scoring:
  • Clarified weighted confidence scoring (50% Method Confidence + 50% Type Similarity) in comments.
  • Explicitly labeled "Type Similarity" as "user-provided" in code comments to remove ambiguity.
• Refactoring:
  • Fixed orchestrator lazy property initialization and configuration normalization logic in Orchestrator.
  • Verified and aligned FileObject.text property usage in GraphRAG notebooks for consistent content decoding.

Fixed

• Critical Fixes:
  • Resolved NameError in extraction_validator.py by adding missing Union import.
  • Resolved issues where extractors would return empty lists for valid input text when primary extraction methods failed.
  • Fixed metadata initialization issue in batch processing where batch_index and document_id were occasionally missing from extracted items.
  • Ensured LLMExtraction methods (enhance_entities, enhance_relations) return original input instead of failing or returning empty results when LLM providers are unavailable.
• Component Fixes:
  • Fixed model switching bug in TextEmbedder where internal state was not cleared, preventing dynamic updates between fastembed and sentence_transformers (#160).
  • Implemented model-intrinsic embedding dimension detection in TextEmbedder to ensure consistency between models and vector databases.
  • Updated set_model to properly refresh configuration and dimensions during model switches.
  • Fixed TypeError: unhashable type: 'Entity' in GraphAnalyzer when processing graphs with raw Entity objects or dictionaries in relationships (#159).
  • Resolved AssertionError in orchestrator tests by aligning test mocks with production component usage.
  • Fixed dependency compatibility issues by pinning protobuf==4.25.3 and grpcio==1.67.1.
  • Fixed a bug in TripletExtractor where the validate_triplets method was shadowed by an internal attribute.
  • Fixed incorrect TextSplitter import path in the semantic_extract.methods module.

[0.1.1] - 2026-01-05

Added

• Exported DoclingParser and DoclingMetadata from semantica.parse for easier access.
• Added comprehensive DoclingParser usage examples to README and documentation.
• Added Windows-specific troubleshooting note for PyTorch DLL issues.

Fixed

• Fixed DoclingParser import/export issues across platforms (Windows, Linux, Google Colab).
• Improved error messaging when optional docling dependency is missing.
• Fixed versioning inconsistencies across the framework.

[0.1.0] - 2025-12-31

Added

• New command-line interface (semantica CLI) with support for knowledge base building and info commands.
• Integrated FastAPI-based REST API server for remote access to framework functionality.
• Dedicated background worker component for scalable task processing and pipeline execution.
• Framework-level versioning configuration for PyPI distribution.
• Automated release workflow with Trusted Publishing support.

Changed

• Updated versioning across the framework to 0.1.0.
• Refined entry point configurations in pyproject.toml.
• Improved lazy module loading for core framework components.

[0.0.5] - 2025-11-26

Changed

• Configured Trusted Publishing for secure automated PyPI deployments

[0.0.4] - 2025-11-26

Changed

• Fixed PyPI deployment issues from v0.0.3

[0.0.3] - 2025-11-25

Changed

• Simplified CI/CD workflows - removed failing tests and strict linting
• Combined release and PyPI publishing into single workflow
• Simplified security scanning to weekly pip-audit only
• Streamlined GitHub Actions configuration

Added

• Comprehensive issue templates (Bug, Feature, Documentation, Support, Grant/Partnership)
• Updated pull request template with clear guidelines
• Community support documentation (SUPPORT.md)
• Funding and sponsorship configuration (FUNDING.yml)
• GitHub configuration README for maintainers
• 10+ new domain-specific cookbook examples (Finance, Healthcare, Cybersecurity, etc.)

Removed

• Redundant scripts folder (8 shell/PowerShell scripts)
• Unnecessary automation workflows (label-issues, mark-answered)
• Excessive issue templates

[0.0.2] - 2025-11-25

Changed

• Updated README with streamlined content and better examples
• Added more notebooks to cookbook
• Improved documentation structure

[0.0.1] - 2024-01-XX

Added

• Core framework architecture
• Universal data ingestion (multiple file formats)
• Semantic intelligence engine (NER, relation extraction, event detection)
• Knowledge graph construction with entity resolution
• 6-stage ontology generation pipeline
• GraphRAG engine for hybrid retrieval
• Multi-agent system infrastructure
• Production-ready quality assurance modules
• Comprehensive documentation with MkDocs
• Cookbook with interactive tutorials
• Support for multiple vector stores (Weaviate, Qdrant, FAISS)
• Support for multiple graph databases (Neo4j, NetworkX, RDFLib)
• Temporal knowledge graph support
• Conflict detection and resolution
• Deduplication and entity merging
• Schema template enforcement
• Seed data management
• Multi-format export (RDF, JSON-LD, CSV, GraphML)
• Visualization tools
• Pipeline orchestration
• Streaming support (Kafka, RabbitMQ, Kinesis)
• Context engineering for AI agents
• Reasoning and inference engine

Documentation

• Getting started guide
• API reference for all modules
• Concepts and architecture documentation
• Use case examples
• Cookbook tutorials
• Community projects showcase

---

Types of Changes

• Added for new features
• Changed for changes in existing functionality
• Deprecated for soon-to-be removed features
• Removed for now removed features
• Fixed for any bug fixes
• Security for vulnerability fixes

Migration Guides

When breaking changes are introduced, migration guides will be provided in the release notes and documentation.

---

For detailed release notes, see GitHub Releases.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

• Fixed: PolicyEngine latest version selection on ContextGraph; AgentContext fallback robustness and secure logging (PR #TBD by @KaifAhmad1)

• Tests: Added ContextGraph fallback and AgentContext smoke tests; full suite passing

• Context Engineering Enhancement (PR #307 by @KaifAhmad1):

  • Comprehensive decision tracking system with full lifecycle management (record → analyze → query → precedent → influence)
  • Advanced KG algorithm integration: centrality analysis, community detection, node embeddings with ContextGraph
  • Enhanced AgentContext with granular feature flags for decision tracking, KG algorithms, and vector store features
  • PolicyException model replacing conflicting Exception name for meaningful business domain modeling
  • GraphStore validation preventing runtime failures with explicit capability checking
  • Hybrid search combining semantic, structural, and category similarity with configurable weights
  • Decision influence analysis with centrality measures and causal chain tracking
  • Policy management with versioning, compliance checking, and exception handling
  • Production-ready architecture with audit trails, security, and scalability features
  • 9 critical bug fixes: logging, security, audit trails, API compatibility, Cypher queries, centrality access, validation, naming
  • Comprehensive documentation with usage guides, production examples, and API references
  • 100% test coverage with all validation tests passing (9/9 tests)
  • Enterprise-grade features for financial services, healthcare, legal, and business domains
  • Complete backward compatibility with existing semantica components
  • Performance optimizations: caching, indexing, and efficient graph operations

• Added PgVector Store Support (PR #303 by @Sameer6305, @KaifAhmad1):

  • Native PostgreSQL vector storage using pgvector extension with full integration
  • Multiple distance metrics: cosine, L2/Euclidean, inner product with automatic score normalization
  • Advanced indexing: HNSW and IVFFlat for approximate nearest neighbor search with tunable parameters
  • JSONB metadata storage with flexible filtering capabilities and batch operations
  • Connection pooling support with psycopg3/psycopg2 fallback and efficient resource management
  • Comprehensive VectorStore integration with backend delegation and unified API
  • Idempotent index creation and table management with safe migration support
  • Production-ready security: SQL injection protection with psycopg_sql.SQL() and input validation
  • Performance optimizations: UUID4-based IDs, batch executemany operations, connection pooling
  • Full backward compatibility with existing vector store implementations
  • 36+ comprehensive test cases with Docker integration and dependency skipping
  • Complete documentation with setup guides, examples, and performance tuning
  • CI/CD integration: resolved benchmark compatibility and fixed documentation links

• Improved Vector Store for Decision Tracking (PR #293 by @KaifAhmad1):

  • Comprehensive decision tracking capabilities with hybrid search combining semantic and structural embeddings
  • New DecisionEmbeddingPipeline for generating semantic and structural embeddings with KG algorithm integration
  • HybridSimilarityCalculator with configurable weights (semantic: 0.7, structural: 0.3)
  • DecisionContext high-level interface for decision management with explainable AI features
  • ContextRetriever with hybrid precedent search and multi-hop reasoning
  • User-friendly convenience API: quick_decision(), find_precedents(), explain(), similar_to(), batch_decisions(), filter_decisions()
  • Knowledge Graph algorithm integration: Node2Vec, PathFinder, CommunityDetector, CentralityCalculator, SimilarityCalculator, ConnectivityAnalyzer
  • Explainable AI with path tracing, confidence scoring, and comprehensive decision explanations
  • Performance optimizations: 0.028s per decision processing, 0.031s search performance, ~0.8KB per decision memory usage
  • 100% backward compatibility maintained with existing VectorStore functionality
  • 34+ comprehensive tests covering all functionality including end-to-end scenarios and performance benchmarks
  • Real-world validation examples for banking and insurance domains
  • Documentation with clear imports, examples, and API references

• Improved Graph Algorithms in KG Module (PR #292 by @KaifAhmad1):

  • Complete algorithm suite with 30+ graph algorithms across 7 categories
  • Node Embeddings: Node2Vec, DeepWalk, Word2Vec for structural similarity analysis
  • Similarity Analysis: Cosine, Euclidean, Manhattan, Correlation metrics with batch processing
  • Path Finding: Dijkstra, A*, BFS, K-shortest paths for route and network analysis
  • Link Prediction: Preferential attachment, Jaccard, Adamic-Adar for network completion
  • Centrality Analysis: Degree, Betweenness, Closeness, PageRank for importance ranking
  • Community Detection: Louvain, Leiden, Label propagation for clustering analysis
  • Connectivity Analysis: Components, bridges, density for network robustness
  • Unified provenance tracking system with GraphBuilderWithProvenance and AlgorithmTrackerWithProvenance
  • Complete execution tracking with metadata, timestamps, and reproducibility IDs
  • Comprehensive test coverage with 5 test suites and 40+ test methods
  • Professional documentation overhaul for all modules and reference documentation
  • Enterprise-ready functionality with error handling and NetworkX compatibility
  • Performance optimizations with sparse matrix operations and batch processing
  • Full backward compatibility maintained with gradual migration support

• Improved Security Configuration with Dependabot:

  • Configured bi-weekly security updates with manual review by @KaifAhmad1
  • Implemented automated security scans (Monday & Thursday at 7 AM IST) with Bandit, Safety, Semgrep
  • Added security-critical package grouping (cryptography, requests, urllib3, certifi, pyopenssl)
  • Enterprise-grade security with audit trail, compliance features, and zero auto-merge
  • Optimized IST timezone scheduling (Security scans: 7 AM IST, PRs: 9 AM IST)
  • Aligned with new Dependabot features: open-source proxy support, smart dependency grouping for Snowflake/Arrow/benchmark features, private registry support, semantic commit prefixes, and latest GitHub security best practices

• ResourceScheduler Deadlock Fix and Performance Improvements (PR #299, #301 by @d4ndr4d3, @KaifAhmad1):

  • Fixed critical deadlock in ResourceScheduler by replacing threading.Lock() with threading.RLock()
  • Resolved nested lock acquisition issue in allocate_resources() → allocate_cpu/memory/gpu() calls
  • Added allocation validation with ValidationError when no resources can be allocated
  • Improved performance by moving progress tracking updates outside lock scope
  • Implemented comprehensive resource cleanup on allocation failures to prevent leaks
  • Added complete regression test suite (6 tests) for deadlock prevention and edge cases
  • Improved error handling and documentation for better operator visibility
  • Zero breaking changes, maintains thread safety and backward compatibility

[0.2.7] - 2026-02-09

Added / Changed

• Snowflake Connector for Data Ingestion (PR #276 by @Sameer6305):

  • Native Snowflake connector with multi-authentication (password, OAuth, key-pair, SSO)
  • Table and query ingestion with pagination, schema introspection, batch processing
  • SQL injection prevention via identifier escaping, OAuth token validation
  • Progress tracking integration, context manager support, document export
  • 24 comprehensive unit tests with mocking, complete documentation and examples
  • Added as optional dependency db-snowflake with snowflake-connector-python>=3.0.0

• Apache Arrow Export Support (PR #273 by @Sameer6305):

  • Added Apache Arrow exporter with explicit schemas, entity/relationship export, compression support
  • Integrated with export module and method registry, Pandas/DuckDB compatible
  • 20 unit tests + 1 integration test, complete documentation with examples

• Comprehensive Benchmark Suite with Regression CLI (PR #289 by @ZohaibHassan16, @KaifAhmad1):

  • 137+ benchmarks across all 10 Semantica modules (Input, Core, Storage, Context, QA, Ontology, etc.)
  • Environment-agnostic design with robust mocking system for CI/CD compatibility
  • Statistical regression detection using Z-score analysis with configurable thresholds
  • Automated performance auditing via GitHub Actions workflow
  • Comprehensive documentation suite (benchmarks.md, architecture guides, usage examples)
  • Zero breaking changes, production-ready with ultra-fast text processing (>10,000 ops/s)
  • Added benchmark runner CLI: python benchmarks/benchmark_runner.py

[0.2.6] - 2026-02-03

Added / Changed

• W3C PROV-O Compliant Provenance Tracking (#254, #246):

  • Comprehensive provenance tracking system with W3C PROV-O compliance across all 17 Semantica modules
  • Core Module: ProvenanceManager, W3C PROV-O schemas, storage backends (InMemory, SQLite), SHA-256 integrity verification
  • Module Integrations: Semantic Extract, LLMs (Groq, OpenAI, HuggingFace, LiteLLM), Pipeline, Context, Ingest, Embeddings, Graph/Vector/Triplet stores, Reasoning, Conflicts, Deduplication, Export, Parse, Normalize, Ontology, Visualization
  • Features: Complete lineage tracking (Document → Chunk → Entity → Relationship → Graph), LLM tracking (tokens, costs, latency), source tracking, bridge axioms for domain transformations
  • Compliance Infrastructure: W3C PROV-O, FDA 21 CFR Part 11, SOX, HIPAA, TNFD
  • Testing: 237 tests covering core functionality, all 17 module integrations, edge cases, backward compatibility
  • Design: Opt-in with provenance=False by default, zero breaking changes, no new dependencies
  • Contributed by @KaifAhmad1

• Enhanced Change Management Module (#248, #243):

  • Enterprise-grade version control for knowledge graphs and ontologies with persistent storage and audit trails
  • Core Classes: TemporalVersionManager (KG versioning), OntologyVersionManager (ontology versioning), ChangeLogEntry (metadata)
  • Storage: SQLite (persistent) and in-memory backends with thread-safe operations
  • Features: SHA-256 checksums, detailed entity/relationship diffs, structural ontology comparison, email validation
  • Compliance Infrastructure: HIPAA, SOX, FDA 21 CFR Part 11 with immutable audit trails
  • Testing: 104 tests (100% pass) - unit, integration, compliance, performance, edge cases
  • Performance: 17.6ms for 10k entities, 510+ ops/sec concurrent, handles 5k+ entity graphs
  • Migration: Backward compatible, simplified class names, zero external dependencies
  • Contributed by @KaifAhmad1

• CSV Ingestion Enhancements (PR #244 by @saloni0318)

  • Auto-detect CSV encoding (chardet) and delimiter (csv.Sniffer)
  • Tolerant decoding and malformed-row handling (on_bad_lines='warn')
  • Optional chunked reading for large files; metadata tracks detected values
  • Expanded unit tests covering delimiters, quoted/multiline fields, header overrides, chunks, and NaN preservation

• Tests: Comprehensive units for TextNormalizer (PR #242 by @ZohaibHassan16)

  • Added focused test coverage for TextNormalizer behavior across inputs

• Tests: Register integration mark and tidy ingest test warnings (PR #241 by @KaifAhmad1)

  • Introduced integration test marker and reduced noisy warnings in ingest tests

• Ingest Unit Tests (#239, #232):

  • Comprehensive unit tests for ingestion modules (file, web, and feed ingestors)
  • Coverage: File scanning (local/cloud S3/GCS/Azure), web ingestion (URL/sitemap/robots.txt), RSS/Atom feed parsing
  • Testing: 998 lines of test code with mocked external dependencies for fast, isolated execution
  • Results: file_ingestor (86%), web_ingestor (86%), feed_ingestor (80%) coverage
  • Covers happy paths, edge cases, and error handling
  • Contributed by @Mohammed2372

Fixed

• Temperature Compatibility Fix (#256, #252):

  • Fixed hardcoded temperature=0.3 that broke compatibility with models requiring specific temperature values (e.g., gpt-5-mini)
  • Added _add_if_set helper method to BaseProvider that only passes parameters when explicitly set
  • When temperature=None, parameter is omitted allowing APIs to use model defaults
  • Updated all 5 providers: OpenAI, Groq, Gemini, Ollama, DeepSeek
  • Reduced code by ~85 lines with cleaner parameter handling
  • Comprehensive test coverage added (10 temperature tests, all passing)
  • Backward compatible - no breaking changes
  • Contributed by @F0rt1s and @IGES-Institut

• JenaStore Empty Graph Bug (#257, #258):

  • Fixed ProcessingError: Graph not initialized when operating on empty (but initialized) graphs
  • Replaced implicit if not self.graph: checks with explicit if self.graph is None: validation in 5 methods (add_triplets, get_triplets, delete_triplet, execute_sparql, serialize)
  • Properly distinguishes None (uninitialized) from empty graphs (initialized with 0 triplets)
  • Unblocks benchmarking suite, fresh deployments, and testing workflows
  • Contributed by @ZohaibHassan16

[0.2.5] - 2026-01-27

Added

• Pinecone Vector Store Support:
  • Implemented native Pinecone support (PineconeStore) with full CRUD capabilities.
  • Added support for serverless and pod-based indexes, namespaces, and metadata filtering.
  • Integrated with VectorStore unified interface and registry.
  • (Closes #219, Resolves #220)
• Configurable LLM Retry Logic:
  • Exposed max_retries parameter in NERExtractor, RelationExtractor, TripletExtractor and low-level extraction methods (extract_entities_llm, extract_relations_llm, extract_triplets_llm).
  • Defaults to 3 retries to prevent infinite loops during JSON validation failures or API timeouts.
  • Propagated retry configuration through chunked processing helpers to ensure consistent behavior for long documents.
  • Updated 03_Earnings_Call_Analysis.ipynb to use max_retries=3 by default.

Added

• Bring Your Own Model (BYOM) Support:
  • Enabled full support for custom Hugging Face models in NERExtractor, RelationExtractor, and TripletExtractor.
  • Added support for custom tokenizers in HuggingFaceModelLoader to handle models with non-standard tokenization requirements.
  • Implemented robust fallback logic for model selection: runtime options (extract(model=...)) now correctly override configuration defaults.
• Enhanced NER Implementation:
  • Added configurable aggregation strategies (simple, first, average, max) to extract_entities_huggingface for better sub-word token handling.
  • Implemented robust IOB/BILOU parsing to reconstruct entities from raw model outputs when structured output is unavailable.
  • Added confidence scoring for aggregated entities.
• Relation Extraction Improvements:
  • Implemented standard entity marker technique (wrapping subject/object with <subj>, <obj> tags) in extract_relations_huggingface for compatibility with sequence classification models.
  • Added structured output parsing to convert raw model predictions into validated Relation objects.
• Triplet Extraction Completion:
  • Added specialized parsing for Seq2Seq models (e.g., REBEL) in extract_triplets_huggingface to generate structured triplets directly from text.
  • Implemented post-processing logic to clean and validate generated triplets.

Fixed

• LLM Extraction Stability:
  • Fixed infinite retry loops in BaseProvider by strictly enforcing max_retries limit during structured output generation.
  • Resolved stuck execution in earnings call analysis notebooks when using smaller models (e.g., Llama 3 8B) that frequently produce invalid JSON.
• Model Parameter Precedence:
  • Fixed issue where configuration defaults took precedence over runtime arguments in Hugging Face extractors. Runtime options now correctly override config values.
• Import Handling:
  • Fixed circular import issues in test suites by implementing robust mocking strategies.

[0.2.4] - 2026-01-22

Added

• Ontology Ingestion Module:
  • Implemented OntologyIngestor in semantica.ingest for parsing RDF/OWL files (Turtle, RDF/XML, JSON-LD, N3) into standardized OntologyData objects.
  • Added ingest_ontology convenience function and integrated it into the unified ingest(source_type="ontology") interface.
  • Added recursive directory scanning support for batch ontology ingestion.
  • Exposed ingestion tools in semantica.ontology for better discoverability.
  • Added OntologyData dataclass for consistent metadata handling (source path, format, timestamps).
• Documentation:
  • Ontology Usage Guide: Updated ontology_usage.md with comprehensive examples for single-file and directory ingestion.
  • API Reference: Updated ontology.md with OntologyIngestor class documentation and method details.
• Tests:
  • Comprehensive Test Suite: Added tests/ingest/test_ontology_ingestor.py covering all supported formats, error handling, and unified interface integration.
  • Demo Script: Added examples/demo_ontology_ingest.py for end-to-end usage demonstration.

[0.2.3] - 2026-01-20

Fixed

• LLM Relation Extraction Parsing:
  • Fixed relation extraction returning zero relations despite successful API calls to Groq and other providers
  • Normalized typed responses from instructor/OpenAI/Groq to consistent dict format before parsing
  • Added structured JSON fallback when typed generation yields zero relations to avoid silent empty outputs
  • Removed acceptance of extra kwargs (max_tokens, max_entities_prompt) from relation extraction internals
  • Filtered kwargs passed to provider LLM calls to only temperature and verbose
• API Parameter Handling:
  • Limited kwargs forwarded in chunked extraction helper to prevent parameter leakage
  • Ensured minimal, safe parameters are passed to provider calls
• Pipeline Circular Import (Issues #192, #193):
  • Fixed circular import between pipeline_builder and pipeline_validator triggered during semantica.pipeline import
  • Lazy-loaded PipelineValidator inside PipelineBuilder.__init__ and guarded type hints with TYPE_CHECKING
  • Ensured from semantica.deduplication import DuplicateDetector no longer fails even when pipeline module is imported
• JupyterLab Progress Output (Issue #181):
  • Added SEMANTICA_DISABLE_JUPYTER_PROGRESS environment variable to disable rich Jupyter/Colab progress tables
  • When enabled, progress falls back to console-style output, preventing infinite scrolling and JupyterLab out-of-memory errors

Added

• Comprehensive Test Suite:
• • Added unit tests (tests/test_relations_llm.py) with mocked LLM provider covering both typed and structured response paths
• • Added integration tests (tests/integration/test_relations_groq.py) for real Groq API calls with environment variable API key
• • Tests validate relation extraction completion and result parsing across different response formats
• Amazon Neptune Dev Environment:
• • Added CloudFormation template (cookbook/introduction/neptune-setup.yaml) to provision a dev Neptune cluster with public endpoint and IAM auth enabled
• • Documented deployment, cost estimates, and IAM User vs IAM Role best practices in cookbook/introduction/21_Amazon_Neptune_Store.ipynb
• • Added cfn-lint to .pre-commit-config.yaml for validating CloudFormation templates while excluding neptune-setup.yaml from generic YAML linters
• Vector Store High-Performance Ingestion:
• • Added VectorStore.add_documents for high-throughput ingestion with automatic embedding generation, batching, and parallel processing
• • Added VectorStore.embed_batch helper for generating embeddings for lists of texts without immediately storing them
• • Enabled default parallel ingestion in VectorStore with max_workers=6 for common workloads
• • Added dedicated documentation page docs/vector_store_usage.md describing high-performance vector store usage and configuration
• • Added tests/vector_store/test_vector_store_parallel.py covering parallel vs sequential performance, error handling, and edge cases for add_documents and embed_batch

Changed

• Relation Extraction API:
• • Simplified parameter interface by removing unused kwargs that were previously ignored
• • Improved error handling and verbose logging for debugging relation extraction issues
• • Enhanced robustness of post-response parsing across different LLM providers
• Vector Store Defaults and Examples:
• • Standardized VectorStore default concurrency to max_workers=6 for parallel ingestion
• • Updated vector store reference documentation and usage guides to rely on implicit defaults instead of requiring manual max_workers configuration in examples

[0.2.2] - 2026-01-15

Added

• Parallel Extraction Engine:
  • Implemented high-throughput parallel batch processing across all core extractors (NERExtractor, RelationExtractor, TripletExtractor, EventDetector, SemanticNetworkExtractor) using concurrent.futures.ThreadPoolExecutor.
  • Added max_workers configuration parameter (default: 1) to all extractor extract() methods, allowing users to tune concurrency based on available CPU cores or API rate limits.
  • Parallel Chunking: Implemented parallel processing for large document chunking in _extract_entities_chunked and _extract_relations_chunked, significantly reducing latency for long-form text analysis.
  • Thread-Safe Progress Tracking: Enhanced ProgressTracker to handle concurrent updates from multiple threads without race conditions during batch processing.
• Semantic Extract Performance & Regression:
  • Added edge-case regression suite covering max worker defaults, LLM prompt entity filtering, and extractor reuse.
  • Added a runnable real-use-case benchmark script for batch latency across NERExtractor, RelationExtractor, TripletExtractor, EventDetector, SemanticAnalyzer, and SemanticNetworkExtractor.
  • Added Groq LLM smoke tests that exercise LLM-based entities/relations/triplets when GROQ_API_KEY is available via environment configuration.

Security

• Credential Sanitization:
  • Removed hardcoded API keys from 8 cookbook notebooks to prevent secret leakage.
  • Enforced environment variable usage for GROQ_API_KEY across all examples.
• Secure Caching:
  • Updated ExtractionCache to exclude sensitive parameters (e.g., api_key, token, password) from cache key generation, preventing secret leakage and enabling safe cache sharing.
  • Upgraded cache key hashing algorithm from MD5 to SHA-256 for enhanced collision resistance and security.

Changed

• Gemini SDK Migration:
  • Migrated GeminiProvider to use the new google-genai SDK (v0.1.0+) to address deprecation warnings.
  • Implemented graceful fallback to google.generativeai for backward compatibility.
• Dependency Resolution:
  • Pinned opentelemetry-api and opentelemetry-sdk to 1.37.0 to resolve pip conflicts.
  • Updated protobuf and grpcio constraints for better stability.
• Entity Filtering Scope:
  • Removed entity filtering from non-LLM extraction flows to avoid accuracy regressions.
  • Applied entity downselection only to LLM relation prompt construction, while matching returned entities against the full original entity list.
• Batch Concurrency Defaults:
  • Standardized max_workers defaulting across semantic_extract and tuned for low-latency: ML-backed methods default to single-worker, while pattern/regex/rules/LLM/huggingface methods use a higher parallelism default capped by CPU.
  • Raised the global optimization.max_workers default to 8 for better throughput on batch workloads.

Performance

• Bottleneck Optimization (GitHub Issue #186):
  • Resolved Bottleneck #1 (Sequential Processing): Replaced sequential for loops with parallel execution for both document-level batches and intra-document chunks.
  • Performance Gains: Achieved ~1.89x speedup in real-world extraction scenarios (tested with Groq llama-3.3-70b-versatile on standard datasets).
  • Initialization Optimization: Refactored test suite to use class-level setUpClass for LLM provider initialization, eliminating redundant API client creation overhead.
• Low-Latency Entity Matching:
  • Avoided heavyweight embedding stack imports on common matches by improving fast matching heuristics and short-circuiting before embedding similarity.
  • Optimized entity matching to prioritize exact/substring/word-boundary matches and only fall back to embedding similarity when needed, reducing CPU overhead in LLM relation/triplet mapping.

[0.2.1] - 2026-01-12

Fixed

• LLM Output Stability (Bug #176):
  • Fixed incomplete JSON output issues by correctly propagating max_tokens parameter in extract_relations_llm.
  • Implemented automatic error handling that halves chunk sizes and retries when LLM context or output limits are exceeded.
  • Fixed AttributeError in provider integration by ensuring consistent parameter passing via **kwargs.
• Constraint Relaxations:
  • Removed hardcoded max_length constraints from Entity, Relation, and Triplet classes to support long-form semantic extraction (e.g., long descriptions or names).
• Fixed orchestrator lazy property initialization and configuration normalization logic in Orchestrator.
• Resolved AssertionError in orchestrator tests by aligning test mocks with production component usage.
• Fixed dependency compatibility issues by pinning protobuf>=5.29.1,<7.0 and grpcio>=1.71.2.
• Added missing dependencies GitPython and chardet to pyproject.toml.
• Verified and aligned FileObject.text property usage in GraphRAG notebooks for consistent content decoding.

Changed

• Chunking Defaults:
  • Increased default max_text_length for auto-chunking to 64,000 characters (from 32k/16k) for OpenAI, Anthropic, Gemini, Groq, and DeepSeek providers.
  • Unified chunking logic across extract_entities_llm, extract_relations_llm, and extract_triplets_llm.
• Groq Support:
  • Standardized Groq provider defaults to use llama-3.3-70b-versatile with a 64k context window.
  • Added native support for max_tokens and max_completion_tokens to prevent output truncation.

Added

• Testing:
  • Added tests/reproduce_issue_176.py to validate max_tokens propagation and chunking behavior across all extractors.

[0.2.0] - 2026-01-10

Added

• Amazon Neptune Support:
  • Added AmazonNeptuneStore providing Amazon Neptune graph database integration via Bolt protocol and OpenCypher.
  • Implemented NeptuneAuthTokenManager extending Neo4j AuthManager for AWS IAM SigV4 signing with automatic token refresh.
  • Added robust connection handling: retry logic with backoff for transient errors (signature expired, connection closed) and driver recreation.
  • Added graph-amazon-neptune optional dependency group (boto3, neo4j).
  • Comprehensive test suite covering all GraphStore interface methods.
• Docling Integration:
  • Added DoclingParser in semantica.parse for high-fidelity document parsing using the Docling library.
  • Supports multi-format parsing (PDF, DOCX, PPTX, XLSX, HTML, images) with superior table extraction and structure understanding.
  • Implemented as a standalone parser supporting local execution, OCR, and multiple export formats (Markdown, HTML, JSON).
• Robust Extraction Fallbacks:
  • Implemented comprehensive fallback chains ("ML/LLM" -> "Pattern" -> "Last Resort") across NERExtractor, RelationExtractor, and TripletExtractor to prevent empty result lists.
  • Added "Last Resort" pattern matching in NERExtractor to identify capitalized words as generic entities when all other methods fail.
  • Added "Last Resort" adjacency-based relation extraction in RelationExtractor to create weak connections between adjacent entities if no relations are found.
  • Added fallback logic in TripletExtractor to convert relations to triplets or use rule-based extraction if standard methods fail.
• Provenance & Tracking:
  • Added count tracking to batch processing logs in NERExtractor, RelationExtractor, and TripletExtractor.
  • Added batch_index and document_id to the metadata of all extracted entities, relations, triplets, semantic roles, and clusters for better traceability.
• Semantic Extract Improvements:
  • Introduced auto-chunking for long text processing in LLM extraction methods (extract_entities_llm, extract_relations_llm, extract_triplets_llm).
  • Added silent_fail parameter to LLM extraction methods for configurable error handling.
  • Implemented robust JSON parsing and automatic retry logic (3 attempts with exponential backoff) in BaseProvider for all LLM providers.
  • Enhanced GroqProvider with better diagnostics and connectivity testing.
  • Added comprehensive entity, relation, and triplet deduplication for chunked extraction.
  • Added semantica/semantic_extract/schemas.py with canonical Pydantic models for consistent structured output.
• Testing:
  • Added comprehensive robustness test suite tests/semantic_extract/test_robustness_fallback.py for validating extraction fallbacks and metadata propagation.
  • Added comprehensive unit test suite tests/embeddings/test_model_switching.py for verifying dynamic model transitions and dimension updates.
  • Added end-to-end integration test suite for Knowledge Graph pipeline validation (GraphBuilder -> EntityResolver -> GraphAnalyzer).
• Other:
  • Added missing dependencies GitPython and chardet to pyproject.toml.
  • Robustified ID extraction across CentralityCalculator, CommunityDetector, and ConnectivityAnalyzer to handle various entity formats.
  • Improved Entity class hashability and equality logic in utils/types.py.

Changed

• Deduplication & Conflict Logic:
  • Removed internal deduplication logic from NERExtractor, RelationExtractor, and TripletExtractor.
  • Removed consistency/conflict checking from ExtractionValidator to defer to dedicated semantica/conflicts module.
  • Removed _deduplicate_* methods from semantica/semantic_extract/methods.py.
• Batch Processing & Consistency:
  • Standardized batch processing across all extractors (NERExtractor, RelationExtractor, TripletExtractor, SemanticNetworkExtractor, EventDetector, SemanticAnalyzer, CoreferenceResolver) using a unified extract/analyze/resolve method pattern with progress tracking.
  • Added provenance metadata (batch_index, document_id) to SemanticNetwork nodes/edges, Event objects, SemanticRole results, CoreferenceChain mentions, and SemanticCluster (tracking source document_ids).
  • Updated SemanticClusterer.cluster and SemanticAnalyzer.cluster_semantically to accept list of dictionaries (with content and id keys) for better document tracking during clustering.
  • Removed legacy check_triplet_consistency from TripletExtractor.
  • Removed validate_consistency and _check_consistency from ExtractionValidator.
• Weighted Scoring:
  • Clarified weighted confidence scoring (50% Method Confidence + 50% Type Similarity) in comments.
  • Explicitly labeled "Type Similarity" as "user-provided" in code comments to remove ambiguity.
• Refactoring:
  • Fixed orchestrator lazy property initialization and configuration normalization logic in Orchestrator.
  • Verified and aligned FileObject.text property usage in GraphRAG notebooks for consistent content decoding.

Fixed

• Critical Fixes:
  • Resolved NameError in extraction_validator.py by adding missing Union import.
  • Resolved issues where extractors would return empty lists for valid input text when primary extraction methods failed.
  • Fixed metadata initialization issue in batch processing where batch_index and document_id were occasionally missing from extracted items.
  • Ensured LLMExtraction methods (enhance_entities, enhance_relations) return original input instead of failing or returning empty results when LLM providers are unavailable.
• Component Fixes:
  • Fixed model switching bug in TextEmbedder where internal state was not cleared, preventing dynamic updates between fastembed and sentence_transformers (#160).
  • Implemented model-intrinsic embedding dimension detection in TextEmbedder to ensure consistency between models and vector databases.
  • Updated set_model to properly refresh configuration and dimensions during model switches.
  • Fixed TypeError: unhashable type: 'Entity' in GraphAnalyzer when processing graphs with raw Entity objects or dictionaries in relationships (#159).
  • Resolved AssertionError in orchestrator tests by aligning test mocks with production component usage.
  • Fixed dependency compatibility issues by pinning protobuf==4.25.3 and grpcio==1.67.1.
  • Fixed a bug in TripletExtractor where the validate_triplets method was shadowed by an internal attribute.
  • Fixed incorrect TextSplitter import path in the semantic_extract.methods module.

[0.1.1] - 2026-01-05

Added

• Exported DoclingParser and DoclingMetadata from semantica.parse for easier access.
• Added comprehensive DoclingParser usage examples to README and documentation.
• Added Windows-specific troubleshooting note for PyTorch DLL issues.

Fixed

• Fixed DoclingParser import/export issues across platforms (Windows, Linux, Google Colab).
• Improved error messaging when optional docling dependency is missing.
• Fixed versioning inconsistencies across the framework.

[0.1.0] - 2025-12-31

Added

• New command-line interface (semantica CLI) with support for knowledge base building and info commands.
• Integrated FastAPI-based REST API server for remote access to framework functionality.
• Dedicated background worker component for scalable task processing and pipeline execution.
• Framework-level versioning configuration for PyPI distribution.
• Automated release workflow with Trusted Publishing support.

Changed

• Updated versioning across the framework to 0.1.0.
• Refined entry point configurations in pyproject.toml.
• Improved lazy module loading for core framework components.

[0.0.5] - 2025-11-26

Changed

• Configured Trusted Publishing for secure automated PyPI deployments

[0.0.4] - 2025-11-26

Changed

• Fixed PyPI deployment issues from v0.0.3

[0.0.3] - 2025-11-25

Changed

• Simplified CI/CD workflows - removed failing tests and strict linting
• Combined release and PyPI publishing into single workflow
• Simplified security scanning to weekly pip-audit only
• Streamlined GitHub Actions configuration

Added

• Comprehensive issue templates (Bug, Feature, Documentation, Support, Grant/Partnership)
• Updated pull request template with clear guidelines
• Community support documentation (SUPPORT.md)
• Funding and sponsorship configuration (FUNDING.yml)
• GitHub configuration README for maintainers
• 10+ new domain-specific cookbook examples (Finance, Healthcare, Cybersecurity, etc.)

Removed

• Redundant scripts folder (8 shell/PowerShell scripts)
• Unnecessary automation workflows (label-issues, mark-answered)
• Excessive issue templates

[0.0.2] - 2025-11-25

Changed

• Updated README with streamlined content and better examples
• Added more notebooks to cookbook
• Improved documentation structure

[0.0.1] - 2024-01-XX

Added

• Core framework architecture
• Universal data ingestion (multiple file formats)
• Semantic intelligence engine (NER, relation extraction, event detection)
• Knowledge graph construction with entity resolution
• 6-stage ontology generation pipeline
• GraphRAG engine for hybrid retrieval
• Multi-agent system infrastructure
• Production-ready quality assurance modules
• Comprehensive documentation with MkDocs
• Cookbook with interactive tutorials
• Support for multiple vector stores (Weaviate, Qdrant, FAISS)
• Support for multiple graph databases (Neo4j, NetworkX, RDFLib)
• Temporal knowledge graph support
• Conflict detection and resolution
• Deduplication and entity merging
• Schema template enforcement
• Seed data management
• Multi-format export (RDF, JSON-LD, CSV, GraphML)
• Visualization tools
• Pipeline orchestration
• Streaming support (Kafka, RabbitMQ, Kinesis)
• Context engineering for AI agents
• Reasoning and inference engine

Documentation

• Getting started guide
• API reference for all modules
• Concepts and architecture documentation
• Use case examples
• Cookbook tutorials
• Community projects showcase

---

Types of Changes

• Added for new features
• Changed for changes in existing functionality
• Deprecated for soon-to-be removed features
• Removed for now removed features
• Fixed for any bug fixes
• Security for vulnerability fixes

Migration Guides

When breaking changes are introduced, migration guides will be provided in the release notes and documentation.

---

For detailed release notes, see GitHub Releases.