Merge pull request #180 from Hawksight-AI/docs

Release v0.2.1: Stability Fixes

CHANGELOG.md

@@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.2.1] - 2026-01-12
+
+### Fixed
+- **LLM Output Stability (Bug #176)**:
+    - Fixed incomplete JSON output issues by correctly propagating `max_tokens` parameter in `extract_relations_llm`.
+    - Implemented automatic error handling that halves chunk sizes and retries when LLM context or output limits are exceeded.
+    - Fixed `AttributeError` in provider integration by ensuring consistent parameter passing via `**kwargs`.
+- **Constraint Relaxations**:
+    - Removed hardcoded `max_length` constraints from `Entity`, `Relation`, and `Triplet` classes to support long-form semantic extraction (e.g., long descriptions or names).
+
+### Changed
+- **Chunking Defaults**:
+    - Increased default `max_text_length` for auto-chunking to **64,000 characters** (from 32k/16k) for OpenAI, Anthropic, Gemini, Groq, and DeepSeek providers.
+    - Unified chunking logic across `extract_entities_llm`, `extract_relations_llm`, and `extract_triplets_llm`.
+- **Groq Support**:
+    - Standardized Groq provider defaults to use `llama-3.3-70b-versatile` with a 64k context window.
+    - Added native support for `max_tokens` and `max_completion_tokens` to prevent output truncation.
+
+### Added
+- **Testing**:
+    - Added `tests/reproduce_issue_176.py` to validate `max_tokens` propagation and chunking behavior across all extractors.
+
+
 ## [0.2.0] - 2026-01-10
 
 ### Added

README.md

@@ -6,7 +6,7 @@
 
 [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyPI version](https://badge.fury.io/py/semantica.svg)](https://pypi.org/project/semantica/)
+[![PyPI version](https://img.shields.io/pypi/v/semantica.svg)](https://pypi.org/project/semantica/)
 [![Monthly Downloads](https://img.shields.io/pypi/dm/semantica)](https://pypi.org/project/semantica/)
 [![Total Downloads](https://static.pepy.tech/badge/semantica)](https://pepy.tech/project/semantica)
 [![Discord](https://img.shields.io/badge/Discord-Join%20Us-7289da?style=flat&logo=discord&logoColor=white)](https://discord.gg/pMHguUzG)
@@ -28,7 +28,7 @@
 
 *The missing fabric between raw data and AI engineering. A comprehensive open-source framework for building semantic layers and knowledge engineering systems that transform unstructured data into AI-ready knowledge — powering Knowledge Graph-Powered RAG (GraphRAG), AI Agents, Multi-Agent Systems, and AI applications with structured semantic knowledge.*
 
-**100% Open Source** • **MIT Licensed** • **Latest Version: 0.2.0** • **Production Ready** • **Community Driven**
+**100% Open Source** • **MIT Licensed** • **Latest Version: 0.2.1** • **Production Ready** • **Community Driven**
 
 [**Discord**](https://discord.gg/pMHguUzG)
 

RELEASE.md

@@ -26,10 +26,10 @@ Before releasing, ensure:
 
 The project uses GitHub Actions for automated releases to PyPI.
 
-1. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.0`).
+1. **Tag the commit**: Create a new git tag for the version (e.g., `v0.2.1`).
    ```bash
-   git tag -a v0.2.0 -m "Release v0.2.0"
-   git push origin v0.2.0
+   git tag -a v0.2.1 -m "Release v0.2.1"
+   git push origin v0.2.1
    ```
 2. **GitHub Action**: The `Release` workflow will automatically trigger, build the package, create a GitHub Release, and publish to PyPI using Trusted Publishing.
 

RELEASE_NOTES_v0.2.0.md

@@ -0,0 +1,95 @@
+# Semantica v0.2.0 Release Notes
+
+We are excited to announce the release of Semantica v0.2.0! This release brings major enhancements to graph database support, document parsing, extraction robustness, and provenance tracking.
+
+## 🚀 Highlights
+
+### Amazon Neptune Support
+- **Native Integration**: Added `AmazonNeptuneStore` for full integration with Amazon Neptune via Bolt and OpenCypher.
+- **Enterprise Security**: Implemented `NeptuneAuthTokenManager` for AWS IAM SigV4 signing with automatic token refresh.
+- **Resilience**: Added robust connection handling with retry logic and backoff for transient errors.
+
+### Docling Integration
+- **High-Fidelity Parsing**: New `DoclingParser` in `semantica.parse` leverages the Docling library for superior document understanding.
+- **Multi-Format Support**: Parse PDF, DOCX, PPTX, XLSX, HTML, and images with state-of-the-art table extraction.
+
+### Robust Extraction Fallbacks
+- **No More Empty Results**: Implemented a "ML/LLM -> Pattern -> Last Resort" fallback chain across all extractors.
+- **Last Resort Strategies**:
+  - **NER**: Identifies capitalized words as generic entities when models fail.
+  - **Relations**: Infers weak connections between adjacent entities.
+
+### Provenance & Tracking
+- **Traceability**: Added `batch_index` and `document_id` metadata to all extracted elements (entities, relations, triplets).
+- **Transparency**: Added count tracking to batch processing logs.
+
+## 📋 Changelog
+
+### Added
+- **Amazon Neptune Support**:
+    - Added `AmazonNeptuneStore` providing Amazon Neptune graph database integration via Bolt protocol and OpenCypher.
+    - Implemented `NeptuneAuthTokenManager` extending Neo4j AuthManager for AWS IAM SigV4 signing with automatic token refresh.
+    - Added robust connection handling: retry logic with backoff for transient errors (signature expired, connection closed) and driver recreation.
+    - Added `graph-amazon-neptune` optional dependency group (boto3, neo4j).
+    - Comprehensive test suite covering all GraphStore interface methods.
+- **Docling Integration**:
+    - Added `DoclingParser` in `semantica.parse` for high-fidelity document parsing using the Docling library.
+    - Supports multi-format parsing (PDF, DOCX, PPTX, XLSX, HTML, images) with superior table extraction and structure understanding.
+    - Implemented as a standalone parser supporting local execution, OCR, and multiple export formats (Markdown, HTML, JSON).
+- **Robust Extraction Fallbacks**:
+    - Implemented comprehensive fallback chains ("ML/LLM" -> "Pattern" -> "Last Resort") across `NERExtractor`, `RelationExtractor`, and `TripletExtractor` to prevent empty result lists.
+    - Added "Last Resort" pattern matching in `NERExtractor` to identify capitalized words as generic entities when all other methods fail.
+    - Added "Last Resort" adjacency-based relation extraction in `RelationExtractor` to create weak connections between adjacent entities if no relations are found.
+    - Added fallback logic in `TripletExtractor` to convert relations to triplets or use rule-based extraction if standard methods fail.
+- **Provenance & Tracking**:
+    - Added count tracking to batch processing logs in `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Added `batch_index` and `document_id` to the metadata of all extracted entities, relations, triplets, semantic roles, and clusters for better traceability.
+- **Semantic Extract Improvements**:
+    - Introduced `auto-chunking` for long text processing in LLM extraction methods (`extract_entities_llm`, `extract_relations_llm`, `extract_triplets_llm`).
+    - Added `silent_fail` parameter to LLM extraction methods for configurable error handling.
+    - Implemented robust JSON parsing and automatic retry logic (3 attempts with exponential backoff) in `BaseProvider` for all LLM providers.
+    - Enhanced `GroqProvider` with better diagnostics and connectivity testing.
+    - Added comprehensive entity, relation, and triplet deduplication for chunked extraction.
+    - Added `semantica/semantic_extract/schemas.py` with canonical Pydantic models for consistent structured output.
+- **Testing**:
+    - Added comprehensive robustness test suite `tests/semantic_extract/test_robustness_fallback.py` for validating extraction fallbacks and metadata propagation.
+    - Added comprehensive unit test suite `tests/embeddings/test_model_switching.py` for verifying dynamic model transitions and dimension updates.
+    - Added end-to-end integration test suite for Knowledge Graph pipeline validation (GraphBuilder -> EntityResolver -> GraphAnalyzer).
+- **Other**:
+    - Added missing dependencies `GitPython` and `chardet` to `pyproject.toml`.
+    - Robustified ID extraction across `CentralityCalculator`, `CommunityDetector`, and `ConnectivityAnalyzer` to handle various entity formats.
+    - Improved `Entity` class hashability and equality logic in `utils/types.py`.
+
+### Changed
+- **Deduplication & Conflict Logic**:
+    - Removed internal deduplication logic from `NERExtractor`, `RelationExtractor`, and `TripletExtractor`.
+    - Removed consistency/conflict checking from `ExtractionValidator` to defer to dedicated `semantica/conflicts` module.
+    - Removed `_deduplicate_*` methods from `semantica/semantic_extract/methods.py`.
+- **Batch Processing & Consistency**:
+    - Standardized batch processing across all extractors (`NERExtractor`, `RelationExtractor`, `TripletExtractor`, `SemanticNetworkExtractor`, `EventDetector`, `SemanticAnalyzer`, `CoreferenceResolver`) using a unified `extract`/`analyze`/`resolve` method pattern with progress tracking.
+    - Added provenance metadata (`batch_index`, `document_id`) to `SemanticNetwork` nodes/edges, `Event` objects, `SemanticRole` results, `CoreferenceChain` mentions, and `SemanticCluster` (tracking source `document_ids`).
+    - Updated `SemanticClusterer.cluster` and `SemanticAnalyzer.cluster_semantically` to accept list of dictionaries (with `content` and `id` keys) for better document tracking during clustering.
+    - Removed legacy `check_triplet_consistency` from `TripletExtractor`.
+    - Removed `validate_consistency` and `_check_consistency` from `ExtractionValidator`.
+- **Weighted Scoring**:
+    - Clarified weighted confidence scoring (50% Method Confidence + 50% Type Similarity) in comments.
+    - Explicitly labeled "Type Similarity" as "user-provided" in code comments to remove ambiguity.
+- **Refactoring**:
+    - Fixed orchestrator lazy property initialization and configuration normalization logic in `Orchestrator`.
+    - Verified and aligned `FileObject.text` property usage in GraphRAG notebooks for consistent content decoding.
+
+### Fixed
+- **Critical Fixes**:
+    - Resolved `NameError` in `extraction_validator.py` by adding missing `Union` import.
+    - Resolved issues where extractors would return empty lists for valid input text when primary extraction methods failed.
+    - Fixed metadata initialization issue in batch processing where `batch_index` and `document_id` were occasionally missing from extracted items.
+    - Ensured `LLMExtraction` methods (`enhance_entities`, `enhance_relations`) return original input instead of failing or returning empty results when LLM providers are unavailable.
+- **Component Fixes**:
+    - Fixed model switching bug in `TextEmbedder` where internal state was not cleared, preventing dynamic updates between `fastembed` and `sentence_transformers` (#160).
+    - Implemented model-intrinsic embedding dimension detection in `TextEmbedder` to ensure consistency between models and vector databases.
+    - Updated `set_model` to properly refresh configuration and dimensions during model switches.
+    - Fixed `TypeError: unhashable type: 'Entity'` in `GraphAnalyzer` when processing graphs with raw `Entity` objects or dictionaries in relationships (#159).
+    - Resolved `AssertionError` in orchestrator tests by aligning test mocks with production component usage.
+    - Fixed dependency compatibility issues by pinning `protobuf==4.25.3` and `grpcio==1.67.1`.
+    - Fixed a bug in `TripletExtractor` where the `validate_triplets` method was shadowed by an internal attribute.
+    - Fixed incorrect `TextSplitter` import path in the `semantic_extract.methods` module.

SECURITY.md

@@ -6,6 +6,7 @@ We actively support the following versions of Semantica with security updates:
 
 | Version | Supported          |
 | ------- | ------------------ |
+| 0.2.1   | :white_check_mark: |
 | 0.2.0   | :white_check_mark: |
 | 0.1.1   | :white_check_mark: |
 | 0.1.0   | :white_check_mark: |

docs/citation.md

@@ -12,22 +12,22 @@ How to cite Semantica in academic papers and research.
   author = {Hawksight AI},
   year = {2026},
   url = {https://github.com/Hawksight-AI/semantica},
-  version = {0.2.0},
+  version = {0.2.1},
   doi = {10.5281/zenodo.XXXXXXX}
 }
 ```
 
 ### APA
-Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.0) [Computer software]. https://github.com/Hawksight-AI/semantica
+Hawksight AI. (2026). *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering* (Version 0.2.1) [Computer software]. https://github.com/Hawksight-AI/semantica
 
 ### MLA
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.0, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.1, GitHub, 2026, https://github.com/Hawksight-AI/semantica.
 
 ### Chicago
-Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.0. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
+Hawksight AI. *Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering*. Version 0.2.1. GitHub, 2026. https://github.com/Hawksight-AI/semantica.
 
 ### IEEE
-Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.0, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
+Hawksight AI, "Semantica: An Open Source Framework for Semantic Layers and Knowledge Engineering," Version 0.2.1, GitHub, 2026. [Online]. Available: https://github.com/Hawksight-AI/semantica
 
 ---
 

docs/reference/semantic_extract.md

@@ -185,7 +185,8 @@ Core entity extraction implementation used by notebooks and lower-level integrat
 |-----------|------|---------|-------------|
 | `method` | str or list | `"ml"` | Method(s): "ml", "llm", "pattern", "regex", "huggingface" |
 | `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
+| `max_text_length` | int | `64000` | Max text length for auto-chunking (LLM only) |
+| `max_tokens` | int | `None` | Max output tokens for LLM generation |
 | `**config` | dict | `{}` | Method-specific config (e.g., `model`, `provider`) |
 
 **Methods:**
@@ -204,11 +205,12 @@ from semantica.semantic_extract import NERExtractor
 extractor = NERExtractor(method="ml", model="en_core_web_trf")
 entities = extractor.extract("Elon Musk leads SpaceX.")
 
-# 2. LLM (OpenAI/Gemini/etc)
+# 2. LLM (OpenAI/Gemini/Groq/etc)
 extractor = NERExtractor(
     method="llm", 
-    provider="openai", 
-    model="gpt-4",
+    provider="groq", 
+    model="llama-3.3-70b-versatile",
+    max_tokens=2048, # Increased output limit
     temperature=0.0
 )
 
@@ -340,7 +342,8 @@ Extracts RDF triplets (Subject-Predicate-Object).
 | `include_provenance` | bool | `False` | Track source sentences |
 | `method` | str | `"pattern"` | Extraction method ("pattern", "rules", "huggingface", "llm") |
 | `silent_fail` | bool | `False` | Return empty list on error instead of raising (LLM only) |
-| `max_text_length` | int | `None` | Max text length for auto-chunking (LLM only) |
+| `max_text_length` | int | `64000` | Max text length for auto-chunking (LLM only) |
+| `max_tokens` | int | `None` | Max output tokens for LLM generation |
 
 **Methods:**
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "semantica"
-version = "0.2.0"
+version = "0.2.1"
 description = "🧠 Semantica - An Open Source Framework for building Semantic Layers and Knowledge Engineering "
 readme = "README.md"
 license = {text = "MIT"}

semantica/__init__.py

@@ -10,7 +10,7 @@ Main exports:
     - Config: Configuration management
 """
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 __author__ = "Semantica Contributors"
 __license__ = "MIT"
 

semantica/semantic_extract/methods.py

@@ -642,12 +642,12 @@ def extract_entities_llm(
     if max_text_length is None:
         # Provider-specific defaults
         max_text_length = {
-            "groq": 8000,
-            "openai": 4000,
-            "gemini": 16000,
-            "anthropic": 16000,
-            "deepseek": 16000,
-        }.get(provider.lower(), 4000)
+            "groq": 64000,
+            "openai": 64000,
+            "gemini": 64000,
+            "anthropic": 64000,
+            "deepseek": 64000,
+        }.get(provider.lower(), 32000)
     
     if len(text) > max_text_length:
         logger.info(f"Text length ({len(text)}) exceeds limit ({max_text_length}). Chunking...")
@@ -701,7 +701,7 @@ Text to extract from:
 {text}"""
         
         # Use typed generation with Pydantic schema
-        result_obj = llm.generate_typed(prompt, schema=EntitiesResponse)
+        result_obj = llm.generate_typed(prompt, schema=EntitiesResponse, **kwargs)
         
         # Convert back to internal Entity format
         entities = []
@@ -723,6 +723,25 @@ Text to extract from:
         return entities
         
     except Exception as e:
+        # Check for length/token limit errors
+        error_msg_str = str(e).lower()
+        if "length" in error_msg_str or "max_tokens" in error_msg_str:
+            logger.warning(f"LLM output truncated due to length limit. Reducing chunk size and retrying... ({e})")
+            
+            # Determine new chunk size (halve it)
+            current_max = max_text_length or len(text)
+            new_max = current_max // 2
+            
+            if new_max > 100: # Minimum viable chunk size check
+                return _extract_entities_chunked(
+                    text, 
+                    provider=provider, 
+                    model=model, 
+                    silent_fail=silent_fail, 
+                    max_text_length=new_max,
+                    **kwargs
+                )
+
         error_msg = f"LLM entity extraction failed ({provider}/{model}): {e}"
         logger.error(error_msg, exc_info=True)
         if not silent_fail:
@@ -1350,13 +1369,14 @@ def extract_relations_llm(
 
     # 3. TEXT LENGTH CHECK AND CHUNKING
     if max_text_length is None:
+        # Default limits for chunking only - NOT for LLM generation
         max_text_length = {
-            "groq": 8000,
-            "openai": 4000,
-            "gemini": 16000,
-            "anthropic": 16000,
-            "deepseek": 16000,
-        }.get(provider.lower(), 4000)
+            "groq": 64000,
+            "openai": 64000,
+            "gemini": 64000,
+            "anthropic": 64000,
+            "deepseek": 64000,
+        }.get(provider.lower(), 32000)
     
     if len(text) > max_text_length:
         logger.info(f"Text length ({len(text)}) exceeds limit for relations. Chunking...")
@@ -1408,7 +1428,8 @@ Entities found in text: {entities_str}"""
 
     try:
         # Use typed generation with Pydantic schema
-        result_obj = llm.generate_typed(prompt, schema=RelationsResponse)
+        # Pass kwargs to allow max_tokens and other parameters to be used
+        result_obj = llm.generate_typed(prompt, schema=RelationsResponse, **kwargs)
         
         # Convert back to internal Relation format
         relations = []
@@ -1441,6 +1462,23 @@ Entities found in text: {entities_str}"""
         return relations
         
     except Exception as e:
+        # Check for length/token limit errors
+        error_msg_str = str(e).lower()
+        if "length" in error_msg_str or "max_tokens" in error_msg_str:
+            logger.warning(f"LLM output truncated due to length limit. Reducing chunk size and retrying... ({e})")
+            
+            # Determine new chunk size (halve it)
+            current_max = max_text_length or len(text)
+            new_max = current_max // 2
+            
+            if new_max > 100: # Minimum viable chunk size check
+                return _extract_relations_chunked(
+                    text, entities, provider=provider, model=model,
+                    silent_fail=silent_fail, max_text_length=new_max,
+                    structured_output_mode=structured_output_mode,
+                    **kwargs
+                )
+
         error_msg = f"LLM relation extraction failed ({provider}/{model}): {e}"
         logger.error(error_msg, exc_info=True)
         if not silent_fail:
@@ -1666,7 +1704,7 @@ def extract_triplets_huggingface(
     """HuggingFace triplet extraction."""
     loader = HuggingFaceModelLoader(device=device)
     model_obj = loader.load_triplet_model(model)
-    results = loader.extract_triplets(model_obj, text)
+    results = loader.extract_triplets(model_obj, text, **kwargs)
 
     triplets = []
     for result in results:
@@ -1742,13 +1780,14 @@ def extract_triplets_llm(
 
     # 3. TEXT LENGTH CHECK AND CHUNKING
     if max_text_length is None:
+        # Default limits for chunking only - NOT for LLM generation
         max_text_length = {
-            "groq": 8000,
-            "openai": 4000,
-            "gemini": 16000,
-            "anthropic": 16000,
-            "deepseek": 16000,
-        }.get(provider.lower(), 4000)
+            "groq": 64000,
+            "openai": 64000,
+            "gemini": 64000,
+            "anthropic": 64000,
+            "deepseek": 64000,
+        }.get(provider.lower(), 32000)
     
     if len(text) > max_text_length:
         logger.info(f"Text length ({len(text)}) exceeds limit for triplets. Chunking...")
@@ -1797,7 +1836,7 @@ Text to extract from:
 
     try:
         # Use typed generation with Pydantic schema
-        result_obj = llm.generate_typed(prompt, schema=TripletsResponse)
+        result_obj = llm.generate_typed(prompt, schema=TripletsResponse, **kwargs)
         
         # Convert back to internal Triplet format
         triplets = []
@@ -1818,6 +1857,22 @@ Text to extract from:
         return triplets
         
     except Exception as e:
+        # Check for length/token limit errors
+        error_msg_str = str(e).lower()
+        if "length" in error_msg_str or "max_tokens" in error_msg_str:
+            logger.warning(f"LLM output truncated due to length limit. Reducing chunk size and retrying... ({e})")
+            
+            # Determine new chunk size (halve it)
+            current_max = max_text_length or len(text)
+            new_max = current_max // 2
+            
+            if new_max > 100: # Minimum viable chunk size check
+                return _extract_triplets_chunked(
+                    text, provider=provider, model=model, 
+                    silent_fail=silent_fail, max_text_length=new_max, 
+                    **kwargs
+                )
+
         error_msg = f"LLM triplet extraction failed ({provider}/{model}): {e}"
         logger.error(error_msg, exc_info=True)
         if not silent_fail:

semantica/semantic_extract/providers.py

@@ -318,6 +318,11 @@ class BaseProvider:
                         "temperature": kwargs.get("temperature", 0.1), # Low temp for structured
                     }
                     
+                    # Pass through other common parameters
+                    for param in ["max_tokens", "max_completion_tokens", "top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user", "top_k"]:
+                        if param in kwargs:
+                            create_kwargs[param] = kwargs[param]
+
                     # Add provider-specific params
                     if provider_name == "GroqProvider":
                         create_kwargs["response_format"] = {"type": "json_object"}
@@ -464,11 +469,24 @@ class OpenAIProvider(BaseProvider):
                 "OpenAI client not initialized. Set OPENAI_API_KEY or pass api_key."
             )
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        
+        # Support max_tokens and max_completion_tokens (for o1 models)
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+            
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+            
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -476,12 +494,25 @@ class OpenAIProvider(BaseProvider):
         if not self.client:
             raise ProcessingError("OpenAI client not initialized.")
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            response_format={"type": "json_object"},
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "response_format": {"type": "json_object"},
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        
+        # Support max_tokens and max_completion_tokens
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "logit_bias", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        response = self.client.chat.completions.create(**create_kwargs)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -526,8 +557,17 @@ class GeminiProvider(BaseProvider):
                 "Gemini client not initialized. Set GEMINI_API_KEY or pass api_key."
             )
 
+        generation_config = {"temperature": kwargs.get("temperature", 0.3)}
+        if "max_tokens" in kwargs:
+            generation_config["max_output_tokens"] = kwargs["max_tokens"]
+            
+        # Pass through other common parameters
+        for param in ["top_p", "top_k", "stop_sequences", "candidate_count"]:
+            if param in kwargs:
+                generation_config[param] = kwargs[param]
+
         response = self.client.generate_content(
-            prompt, generation_config={"temperature": kwargs.get("temperature", 0.3)}
+            prompt, generation_config=generation_config
         )
         return response.text
 
@@ -621,11 +661,24 @@ class GroqProvider(BaseProvider):
                 "Groq client not initialized. Set GROQ_API_KEY or pass api_key."
             )
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        
+        # Support max_tokens and max_completion_tokens
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+            
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -638,12 +691,25 @@ class GroqProvider(BaseProvider):
         if "json" not in prompt.lower():
             json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
 
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": json_prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-            response_format={"type": "json_object"},
-        )
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": json_prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+            "response_format": {"type": "json_object"},
+        }
+        
+        # Support max_tokens and max_completion_tokens
+        if "max_completion_tokens" in kwargs:
+            create_kwargs["max_completion_tokens"] = kwargs["max_completion_tokens"]
+        elif "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+            
+        # Pass through other common parameters
+        for param in ["top_p", "frequency_penalty", "presence_penalty", "seed", "stop", "user"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        response = self.client.chat.completions.create(**create_kwargs)
         try:
             return self._parse_json(response.choices[0].message.content)
         except Exception as e:
@@ -690,11 +756,23 @@ class AnthropicProvider(BaseProvider):
                 "Anthropic client not initialized. Set ANTHROPIC_API_KEY or pass api_key."
             )
 
-        response = self.client.messages.create(
-            model=kwargs.get("model", self.model),
-            max_tokens=kwargs.get("max_tokens", 4096),
-            messages=[{"role": "user", "content": prompt}],
-        )
+        # Anthropic requires max_tokens. 
+        # We rely on kwargs, but fallback to 8192 (safe max for newer models) if not provided.
+        max_tokens = kwargs.get("max_tokens", 8192)
+        
+        # Prepare arguments
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "max_tokens": max_tokens,
+            "messages": [{"role": "user", "content": prompt}],
+        }
+        
+        # Pass through other common parameters
+        for param in ["temperature", "top_p", "top_k", "stop_sequences", "system", "metadata"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+
+        response = self.client.messages.create(**create_kwargs)
         return response.content[0].text
 
     def generate_structured(self, prompt: str, **kwargs) -> dict:
@@ -703,11 +781,23 @@ class AnthropicProvider(BaseProvider):
             raise ProcessingError("Anthropic client not initialized.")
 
         json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
-        response = self.client.messages.create(
-            model=kwargs.get("model", self.model),
-            max_tokens=kwargs.get("max_tokens", 4096),
-            messages=[{"role": "user", "content": json_prompt}],
-        )
+        
+        # Anthropic requires max_tokens. 
+        max_tokens = kwargs.get("max_tokens", 8192)
+        
+        # Prepare arguments
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "max_tokens": max_tokens,
+            "messages": [{"role": "user", "content": json_prompt}],
+        }
+        
+        # Pass through other common parameters
+        for param in ["temperature", "top_p", "top_k", "stop_sequences", "system", "metadata"]:
+            if param in kwargs:
+                create_kwargs[param] = kwargs[param]
+        
+        response = self.client.messages.create(**create_kwargs)
         try:
             return self._parse_json(response.content[0].text)
         except Exception as e:
@@ -758,10 +848,25 @@ class OllamaProvider(BaseProvider):
                 "Ollama client not initialized. Make sure Ollama is running."
             )
 
+        options = {"temperature": kwargs.get("temperature", 0.3)}
+        
+        if "max_tokens" in kwargs:
+            options["num_predict"] = kwargs["max_tokens"]
+            
+        if "num_ctx" in kwargs:
+            options["num_ctx"] = kwargs["num_ctx"]
+        elif "context_window" in kwargs:
+            options["num_ctx"] = kwargs["context_window"]
+        
+        # Pass through other common options
+        for param in ["top_p", "top_k", "repeat_penalty", "seed"]:
+            if param in kwargs:
+                options[param] = kwargs[param]
+
         response = self.client.generate(
             model=kwargs.get("model", self.model),
             prompt=prompt,
-            options={"temperature": kwargs.get("temperature", 0.3)},
+            options=options,
         )
         return response.get("response", "")
 
@@ -771,10 +876,26 @@ class OllamaProvider(BaseProvider):
             raise ProcessingError("Ollama client not initialized.")
 
         json_prompt = f"{prompt}\n\nReturn the response as valid JSON only."
+        
+        options = {"temperature": kwargs.get("temperature", 0.3)}
+        
+        if "max_tokens" in kwargs:
+            options["num_predict"] = kwargs["max_tokens"]
+            
+        if "num_ctx" in kwargs:
+            options["num_ctx"] = kwargs["num_ctx"]
+        elif "context_window" in kwargs:
+            options["num_ctx"] = kwargs["context_window"]
+
+        # Pass through other common options
+        for param in ["top_p", "top_k", "repeat_penalty", "seed"]:
+            if param in kwargs:
+                options[param] = kwargs[param]
+
         response = self.client.generate(
             model=kwargs.get("model", self.model),
             prompt=json_prompt,
-            options={"temperature": kwargs.get("temperature", 0.3)},
+            options=options,
         )
         try:
             return self._parse_json(response.get("response", "{}"))
@@ -808,11 +929,16 @@ class DeepSeekProvider(BaseProvider):
     def generate(self, prompt: str, **kwargs) -> str:
         if not self.client:
             raise ProcessingError("DeepSeek client not initialized. Set DEEPSEEK_API_KEY or pass api_key.")
-        response = self.client.chat.completions.create(
-            model=kwargs.get("model", self.model),
-            messages=[{"role": "user", "content": prompt}],
-            temperature=kwargs.get("temperature", 0.3),
-        )
+            
+        create_kwargs = {
+            "model": kwargs.get("model", self.model),
+            "messages": [{"role": "user", "content": prompt}],
+            "temperature": kwargs.get("temperature", 0.3),
+        }
+        if "max_tokens" in kwargs:
+            create_kwargs["max_tokens"] = kwargs["max_tokens"]
+
+        response = self.client.chat.completions.create(**create_kwargs)
         return response.choices[0].message.content
     def generate_structured(self, prompt: str, **kwargs) -> Union[dict, list]:
         """Generate structured output."""
@@ -879,11 +1005,27 @@ class HuggingFaceLLMProvider(BaseProvider):
             raise ProcessingError("HuggingFace model not initialized.")
 
         inputs = self.tokenizer.encode(prompt, return_tensors="pt").to(self.device)
+        
+        # Use max_new_tokens if available, otherwise fallback to max_length with a safe default
+        generate_kwargs = {
+            "temperature": kwargs.get("temperature", 0.7),
+            "do_sample": True,
+        }
+        
+        if "max_new_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_new_tokens"]
+        elif "max_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_tokens"]
+            
+        # Support legacy max_length if explicitly provided
+        if "max_length" in kwargs:
+            generate_kwargs["max_length"] = kwargs["max_length"]
+            # Remove max_new_tokens if max_length is set to avoid conflict
+            generate_kwargs.pop("max_new_tokens", None)
+
         outputs = self.model.generate(
             inputs,
-            max_length=kwargs.get("max_length", 100),
-            temperature=kwargs.get("temperature", 0.7),
-            do_sample=True,
+            **generate_kwargs
         )
         generated_text = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
         # Remove the original prompt from the response
@@ -1007,16 +1149,33 @@ class HuggingFaceModelLoader:
         # This would need to be customized based on the model architecture
         return model(text)
 
-    def extract_triplets(self, model, text: str) -> List[Dict]:
+    def extract_triplets(self, model, text: str, **kwargs) -> List[Dict]:
         """Extract triplets using loaded model."""
         tokenizer = model["tokenizer"]
         model_obj = model["model"]
         device = model["device"]
 
+        # Use kwargs for max_length, default to 512 for input and 128 for output if not specified
+        max_input_length = kwargs.get("max_input_length", 512)
+        max_length = kwargs.get("max_length", 128)
+        
+        # Allow max_new_tokens as well
+        generate_kwargs = {"max_length": max_length}
+        if "max_new_tokens" in kwargs:
+            generate_kwargs["max_new_tokens"] = kwargs["max_new_tokens"]
+            # If max_new_tokens is set, we might want to remove max_length or ensure they don't conflict
+            # For Seq2Seq, max_length usually refers to the total length of the target sequence
+            
+        # Pass other generation args
+        for param in ["num_beams", "temperature", "top_p", "top_k", "do_sample"]:
+            if param in kwargs:
+                generate_kwargs[param] = kwargs[param]
+
         inputs = tokenizer(
-            text, return_tensors="pt", truncation=True, max_length=512
+            text, return_tensors="pt", truncation=True, max_length=max_input_length
         ).to(device)
-        outputs = model_obj.generate(**inputs, max_length=128)
+        
+        outputs = model_obj.generate(**inputs, **generate_kwargs)
         decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
 
         # Parse decoded output (format depends on model)

semantica/semantic_extract/semantic_extract_usage.md

@@ -120,9 +120,22 @@ entities = extractor.extract(
     provider="openai", 
     model="gpt-4",
     silent_fail=False,      # Raise ProcessingError on failure (default)
-    max_text_length=4000     # Auto-chunking for long text
+    max_text_length=4000,   # Auto-chunking for long text (default: 64k for major providers)
+    max_tokens=4096,        # Explicitly control generation output length
+    temperature=0.0
 )
 print(f"LLM method: {len(entities)} entities")
+
+# Groq extraction with long context support
+# Groq defaults to 64k chunking limit for models like llama-3.3-70b
+groq_extractor = NERExtractor(method="llm")
+groq_entities = groq_extractor.extract(
+    text,
+    provider="groq",
+    model="llama-3.3-70b-versatile",
+    max_tokens=8000 # Passed directly to Groq API
+)
+print(f"Groq method: {len(groq_entities)} entities")
 ```
 
 ### Using NERExtractor Directly
@@ -221,6 +234,8 @@ relations = extractor.extract(
     text, 
     entities=entities, 
     provider="openai",
+    model="gpt-4",
+    max_tokens=2048, # Increased output limit for many relations
     silent_fail=True  # Return empty list if extraction fails
 )
 ```
@@ -284,7 +299,8 @@ triplets = extractor.extract_triplets(
     text, 
     provider="openai", 
     model="gpt-4",
-    max_text_length=2000  # Force chunking for long text
+    max_text_length=64000, # Large default chunk size supported
+    max_tokens=4096 # Ensure enough tokens for all triplets
 )
 ```
 

tests/reproduce_issue_176.py

@@ -0,0 +1,100 @@
+
+import unittest
+from unittest.mock import MagicMock, patch
+from semantica.semantic_extract.methods import extract_relations_llm, extract_entities_llm, extract_triplets_llm
+from semantica.semantic_extract.ner_extractor import Entity
+
+class TestMaxTokensPropagation(unittest.TestCase):
+    @patch("semantica.semantic_extract.methods.create_provider")
+    def test_max_tokens_propagation_relations(self, mock_create_provider):
+        """Test that max_tokens is passed to generate_typed in extract_relations_llm."""
+        # Setup mock
+        mock_llm = MagicMock()
+        mock_create_provider.return_value = mock_llm
+        mock_llm.is_available.return_value = True
+        
+        # Setup return value to avoid pydantic validation errors
+        mock_response = MagicMock()
+        mock_response.relations = []
+        mock_llm.generate_typed.return_value = mock_response
+
+        # Create dummy entities
+        entities = [Entity(text="Foo", label="ORG", start_char=0, end_char=3)]
+
+        # Call the function with max_tokens
+        extract_relations_llm(
+            text="some text",
+            entities=entities,
+            provider="openai",
+            model="gpt-4",
+            max_tokens=128000
+        )
+
+        # Check if generate_typed was called with max_tokens
+        args, kwargs = mock_llm.generate_typed.call_args
+        
+        print(f"Relations Call kwargs: {kwargs}")
+        
+        self.assertIn("max_tokens", kwargs)
+        self.assertEqual(kwargs["max_tokens"], 128000)
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    def test_max_tokens_propagation_entities(self, mock_create_provider):
+        """Test that max_tokens is passed to generate_typed in extract_entities_llm."""
+        # Setup mock
+        mock_llm = MagicMock()
+        mock_create_provider.return_value = mock_llm
+        mock_llm.is_available.return_value = True
+        
+        # Setup return value to avoid pydantic validation errors
+        mock_response = MagicMock()
+        mock_response.entities = []
+        mock_llm.generate_typed.return_value = mock_response
+
+        # Call the function with max_tokens
+        extract_entities_llm(
+            text="some text",
+            provider="openai",
+            model="gpt-4",
+            max_tokens=128000
+        )
+
+        # Check if generate_typed was called with max_tokens
+        args, kwargs = mock_llm.generate_typed.call_args
+        
+        print(f"Entities Call kwargs: {kwargs}")
+        
+        self.assertIn("max_tokens", kwargs)
+        self.assertEqual(kwargs["max_tokens"], 128000)
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    def test_max_tokens_propagation_triplets(self, mock_create_provider):
+        """Test that max_tokens is passed to generate_typed in extract_triplets_llm."""
+        # Setup mock
+        mock_llm = MagicMock()
+        mock_create_provider.return_value = mock_llm
+        mock_llm.is_available.return_value = True
+        
+        # Setup return value to avoid pydantic validation errors
+        mock_response = MagicMock()
+        mock_response.triplets = []
+        mock_llm.generate_typed.return_value = mock_response
+
+        # Call the function with max_tokens
+        extract_triplets_llm(
+            text="some text",
+            provider="openai",
+            model="gpt-4",
+            max_tokens=128000
+        )
+
+        # Check if generate_typed was called with max_tokens
+        args, kwargs = mock_llm.generate_typed.call_args
+        
+        print(f"Triplets Call kwargs: {kwargs}")
+        
+        self.assertIn("max_tokens", kwargs)
+        self.assertEqual(kwargs["max_tokens"], 128000)
+
+if __name__ == "__main__":
+    unittest.main()

tests/semantic_extract/test_provider_limits.py

@@ -0,0 +1,157 @@
+
+import unittest
+from unittest.mock import patch, MagicMock
+import sys
+import os
+
+# Ensure we test the local code, not the installed package
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')))
+
+from semantica.semantic_extract.ner_extractor import Entity
+from semantica.semantic_extract.relation_extractor import Relation
+from semantica.semantic_extract.triplet_extractor import Triplet
+from semantica.semantic_extract.methods import extract_entities_llm
+# We import providers later inside tests to allow patching
+
+class TestSemanticClasses:
+    """Test that core semantic classes do not have hardcoded max lengths."""
+
+    def test_entity_no_max_length(self):
+        long_text = "a" * 10000
+        entity = Entity(text=long_text, label="TEST", start_char=0, end_char=10000)
+        assert entity.text == long_text
+        assert len(entity.text) == 10000
+
+    def test_relation_no_max_length(self):
+        long_text = "a" * 10000
+        e1 = Entity(text="s", label="S", start_char=0, end_char=1)
+        e2 = Entity(text="o", label="O", start_char=0, end_char=1)
+        relation = Relation(subject=e1, predicate=long_text, object=e2)
+        assert relation.predicate == long_text
+
+    def test_triplet_no_max_length(self):
+        long_text = "a" * 10000
+        triplet = Triplet(subject=long_text, predicate="r", object="t")
+        assert triplet.subject == long_text
+
+
+class TestProviderLimits:
+    """Test that providers pass through correct length parameters."""
+
+    def test_openai_max_completion_tokens(self):
+        from semantica.semantic_extract.providers import OpenAIProvider
+        
+        # Patch _init_client to avoid real client creation and import issues
+        with patch.object(OpenAIProvider, '_init_client', return_value=None):
+            provider = OpenAIProvider(api_key="fake")
+            
+            # Manually mock client
+            mock_client = MagicMock()
+            mock_response = MagicMock()
+            mock_response.choices[0].message.content = "result"
+            mock_client.chat.completions.create.return_value = mock_response
+            provider.client = mock_client
+            
+            provider.generate("prompt", max_completion_tokens=12345, top_p=0.9)
+
+            call_kwargs = mock_client.chat.completions.create.call_args[1]
+            assert call_kwargs["max_completion_tokens"] == 12345
+            assert call_kwargs["top_p"] == 0.9
+            assert "max_tokens" not in call_kwargs
+
+    def test_anthropic_max_tokens_defaults(self):
+        from semantica.semantic_extract.providers import AnthropicProvider
+        
+        with patch.object(AnthropicProvider, '_init_client', return_value=None):
+            provider = AnthropicProvider(api_key="fake")
+            
+            mock_client = MagicMock()
+            mock_response = MagicMock()
+            mock_response.content = [MagicMock(text="result")]
+            mock_client.messages.create.return_value = mock_response
+            provider.client = mock_client
+            
+            provider.generate("prompt")
+            
+            # Verify default is 8192 (new limit)
+            call_kwargs = mock_client.messages.create.call_args[1]
+            assert call_kwargs["max_tokens"] == 8192
+
+            # Test override
+            provider.generate("prompt", max_tokens=9999)
+            call_kwargs = mock_client.messages.create.call_args[1]
+            assert call_kwargs["max_tokens"] == 9999
+
+    def test_groq_max_completion_tokens(self):
+        from semantica.semantic_extract.providers import GroqProvider
+        
+        with patch.object(GroqProvider, '_init_client', return_value=None):
+            provider = GroqProvider(api_key="fake")
+            
+            mock_client = MagicMock()
+            mock_response = MagicMock()
+            mock_response.choices[0].message.content = "result"
+            mock_client.chat.completions.create.return_value = mock_response
+            provider.client = mock_client
+            
+            provider.generate("prompt", max_completion_tokens=5000)
+
+            # Verify
+            call_kwargs = mock_client.chat.completions.create.call_args[1]
+            assert call_kwargs["max_completion_tokens"] == 5000
+
+    def test_gemini_params(self):
+        from semantica.semantic_extract.providers import GeminiProvider
+        
+        with patch.object(GeminiProvider, '_init_client', return_value=None):
+            provider = GeminiProvider(api_key="fake")
+            
+            mock_model = MagicMock()
+            mock_response = MagicMock()
+            mock_response.text = "result"
+            mock_model.generate_content.return_value = mock_response
+            provider.client = mock_model
+            
+            provider.generate("prompt", top_k=10, candidate_count=2)
+
+            # Verify
+            call_kwargs = mock_model.generate_content.call_args[1]
+            gen_config = call_kwargs["generation_config"]
+            assert gen_config["top_k"] == 10
+            assert gen_config["candidate_count"] == 2
+
+class TestChunkingDefaults:
+    """Test that chunking defaults have been increased."""
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    @patch("semantica.semantic_extract.methods._extract_entities_chunked")
+    def test_openai_chunking_limit(self, mock_chunked, mock_create_provider):
+        # Setup
+        mock_llm = MagicMock()
+        mock_llm.is_available.return_value = True
+        mock_create_provider.return_value = mock_llm
+        
+        # Text length = 10000 (Greater than old 4000, less than new 64000)
+        long_text = "a" * 10000
+        
+        # Call without explicit max_text_length
+        extract_entities_llm(long_text, provider="openai", api_key="fake")
+        
+        # Should NOT call chunked extraction because default is now 64000
+        mock_chunked.assert_not_called()
+
+    @patch("semantica.semantic_extract.methods.create_provider")
+    @patch("semantica.semantic_extract.methods._extract_entities_chunked")
+    def test_groq_chunking_limit(self, mock_chunked, mock_create_provider):
+        # Setup
+        mock_llm = MagicMock()
+        mock_llm.is_available.return_value = True
+        mock_create_provider.return_value = mock_llm
+        
+        # Text length = 10000 (Greater than old 8000, less than new 64000)
+        long_text = "a" * 10000
+        
+        extract_entities_llm(long_text, provider="groq", api_key="fake")
+        
+        # Should NOT call chunked extraction because default is now 64000
+        mock_chunked.assert_not_called()