Commit Graph
89 Commits
Author SHA1 Message Date
Sameer6305 0a66e1c6ea fix: address Copilot review feedback for Snowflake ingestor 2026-02-04 00:17:25 +05:30
Sameer6305 06d5fad6b9 feat: add Snowflake ingestor for native data warehouse ingestion 2026-02-03 23:27:25 +05:30
Mohd Kaif e9dfcff873 Merge pull request #273 from Sameer6305/feature/arrow-exporter
feat: add Apache Arrow exporter
2026-02-03 21:29:45 +05:30
KaifAhmad1 804de2c13c Fix critical test issues and add JenaStore empty graph tests
- Fixed provenance test KeyError: changed lineage['source'] to lineage['source_documents']
- Fixed import error in test_llm_extraction_fixes.py by removing problematic reload
- Added comprehensive JenaStore empty graph test suite (22 tests)
  - Tests empty graph initialization and operations
  - Validates distinction between None (uninitialized) and empty (0 triplets)
  - Covers all 5 fixed methods: add_triplets, get_triplets, delete_triplet, execute_sparql, serialize
  - Includes edge cases: concurrent operations, benchmarking scenarios, Unicode handling

All 575 tests now passing. Ready for release.
2026-02-02 23:50:00 +05:30
Sameer6305 4ab8b4d72b feat: add Apache Arrow exporter 2026-02-02 22:56:53 +05:30
KaifAhmad1 8a295f97ce Fix(tests): Update temporal tracking assertion to align with auto-parenting logic 2026-02-02 22:17:08 +05:30
KaifAhmad1 8faeb606d7 Fix(provenance): Resolve backward compatibility and metadata issues
- Fix: Provide versioned source history in ProvenanceManager.track_entity to support correct get_all_sources behavior.
- Fix: Ensure get_lineage aggregates and returns metadata fields correctly.
- Fix: Update 	est_real_module_integration.py and 	est_semantic_extract_provenance.py to match correct 	rack_relationship API signature.
- Verified: All provenance tests passed (237/237).
2026-02-02 21:36:29 +05:30
Mohd Kaif bce53a9fe3 Merge pull request #252 from F0rt1s/fix/temperature-compatibility
fix: allow temperature=None to use model defaults
2026-02-01 15:40:05 +05:30
Steffen John 78664ec5f6 test: add tests for temperature=None behavior
Verify that temperature parameter is omitted from API calls when None,
allowing models to use their defaults. Tests cover OpenAI, Groq, Gemini,
Ollama, and DeepSeek providers.
2026-01-31 22:16:09 +01:00
KaifAhmad1 6391dcdf72 Add comprehensive W3C PROV-O compliant provenance tracking module
- Implemented provenance tracking across all 17 Semantica modules
- Added W3C PROV-O compliant schemas (prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom)
- Created ProvenanceManager with InMemory and SQLite storage backends
- Implemented SHA-256 integrity verification for tamper detection
- Added bridge axiom support for domain transformations (L1→L2→L3)
- Created provenance-enabled versions of all modules (opt-in with provenance=True)
- Added comprehensive test suite (237 tests covering edge cases and real scenarios)
- Updated README with accurate claims and compliance disclaimers
- Added complete documentation (usage guide and API reference)
- Zero breaking changes - fully backward compatible
2026-01-31 20:11:38 +05:30
KaifAhmad1 b3f3ac413c Add Enhanced Change Management Module with comprehensive testing and documentation
- New semantica.change_management module with persistent version storage
- Core classes: TemporalVersionManager, OntologyVersionManager, ChangeLogEntry
- Storage backends: SQLite (persistent) and InMemory (fast)
- Features: SHA-256 checksums, detailed entity/relationship diffs, email validation
- Compliance: HIPAA, SOX, FDA 21 CFR Part 11 support with audit trails
- Testing: 104 tests (100% pass) - unit, integration, compliance, performance
- Performance: 17.6ms for 10k entities, 510+ ops/sec concurrent
- Documentation: Complete usage guide and API reference
- Backward compatible with simplified class names
2026-01-30 15:51:07 +05:30
KaifAhmad1 979653e498 Finalize CSV tests after conflict resolution 2026-01-29 00:32:07 +05:30
KaifAhmad1 1a95b0d35f Resolve merge conflicts for PR #244: keep corrected PandasIngestor.from_csv implementation and expanded CSV tests 2026-01-29 00:30:14 +05:30
KaifAhmad1 95ea8de455 CSV ingestion: fix header handling and duplicate header kwarg; add edge-case tests (tab, quoted, multiline, chunksize, NaN) 2026-01-29 00:21:40 +05:30
saloni 327792c830 test_ingest_from_csv().py 2026-01-29 00:09:12 +05:30
saloni e9ec904d87 Create test_ingest_from_csv().py 2026-01-28 23:39:19 +05:30
KaifAhmad1 274114ae67 Merge PR #242: add comprehensive tests for TextNormalizer; adjust punctuation normalization and preserve-case expectation 2026-01-28 20:25:34 +05:30
KaifAhmad1 4ec94b6a5d test(normalize): fix preserve-case expectation; feat(normalize): use explicit unicode mappings for punctuation normalization 2026-01-28 20:25:21 +05:30
ZohaibHassan16 eb1886bee3 test: add compre test units for TextNormalizer 2026-01-28 18:00:18 +05:00
KaifAhmad1 bc875450fa test(ingest): add unit tests for file, web, and feed ingestors (#239) 2026-01-28 14:15:15 +05:30
KaifAhmad1 400a70986d test(ingest): mock boto3 client in tests; fix FeedParser._parse_date to raise ValueError on invalid input 2026-01-28 14:13:15 +05:30
Mohammed237 15b32f49be test(ingest): add unit tests for file, web, and feed ingestors 2026-01-27 18:53:05 +02:00
KaifAhmad1 3968a450a8 chore: release v0.2.5 2026-01-27 22:01:25 +05:30
KaifAhmad1 57d9c2006e feat: enhance Hugging Face integration with robust BYOM support and improved triplet/relation extraction 2026-01-27 21:55:22 +05:30
KaifAhmad1 ebefa61745 Merge branch 'abhiishekk31/main' into pr-fix: Resolve conflicts in Pinecone store implementation 2026-01-26 21:27:46 +05:30
KaifAhmad1 390835ec80 fix: Apply code review fixes for Pinecone integration (PR #220)
- Fix variable shadowing in fetch_vectors (use vector_id instead of id)
- Remove redundant PINECONE_AVAILABLE check in create_index
- Add Pinecone imports and exports to __init__.py
- Add 'pinecone' to SUPPORTED_BACKENDS in vector_store.py
- Add vectorstore-pinecone dependency group to pyproject.toml
- Create vectorstore-all optional dependency group
- Fix duplicate MagicMock import in test_pinecone_store.py
- Update test_pinecone_removal.py with explanatory comment
- Update all docstrings to include Pinecone in supported backends

All fixes address code review feedback and ensure proper integration.
2026-01-26 20:49:29 +05:30
Abhishek Hede 5443a221a0 Added pinecone support with required interface code 2026-01-26 09:53:21 +00:00
KaifAhmad1 bc55dcc57a Fix stuck retries in extraction and enable configurable retry limit. Resolves #207 2026-01-25 21:27:03 +05:30
KaifAhmad1 b3a239ccb1 feat: enhance semantic extraction with BYOM support, NER aggregation, RE implementation, and Triplet improvements
- Implemented 'Bring Your Own Model' (BYOM) support for NER, Relation, and Triplet extraction
- Added NER aggregation strategies (simple, max, average)
- Implemented Relation Extraction via Sequence Classification with entity markers
- Enhanced Triplet Extraction with REBEL post-processing and lazy loading
- Updated all extractors to prioritize runtime options over config defaults
- Added extensive tests and examples (huggingface_demo.py)
- Updated documentation and CHANGELOG
2026-01-24 20:53:21 +05:30
KaifAhmad1 b96e71ae72 feat(ontology): Implement OntologyIngestor and update exports
- Added OntologyIngestor in semantica/ingest/ontology_ingestor.py
- Updated semantica/ontology/__init__.py to export OntologyIngestor
- Updated semantica/ingest/methods.py to use OntologyIngestor
- Added tests for ontology ingestion
- Cleaned up temporary files
2026-01-21 13:46:46 +05:30
KaifAhmad1 1568237ce7 Add high-performance VectorStore ingestion and docs 2026-01-19 13:32:16 +05:30
KaifAhmad1 9f77dfb761 Fix GraphBuilder external relationships; refs #208 #206 2026-01-18 22:10:02 +05:30
KaifAhmad1 8671385cbf fix: break pipeline circular import (#192, #193) and update changelog 2026-01-17 16:02:21 +05:30
KaifAhmad1 96dc5d754a Fix LLM relation extraction parsing and add tests
- Harden LLM relation extraction result handling to parse instructor/OpenAI/Groq variations
- Add structured JSON fallback when typed generation yields zero relations
- Strip acceptance of extra kwargs like max_tokens/max_entities_prompt in relation extraction internals
- Add comprehensive unit tests with mocked LLM provider
- Add integration tests for Groq provider with environment variable API key
- Ensure relation extraction completes and returns results when model identifies relations
2026-01-16 01:19:33 +05:30
KaifAhmad1 c7814616a9 Improve semantic_extract performance and add Groq LLM smoke tests 2026-01-14 17:11:26 +05:30
KaifAhmad1 d2e599bcb0 [SECURITY] Enhance caching security by excluding sensitive keys and using SHA-256 2026-01-14 00:22:41 +05:30
KaifAhmad1 dd7fcd3ddb [FEATURE] Performance Bottlenecks and Scaling Limitations in semantic_extract #186
- Implemented high-throughput parallel batch processing across all core extractors (NERExtractor, RelationExtractor, TripletExtractor, EventDetector, SemanticNetworkExtractor) using ThreadPoolExecutor.

- Added max_workers configuration parameter (default: 1) to all extractor extract() methods.

- Implemented parallel processing for large document chunking in _extract_entities_chunked and _extract_relations_chunked.

- Enhanced ProgressTracker to be thread-safe.

- Optimized setUpClass in tests to reduce Groq LLM initialization overhead.

- Updated documentation and usage examples.
2026-01-14 00:11:30 +05:30
KaifAhmad1 8c3f7f1f0a fix(semantic-extract): resolve incomplete output #176, relax constraints, and add Groq support 2026-01-12 17:15:21 +05:30
KaifAhmad1 87a08e0240 chore: Prepare release v0.2.0 2026-01-10 23:32:10 +05:30
KaifAhmad1 f338b66274 feat: Add provenance metadata and robust fallback chains to semantic extraction
- Implemented ML/LLM -> Pattern -> Last Resort fallback chains for NER, Relation, and Triplet extractors to prevent empty results.
- Added provenance metadata (batch_index, document_id) to all extraction schemas (Entity, Relation, Triplet, etc.).
- Unified batch processing API with progress tracking across all extractors.
- Updated documentation (module usage and reference docs) to reflect new features.
- Added robustness and batch provenance tests.
2026-01-10 20:43:09 +05:30
Mohd Kaif 8b1cd47f51 Merge pull request #167 from don-simpson/feature/amazon-neptune-graph-store
feat: Add Amazon Neptune Database Graph Store Support
2026-01-09 19:27:44 +05:30
KaifAhmad1 a5c531fd29 Fix semantic extraction empty returns, schema validation, and update docs 2026-01-09 18:39:09 +05:30
Don Simpson 976a20496d feat: Add Amazon Neptune Database Graph Store Support
- Implement NeptuneAuthTokenManager extending Neo4j AuthManager for IAM SigV4 signing
- Add automatic token refresh and security exception handling
- Add retry logic with backoff for transient errors (signature expired, connection closed)
- Add connection recovery with driver recreation
- Add NeptuneDriver, NeptuneSession, NeptuneTransaction wrapper classes
- Use native Neptune ~id via id() function for all CRUD operations
- Add graph-amazon-neptune optional dependency group (boto3, neo4j)
- Update cookbook with Amazon Neptune Graph Store examples
- Add comprehensive tests (61 tests covering all GraphStore interface methods)

Closes #151
2026-01-08 20:13:28 -05:00
KaifAhmad1 957c122116 docs: add Docling integration guide, clear code example, and fix parser consistency issues 2026-01-08 21:27:48 +05:30
KaifAhmad1 2bd1d06eb2 fix: resolve model switching bug and implement intrinsic dimension detection in TextEmbedder 2026-01-08 18:56:42 +05:30
KaifAhmad1 58707ff721 fix(kg): resolve 'unhashable type: Entity' in GraphAnalyzer #159
- Robust ID extraction in CentralityCalculator, CommunityDetector, and ConnectivityAnalyzer
- Support for direct Entity objects and dictionaries as node identifiers
- Improved Entity hashability in utils/types.py
- Added integration test to verify fix and prevent regression
2026-01-08 17:23:31 +05:30
KaifAhmad1 e3b53998c3 Fix dependency issues, align GraphRAG notebook, and update changelog 2026-01-07 19:00:30 +05:30
KaifAhmad1 dcd6f25f87 docs: fix KnowledgeGraph mismatch and update tests for issue #144 2026-01-07 15:01:47 +05:30
KaifAhmad1 e712949872 Enhance LLM extraction methods with auto-chunking, robust parsing and improved diagnostics (#149) 2026-01-07 03:03:19 +05:30
KaifAhmad1 eaa1fbefa6 feat(reasoning): add dedicated reasoning tests and fix critical reasoning bugs
- Added tests/reasoning/ directory with unit and integration tests
- Fixed indentation bug in Reasoner.add_fact for dictionary-based relationships
- Fixed regex variable matching in Reasoner._match_pattern
- Fixed variable handling in SPARQLReasoner query expansion
- Cleaned up cookbook and documentation references
2025-12-23 21:26:26 +05:30