Commit Graph
117 Commits
Author SHA1 Message Date
KaifAhmad1 a60791d815 Expand e2e tests with realistic cross-system data sources 2026-02-18 14:44:33 +05:30
KaifAhmad1 c31df5c4d7 Add end-to-end context graph feature test suite 2026-02-18 14:43:07 +05:30
KaifAhmad1 a785247b98 Sanitize cross-system capture errors in returned payload 2026-02-18 12:54:59 +05:30
KaifAhmad1 8bd4df74e1 Apply entity scoping in ContextGraph policy fallback 2026-02-18 12:50:08 +05:30
KaifAhmad1 f9f19f343e Handle FalkorDB policy rows in applicability parsing 2026-02-18 12:37:32 +05:30
KaifAhmad1 41530da25f Strengthen decision trace test assertions 2026-02-18 00:50:25 +05:30
KaifAhmad1 17b0a24257 Log legacy policy constraint drop failures 2026-02-18 00:48:12 +05:30
KaifAhmad1 a98f21e5d3 Log immutable trace lookup failures before fallback 2026-02-18 00:46:13 +05:30
KaifAhmad1 bcb9a65a20 Improve non-persistent decision trace audit logging 2026-02-18 00:44:14 +05:30
KaifAhmad1 3872ea75e1 Make policy application version-aware and deterministic 2026-02-18 00:42:05 +05:30
KaifAhmad1 20b5f7c0ab Fix execute_query wrapper handling in context queries 2026-02-18 00:37:50 +05:30
KaifAhmad1 1a5e34dee8 Harden decision trace capture compatibility paths 2026-02-18 00:27:32 +05:30
KaifAhmad1 ff957be6a8 Enhance context decision tracing and schema compatibility 2026-02-18 00:07:36 +05:30
KaifAhmad1 33c90d8277 Fix Context Graph features - resolve method conflicts and integration issues
- Fix method name conflicts: add_decision -> add_decision_simple, find_precedents -> find_precedents_by_scenario
- Fix Decision ID handling: align tests with Decision model UUID generation behavior
- Fix AgentContext integration: proper handling of context_graph backend in get_causal_chain
- Fix Policy engine: remove invalid auto_generate_id parameter from deserialization
- Fix node type consistency: handle lowercase 'decision' type across all methods
- Fix timestamp handling: proper conversion for string and datetime objects
- Update documentation: correct method names and Decision model usage in examples
- All 62 Context Graph tests passing successfully
- Production ready with comprehensive verification
2026-02-17 13:43:08 +05:30
KaifAhmad1 dcb4f77efc Fix PolicyException naming and auto-ID masking bugs
Bug Fixes:
1. PolicyException naming conflicts:
   - Replace Exception with PolicyException in DecisionRecorder.record_exception()
   - Update _store_exception_node type annotation to PolicyException
   - Fix test imports in test_decision_recorder.py
   - Resolves runtime TypeError from conflicting Exception class name

2. Auto-ID masking missing IDs:
   - Add auto_generate_id parameter to all model __post_init__ methods
   - Update dict-to-model helpers to require IDs (data['decision_id'] vs data.get())
   - Set auto_generate_id=False for deserialization to prevent silent UUID generation
   - Makes missing IDs visible as KeyError instead of masked with auto-generated UUIDs

Files Changed:
- semantica/context/decision_recorder.py: PolicyException usage fixes
- semantica/context/decision_models.py: Auto-ID control parameter
- semantica/context/decision_query.py: Strict ID requirements
- semantica/context/policy_engine.py: Strict ID requirements
- semantica/context/causal_analyzer.py: Strict ID requirements
- tests/context/test_decision_recorder.py: Import fixes

Impact:
- Resolves PolicyException runtime failures
- Prevents silent data corruption from missing IDs
- Maintains backward compatibility for new object creation
- Improves data integrity for deserialization operations
2026-02-16 23:32:58 +05:30
KaifAhmad1 28dc1ed4e9 Fix PolicyException naming conflicts in decision models
- Replace conflicting Exception class name with PolicyException in decision_models.py
- Update all test imports to use PolicyException instead of Exception
- Fix auto ID generation to handle empty strings, not just None
- Resolves import errors in decision tracking test suites
- Maintains backward compatibility while fixing naming conflicts

Fixes: PolicyException naming conflicts preventing test execution
Tests: All decision model tests now pass (19/19)
2026-02-16 23:14:57 +05:30
KaifAhmad1 14f5e05336 Update context documentation with user-friendly approach and strategic emoji placement
- Enhanced README.md with strategic emojis for better visual appeal
- Updated context_usage.md with detailed, user-friendly examples
- Improved docs/reference/context.md with accessible language
- Added AgentContext sections with progressive learning approach
- Maintained professional appearance while improving readability
- Consistent documentation across all context module files
2026-02-16 17:40:50 +05:30
Sameer Kadam 20755e69e2 feat(graph): add Apache AGE backend integration with configuration, registration, tests and documentation (#311) 2026-02-15 15:57:07 +05:30
Mohd Kaif d2f8992ca9 Fix Context Graphs Decision Tracking & Add Comprehensive Tests (#315)
* context_fixes

* context_compliance_fixes

* Delete PR_CONTEXT.md

* Fix Context Graphs decision tracking and add comprehensive tests

- Fix empty/None decision ID handling in ContextGraph.add_decision()
- Fix None metadata handling to prevent TypeError
- Fix causal chain depth logic and node exclusion
- Fix nonexistent node handling in add_causal_relationship()
- Add missing properties field in to_dict serialization
- Add missing from_dict method for graph deserialization
- Fix precedent search direction in find_precedents()
- Fix UUID generation logic in all decision models
- Add comprehensive test suite with 9 tests covering all features
- Test coverage: decision tracking, graph analytics, use cases, performance
- All 71 context tests now passing (100% success rate)

Resolves critical bugs in Context Graphs feature (#290) implementation
2026-02-15 13:20:34 +05:30
Mohd Kaif 5d712d5a62 Context: PolicyEngine fixes, new context tests, cleanup — all tests passing (#312)
* context_fixes

* context_compliance_fixes

* Delete PR_CONTEXT.md
2026-02-14 18:25:24 +05:30
KaifAhmad1 7ad48df600 feat: Add comprehensive context engineering with decision tracking, KG algorithms, and context graphs
- Add decision tracking system with DecisionRecorder, DecisionQuery, CausalChainAnalyzer, PolicyEngine
- Implement KG algorithm integration with centrality, community detection, embeddings, path finding
- Add vector store integration with hybrid search and custom similarity weights
- Enhance context graphs with advanced analytics and decision support
- Update documentation with comprehensive context module reference
- Add production examples for banking and healthcare use cases
- Update README to highlight context graph framework capabilities
- Add comprehensive test suite for all new features
2026-02-12 23:04:29 +05:30
KaifAhmad1 64ce8497f4 resolve(vector_store): Merge conflict resolution for pgvector integration
- Keep pgvector backend integration with _init_backend_store method
- Preserve decision-specific components from main branch
- Maintain both VectorStore backend support and decision pipeline functionality
- Fix duplicate initialization and proper component placement
2026-02-12 12:31:26 +05:30
KaifAhmad1 0254843fa3 [FEATURE] Enhanced Vector Store for Decision Tracking #293
Implement comprehensive decision tracking capabilities with hybrid search, multi-embedding support, and optimized indexing for precedent search.

## Features Implemented

### Enhanced VectorStore Class
- Decision-specific embedding storage with metadata
- Hybrid precedent search combining semantic + structural embeddings
- Configurable weights for semantic (0.7) and structural (0.3) similarity
- Decision metadata filtering and natural language queries
- Batch processing capabilities for multiple decisions
- 100% backward compatibility with existing VectorStore functionality

### New Components
- DecisionEmbeddingPipeline: Generates semantic and structural embeddings
- HybridSimilarityCalculator: Combines embeddings with configurable weights
- DecisionContext: High-level interface for decision management
- DecisionVectorMethods: Convenience functions for one-liner operations

### Enhanced ContextRetriever
- Hybrid precedent search with semantic fallback
- Multi-hop reasoning with configurable depth
- KG algorithm integration (Node2Vec, PathFinder, CommunityDetector, etc.)
- Context expansion with entity relationships

### User-Friendly API
- quick_decision(): One-liner decision recording
- find_precedents(): Effortless precedent search
- explain(): Explainable AI with path tracing
- similar_to(): Find similar decisions
- batch_decisions(): Process multiple decisions
- filter_decisions(): Smart filtering with natural language

### KG Algorithm Integration
- Node2Vec: Structural embeddings from graph topology
- PathFinder: Shortest path algorithms for multi-hop reasoning
- CommunityDetector: Community detection for contextual relationships
- CentralityCalculator: Centrality measures for entity importance
- SimilarityCalculator: Graph-based similarity calculations
- ConnectivityAnalyzer: Graph connectivity analysis

### Explainable AI
- Path tracing through decision relationships
- Confidence scoring with semantic/structural weights
- Comprehensive decision explanations
- Multi-hop context analysis

### Performance Optimizations
- Efficient batch processing (0.028s per decision)
- Optimized vector indexing with padding for inhomogeneous shapes
- Memory-efficient operations (~0.8KB per decision)
- Scalable architecture supporting 1000+ decisions

### Testing & Quality Assurance
- 34+ comprehensive tests covering all functionality
- 100% backward compatibility verification
- End-to-end testing with real-world scenarios
- Performance benchmarking and stress testing
- KG algorithm integration testing

## Backward Compatibility
- All existing VectorStore functionality preserved
- No breaking changes to existing APIs
- Same performance characteristics maintained
- Seamless integration with existing code

## Dependencies
- scipy>=1.9.0 (similarity calculations)
- numpy>=1.21.0 (numerical operations)
- Existing semantica.embeddings and semantica.graph_store

## Files Added/Modified
- semantica/context/decision_context.py (NEW)
- semantica/vector_store/decision_embedding_pipeline.py (NEW)
- semantica/vector_store/hybrid_similarity.py (NEW)
- semantica/vector_store/decision_vector_methods.py (NEW)
- Enhanced semantica/context/context_retriever.py
- Enhanced semantica/vector_store/vector_store.py
- Updated semantica/context/__init__.py and semantica/vector_store/__init__.py
- Enhanced documentation with clear imports and examples
- Comprehensive test suite with >90% coverage

## Acceptance Criteria Met
 VectorStore class enhanced with decision embedding support
 Hybrid precedent search combines semantic + structural embeddings effectively
 HybridSimilarityCalculator works with configurable weights
 DecisionEmbeddingPipeline generates both embedding types
 ContextRetriever supports hybrid precedent search with semantic fallback
 100% backward compatibility maintained
 All tests pass with >90% coverage
 Performance meets targets for precedent search

This implementation provides a comprehensive solution for decision tracking with hybrid search, explainable AI, and KG algorithm integration while maintaining full backward compatibility.
2026-02-11 19:02:33 +05:30
Sameer6305 b473285dcb fix(pgvector): address Copilot review feedback 2026-02-11 18:15:23 +05:30
Sameer6305 95322df8e0 fix(pgvector): address security, reliability, and test issues from review 2026-02-11 17:57:00 +05:30
Sameer6305 163318da1f test(vector_store): Add comprehensive tests for PgVectorStore
- CRUD unit tests
- Similarity search tests with filters
- Index creation tests (HNSW, IVFFlat)
- Docker-based PostgreSQL + pgvector support
- Tests skip if DB unavailable
2026-02-11 14:26:52 +05:30
KaifAhmad1 6fc69aef2e [FEATURE] Enhanced Graph Algorithms in KG Module #292
This commit introduces comprehensive enhancements to the Knowledge Graph (KG) module with:

Major Enhancements:
- Complete algorithm suite with 30+ graph algorithms
- Unified provenance tracking system for all operations
- Comprehensive documentation and test coverage
- Enterprise-grade functionality

New Algorithm Components:
- NodeEmbedder: Node2Vec, DeepWalk, Word2Vec algorithms
- SimilarityCalculator: Cosine, Euclidean, Manhattan, Correlation metrics
- PathFinder: Dijkstra, A*, BFS, K-shortest paths
- LinkPredictor: Preferential attachment, Jaccard, Adamic-Adar
- CentralityCalculator: Degree, Betweenness, Closeness, PageRank
- CommunityDetector: Louvain, Leiden, Label propagation
- ConnectivityAnalyzer: Components, bridges, density analysis

Provenance System:
- GraphBuilderWithProvenance: Graph construction with tracking
- AlgorithmTrackerWithProvenance: Algorithm execution tracking
- Execution IDs and metadata tracking for reproducibility

Test Coverage:
- 5 comprehensive test suites with 40+ test methods
- End-to-end testing for all algorithms
- Real-world scenario testing
- Provenance integration testing

Documentation:
- Updated all module documentation with algorithm listings
- Enhanced KG reference documentation
- Comprehensive usage examples and API documentation

Technical Improvements:
- Unified provenance system integration
- Enhanced error handling and recovery
- Performance optimizations
- NetworkX compatibility with fallback implementations

Resolves: #292
Parent: Context Graphs feature
2026-02-10 21:49:11 +05:30
KaifAhmad1 5e23007658 fix: use RLock in ResourceScheduler to prevent deadlock
- Change threading.Lock() to threading.RLock() in ResourceScheduler.__init__
- Fixes deadlock in allocate_resources() when it calls allocate_cpu/memory/gpu
- Each allocate_* method also acquires the same lock, causing re-entrancy issue
- RLock allows same thread to re-enter lock without blocking itself
- Resolves build_knowledge_base() hanging indefinitely

Test fixes and improvements:
- Add allocation validation to prevent silent failures
- Move progress tracking updates outside lock for better performance
- Add comprehensive regression tests
- Add explanatory comment for RLock usage

Addresses Qodo review concerns:
 Silent allocation failure - now raises ValidationError
 Lock held during progress updates - moved outside lock
 Deadlock prevention - RLock allows re-entrant acquisition

Resolves: #299
2026-02-10 12:22:16 +05:30
Sameer6305 0a66e1c6ea fix: address Copilot review feedback for Snowflake ingestor 2026-02-04 00:17:25 +05:30
Sameer6305 06d5fad6b9 feat: add Snowflake ingestor for native data warehouse ingestion 2026-02-03 23:27:25 +05:30
Mohd Kaif e9dfcff873 Merge pull request #273 from Sameer6305/feature/arrow-exporter
feat: add Apache Arrow exporter
2026-02-03 21:29:45 +05:30
KaifAhmad1 804de2c13c Fix critical test issues and add JenaStore empty graph tests
- Fixed provenance test KeyError: changed lineage['source'] to lineage['source_documents']
- Fixed import error in test_llm_extraction_fixes.py by removing problematic reload
- Added comprehensive JenaStore empty graph test suite (22 tests)
  - Tests empty graph initialization and operations
  - Validates distinction between None (uninitialized) and empty (0 triplets)
  - Covers all 5 fixed methods: add_triplets, get_triplets, delete_triplet, execute_sparql, serialize
  - Includes edge cases: concurrent operations, benchmarking scenarios, Unicode handling

All 575 tests now passing. Ready for release.
2026-02-02 23:50:00 +05:30
Sameer6305 4ab8b4d72b feat: add Apache Arrow exporter 2026-02-02 22:56:53 +05:30
KaifAhmad1 8a295f97ce Fix(tests): Update temporal tracking assertion to align with auto-parenting logic 2026-02-02 22:17:08 +05:30
KaifAhmad1 8faeb606d7 Fix(provenance): Resolve backward compatibility and metadata issues
- Fix: Provide versioned source history in ProvenanceManager.track_entity to support correct get_all_sources behavior.
- Fix: Ensure get_lineage aggregates and returns metadata fields correctly.
- Fix: Update 	est_real_module_integration.py and 	est_semantic_extract_provenance.py to match correct 	rack_relationship API signature.
- Verified: All provenance tests passed (237/237).
2026-02-02 21:36:29 +05:30
Mohd Kaif bce53a9fe3 Merge pull request #252 from F0rt1s/fix/temperature-compatibility
fix: allow temperature=None to use model defaults
2026-02-01 15:40:05 +05:30
Steffen John 78664ec5f6 test: add tests for temperature=None behavior
Verify that temperature parameter is omitted from API calls when None,
allowing models to use their defaults. Tests cover OpenAI, Groq, Gemini,
Ollama, and DeepSeek providers.
2026-01-31 22:16:09 +01:00
KaifAhmad1 6391dcdf72 Add comprehensive W3C PROV-O compliant provenance tracking module
- Implemented provenance tracking across all 17 Semantica modules
- Added W3C PROV-O compliant schemas (prov:Entity, prov:Activity, prov:Agent, prov:wasDerivedFrom)
- Created ProvenanceManager with InMemory and SQLite storage backends
- Implemented SHA-256 integrity verification for tamper detection
- Added bridge axiom support for domain transformations (L1→L2→L3)
- Created provenance-enabled versions of all modules (opt-in with provenance=True)
- Added comprehensive test suite (237 tests covering edge cases and real scenarios)
- Updated README with accurate claims and compliance disclaimers
- Added complete documentation (usage guide and API reference)
- Zero breaking changes - fully backward compatible
2026-01-31 20:11:38 +05:30
KaifAhmad1 b3f3ac413c Add Enhanced Change Management Module with comprehensive testing and documentation
- New semantica.change_management module with persistent version storage
- Core classes: TemporalVersionManager, OntologyVersionManager, ChangeLogEntry
- Storage backends: SQLite (persistent) and InMemory (fast)
- Features: SHA-256 checksums, detailed entity/relationship diffs, email validation
- Compliance: HIPAA, SOX, FDA 21 CFR Part 11 support with audit trails
- Testing: 104 tests (100% pass) - unit, integration, compliance, performance
- Performance: 17.6ms for 10k entities, 510+ ops/sec concurrent
- Documentation: Complete usage guide and API reference
- Backward compatible with simplified class names
2026-01-30 15:51:07 +05:30
KaifAhmad1 979653e498 Finalize CSV tests after conflict resolution 2026-01-29 00:32:07 +05:30
KaifAhmad1 1a95b0d35f Resolve merge conflicts for PR #244: keep corrected PandasIngestor.from_csv implementation and expanded CSV tests 2026-01-29 00:30:14 +05:30
KaifAhmad1 95ea8de455 CSV ingestion: fix header handling and duplicate header kwarg; add edge-case tests (tab, quoted, multiline, chunksize, NaN) 2026-01-29 00:21:40 +05:30
saloni 327792c830 test_ingest_from_csv().py 2026-01-29 00:09:12 +05:30
saloni e9ec904d87 Create test_ingest_from_csv().py 2026-01-28 23:39:19 +05:30
KaifAhmad1 274114ae67 Merge PR #242: add comprehensive tests for TextNormalizer; adjust punctuation normalization and preserve-case expectation 2026-01-28 20:25:34 +05:30
KaifAhmad1 4ec94b6a5d test(normalize): fix preserve-case expectation; feat(normalize): use explicit unicode mappings for punctuation normalization 2026-01-28 20:25:21 +05:30
ZohaibHassan16 eb1886bee3 test: add compre test units for TextNormalizer 2026-01-28 18:00:18 +05:00
KaifAhmad1 bc875450fa test(ingest): add unit tests for file, web, and feed ingestors (#239) 2026-01-28 14:15:15 +05:30
KaifAhmad1 400a70986d test(ingest): mock boto3 client in tests; fix FeedParser._parse_date to raise ValueError on invalid input 2026-01-28 14:13:15 +05:30
Mohammed237 15b32f49be test(ingest): add unit tests for file, web, and feed ingestors 2026-01-27 18:53:05 +02:00