264 Commits
Author SHA1 Message Date
KaifAhmad1 47809f2ef9 Fix: Make PyTorch import lazy to avoid DLL errors on Windows
- Remove top-level torch import from providers.py
- Add lazy imports in HuggingFaceLLMProvider and HuggingFaceModelLoader
- Remove hardcoded API key from notebook
- PyTorch now only loads when HuggingFace providers are instantiated

Fixes #129
2026-01-03 20:39:37 +05:30
KaifAhmad1 96a98fa037 [FEATURE] Integrate Docling for Enhanced Document Parsing
- Added DoclingParser class in semantica/parse/ module
- Created earnings call analysis notebook with Docling integration
- Added docling to pyproject.toml as optional dependency
- Maintained backward compatibility with existing parsers

Closes #124
2026-01-03 18:46:25 +05:30
KaifAhmad1 8cde40d753 Remove trading notebooks and supply chain risk management notebook
- Deleted cookbook/use_cases/trading/01_Risk_Assessment.ipynb
- Deleted cookbook/use_cases/trading/02_News_Sentiment_Analysis.ipynb
- Deleted cookbook/use_cases/supply_chain/02_Supply_Chain_Risk_Management.ipynb
- Removed empty trading directory
- Updated documentation to reflect 14 cookbooks (down from 15)
- Removed all references from README.md, docs/cookbook.md, docs/use-cases.md, docs/index.md, and STRATEGIES_SUMMARY.md
2025-12-30 23:33:27 +05:30
KaifAhmad1 9ef8a7aa18 Update Energy Market Analysis notebook: simplify code, use Semantica effectively, remove redirect_stderr, fix entity/relationship extraction 2025-12-30 21:44:53 +05:30
KaifAhmad1 af17585087 Remove Smart Grid Management notebook and update cookbook count to 15 2025-12-30 20:20:11 +05:30
KaifAhmad1 7b9bd42790 Clean up intelligence analysis notebook: remove unnecessary imports and with blocks, use Semantica built-in methods properly 2025-12-30 20:10:02 +05:30
KaifAhmad1 d8f78cd49e Refactor Criminal Network Analysis notebook: simplify code, use Semantica modules effectively, add interactive visualization, fix GraphRAG queries 2025-12-30 18:19:57 +05:30
KaifAhmad1 f4016237bd Remove healthcare use case: Drug Interactions Analysis
- Deleted cookbook/use_cases/healthcare/02_Drug_Interactions_Analysis.ipynb
- Removed Healthcare section from README.md
- Removed Healthcare section from docs/cookbook.md
- Removed Drug Interactions references from STRATEGIES_SUMMARY.md
- Updated cookbook count from 18 to 17 in all documentation
- Updated docs/index.md to reflect 17 cookbooks
2025-12-30 15:27:21 +05:30
KaifAhmad1 0c5e12f5c9 Remove Clinical Reports Processing notebook and all references
- Delete cookbook/use_cases/healthcare/01_Clinical_Reports_Processing.ipynb
- Delete cookbook/use_cases/healthcare/data/clinical_report.txt
- Remove references from README.md Healthcare section
- Remove Medical Record Analysis card from docs/use-cases.md
- Remove Clinical Reports Processing card from docs/cookbook.md
- Remove entries from STRATEGIES_SUMMARY.md table and rationale
2025-12-30 14:32:55 +05:30
KaifAhmad1 9c97d3236c Fix Fraud Detection notebook: Add real data sources, fix errors, enhance GraphRAG with Context Graph
- Add real CSV and JSON data sources for transactions and accounts
- Fix ConflictDetector, TemporalGraphQuery, and Reasoner errors
- Simplify code to use Semantica modules properly
- Enhance GraphRAG section with Context Graph and Groq LLM
- Add temporal interactive visualization using TemporalVisualizer
- Fix CSV export to use CSVExporter instead of GraphExporter
- Update README.md to mention Context Graph and Context Retriever
2025-12-30 13:45:56 +05:30
KaifAhmad1 599372a50f Update financial data integration notebook:
- Switch entity and relation extraction to ML-based methods (spaCy)
- Fix conflict detection to use detect_temporal_conflicts directly
- Fix graph building to use correct Relation attributes (subject/object/predicate)
- Improve GraphRAG with LLM-based multi-hop reasoning
- Enhance graph analytics output to show all entity types
- Update markdown descriptions with concise bullet points
2025-12-29 23:09:20 +05:30
KaifAhmad1 9f31f825ff Fix Threat Intelligence Hybrid RAG notebook: Update conflict detection, GraphRAG queries, reasoning, and visualization 2025-12-29 22:32:30 +05:30
KaifAhmad1 a674e8c039 fix: resolve Entity TypeError by adding required start_char and end_char fields across cookbook notebooks 2025-12-29 21:35:30 +05:30
KaifAhmad1 d2d6adafdb Add comprehensive progress tracking with Jupyter/Colab support
- Enhanced progress tracker with automatic Jupyter/Colab detection
- Added detailed progress tracking to all deduplication modules
- Added detailed progress tracking to all semantic_extract modules
- Progress tracker now always enabled automatically
- Shows remaining items, percentages, ETA, and processing rates
- Works in both Jupyter notebooks and Google Colab
- Dynamic update intervals based on dataset size
- Improved display handling for Colab compatibility
2025-12-29 13:11:22 +05:30
KaifAhmad1 00575b135e Add progress tracker enable check to all modules
- Added enable check to normalize module (8 files)
- Added enable check to ontology module (16 files)
- Added enable check to ingest module (4 files)
- Added enable check to graph_store module (3 files)
- Ensures progress tracking is enabled by default in all modules
- Total: 112 files updated across the codebase
2025-12-28 22:13:11 +05:30
KaifAhmad1 53db5bbdc0 Add progress tracking with ETA to all long-running operations
- Fixed ConflictDetector to use update_progress() with counts/ETA for type, temporal, and logical conflict detection
- Fixed NERExtractor batch operations to show progress with ETA
- Fixed RelationExtractor batch operations to show progress with ETA
- All modules now display clear progress bars with percentage, counts, and estimated time remaining
2025-12-28 20:32:56 +05:30
KaifAhmad1 c7559afdc5 Add progress tracking with ETA to all modules
- Enhanced ProgressItem with ETA fields (progress_percentage, total_items, processed_items, estimated_remaining)
- Added update_progress() and _calculate_eta() methods to ProgressTracker
- Updated ConsoleProgressDisplay and JupyterProgressDisplay to show progress with ETA
- Added progress tracking to deduplication modules (DuplicateDetector, EntityMerger, SimilarityCalculator, ClusterBuilder)
- Added progress tracking to conflicts modules (ConflictDetector, ConflictResolver)
- Added progress tracking to ingest, parse, kg, core, embeddings, and triplet_store modules
- All modules now display progress percentage, item counts, ETA, and processing rate
2025-12-28 19:26:58 +05:30
KaifAhmad1 40e5c5110c Optimize GraphBuilder entity processing performance
- Add fast path for dictionary entities/relationships to bypass _process_item overhead
- Improve entity recognition to handle 'text' and 'type' fields directly
- Significantly improve processing speed from ~0.8/s to thousands/s
- Fixes performance bottleneck in knowledge graph building
2025-12-28 17:25:28 +05:30
KaifAhmad1 c3555e0cfd Add LLM providers module and GraphRAG reasoning features
- Add semantica.llms module with Groq, OpenAI, HuggingFace, and LiteLLM providers
- Add query_with_reasoning() method for multi-hop reasoning with LLM-generated responses
- Update ContextRetriever and AgentContext with reasoning capabilities
- Add comprehensive documentation for LLM providers and GraphRAG reasoning
- Update README and docs with new features
- Update notebook examples to use new query_with_reasoning() method
2025-12-27 23:23:32 +05:30
KaifAhmad1 94ddcc4d33 Fix blockchain transaction network analysis notebook
- Fix TemporalGraphQuery: Change detect_temporal_patterns to query_temporal_pattern
- Fix GraphAnalyzer: Replace find_paths with direct relationship queries and BFS implementation
- Fix KGVisualizer: Change visualize() to visualize_network() with interactive visualization
- Fix GraphExporter: Remove unsupported CSV format, use export_csv for CSV export
- Add proper imports and improve error handling
- Enhance visualization with force-directed layout and better interactivity
2025-12-27 21:48:35 +05:30
KaifAhmad1 7a652cf227 Fix GraphBuilder progress tracking for list of dict sources
- Added support for detecting and merging list of dict sources with entities/relationships
- Progress tracking now shows ETA and remaining items when sources is a list
- Fixes issue where progress wasn't displayed when passing list of dicts to build()
2025-12-27 19:21:07 +05:30
KaifAhmad1 f53935e0a1 Add progress tracking and ETA to GraphBuilder
- Enhanced GraphBuilder with real-time progress updates showing percentage, ETA, and processing rate
- Added time tracking for entity processing, relationship processing, entity resolution, and graph structure building
- Added final summary with total build time
- Simplified notebook cell to rely on Semantica's built-in progress tracking instead of manual Python code
2025-12-27 18:46:27 +05:30
KaifAhmad1 b9ffba67ea Update DeFi Protocol Intelligence notebook:
- Use ML-only approach for entity extraction (spaCy)
- Improve knowledge graph visualization with interactive layout
- Fix ontology export to use RDFExporter for TTL format
- Enhance visualization with better interactivity and explanations
2025-12-27 16:57:24 +05:30
KaifAhmad1 d4f008a183 Fix AttributeError issues in TripletStore and ContextRetriever
- Fix LoadProgress attribute access in TripletStore (use loaded_triplets instead of processed_triplets)
- Fix None source handling in ContextRetriever RetrievedContext objects
- Add error handling for Blazegraph connection in notebook
- Ensure source field always has a default value in vector/memory retrieval
2025-12-27 15:34:04 +05:30
KaifAhmad1 f7fcfa3691 Fix LLM-based entity and relation extraction
- Updated extract_entities_llm to use custom entity_types in prompts
- Updated extract_relations_llm to use custom relation_types in prompts
- Made entity type filtering case-insensitive and flexible
- Added verbose mode to RelationExtractor for progress tracking
- Improved error handling and progress reporting in notebook
- Made prompts more flexible to accept variations of entity/relation types
2025-12-26 23:00:14 +05:30
KaifAhmad1 8289d56d89 Update notebook and other changes 2025-12-26 19:41:55 +05:30
KaifAhmad1 bde4ef18a3 Update Genomic Variant Analysis notebook and fix temporal query issues
- Fixed analyze_evolution metrics None handling in temporal_query.py
- Updated 02_Genomic_Variant_Analysis.ipynb with simplified code using Semantica effectively
- Added temporal visualization support
- Fixed GraphBuilder conflict resolution (set resolve_conflicts=False when conflicts already handled)
- Simplified pathway analysis and disease association cells
- Updated visualization to use interactive Plotly graphs instead of HTML
- Added temporal dashboard visualization
2025-12-26 19:01:32 +05:30
KaifAhmad1 1542b2dafb Improve GraphRAG accuracy with semantic matching and domain-agnostic query intent
- Replace keyword matching with semantic similarity using embeddings
- Add domain-agnostic query intent extraction
- Improve ranking with hybrid_alpha weighting and context boosting
- Enhance content generation from graph structures
- Update ContextRetriever documentation
- Fix KGVisualizer method call in notebook
2025-12-26 18:07:44 +05:30
KaifAhmad1 76def647d8 Fix Drug Discovery Pipeline notebook: resolve conflicts, simplify GraphBuilder, update documentation
- Fixed NameError: Changed merged_entities to all_entities in conflict detection and GraphBuilder cells
- Fixed TypeError: Updated conflict detection to use detect_relationship_conflicts() directly
- Simplified code: Reduced manual Python code, better utilize Semantica's built-in features
- Updated markdown: Converted to bullet points, reflect credibility-weighted strategy
- Improved GraphBuilder: Use automatic object handling instead of manual conversion
2025-12-26 13:13:23 +05:30
KaifAhmad1 c15ee40cdf feat: Diversify deduplication and conflict resolution across 16 use case notebooks
Implement domain-appropriate strategies for all notebooks:

Deduplication Methods (9): pairwise, batch, incremental, group, graph_based,
hierarchical, exact, semantic, fuzzy

Merge Strategies (5): keep_first, keep_last, keep_most_complete,
keep_highest_confidence, merge_all

Conflict Detection (6): value, type, entity, relationship, temporal, logical

Conflict Resolution (6): voting, credibility_weighted, most_recent,
first_seen, highest_confidence, expert_review

Key patterns:
- Real-time: pairwise + keep_first + first_seen
- Time-sensitive: temporal + most_recent
- Multi-source: batch + merge_all + voting
- Medical/Research: credibility_weighted
- Fraud/Security: graph_based + logical + expert_review

Added STRATEGIES_SUMMARY.md documentation.
Removed temporary update scripts.
2025-12-25 22:38:03 +05:30
KaifAhmad1 068d0d489a refactor(trading): rebuild risk assessment and sentiment analysis notebooks
- Refactor 01_Risk_Assessment.ipynb with GraphStore, DBIngestor, conflict detection
- Refactor 02_News_Sentiment_Analysis.ipynb with TripletStore, StreamIngestor, deduplication
- Complete all 8 phases in both notebooks with different module approaches
- Add comprehensive graph analytics, ontology generation, and export functionality
2025-12-25 16:54:35 +05:30
KaifAhmad1 7a82b7b597 docs(supply-chain): update risk management notebook 2025-12-25 16:30:23 +05:30
KaifAhmad1 96e784509e Refactor renewable energy notebooks: modular architecture with unique module combinations
- Rebuilt 01_Energy_Market_Analysis.ipynb with temporal pattern detection, trend prediction, and seed data integration
- Rebuilt 02_Smart_Grid_Management.ipynb with stream processing, real-time monitoring, and anomaly detection
- Removed core orchestrator usage, implemented cell-specific imports
- Added comprehensive data sources and Mermaid pipeline diagrams
- Minimal print statements, proper error handling with redirect_stderr
- Unique module combinations per use case for differentiation
2025-12-25 15:51:54 +05:30
KaifAhmad1 bd594f9c41 Refactor intelligence notebooks: Use modular architecture with unique approaches per use case
- Rebuilt 01_Criminal_Network_Analysis.ipynb with graph analytics and centrality focus (31 cells)
- Rebuilt 02_Intelligence_Analysis_Orchestrator_Worker.ipynb with multi-source integration and temporal analysis focus (31 cells)
- Added comprehensive data sources (OSINT feeds, threat intelligence, geospatial data, intelligence agency feeds)
- Implemented cell-specific imports and minimal print statements
- Each notebook uses different module combinations to showcase uniqueness:
  - Criminal Network: CentralityCalculator, CommunityDetector, GraphAnalyzer, entity-aware chunking
  - Intelligence Analysis: StreamIngestor, ConflictDetector, Reasoner, TemporalGraphQuery, sentence chunking
- Removed empty markdown cells
- Added Mermaid pipeline flow diagrams
- No phase/step numbers - descriptive section headers
2025-12-25 15:19:31 +05:30
KaifAhmad1 b5f895e289 Refactor healthcare notebooks: Use modular architecture with unique approaches per use case
- Rebuilt 01_Clinical_Reports_Processing.ipynb with EHR integration and triplet store focus (32 cells)
- Rebuilt 02_Drug_Interactions_Analysis.ipynb with ontology generation and reasoning focus (35 cells)
- Added comprehensive data sources (EHR APIs, HL7/FHIR feeds, FDA RSS, PubMed, drug databases)
- Implemented cell-specific imports and minimal print statements
- Each notebook uses different module combinations to showcase uniqueness:
  - Clinical Reports: TripletStore, SeedDataManager, TemporalGraphQuery, DocumentParser
  - Drug Interactions: OntologyGenerator, Reasoner, ConflictDetector, TemporalPatternDetector, CommunityDetector
- Removed empty markdown cells
- Added Mermaid pipeline flow diagrams
- No phase/step numbers - descriptive section headers
2025-12-25 14:40:58 +05:30
KaifAhmad1 7d07691d99 Refactor finance notebooks: Use modular architecture with unique approaches per use case
- Rebuilt 01_Financial_Data_Integration_MCP.ipynb with MCP and seed data focus (31 cells)
- Rebuilt 02_Fraud_Detection.ipynb with temporal analysis and pattern detection focus (38 cells)
- Added comprehensive data sources (APIs, RSS feeds, streams, databases)
- Implemented cell-specific imports and minimal print statements
- Each notebook uses different module combinations to showcase uniqueness:
  - Financial Data Integration: MCPIngestor, SeedDataManager, GraphAnalyzer, CentralityCalculator
  - Fraud Detection: StreamIngestor, TemporalGraphQuery, TemporalPatternDetector, ConflictDetector
- Removed empty markdown cells
- Added Mermaid pipeline flow diagrams
- No phase/step numbers - descriptive section headers
2025-12-25 14:12:27 +05:30
KaifAhmad1 8740379df6 Refactor cybersecurity notebooks: Use modular architecture with comprehensive data sources and all Semantica modules
- Rebuilt 01_Real_Time_Anomaly_Detection.ipynb with 20+ sections
- Rebuilt 02_Threat_Intelligence_Hybrid_RAG.ipynb with 20+ sections
- Added comprehensive data sources (RSS feeds, APIs, databases, IOC sources)
- Implemented cell-specific imports and minimal print statements
- Integrated all relevant Semantica modules (parse, embeddings, vector_store, graph_store, temporal queries, etc.)
- Removed empty markdown cells
- Added Mermaid pipeline flow diagrams
- No phase/step numbers - descriptive section headers
2025-12-25 13:38:35 +05:30
KaifAhmad1 5e220dc341 Refactor blockchain notebooks: Use modular architecture with comprehensive data sources and all Semantica modules
- Rebuilt 01_DeFi_Protocol_Intelligence.ipynb with 20+ sections
- Rebuilt 02_Transaction_Network_Analysis.ipynb with 21+ sections
- Added comprehensive data sources (RSS feeds, APIs, databases)
- Implemented cell-specific imports and minimal print statements
- Integrated all relevant Semantica modules (parse, embeddings, vector_store, graph_store, triplet_store, context, etc.)
- Removed empty markdown cells
- Added Mermaid pipeline flow diagrams
- No phase/step numbers - descriptive section headers
2025-12-25 13:08:13 +05:30
KaifAhmad1 3f74040509 Refactor biomedical notebooks: Use modular architecture with cell-specific imports and comprehensive data sources 2025-12-25 12:48:51 +05:30
KaifAhmad1 a54959d277 Refactor Drug Discovery Pipeline notebook: break down dense cells, remove unnecessary markdown headings, add bullet points 2025-12-25 00:20:06 +05:30
Mohd Kaif 2bbe36400a Delete cookbook/use_cases/USE_CASES_CATALOG.md 2025-12-24 16:48:07 +05:30
KaifAhmad1 106e817ad8 docs: Update all documentation with 18 enhanced domain-specific cookbooks
- Enhanced 18 cookbooks across 9 domains with real data sources, advanced chunking, temporal KGs, and GraphRAG
- Updated docs/cookbook.md with all 18 cookbook links and enhanced descriptions
- Updated docs/use-cases.md with corrected links and removed duplicates
- Updated README.md with comprehensive Industry Use Cases section
- Fixed all outdated notebook links and ensured consistency across all docs
- Added real data ingestion (RSS feeds, APIs, MCP servers, streams)
- Integrated advanced chunking strategies (entity-aware, relation-aware, ontology-aware, semantic_transformer, etc.)
- Added temporal knowledge graphs, GraphRAG, deduplication, conflict detection, and other Semantica modules
2025-12-24 16:33:36 +05:30
KaifAhmad1 d5ec639d3c Fix GraphRAG notebooks: update LLM model to llama-3.3-70b-versatile, fix embedding dimension check, fix file ingestion, fix graph metrics access, fix export methods, fix conflict detection, and add side-by-side comparison 2025-12-24 13:37:17 +05:30
KaifAhmad1 cd437a9cfb Fix GraphRAG notebook: embedding dimension check, update LLM model to llama-3.3-70b-versatile, fix file ingestion, graph metrics access, and export methods 2025-12-24 13:00:10 +05:30
KaifAhmad1 bd466b6016 Fix GraphRAG notebook issues: embedding dimension check, update LLM model to llama-3.3-70b-versatile, fix file ingestion, and fix graph metrics access 2025-12-24 12:52:55 +05:30
KaifAhmad1 ef0797e03e Fix semantic extraction pipeline errors and enhance LLM JSON parsing
Summary of changes:
- Fixed TripletExtractor method name (extract -> extract_triplets)
- Fixed Event attribute name (type -> event_type)
- Improved LLM JSON parsing in providers.py (handles trailing commas, unclosed structures)
- Fixed CentralityCalculator TypeError in GraphRAG notebook
- Corrected CommunityDetector logic and imports in notebook
2025-12-23 23:23:46 +05:30
KaifAhmad1 eaa1fbefa6 feat(reasoning): add dedicated reasoning tests and fix critical reasoning bugs
- Added tests/reasoning/ directory with unit and integration tests
- Fixed indentation bug in Reasoner.add_fact for dictionary-based relationships
- Fixed regex variable matching in Reasoner._match_pattern
- Fixed variable handling in SPARQLReasoner query expansion
- Cleaned up cookbook and documentation references
2025-12-23 21:26:26 +05:30
KaifAhmad1 9cc096dcd5 Refactor GraphRAG notebooks: remove empty cells, step numbers, reorganize imports, and make markdown concise 2025-12-23 18:19:23 +05:30
KaifAhmad1 07bd371e7d Expand GraphRAG notebooks with more cells, less dense code, and improved markdown documentation 2025-12-23 17:10:28 +05:30
KaifAhmad1 7046c92b3a Merge main and resolve conflicts by prioritizing audited reasoning refactor 2025-12-23 15:24:07 +05:30