Commit Graph
184 Commits
Author SHA1 Message Date
KaifAhmad1 8cecef9e58 feat: Standardize kg and kg_qa modules with methods, registry, config
- Add methods.py, registry.py, config.py to kg module
- Add methods.py, registry.py, config.py to kg_qa module
- Update __init__.py files with comprehensive documentation
- Add kg_usage.md and kg_qa_usage.md with comprehensive guides
- Enhance semantic_extract_usage.md and split_usage.md
2025-11-17 13:47:04 +05:30
KaifAhmad1 eff719610d feat: Add methods, registry, config to kg module and enhance usage docs
- Add methods.py with wrapper functions for all KG operations
- Add registry.py with MethodRegistry class for custom method registration
- Add config.py with KGConfig class for configuration management
- Update kg/__init__.py with comprehensive documentation and exports
- Add kg_usage.md with comprehensive usage guide
- Enhance semantic_extract_usage.md and split_usage.md with improved introductions
2025-11-17 12:40:30 +05:30
KaifAhmad1 6f1f845c04 Organize ingest folder structure with config, registry, and methods
- Add config.py for configuration management with environment variable support
- Add registry.py for method registry extensibility
- Add methods.py with reusable wrapper functions for all ingestion operations
- Update __init__.py to export config, registry, and methods with comprehensive documentation
- Add ingest_usage.md with complete usage guide and examples

Follows the standardized module structure pattern from conflicts, context, core, deduplication, embeddings, and export modules.
Includes comprehensive documentation of all algorithms used in file, web, feed, stream, repo, email, and database ingestion.
2025-11-16 18:22:09 +05:30
KaifAhmad1 a0a4fdfb13 Organize export folder structure and add LPG export support
- Add config.py for configuration management with environment variable support
- Add registry.py for method registry extensibility
- Add lpg_exporter.py for LPG (Labeled Property Graph) export to Neo4j/Memgraph
- Add methods.py with reusable wrapper functions for all export operations
- Update __init__.py to export config, registry, methods, and LPG exporter with comprehensive documentation
- Add export_usage.md with complete usage guide and examples

Follows the standardized module structure pattern from conflicts, context, core, deduplication, and embeddings modules.
Includes LPG export functionality for graph database integration.
2025-11-16 17:52:38 +05:30
KaifAhmad1 15fc644093 Organize embeddings folder structure
- Add config.py for configuration management with environment variable support
- Add registry.py for method registry extensibility
- Add methods.py with reusable wrapper functions for all embedding operations
- Update __init__.py to export config, registry, and methods with comprehensive documentation
- Add embeddings_usage.md with complete usage guide and examples

Follows the standardized module structure pattern from conflicts, context, core, and deduplication modules.
2025-11-16 15:41:25 +05:30
KaifAhmad1 cf0c4dafca feat(context): Reorganize context module with registry, methods, and config
- Add registry.py for method registration system (graph, memory, retrieval, linking)
- Add methods.py with reusable context methods (build_context_graph, store_memory, retrieve_context, link_entities)
- Add config.py for context module configuration management
- Update __init__.py with comprehensive documentation, exports, and build_context() convenience function
- Enhance docstrings in agent_memory.py, context_graph.py, context_retriever.py, entity_linker.py with algorithms and features
- Create context_usage.md with comprehensive usage examples
- Fix missing get_progress_tracker import in context_graph.py

This reorganization standardizes the context module structure to match other reorganized modules (semantic_extract, split, deduplication, conflicts, core).
2025-11-15 21:49:10 +05:30
KaifAhmad1 612dc53729 feat(conflicts): reorganize conflicts module with registry, methods, and config
- Add registry.py for custom conflict method registration
- Add methods.py with reusable conflict functions
- Add config.py for configuration management
- Add conflicts_usage.md with comprehensive usage guide
- Enhance __init__.py with algorithms documentation and convenience functions
- Enhance all existing files with detailed algorithms documentation
- Add detect_and_resolve() convenience function
- Support environment variables and config files for configuration
2025-11-15 19:17:27 +05:30
KaifAhmad1 24c6ceadbc Reorganize deduplication module and add usage documentation
- Add registry.py for method registration system
- Add methods.py with reusable deduplication functions
- Add config.py for deduplication configuration management
- Enhance __init__.py with comprehensive exports and deduplicate() convenience function
- Update all existing files with enhanced docstrings including detailed algorithms
- Add deduplication_usage.md with comprehensive usage examples
- Add semantic_extract_usage.md with comprehensive usage examples
- Add split_usage.md with comprehensive usage examples
- Follow same structure pattern as semantic_extract, split, and core modules
2025-11-15 17:40:38 +05:30
KaifAhmad1 985eecd14b Update core module structure to match semantic_extract and split patterns
- Add registry.py for method registration system
- Add methods.py with reusable orchestration functions
- Enhance __init__.py with comprehensive exports and build() convenience function
- Update config_manager.py with enhanced documentation
- Follow same structure pattern as semantic_extract and split modules
2025-11-15 16:39:07 +05:30
Mohd Kaif 107c0cb311 Merge pull request #20 from Hawksight-AI/staging
Add multiple splitting and chunking methods with KG/ontology su…
2025-11-14 23:07:58 +05:30
KaifAhmad1 3c31475fdc feat: Add multiple splitting and chunking methods with KG/ontology support
- Add unified TextSplitter with method parameter support
- Implement 20+ splitting methods (standard + KG/ontology)
- Add entity-aware and relation-aware chunking for GraphRAG
- Add graph-based, ontology-aware, and hierarchical chunking
- Add configuration management and plugin registry
- Add specialized KG chunkers (EntityAware, RelationAware, etc.)
- Update pyproject.toml with optional dependencies
- Maintain backward compatibility with existing chunkers
2025-11-14 23:03:41 +05:30
Mohd Kaif 2df4907d1a Merge pull request #19 from Hawksight-AI/staging
Add multiple extraction methods and unified LLM provider support
2025-11-14 18:45:17 +05:30
KaifAhmad1 0a1821d586 feat: Add multiple extraction methods and LLM provider support
- Add pattern, regex, rules, ML, HuggingFace, and LLM extraction methods
- Implement unified LLM provider system (OpenAI, Gemini, Groq, Anthropic, Ollama)
- Add HuggingFace model loader for NER, relation, and triplet extraction
- Enhance all extractors with method parameter and fallback chain support
- Add comprehensive documentation with algorithms and supported methods
- Add plugin registry system for custom providers and methods
- Add configuration management with environment variable support
2025-11-14 18:42:24 +05:30
Mohd Kaif b6c8483529 Delete cookbook/REAL_DATA_SOURCES.md 2025-11-13 21:28:44 +05:30
Mohd Kaif 52300461a0 Merge pull request #18 from Hawksight-AI/staging
Add Blockchain and Biomedical Use Cases in Coockbook
2025-11-13 19:53:12 +05:30
KaifAhmad1 8286a8b91e Add blockchain and biomedical use cases: Transaction Network Analysis, DeFi Protocol Intelligence, Genomic Variant Analysis, and Drug Discovery Pipeline 2025-11-13 19:50:42 +05:30
KaifAhmad1 a3c461a0d6 Merge branch 'main' of https://github.com/Hawksight-AI/semantica 2025-11-13 18:28:34 +05:30
KaifAhmad1 6100d5e0d5 Complete cookbook implementation: Add all use case notebooks with real data sources
- Implemented all 25 use case notebooks across 5 domains (Trading, Healthcare, Finance, Cybersecurity, Renewable Energy)
- Added real data sources: APIs, RSS feeds, database connections, streaming sources
- Each notebook uses 20+ Semantica modules with complete end-to-end pipelines
- Multi-source ingestion with lists of URLs for feeds and APIs
- Complete workflows: ingestion → parsing → extraction → KG building → analysis → export → visualization
- All notebooks include realistic data structures and error handling
- Updated renewable energy notebooks: Environmental_Impact, Grid_Management, Resource_Optimization, Supply_Chain_Analysis
2025-11-13 18:27:33 +05:30
Mohd Kaif 817811dad6 Delete cookbook/specialized_applications directory 2025-11-13 16:54:34 +05:30
Mohd Kaif a5d00149d7 Delete cookbook/core_workflows directory 2025-11-13 16:53:39 +05:30
KaifAhmad1 116bbbd700 Update cookbook notebooks with real data sources
- Replace mock data with real feed URLs, APIs, and database patterns
- Add real threat intelligence feeds (CISA, US-CERT, Security Week, Dark Reading)
- Add real financial feeds (Reuters, CNN Money, Bloomberg, Financial Times)
- Add real healthcare feeds (CDC, WHO)
- Add real API endpoints (MITRE ATT&CK, NVD CVE API, Polygon.io, Alpha Vantage, FHIR APIs)
- Add realistic database connection patterns with SQL queries
- Add Kafka/RabbitMQ streaming configurations
- Update all cybersecurity notebooks (5/5) with real sources
- Update finance notebooks (2/2) with real sources
- Update healthcare notebooks (1/1) with real sources
- Create REAL_DATA_SOURCES.md documentation
- Improve error handling with try-except blocks
- Add batch processing for multiple feed URLs
2025-11-13 16:52:51 +05:30
KaifAhmad1 dc7d239903 Update Unstructured_to_Ontology notebook to use OntologyValidator module from semantica 2025-11-13 13:54:39 +05:30
KaifAhmad1 4cca9c6ac5 Add progress tracking to vector_store and visualization modules
- Added progress tracking to namespace_manager methods (create, delete, list, add/remove vectors, get stats)
- Added progress tracking to embedding_visualizer methods (2D/3D projection, similarity heatmap, clustering, multimodal comparison, quality metrics)
- Added progress tracking to quality_visualizer dashboard method
- Added progress tracking to temporal_visualizer timeline method
- Added progress tracking to semantic_network_visualizer network visualization
- All methods now include start_tracking, update_tracking, and stop_tracking with proper error handling
2025-11-12 18:23:03 +05:30
KaifAhmad1 05caff91fa Add progress tracking to seed, semantic_extract, and split modules
- Added progress tracking to seed module (seed_manager.py)
- Added progress tracking to all semantic_extract files (10 files):
  * triple_extractor.py
  * semantic_network_extractor.py
  * relation_extractor.py
  * semantic_analyzer.py
  * ner_extractor.py
  * named_entity_recognizer.py
  * llm_enhancer.py
  * extraction_validator.py
  * event_detector.py
  * coreference_resolver.py
- Added progress tracking to all split files (6 files):
  * semantic_chunker.py
  * sliding_window_chunker.py
  * structural_chunker.py
  * table_chunker.py
  * chunk_validator.py
  * provenance_tracker.py
- All methods now use start_tracking, update_tracking, and stop_tracking
- Consistent error handling with try-except blocks across all modules
2025-11-12 17:56:45 +05:30
KaifAhmad1 9b7a3275c1 Add progress tracking to all modules
- Added progress tracking to all ontology module files (16 files)
- Added progress tracking to all pipeline module files (7 files)
- Added progress tracking to all reasoning module files (7 files)
- Implemented consistent progress tracking pattern across all modules
- All methods now use start_tracking, update_tracking, and stop_tracking
2025-11-12 17:45:55 +05:30
KaifAhmad1 d97d6276e2 Add progress tracking to all methods in GraphBuilder
- Add progress tracking to add_temporal_edge method
- Add progress tracking to create_temporal_snapshot method
- Add progress tracking to query_temporal method
- Add progress tracking to load_from_neo4j method
- Include intermediate progress updates for long-running operations
- Add proper error handling with progress tracking in all methods
2025-11-12 17:11:53 +05:30
KaifAhmad1 8e35bf89f6 Add progress tracking to core, deduplication, and embeddings modules
- Integrated progress tracking into all core modules:
  * config_manager.py - Added tracking to load_from_file()
  * lifecycle.py - Added tracking to startup() and shutdown()
  * plugin_registry.py - Added tracking to load_plugin()

- Integrated progress tracking into all deduplication modules:
  * duplicate_detector.py - Added tracking to detect_duplicates() and detect_duplicate_groups()
  * entity_merger.py - Added tracking to merge_duplicates()
  * similarity_calculator.py - Added tracking to calculate_similarity()
  * cluster_builder.py - Added tracking to build_clusters()
  * merge_strategy.py - Added tracking to merge_entities()

- Integrated progress tracking into all embeddings modules:
  * text_embedder.py - Added tracking to embed_text()
  * image_embedder.py - Added tracking to embed_image()
  * audio_embedder.py - Added tracking to embed_audio()
  * multimodal_embedder.py - Added tracking to embed_multimodal()
  * embedding_optimizer.py - Added tracking to compress()
  * context_manager.py - Added tracking to split_into_windows()

- All methods include real-time progress updates with Semantica branding
- Proper error handling with progress tracking status updates
- Automatic module/submodule detection via progress tracker
2025-11-12 16:34:32 +05:30
KaifAhmad1 0f1c0ad5d0 Add progress tracking to conflicts and context modules
- Integrated progress tracking into all conflicts modules:
  * conflict_detector.py - Added tracking to detect_value_conflicts()
  * conflict_resolver.py - Added tracking to resolve_conflict()
  * conflict_analyzer.py - Added tracking to analyze_conflicts()
  * source_tracker.py - Initialized progress tracker
  * investigation_guide.py - Added tracking to generate_guide()

- Integrated progress tracking into all context modules:
  * context_retriever.py - Added tracking to retrieve() with vector/graph/memory updates
  * entity_linker.py - Added tracking to link() and link_entities()
  * context_graph.py - Added tracking to build_from_conversations() and build_from_entities_and_relationships()
  * agent_memory.py - Added tracking to store() and retrieve()

- All methods include real-time progress updates with Semantica branding
- Proper error handling with progress tracking status updates
2025-11-12 16:14:38 +05:30
KaifAhmad1 af65e4e90b Add comprehensive progress tracking to all ingestion and parsing modules
- Implemented ProgressTracker with console, Jupyter, and file display handlers
- Added automatic module/submodule detection using call stack introspection
- Integrated progress tracking into all ingestion modules (FileIngestor, WebIngestor, FeedIngestor, RepoIngestor, EmailIngestor, DBIngestor)
- Integrated progress tracking into all parsing modules (PDFParser, DOCXParser, HTMLParser, XMLParser, ExcelParser, JSONParser, CSVParser, PPTXParser, ImageParser, EmailParser, CodeParser, MediaParser, StructuredDataParser, WebParser)
- Added real-time progress updates with Semantica branding and emojis
- Added progress summaries without action format
- All progress tracking is automatic with zero configuration required
2025-11-12 15:53:52 +05:30
KaifAhmad1 5d1bc5836a Clean up cookbook structure: remove numeric prefixes from folders and files 2025-11-12 14:16:36 +05:30
KaifAhmad1 3fd9f49c67 Implement cookbook notebooks: Add code cells and improve markdown formatting for 01_core_workflows folder 2025-11-11 21:40:15 +05:30
KaifAhmad1 b758eb4554 Implement cookbook notebooks with executable code cells and improved documentation
- Added executable Python code cells to all three getting started notebooks
- Improved markdown formatting with consistent header hierarchy
- Removed excessive comments from code cells for better readability
- Added markdown explanation cells between code sections
- Enhanced documentation structure and clarity
2025-11-11 21:20:54 +05:30
Mohd Kaif 628b09663a Delete cookbook/init 2025-11-10 20:34:55 +05:30
KaifAhmad1 bd7f0e06c7 Resolve merge conflict: move init file to cookbook/init after restructuring 2025-11-10 20:34:17 +05:30
KaifAhmad1 5f0b254552 Restructure cookbook: move notebooks folders directly into cookbook directory
- Moved all folders from cookbook/notebooks/ to cookbook/
- Removed empty notebooks folder
- All 53 notebooks now directly under cookbook/:
  - 00_getting_started/ (3 notebooks)
  - 01_core_workflows/ (12 notebooks)
  - 02_advanced_patterns/ (8 notebooks)
  - 03_use_cases/ (25 notebooks across 5 domains)
  - 04_specialized_applications/ (5 notebooks)
2025-11-10 20:31:08 +05:30
Mohd Kaif 83c93ec89a Create init 2025-11-10 20:24:28 +05:30
KaifAhmad1 9a7759d8b7 Add cookbook folder with 53 Jupyter notebooks organized by category
- Added 00_getting_started/ with 3 notebooks (basics and setup)
- Added 01_core_workflows/ with 12 notebooks (common patterns)
- Added 02_advanced_patterns/ with 8 notebooks (complex integrations)
- Added 03_use_cases/ with 25 notebooks across 5 domains:
  - cybersecurity/ (5 notebooks)
  - healthcare/ (5 notebooks)
  - finance/ (5 notebooks)
  - trading/ (5 notebooks)
  - renewable_energy/ (5 notebooks)
- Added 04_specialized_applications/ with 5 notebooks (cutting-edge features)

All notebooks contain comment cells only (no executable code) as documentation templates.
2025-11-10 20:22:12 +05:30
Mohd Kaif f31a0447c8 Delete cookbook directory 2025-11-10 19:36:46 +05:30
KaifAhmad1 3f795c316d Add Modules Documentation Files which cotains all the modules Info 2025-11-10 18:13:27 +05:30
KaifAhmad1 b8abe13686 update readme 2025-11-06 22:23:00 +05:30
KaifAhmad1 b29f9771fe refactor: standardize conflicts module with comprehensive documentation
- Updated all 6 files in conflicts module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes and methods
- Standardized Author and License information
- Files updated: __init__.py, conflict_analyzer.py, conflict_detector.py, conflict_resolver.py, investigation_guide.py, source_tracker.py
2025-11-06 22:12:00 +05:30
KaifAhmad1 a0581a2e19 refactor: standardize context module with comprehensive documentation
- Updated all 5 files in context module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes and methods
- Standardized Author and License information
- Files updated: __init__.py, agent_memory.py, context_graph.py, context_retriever.py, entity_linker.py
2025-11-06 22:07:16 +05:30
KaifAhmad1 bd7794feec refactor: standardize visualization module with comprehensive documentation
- Updated all 12 files in visualization module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes and methods
- Standardized Author and License information
- Files updated: __init__.py, kg_visualizer.py, embedding_visualizer.py, analytics_visualizer.py, ontology_visualizer.py, quality_visualizer.py, semantic_network_visualizer.py, temporal_visualizer.py, utils/__init__.py, utils/color_schemes.py, utils/export_formats.py, utils/layout_algorithms.py
2025-11-06 22:00:41 +05:30
KaifAhmad1 3664f1a606 refactor: standardize vector_store module with comprehensive documentation
- Updated all 10 files in vector_store module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes and methods
- Standardized Author and License information
- Files updated: __init__.py, vector_store.py, faiss_adapter.py, pinecone_adapter.py, weaviate_adapter.py, qdrant_adapter.py, milvus_adapter.py, hybrid_search.py, metadata_store.py, namespace_manager.py
2025-11-06 21:53:17 +05:30
KaifAhmad1 8a8487aa93 refactor: standardize utils module with comprehensive documentation
- Updated all 7 files in utils module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes, functions, and utilities
- Standardized Author and License information
- Files updated: __init__.py, constants.py, exceptions.py, helpers.py, logging.py, types.py, validators.py
2025-11-06 21:46:59 +05:30
KaifAhmad1 eeae93b1ed refactor: standardize triple_store module with comprehensive documentation
- Updated all 8 files in triple_store module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes and methods
- Standardized Author and License information
- Files updated: __init__.py, triple_manager.py, query_engine.py, bulk_loader.py, jena_adapter.py, blazegraph_adapter.py, rdf4j_adapter.py, virtuoso_adapter.py
2025-11-06 21:40:53 +05:30
KaifAhmad1 88a2178293 refactor: standardize split module with comprehensive documentation
- Updated all 7 files in split module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes and methods
- Standardized Author and License information
- Files updated: __init__.py, chunk_validator.py, provenance_tracker.py, semantic_chunker.py, sliding_window_chunker.py, structural_chunker.py, table_chunker.py
2025-11-06 21:35:52 +05:30
KaifAhmad1 bedacf4573 refactor: standardize semantic_extract module with comprehensive documentation
- Updated all 11 files in semantic_extract module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for all classes and methods
- Standardized Author and License information
- Files updated: __init__.py, coreference_resolver.py, event_detector.py, extraction_validator.py, llm_enhancer.py, named_entity_recognizer.py, ner_extractor.py, relation_extractor.py, semantic_analyzer.py, semantic_network_extractor.py, triple_extractor.py
2025-11-06 21:28:08 +05:30
KaifAhmad1 584f6f4e9e refactor: standardize seed module with comprehensive documentation
- Updated both files in seed module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved docstrings for SeedDataManager, SeedDataSource, and SeedData classes
- Added comprehensive method documentation with Args, Returns, Raises, and Examples
- Standardized Author and License information
- Files updated: __init__.py, seed_manager.py
2025-11-06 21:21:39 +05:30
KaifAhmad1 e11bf266b5 refactor: standardize reasoning module with comprehensive documentation
- Updated all 8 files in reasoning module to match standardized docstring format
- Added Key Features, Main Classes, Example Usage sections to all modules
- Enhanced module-level documentation with detailed descriptions
- Improved type hints and error handling across all reasoning components
- Standardized Author and License information
- Files updated: __init__.py, abductive_reasoner.py, deductive_reasoner.py, explanation_generator.py, inference_engine.py, rete_engine.py, rule_manager.py, sparql_reasoner.py
2025-11-06 21:13:30 +05:30