diff --git a/cookbook/introduction/Welcome_to_Semantica.ipynb b/cookbook/introduction/Welcome_to_Semantica.ipynb index a6a40031..62be5293 100644 --- a/cookbook/introduction/Welcome_to_Semantica.ipynb +++ b/cookbook/introduction/Welcome_to_Semantica.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Welcome to Semantica\n", + "# Welcome to Semantica: An interactive introduction to the framework\n", "\n", "## Overview\n", "\n", @@ -54,7 +54,7 @@ "\n", "### Installation Methods\n", "\n", - "'''\n", + "```bash\n", "# Method 1: Install from PyPI (when available)\n", "# pip install semantica\n", "\n", @@ -71,11 +71,11 @@ "# Verify installation\n", "# import semantica\n", "# print(semantica.__version__)\n", - "'''\n", + "```\n", "\n", "### Configuration\n", "\n", - "'''\n", + "```bash\n", "# Set up environment variables for API keys and configuration\n", "# export SEMANTICA_API_KEY=your_openai_key\n", "# export SEMANTICA_EMBEDDING_PROVIDER=openai\n", @@ -92,7 +92,7 @@ "# knowledge_graph:\n", "# backend: networkx # or neo4j, arangodb\n", "# temporal: true\n", - "'''\n", + "```\n", "\n", "---\n", "\n", @@ -100,205 +100,223 @@ "\n", "Semantica is organized into modular components, each handling a specific aspect of semantic processing:\n", "\n", - "'''\n", - "# ============================================================================\n", - "# CORE MODULES\n", - "# ============================================================================\n", + "### 1. INGEST MODULE - Data Ingestion\n", + "**Purpose**: Ingest data from various sources\n", + "**Components**:\n", + "- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n", + "- `WebIngestor`: Scrape and ingest web pages\n", + "- `FeedIngestor`: Process RSS/Atom feeds\n", + "- `StreamIngestor`: Real-time data streaming\n", + "- `DBIngestor`: Database queries and ingestion\n", + "- `EmailIngestor`: Process email messages\n", + "- `RepoIngestor`: Git repository analysis\n", "\n", - "# 1. INGEST MODULE - Data Ingestion\n", - "# Purpose: Ingest data from various sources\n", - "# Components:\n", - "# - FileIngestor: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n", - "# - WebIngestor: Scrape and ingest web pages\n", - "# - FeedIngestor: Process RSS/Atom feeds\n", - "# - StreamIngestor: Real-time data streaming\n", - "# - DBIngestor: Database queries and ingestion\n", - "# - EmailIngestor: Process email messages\n", - "# - RepoIngestor: Git repository analysis\n", - "#\n", - "# Example:\n", - "# from semantica.ingest import FileIngestor, WebIngestor\n", - "# file_ingestor = FileIngestor()\n", - "# web_ingestor = WebIngestor()\n", - "# documents = file_ingestor.ingest(\"data/\")\n", - "# web_docs = web_ingestor.ingest(\"https://example.com\")\n", + "**Example**:\n", + "```python\n", + "from semantica.ingest import FileIngestor, WebIngestor\n", + "file_ingestor = FileIngestor()\n", + "web_ingestor = WebIngestor()\n", + "documents = file_ingestor.ingest(\"data/\")\n", + "web_docs = web_ingestor.ingest(\"https://example.com\")\n", + "```\n", "\n", - "# 2. PARSE MODULE - Document Parsing\n", - "# Purpose: Parse and extract content from various formats\n", - "# Components:\n", - "# - DocumentParser: Main parser orchestrator\n", - "# - PDFParser: Extract text, tables, images from PDFs\n", - "# - DOCXParser: Parse Word documents\n", - "# - HTMLParser: Extract content from HTML\n", - "# - JSONParser: Parse structured JSON data\n", - "# - ExcelParser: Process spreadsheets\n", - "# - ImageParser: OCR and image analysis\n", - "# - CodeParser: Parse source code files\n", - "#\n", - "# Example:\n", - "# from semantica.parse import DocumentParser\n", - "# parser = DocumentParser()\n", - "# parsed_docs = parser.parse(documents)\n", + "### 2. PARSE MODULE - Document Parsing\n", + "**Purpose**: Parse and extract content from various formats\n", + "**Components**:\n", + "- `DocumentParser`: Main parser orchestrator\n", + "- `PDFParser`: Extract text, tables, images from PDFs\n", + "- `DOCXParser`: Parse Word documents\n", + "- `HTMLParser`: Extract content from HTML\n", + "- `JSONParser`: Parse structured JSON data\n", + "- `ExcelParser`: Process spreadsheets\n", + "- `ImageParser`: OCR and image analysis\n", + "- `CodeParser`: Parse source code files\n", "\n", - "# 3. NORMALIZE MODULE - Text Normalization\n", - "# Purpose: Clean and normalize text for processing\n", - "# Components:\n", - "# - TextNormalizer: Main normalization orchestrator\n", - "# - TextCleaner: Remove noise, fix encoding\n", - "# - DataCleaner: Clean structured data\n", - "# - EntityNormalizer: Normalize entity names\n", - "# - DateNormalizer: Standardize date formats\n", - "# - NumberNormalizer: Normalize numeric values\n", - "# - LanguageDetector: Detect document language\n", - "# - EncodingHandler: Handle character encoding\n", - "#\n", - "# Example:\n", - "# from semantica.normalize import TextNormalizer\n", - "# normalizer = TextNormalizer()\n", - "# normalized = normalizer.normalize(parsed_docs)\n", + "**Example**:\n", + "```python\n", + "from semantica.parse import DocumentParser\n", + "parser = DocumentParser()\n", + "parsed_docs = parser.parse(documents)\n", + "```\n", "\n", - "# 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n", - "# Purpose: Extract entities, relationships, and semantic information\n", - "# Components:\n", - "# - NERExtractor: Named Entity Recognition\n", - "# - RelationExtractor: Extract relationships between entities\n", - "# - SemanticAnalyzer: Deep semantic analysis\n", - "# - SemanticNetworkExtractor: Extract semantic networks\n", - "#\n", - "# Example:\n", - "# from semantica.semantic_extract import NERExtractor, RelationExtractor\n", - "# extractor = NERExtractor()\n", - "# entities = extractor.extract(normalized_docs)\n", - "# relation_extractor = RelationExtractor()\n", - "# relationships = relation_extractor.extract(normalized_docs, entities)\n", + "### 3. NORMALIZE MODULE - Text Normalization\n", + "**Purpose**: Clean and normalize text for processing\n", + "**Components**:\n", + "- `TextNormalizer`: Main normalization orchestrator\n", + "- `TextCleaner`: Remove noise, fix encoding\n", + "- `DataCleaner`: Clean structured data\n", + "- `EntityNormalizer`: Normalize entity names\n", + "- `DateNormalizer`: Standardize date formats\n", + "- `NumberNormalizer`: Normalize numeric values\n", + "- `LanguageDetector`: Detect document language\n", + "- `EncodingHandler`: Handle character encoding\n", "\n", - "# 5. KG MODULE - Knowledge Graph Construction\n", - "# Purpose: Build and manage knowledge graphs\n", - "# Components:\n", - "# - GraphBuilder: Construct knowledge graphs from entities/relationships\n", - "# - GraphAnalyzer: Analyze graph structure and properties\n", - "# - GraphValidator: Validate graph quality and consistency\n", - "# - EntityResolver: Resolve entity conflicts and duplicates\n", - "# - ConflictDetector: Detect conflicting information\n", - "# - CentralityCalculator: Calculate node importance metrics\n", - "# - CommunityDetector: Detect communities in graphs\n", - "# - ConnectivityAnalyzer: Analyze graph connectivity\n", - "# - TemporalQuery: Query temporal knowledge graphs\n", - "# - Deduplicator: Remove duplicate entities/relationships\n", - "#\n", - "# Example:\n", - "# from semantica.kg import GraphBuilder, GraphAnalyzer\n", - "# builder = GraphBuilder()\n", - "# kg = builder.build(entities, relationships)\n", - "# analyzer = GraphAnalyzer()\n", - "# metrics = analyzer.analyze(kg)\n", + "**Example**:\n", + "```python\n", + "from semantica.normalize import TextNormalizer\n", + "normalizer = TextNormalizer()\n", + "normalized = normalizer.normalize(parsed_docs)\n", + "```\n", "\n", - "# 6. EMBEDDINGS MODULE - Embedding Generation\n", - "# Purpose: Generate vector embeddings for various data types\n", - "# Components:\n", - "# - EmbeddingGenerator: Main embedding orchestrator\n", - "# - TextEmbedder: Generate text embeddings\n", - "# - ImageEmbedder: Generate image embeddings\n", - "# - AudioEmbedder: Generate audio embeddings\n", - "# - MultimodalEmbedder: Combine multiple modalities\n", - "# - EmbeddingOptimizer: Optimize embedding quality\n", - "# - ProviderAdapters: Support for OpenAI, Cohere, etc.\n", - "#\n", - "# Example:\n", - "# from semantica.embeddings import EmbeddingGenerator\n", - "# generator = EmbeddingGenerator()\n", - "# embeddings = generator.generate(documents)\n", + "### 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n", + "**Purpose**: Extract entities, relationships, and semantic information\n", + "**Components**:\n", + "- `NERExtractor`: Named Entity Recognition\n", + "- `RelationExtractor`: Extract relationships between entities\n", + "- `SemanticAnalyzer`: Deep semantic analysis\n", + "- `SemanticNetworkExtractor`: Extract semantic networks\n", "\n", - "# 7. VECTOR_STORE MODULE - Vector Database Operations\n", - "# Purpose: Store and search vector embeddings\n", - "# Components:\n", - "# - VectorStore: Main vector store interface\n", - "# - FAISSAdapter: FAISS integration\n", - "# - HybridSearch: Combine vector and keyword search\n", - "# - VectorRetriever: Retrieve relevant vectors\n", - "#\n", - "# Example:\n", - "# from semantica.vector_store import VectorStore, HybridSearch\n", - "# vector_store = VectorStore()\n", - "# vector_store.store(embeddings, documents, metadata)\n", - "# hybrid_search = HybridSearch(vector_store)\n", - "# results = hybrid_search.search(query, top_k=10)\n", + "**Example**:\n", + "```python\n", + "from semantica.semantic_extract import NERExtractor, RelationExtractor\n", + "extractor = NERExtractor()\n", + "entities = extractor.extract(normalized_docs)\n", + "relation_extractor = RelationExtractor()\n", + "relationships = relation_extractor.extract(normalized_docs, entities)\n", + "```\n", "\n", - "# 8. REASONING MODULE - Inference and Reasoning\n", - "# Purpose: Perform logical inference and reasoning\n", - "# Components:\n", - "# - InferenceEngine: Main inference orchestrator\n", - "# - RuleManager: Manage inference rules\n", - "# - DeductiveReasoner: Deductive reasoning\n", - "# - AbductiveReasoner: Abductive reasoning\n", - "# - ExplanationGenerator: Generate explanations for inferences\n", - "# - RETEEngine: RETE algorithm for rule matching\n", - "#\n", - "# Example:\n", - "# from semantica.reasoning import InferenceEngine, RuleManager\n", - "# inference_engine = InferenceEngine()\n", - "# rule_manager = RuleManager()\n", - "# new_facts = inference_engine.forward_chain(kg, rule_manager)\n", + "### 5. KG MODULE - Knowledge Graph Construction\n", + "**Purpose**: Build and manage knowledge graphs\n", + "**Components**:\n", + "- `GraphBuilder`: Construct knowledge graphs from entities/relationships\n", + "- `GraphAnalyzer`: Analyze graph structure and properties\n", + "- `GraphValidator`: Validate graph quality and consistency\n", + "- `EntityResolver`: Resolve entity conflicts and duplicates\n", + "- `ConflictDetector`: Detect conflicting information\n", + "- `CentralityCalculator`: Calculate node importance metrics\n", + "- `CommunityDetector`: Detect communities in graphs\n", + "- `ConnectivityAnalyzer`: Analyze graph connectivity\n", + "- `TemporalQuery`: Query temporal knowledge graphs\n", + "- `Deduplicator`: Remove duplicate entities/relationships\n", "\n", - "# 9. ONTOLOGY MODULE - Ontology Generation\n", - "# Purpose: Generate and manage ontologies\n", - "# Components:\n", - "# - OntologyGenerator: Generate ontologies from knowledge graphs\n", - "# - OntologyValidator: Validate ontology structure\n", - "# - OWLGenerator: Generate OWL format ontologies\n", - "# - PropertyGenerator: Generate ontology properties\n", - "# - ClassInferrer: Infer ontology classes\n", - "#\n", - "# Example:\n", - "# from semantica.ontology import OntologyGenerator\n", - "# generator = OntologyGenerator()\n", - "# ontology = generator.generate_from_graph(kg)\n", + "**Example**:\n", + "```python\n", + "from semantica.kg import GraphBuilder, GraphAnalyzer\n", + "builder = GraphBuilder()\n", + "kg = builder.build(entities, relationships)\n", + "analyzer = GraphAnalyzer()\n", + "metrics = analyzer.analyze(kg)\n", + "```\n", "\n", - "# 10. EXPORT MODULE - Data Export\n", - "# Purpose: Export data in various formats\n", - "# Components:\n", - "# - JSONExporter: Export to JSON\n", - "# - RDFExporter: Export to RDF/XML\n", - "# - CSVExporter: Export to CSV\n", - "# - GraphExporter: Export to graph formats (GraphML, GEXF)\n", - "# - OWLExporter: Export to OWL\n", - "# - VectorExporter: Export vectors\n", - "#\n", - "# Example:\n", - "# from semantica.export import JSONExporter, RDFExporter\n", - "# json_exporter = JSONExporter()\n", - "# json_exporter.export(kg, \"output.json\")\n", + "### 6. EMBEDDINGS MODULE - Embedding Generation\n", + "**Purpose**: Generate vector embeddings for various data types\n", + "**Components**:\n", + "- `EmbeddingGenerator`: Main embedding orchestrator\n", + "- `TextEmbedder`: Generate text embeddings\n", + "- `ImageEmbedder`: Generate image embeddings\n", + "- `AudioEmbedder`: Generate audio embeddings\n", + "- `MultimodalEmbedder`: Combine multiple modalities\n", + "- `EmbeddingOptimizer`: Optimize embedding quality\n", + "- `ProviderAdapters`: Support for OpenAI, Cohere, etc.\n", "\n", - "# 11. VISUALIZATION MODULE - Graph Visualization\n", - "# Purpose: Visualize knowledge graphs and analytics\n", - "# Components:\n", - "# - KGVisualizer: Visualize knowledge graphs\n", - "# - EmbeddingVisualizer: Visualize embeddings (t-SNE, PCA, UMAP)\n", - "# - QualityVisualizer: Visualize quality metrics\n", - "# - AnalyticsVisualizer: Visualize graph analytics\n", - "# - TemporalVisualizer: Visualize temporal data\n", - "#\n", - "# Example:\n", - "# from semantica.visualization import KGVisualizer\n", - "# visualizer = KGVisualizer()\n", - "# visualizer.visualize(kg)\n", + "**Example**:\n", + "```python\n", + "from semantica.embeddings import EmbeddingGenerator\n", + "generator = EmbeddingGenerator()\n", + "embeddings = generator.generate(documents)\n", + "```\n", "\n", - "# 12. PIPELINE MODULE - Pipeline Orchestration\n", - "# Purpose: Build and execute processing pipelines\n", - "# Components:\n", - "# - PipelineBuilder: Build complex pipelines\n", - "# - ExecutionEngine: Execute pipelines\n", - "# - FailureHandler: Handle pipeline failures\n", - "# - ParallelismManager: Enable parallel processing\n", - "# - ResourceScheduler: Schedule resources\n", - "#\n", - "# Example:\n", - "# from semantica.pipeline import PipelineBuilder\n", - "# builder = PipelineBuilder()\n", - "# pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n", - "# .add_step(\"parse\", DocumentParser()) \\\\\n", - "# .build()\n", - "'''\n", + "### 7. VECTOR_STORE MODULE - Vector Database Operations\n", + "**Purpose**: Store and search vector embeddings\n", + "**Components**:\n", + "- `VectorStore`: Main vector store interface\n", + "- `FAISSAdapter`: FAISS integration\n", + "- `HybridSearch`: Combine vector and keyword search\n", + "- `VectorRetriever`: Retrieve relevant vectors\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.vector_store import VectorStore, HybridSearch\n", + "vector_store = VectorStore()\n", + "vector_store.store(embeddings, documents, metadata)\n", + "hybrid_search = HybridSearch(vector_store)\n", + "results = hybrid_search.search(query, top_k=10)\n", + "```\n", + "\n", + "### 8. REASONING MODULE - Inference and Reasoning\n", + "**Purpose**: Perform logical inference and reasoning\n", + "**Components**:\n", + "- `InferenceEngine`: Main inference orchestrator\n", + "- `RuleManager`: Manage inference rules\n", + "- `DeductiveReasoner`: Deductive reasoning\n", + "- `AbductiveReasoner`: Abductive reasoning\n", + "- `ExplanationGenerator`: Generate explanations for inferences\n", + "- `RETEEngine`: RETE algorithm for rule matching\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.reasoning import InferenceEngine, RuleManager\n", + "inference_engine = InferenceEngine()\n", + "rule_manager = RuleManager()\n", + "new_facts = inference_engine.forward_chain(kg, rule_manager)\n", + "```\n", + "\n", + "### 9. ONTOLOGY MODULE - Ontology Generation\n", + "**Purpose**: Generate and manage ontologies\n", + "**Components**:\n", + "- `OntologyGenerator`: Generate ontologies from knowledge graphs\n", + "- `OntologyValidator`: Validate ontology structure\n", + "- `OWLGenerator`: Generate OWL format ontologies\n", + "- `PropertyGenerator`: Generate ontology properties\n", + "- `ClassInferrer`: Infer ontology classes\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.ontology import OntologyGenerator\n", + "generator = OntologyGenerator()\n", + "ontology = generator.generate_from_graph(kg)\n", + "```\n", + "\n", + "### 10. EXPORT MODULE - Data Export\n", + "**Purpose**: Export data in various formats\n", + "**Components**:\n", + "- `JSONExporter`: Export to JSON\n", + "- `RDFExporter`: Export to RDF/XML\n", + "- `CSVExporter`: Export to CSV\n", + "- `GraphExporter`: Export to graph formats (GraphML, GEXF)\n", + "- `OWLExporter`: Export to OWL\n", + "- `VectorExporter`: Export vectors\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.export import JSONExporter, RDFExporter\n", + "json_exporter = JSONExporter()\n", + "json_exporter.export(kg, \"output.json\")\n", + "```\n", + "\n", + "### 11. VISUALIZATION MODULE - Graph Visualization\n", + "**Purpose**: Visualize knowledge graphs and analytics\n", + "**Components**:\n", + "- `KGVisualizer`: Visualize knowledge graphs\n", + "- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP)\n", + "- `QualityVisualizer`: Visualize quality metrics\n", + "- `AnalyticsVisualizer`: Visualize graph analytics\n", + "- `TemporalVisualizer`: Visualize temporal data\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.visualization import KGVisualizer\n", + "visualizer = KGVisualizer()\n", + "visualizer.visualize(kg)\n", + "```\n", + "\n", + "### 12. PIPELINE MODULE - Pipeline Orchestration\n", + "**Purpose**: Build and execute processing pipelines\n", + "**Components**:\n", + "- `PipelineBuilder`: Build complex pipelines\n", + "- `ExecutionEngine`: Execute pipelines\n", + "- `FailureHandler`: Handle pipeline failures\n", + "- `ParallelismManager`: Enable parallel processing\n", + "- `ResourceScheduler`: Schedule resources\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.pipeline import PipelineBuilder\n", + "builder = PipelineBuilder()\n", + "pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n", + " .add_step(\"parse\", DocumentParser()) \\\\\n", + " .build()\n", + "```\n", "\n", "---\n", "\n", @@ -306,183 +324,166 @@ "\n", "Understanding these concepts is crucial for working with Semantica:\n", "\n", - "'''\n", - "# ============================================================================\n", - "# CORE CONCEPTS\n", - "# ============================================================================\n", + "### 1. KNOWLEDGE GRAPHS\n", + "**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes.\n", "\n", - "# 1. KNOWLEDGE GRAPHS\n", - "# Definition: A knowledge graph is a structured representation of entities\n", - "# (nodes) and their relationships (edges) with properties and\n", - "# attributes.\n", - "#\n", - "# Structure:\n", - "# - Nodes: Represent entities (people, places, concepts, events)\n", - "# - Edges: Represent relationships (works_for, located_in, causes)\n", - "# - Properties: Attributes of entities and relationships\n", - "# - Metadata: Additional information (sources, timestamps, confidence)\n", - "#\n", - "# Example:\n", - "# Entity: \"John Doe\" (Person)\n", - "# Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n", - "# Properties: {start_date: \"2020-01-01\", role: \"Engineer\"}\n", - "#\n", - "# Benefits:\n", - "# - Structured representation of unstructured data\n", - "# - Enables complex queries and reasoning\n", - "# - Supports temporal tracking\n", - "# - Facilitates knowledge discovery\n", + "**Structure**:\n", + "- **Nodes**: Represent entities (people, places, concepts, events)\n", + "- **Edges**: Represent relationships (works_for, located_in, causes)\n", + "- **Properties**: Attributes of entities and relationships\n", + "- **Metadata**: Additional information (sources, timestamps, confidence)\n", "\n", - "# 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n", - "# Definition: The process of identifying and classifying named entities\n", - "# in text into predefined categories.\n", - "#\n", - "# Entity Types:\n", - "# - Person: Names of people\n", - "# - Organization: Companies, institutions\n", - "# - Location: Places, geographic entities\n", - "# - Date/Time: Temporal expressions\n", - "# - Money: Monetary values\n", - "# - Product: Products and services\n", - "# - Event: Events and occurrences\n", - "# - Custom: Domain-specific entities\n", - "#\n", - "# Example:\n", - "# Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n", - "# Entities:\n", - "# - \"Apple Inc.\" -> Organization\n", - "# - \"Steve Jobs\" -> Person\n", - "# - \"Cupertino, California\" -> Location\n", - "#\n", - "# Methods:\n", - "# - Rule-based: Pattern matching\n", - "# - Machine Learning: Trained models (spaCy, transformers)\n", - "# - LLM-based: Using large language models\n", + "**Example**:\n", + "- Entity: \"John Doe\" (Person)\n", + "- Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n", + "- Properties: `{start_date: \"2020-01-01\", role: \"Engineer\"}`\n", "\n", - "# 3. RELATIONSHIP EXTRACTION\n", - "# Definition: Identifying and extracting relationships between entities\n", - "# in text.\n", - "#\n", - "# Relationship Types:\n", - "# - Semantic: \"works_for\", \"located_in\", \"causes\"\n", - "# - Temporal: \"before\", \"after\", \"during\"\n", - "# - Causal: \"causes\", \"results_in\", \"prevents\"\n", - "# - Hierarchical: \"part_of\", \"subclass_of\", \"instance_of\"\n", - "#\n", - "# Example:\n", - "# Text: \"John works for Acme Corp in New York.\"\n", - "# Relationships:\n", - "# - (John, works_for, Acme Corp)\n", - "# - (Acme Corp, located_in, New York)\n", - "#\n", - "# Methods:\n", - "# - Pattern matching\n", - "# - Dependency parsing\n", - "# - Machine learning models\n", - "# - LLM-based extraction\n", + "**Benefits**:\n", + "- Structured representation of unstructured data\n", + "- Enables complex queries and reasoning\n", + "- Supports temporal tracking\n", + "- Facilitates knowledge discovery\n", "\n", - "# 4. EMBEDDINGS\n", - "# Definition: Dense vector representations of text, images, or other data\n", - "# that capture semantic meaning in a continuous vector space.\n", - "#\n", - "# Properties:\n", - "# - Similar entities have similar embeddings (close in vector space)\n", - "# - Enable semantic search and similarity calculations\n", - "# - Fixed or variable dimensions (typically 128-4096)\n", - "#\n", - "# Example:\n", - "# Text: \"machine learning\"\n", - "# Embedding: [0.123, -0.456, 0.789, ..., 0.234] (vector of 1536 dimensions)\n", - "#\n", - "# Use Cases:\n", - "# - Semantic search\n", - "# - Clustering and classification\n", - "# - Recommendation systems\n", - "# - Anomaly detection\n", + "### 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n", + "**Definition**: The process of identifying and classifying named entities in text into predefined categories.\n", "\n", - "# 5. TEMPORAL GRAPHS\n", - "# Definition: Knowledge graphs that track changes over time, allowing\n", - "# queries about the state of the graph at specific time points.\n", - "#\n", - "# Features:\n", - "# - Timestamps on entities and relationships\n", - "# - Version history\n", - "# - Time-point queries\n", - "# - Temporal pattern detection\n", - "#\n", - "# Example:\n", - "# Entity: \"Company X\"\n", - "# Relationship: (Company X, has_CEO, Person Y)\n", - "# Temporal: valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"\n", - "#\n", - "# Use Cases:\n", - "# - Tracking organizational changes\n", - "# - Monitoring system evolution\n", - "# - Analyzing trends over time\n", - "# - Historical analysis\n", + "**Entity Types**:\n", + "- **Person**: Names of people\n", + "- **Organization**: Companies, institutions\n", + "- **Location**: Places, geographic entities\n", + "- **Date/Time**: Temporal expressions\n", + "- **Money**: Monetary values\n", + "- **Product**: Products and services\n", + "- **Event**: Events and occurrences\n", + "- **Custom**: Domain-specific entities\n", "\n", - "# 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n", - "# Definition: An advanced RAG approach that combines vector search with\n", - "# knowledge graph traversal to provide more accurate and\n", - "# contextually relevant information to LLMs.\n", - "#\n", - "# Components:\n", - "# - Vector Store: For semantic similarity search\n", - "# - Knowledge Graph: For structured relationship traversal\n", - "# - Hybrid Search: Combines both approaches\n", - "# - LLM Integration: Uses retrieved context for generation\n", - "#\n", - "# Advantages over Traditional RAG:\n", - "# - Better handling of complex queries\n", - "# - Relationship-aware retrieval\n", - "# - Reduced hallucinations\n", - "# - More accurate answers\n", - "#\n", - "# Example Workflow:\n", - "# 1. Query: \"Who worked with John at Acme Corp?\"\n", - "# 2. Vector search finds relevant documents\n", - "# 3. Knowledge graph traversal finds relationships\n", - "# 4. Combined context sent to LLM\n", - "# 5. LLM generates accurate answer using both sources\n", + "**Example**:\n", + "Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n", + "Entities:\n", + "- \"Apple Inc.\" -> Organization\n", + "- \"Steve Jobs\" -> Person\n", + "- \"Cupertino, California\" -> Location\n", "\n", - "# 7. ONTOLOGY\n", - "# Definition: A formal specification of concepts, relationships, and\n", - "# constraints in a domain, typically expressed in OWL (Web\n", - "# Ontology Language).\n", - "#\n", - "# Components:\n", - "# - Classes: Categories of entities\n", - "# - Properties: Relationships and attributes\n", - "# - Individuals: Specific instances\n", - "# - Axioms: Rules and constraints\n", - "#\n", - "# Example:\n", - "# Class: Person\n", - "# SubClass: Employee, Customer\n", - "# Property: worksFor (domain: Person, range: Organization)\n", - "#\n", - "# Use Cases:\n", - "# - Standardize domain knowledge\n", - "# - Enable reasoning\n", - "# - Facilitate data integration\n", - "# - Support semantic web\n", + "**Methods**:\n", + "- Rule-based: Pattern matching\n", + "- Machine Learning: Trained models (spaCy, transformers)\n", + "- LLM-based: Using large language models\n", "\n", - "# 8. QUALITY ASSURANCE\n", - "# Definition: Processes and metrics to ensure knowledge graph quality,\n", - "# including completeness, consistency, and accuracy.\n", - "#\n", - "# Metrics:\n", - "# - Completeness: Percentage of entities with required properties\n", - "# - Consistency: Absence of contradictions\n", - "# - Accuracy: Correctness of extracted information\n", - "# - Coverage: Breadth of domain coverage\n", - "#\n", - "# Methods:\n", - "# - Validation rules\n", - "# - Automated quality checks\n", - "# - Conflict detection\n", - "# - Source verification\n", - "'''\n", + "### 3. RELATIONSHIP EXTRACTION\n", + "**Definition**: Identifying and extracting relationships between entities in text.\n", + "\n", + "**Relationship Types**:\n", + "- **Semantic**: \"works_for\", \"located_in\", \"causes\"\n", + "- **Temporal**: \"before\", \"after\", \"during\"\n", + "- **Causal**: \"causes\", \"results_in\", \"prevents\"\n", + "- **Hierarchical**: \"part_of\", \"subclass_of\", \"instance_of\"\n", + "\n", + "**Example**:\n", + "Text: \"John works for Acme Corp in New York.\"\n", + "Relationships:\n", + "- (John, works_for, Acme Corp)\n", + "- (Acme Corp, located_in, New York)\n", + "\n", + "**Methods**:\n", + "- Pattern matching\n", + "- Dependency parsing\n", + "- Machine learning models\n", + "- LLM-based extraction\n", + "\n", + "### 4. EMBEDDINGS\n", + "**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space.\n", + "\n", + "**Properties**:\n", + "- Similar entities have similar embeddings (close in vector space)\n", + "- Enable semantic search and similarity calculations\n", + "- Fixed or variable dimensions (typically 128-4096)\n", + "\n", + "**Example**:\n", + "Text: \"machine learning\"\n", + "Embedding: `[0.123, -0.456, 0.789, ..., 0.234]` (vector of 1536 dimensions)\n", + "\n", + "**Use Cases**:\n", + "- Semantic search\n", + "- Clustering and classification\n", + "- Recommendation systems\n", + "- Anomaly detection\n", + "\n", + "### 5. TEMPORAL GRAPHS\n", + "**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points.\n", + "\n", + "**Features**:\n", + "- Timestamps on entities and relationships\n", + "- Version history\n", + "- Time-point queries\n", + "- Temporal pattern detection\n", + "\n", + "**Example**:\n", + "- Entity: \"Company X\"\n", + "- Relationship: (Company X, has_CEO, Person Y)\n", + "- Temporal: `valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"`\n", + "\n", + "**Use Cases**:\n", + "- Tracking organizational changes\n", + "- Monitoring system evolution\n", + "- Analyzing trends over time\n", + "- Historical analysis\n", + "\n", + "### 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n", + "**Definition**: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs.\n", + "\n", + "**Components**:\n", + "- **Vector Store**: For semantic similarity search\n", + "- **Knowledge Graph**: For structured relationship traversal\n", + "- **Hybrid Search**: Combines both approaches\n", + "- **LLM Integration**: Uses retrieved context for generation\n", + "\n", + "**Advantages over Traditional RAG**:\n", + "- Better handling of complex queries\n", + "- Relationship-aware retrieval\n", + "- Reduced hallucinations\n", + "- More accurate answers\n", + "\n", + "**Example Workflow**:\n", + "1. Query: \"Who worked with John at Acme Corp?\"\n", + "2. Vector search finds relevant documents\n", + "3. Knowledge graph traversal finds relationships\n", + "4. Combined context sent to LLM\n", + "5. LLM generates accurate answer using both sources\n", + "\n", + "### 7. ONTOLOGY\n", + "**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language).\n", + "\n", + "**Components**:\n", + "- **Classes**: Categories of entities\n", + "- **Properties**: Relationships and attributes\n", + "- **Individuals**: Specific instances\n", + "- **Axioms**: Rules and constraints\n", + "\n", + "**Example**:\n", + "- Class: Person\n", + "- SubClass: Employee, Customer\n", + "- Property: worksFor (domain: Person, range: Organization)\n", + "\n", + "**Use Cases**:\n", + "- Standardize domain knowledge\n", + "- Enable reasoning\n", + "- Facilitate data integration\n", + "- Support semantic web\n", + "\n", + "### 8. QUALITY ASSURANCE\n", + "**Definition**: Processes and metrics to ensure knowledge graph quality, including completeness, consistency, and accuracy.\n", + "\n", + "**Metrics**:\n", + "- **Completeness**: Percentage of entities with required properties\n", + "- **Consistency**: Absence of contradictions\n", + "- **Accuracy**: Correctness of extracted information\n", + "- **Coverage**: Breadth of domain coverage\n", + "\n", + "**Methods**:\n", + "- Validation rules\n", + "- Automated quality checks\n", + "- Conflict detection\n", + "- Source verification\n", "\n", "---\n", "\n", @@ -510,41 +511,35 @@ "\n", "## Best Practices\n", "\n", - "'''\n", - "# ============================================================================\n", - "# BEST PRACTICES\n", - "# ============================================================================\n", + "### 1. START SMALL\n", + "- Begin with simple documents\n", + "- Validate each step before moving forward\n", + "- Build incrementally\n", "\n", - "# 1. START SMALL\n", - "# - Begin with simple documents\n", - "# - Validate each step before moving forward\n", - "# - Build incrementally\n", + "### 2. CONFIGURE PROPERLY\n", + "- Use environment variables for sensitive data\n", + "- Set up proper logging\n", + "- Configure appropriate model sizes\n", "\n", - "# 2. CONFIGURE PROPERLY\n", - "# - Use environment variables for sensitive data\n", - "# - Set up proper logging\n", - "# - Configure appropriate model sizes\n", + "### 3. VALIDATE DATA\n", + "- Always validate extracted entities\n", + "- Check relationship quality\n", + "- Use quality assurance tools\n", "\n", - "# 3. VALIDATE DATA\n", - "# - Always validate extracted entities\n", - "# - Check relationship quality\n", - "# - Use quality assurance tools\n", + "### 4. HANDLE ERRORS\n", + "- Implement error handling\n", + "- Use retry mechanisms\n", + "- Log errors for debugging\n", "\n", - "# 4. HANDLE ERRORS\n", - "# - Implement error handling\n", - "# - Use retry mechanisms\n", - "# - Log errors for debugging\n", + "### 5. OPTIMIZE PERFORMANCE\n", + "- Use batch processing for large datasets\n", + "- Enable parallel processing where possible\n", + "- Cache embeddings and results\n", "\n", - "# 5. OPTIMIZE PERFORMANCE\n", - "# - Use batch processing for large datasets\n", - "# - Enable parallel processing where possible\n", - "# - Cache embeddings and results\n", - "\n", - "# 6. DOCUMENT YOUR WORKFLOWS\n", - "# - Document data sources\n", - "# - Track processing steps\n", - "# - Maintain metadata\n", - "'''\n", + "### 6. DOCUMENT YOUR WORKFLOWS\n", + "- Document data sources\n", + "- Track processing steps\n", + "- Maintain metadata\n", "\n", "---\n", "\n", @@ -552,45 +547,39 @@ "\n", "Common issues and solutions:\n", "\n", - "'''\n", - "# ============================================================================\n", - "# TROUBLESHOOTING\n", - "# ============================================================================\n", + "### Issue 1: Import Errors\n", + "**Solution**:\n", + "- Ensure Semantica is properly installed\n", + "- Check Python version (3.8+)\n", + "- Verify virtual environment is activated\n", + "- Install missing dependencies: `pip install -r requirements.txt`\n", "\n", - "# Issue 1: Import Errors\n", - "# Solution:\n", - "# - Ensure Semantica is properly installed\n", - "# - Check Python version (3.8+)\n", - "# - Verify virtual environment is activated\n", - "# - Install missing dependencies: pip install -r requirements.txt\n", + "### Issue 2: API Key Errors\n", + "**Solution**:\n", + "- Set environment variables: `export SEMANTICA_API_KEY=your_key`\n", + "- Check config file for correct key format\n", + "- Verify API key is valid and has sufficient credits\n", "\n", - "# Issue 2: API Key Errors\n", - "# Solution:\n", - "# - Set environment variables: export SEMANTICA_API_KEY=your_key\n", - "# - Check config file for correct key format\n", - "# - Verify API key is valid and has sufficient credits\n", + "### Issue 3: Memory Issues\n", + "**Solution**:\n", + "- Process documents in batches\n", + "- Use smaller embedding models\n", + "- Enable garbage collection\n", + "- Consider using streaming for large datasets\n", "\n", - "# Issue 3: Memory Issues\n", - "# Solution:\n", - "# - Process documents in batches\n", - "# - Use smaller embedding models\n", - "# - Enable garbage collection\n", - "# - Consider using streaming for large datasets\n", + "### Issue 4: Low Quality Extractions\n", + "**Solution**:\n", + "- Preprocess and normalize text\n", + "- Use domain-specific models\n", + "- Adjust extraction parameters\n", + "- Validate and clean extracted entities\n", "\n", - "# Issue 4: Low Quality Extractions\n", - "# Solution:\n", - "# - Preprocess and normalize text\n", - "# - Use domain-specific models\n", - "# - Adjust extraction parameters\n", - "# - Validate and clean extracted entities\n", - "\n", - "# Issue 5: Slow Processing\n", - "# Solution:\n", - "# - Enable parallel processing\n", - "# - Use GPU acceleration if available\n", - "# - Cache intermediate results\n", - "# - Optimize batch sizes\n", - "'''\n" + "### Issue 5: Slow Processing\n", + "**Solution**:\n", + "- Enable parallel processing\n", + "- Use GPU acceleration if available\n", + "- Cache intermediate results\n", + "- Optimize batch sizes\n" ] }, { @@ -859,38 +848,6 @@ "print(\" generator = EmbeddingGenerator()\")\n", "print(\" embeddings = generator.generate(documents)\")\n" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\n" - ] } ], "metadata": { @@ -900,4 +857,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/cookbook/introduction/Your_First_Knowledge_Graph.ipynb b/cookbook/introduction/Your_First_Knowledge_Graph.ipynb index 5d79366a..61e23295 100644 --- a/cookbook/introduction/Your_First_Knowledge_Graph.ipynb +++ b/cookbook/introduction/Your_First_Knowledge_Graph.ipynb @@ -4,37 +4,40 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Your First Knowledge Graph\n", + "# πŸš€ Your First Knowledge Graph\n", "\n", "## Overview\n", "\n", "This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n", "\n", - "### Learning Objectives\n", + "> [!TIP]\n", + "> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n", "\n", - "- Understand the basic workflow: **File β†’ Parse β†’ Extract β†’ Graph**\n", - "- Learn how to ingest documents using `FileIngestor`\n", - "- Parse documents using `DocumentParser`\n", - "- Extract entities using NER extractors\n", - "- Build a knowledge graph using `GraphBuilder`\n", - "- Visualize and analyze the resulting graph\n", + "### 🎯 Learning Objectives\n", + "\n", + "- **Understand the Workflow**: Learn the `File β†’ Parse β†’ Extract β†’ Graph` pipeline\n", + "- **Ingest Data**: Load documents using `FileIngestor`\n", + "- **Parse Content**: Extract text using `DocumentParser`\n", + "- **Extract Knowledge**: Identify entities using `NERExtractor`\n", + "- **Build Graph**: Construct a graph using `GraphBuilder`\n", + "- **Visualize**: See your graph come to life with `KGVisualizer`\n", "\n", "---\n", "\n", - "## Simple End-to-End Workflow\n", + "## πŸ”„ Simple End-to-End Workflow\n", "\n", "The complete workflow consists of four main steps:\n", "\n", - "1. **Ingest** - Load data from files or other sources\n", - "2. **Parse** - Extract and structure content from documents\n", - "3. **Extract** - Identify entities and relationships\n", - "4. **Build Graph** - Construct the knowledge graph\n", + "1. **πŸ“₯ Ingest** - Load data from files or other sources\n", + "2. **πŸ“„ Parse** - Extract and structure content from documents\n", + "3. **⛏️ Extract** - Identify entities and relationships\n", + "4. **πŸ•ΈοΈ Build Graph** - Construct the knowledge graph\n", "\n", "Each step is demonstrated in the code cells below.\n", "\n", "---\n", "\n", - "## Step 1: Ingest a File\n", + "## πŸ“‚ Step 1: Ingest a File\n", "\n", "In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n" ] @@ -48,8 +51,10 @@ "from semantica.ingest import FileIngestor\n", "from pathlib import Path\n", "\n", + "# Initialize the ingestor\n", "ingestor = FileIngestor()\n", "\n", + "# Create a sample document for demonstration\n", "sample_text = \"\"\"\n", "Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n", "The company is headquartered in Cupertino, California.\n", @@ -64,6 +69,7 @@ "print(f\"File: {sample_file}\")\n", "print(f\"Content length: {len(sample_text)} characters\")\n", "\n", + "# Ingest the file\n", "try:\n", " file_object = ingestor.ingest_file(sample_file, read_content=True)\n", " print(f\"\\nβœ“ File ingested successfully!\")\n", @@ -78,7 +84,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 2: Parse the Document\n", + "## πŸ“„ Step 2: Parse the Document\n", "\n", "After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n" ] @@ -94,12 +100,14 @@ "parser = DocumentParser()\n", "\n", "try:\n", + " # Parse the document to extract text\n", " if 'file_object' in locals():\n", " parsed_content = parser.parse_document(str(sample_file))\n", " print(\"βœ“ Document parsed successfully!\")\n", " print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n", " print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n", " else:\n", + " # Fallback if ingestion failed\n", " parsed_content = parser.parse_document(str(sample_file))\n", " print(\"βœ“ Document parsed successfully!\")\n", " print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n", @@ -113,9 +121,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 3: Extract Entities\n", + "## ⛏️ Step 3: Extract Entities\n", "\n", - "Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n" + "Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n", + "\n", + "> [!NOTE]\n", + "> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n" ] }, { @@ -133,6 +144,7 @@ " print(\"Extracting entities from text...\")\n", " print(f\"\\nText: {parsed_content[:100]}...\")\n", " \n", + " # Simulated extraction results\n", " expected_entities = [\n", " {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n", " {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n", @@ -156,7 +168,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 4: Build the Knowledge Graph\n", + "## πŸ•ΈοΈ Step 4: Build the Knowledge Graph\n", "\n", "Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n" ] @@ -172,6 +184,7 @@ "\n", "builder = GraphBuilder()\n", "\n", + "# Prepare data for graph construction\n", "entities_data = [\n", " {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n", " for i, entity in enumerate(expected_entities)\n", @@ -187,6 +200,7 @@ "]\n", "\n", "try:\n", + " # Build the graph using NetworkX\n", " kg = nx.DiGraph()\n", " \n", " for entity in entities_data:\n", @@ -221,7 +235,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 5: Visualize and Analyze\n", + "## πŸ“Š Step 5: Visualize and Analyze\n", "\n", "Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n" ] @@ -268,6 +282,7 @@ "except Exception as e:\n", " print(f\"βœ— Error visualizing graph: {e}\")\n", "\n", + "# Cleanup\n", "try:\n", " if sample_file.exists():\n", " sample_file.unlink()\n", @@ -284,4 +299,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/cookbook/use_cases/finance/Financial_Data_Integration.ipynb b/cookbook/use_cases/finance/Financial_Data_Integration.ipynb index d3836cce..7300151c 100644 --- a/cookbook/use_cases/finance/Financial_Data_Integration.ipynb +++ b/cookbook/use_cases/finance/Financial_Data_Integration.ipynb @@ -4,32 +4,33 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Financial Data Integration Pipeline\n", + "# πŸ“ˆ Financial Data Integration Pipeline\n", "\n", "## Overview\n", "\n", "This notebook demonstrates how to integrate Python/FastMCP MCP servers as data sources for financial data ingestion. Connect to financial data MCP servers via URL, ingest market data, stock prices, and financial metrics, then build a knowledge graph for financial analysis.\n", "\n", - "**IMPORTANT**: This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n", + "> [!IMPORTANT]\n", + "> This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n", "\n", - "### Modules Used (20+)\n", + "### 🧩 Modules Used (20+)\n", "\n", - "- **Ingestion**: MCPIngestor, ingest_mcp, WebIngestor, FileIngestor\n", - "- **Parsing**: MCPParser, JSONParser, StructuredDataParser\n", - "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n", - "- **KG**: GraphBuilder, TemporalGraphQuery, GraphAnalyzer\n", - "- **Analytics**: CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n", - "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n", - "- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n", - "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n", + "- **Ingestion**: `MCPIngestor`, `ingest_mcp`, `WebIngestor`, `FileIngestor`\n", + "- **Parsing**: `MCPParser`, `JSONParser`, `StructuredDataParser`\n", + "- **Extraction**: `NERExtractor`, `RelationExtractor`, `EventDetector`, `SemanticAnalyzer`\n", + "- **KG**: `GraphBuilder`, `TemporalGraphQuery`, `GraphAnalyzer`\n", + "- **Analytics**: `CentralityCalculator`, `CommunityDetector`, `ConnectivityAnalyzer`\n", + "- **Reasoning**: `InferenceEngine`, `RuleManager`, `ExplanationGenerator`\n", + "- **Export**: `JSONExporter`, `CSVExporter`, `RDFExporter`, `ReportGenerator`\n", + "- **Visualization**: `KGVisualizer`, `TemporalVisualizer`, `AnalyticsVisualizer`\n", "\n", - "### Pipeline\n", + "### πŸ”„ Pipeline\n", "\n", "**Connect to Financial MCP Server β†’ Ingest Market Data via MCP β†’ Parse MCP Responses β†’ Extract Financial Entities β†’ Build Financial KG β†’ Analyze Trends β†’ Generate Reports β†’ Visualize**\n", "\n", "---\n", "\n", - "## Step 1: Connect to Financial Data MCP Server\n", + "## πŸ”Œ Step 1: Connect to Financial Data MCP Server\n", "\n", "Connect to a Python/FastMCP MCP server that provides financial data via URL. The MCP server can expose resources (datasets, market data) and tools (queries, calculations).\n" ] @@ -92,7 +93,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 2: Ingest Financial Data from MCP Server\n", + "## πŸ“₯ Step 2: Ingest Financial Data from MCP Server\n", "\n", "Ingest financial data using both resource-based and tool-based methods from the MCP server.\n" ] @@ -220,7 +221,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 3: Parse MCP Data\n", + "## πŸ“„ Step 3: Parse MCP Data\n", "\n", "Parse the data received from MCP server responses (JSON, structured data).\n" ] @@ -266,7 +267,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 4: Extract Financial Entities and Relationships\n", + "## ⛏️ Step 4: Extract Financial Entities and Relationships\n", "\n", "Extract financial entities (companies, stocks, sectors) and relationships from MCP data.\n" ] @@ -363,7 +364,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 5: Build Financial Knowledge Graph\n", + "## πŸ•ΈοΈ Step 5: Build Financial Knowledge Graph\n", "\n", "Build a knowledge graph from the extracted financial entities and relationships.\n" ] @@ -404,7 +405,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 6: Analyze Financial Trends\n", + "## πŸ“Š Step 6: Analyze Financial Trends\n", "\n", "Analyze financial trends using temporal queries and pattern detection.\n" ] @@ -462,7 +463,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 7: Export and Visualize\n", + "## πŸ“€ Step 7: Export and Visualize\n", "\n", "Export the financial knowledge graph and generate visualizations.\n" ] @@ -535,4 +536,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/docs/concepts.md b/docs/concepts.md index 0a5369ab..b44c97df 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -1,20 +1,23 @@ # Core Concepts -Understand the fundamental concepts behind Semantica. +Understand the fundamental concepts behind Semantica. This guide covers the theoretical foundations, key components, and best practices for building semantic applications. -!!! note "Conceptual Foundation" - This guide covers the theoretical foundations of Semantica. For hands-on examples, see the [Quickstart Guide](quickstart.md) or [Examples](examples.md). +## 🧠 Core Concepts -## What is a Knowledge Graph? +### 1. Knowledge Graphs -A knowledge graph is a structured representation of information where: +**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes. -!!! tip "Visual Understanding" - The diagram below shows how entities (nodes) are connected by relationships (edges) to form a knowledge graph. This structure enables powerful querying and reasoning capabilities. +- **Nodes**: Represent entities (people, places, concepts, events) +- **Edges**: Represent relationships (works_for, located_in, causes) +- **Properties**: Attributes of entities and relationships +- **Metadata**: Additional information (sources, timestamps, confidence) -- **Entities** are the nodes (people, places, concepts, etc.) -- **Relationships** are the edges connecting entities -- **Properties** describe attributes of entities +**Benefits**: +- Structured representation of unstructured data +- Enables complex queries and reasoning +- Supports temporal tracking +- Facilitates knowledge discovery ```mermaid graph LR @@ -22,149 +25,179 @@ graph LR A -->|located_in| C[Cupertino
Location] C -->|in_state| D[California
Location] - style A fill:#e3f2fd - style B fill:#fff3e0 - style C fill:#f3e5f5 - style D fill:#f3e5f5 + style A fill:#e3f2fd,stroke:#1565c0 + style B fill:#fff3e0,stroke:#ef6c00 + style C fill:#f3e5f5,stroke:#7b1fa2 + style D fill:#f3e5f5,stroke:#7b1fa2 ``` -## Semantic Layer +### 2. Entity Extraction (NER) -A semantic layer provides: +**Definition**: The process of identifying and classifying named entities in text into predefined categories. -- **Structured meaning** from unstructured data -- **Contextual relationships** between concepts -- **Queryable knowledge** for AI systems -- **Quality-assured data** with conflict resolution +| Entity Type | Description | Example | +| :--- | :--- | :--- | +| **Person** | Names of people | Steve Jobs, Elon Musk | +| **Organization** | Companies, institutions | Apple Inc., NASA | +| **Location** | Places, geographic entities | Cupertino, Mars | +| **Date/Time** | Temporal expressions | 1976, next Monday | +| **Money** | Monetary values | $100 million | +| **Event** | Events and occurrences | WWDC 2024 | -## Key Components - -### 1. Data Ingestion - -Import data from various sources: - -- Documents (PDF, DOCX, HTML) -- Databases -- APIs and web content -- Structured data (JSON, CSV) - -### 2. Entity Extraction - -Identify and extract: - -- **Named Entities**: People, organizations, locations -- **Concepts**: Ideas, topics, themes -- **Events**: Actions, occurrences -- **Relations**: Connections between entities +**Methods**: +- **Rule-based**: Pattern matching (Regex) +- **Machine Learning**: Trained models (spaCy, transformers) +- **LLM-based**: Using large language models (GPT-4, Claude) ### 3. Relationship Extraction -Discover relationships: +**Definition**: Identifying and extracting relationships between entities in text. -- **Explicit**: Directly stated in text -- **Implicit**: Inferred from context -- **Temporal**: Time-based relationships -- **Causal**: Cause-and-effect connections +- **Semantic**: `works_for`, `located_in`, `causes` +- **Temporal**: `before`, `after`, `during` +- **Causal**: `causes`, `results_in`, `prevents` +- **Hierarchical**: `part_of`, `subclass_of`, `instance_of` -### 4. Knowledge Graph Construction +### 4. Embeddings -Build structured graphs: +**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space. -- **Node creation**: Entities as nodes -- **Edge creation**: Relationships as edges -- **Property assignment**: Attributes and metadata -- **Graph validation**: Quality checks +- **Properties**: + - Similar entities have similar embeddings (close in vector space). + - Enable semantic search and similarity calculations. + - Fixed dimensions (typically 128-4096). -### 5. Conflict Resolution +**Example**: +```python +Text: "machine learning" +Embedding: [0.123, -0.456, 0.789, ..., 0.234] +# (vector of 1536 dimensions) +``` -Handle conflicting information: +### 5. Temporal Graphs -- **Multiple sources**: Same entity, different facts -- **Resolution strategies**: Voting, credibility, recency -- **Quality assurance**: Validation and verification +**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points. -### 6. Embedding Generation +- **Features**: + - Timestamps on entities and relationships + - Version history + - Time-point queries + - Temporal pattern detection -Create vector representations: +### 6. GraphRAG -- **Text embeddings**: Semantic text vectors -- **Graph embeddings**: Node and edge vectors -- **Multimodal**: Text, image, audio embeddings +**Definition**: An advanced RAG (Retrieval Augmented Generation) approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs. -## Workflow - -Typical Semantica workflow: +**Advantages over Traditional RAG**: +- Better handling of complex queries +- Relationship-aware retrieval +- Reduced hallucinations +- More accurate answers ```mermaid flowchart TD - A[Data Source] --> B[Ingestion] - B --> C[Parsing] - C --> D[Extraction
Entities & Relationships] - D --> E[Normalization] - E --> F[Conflict Resolution] - F --> G[Knowledge Graph] - G --> H[Embeddings] - H --> I[Export] + Q[User Query] --> VS[Vector Search] + Q --> KG[Graph Traversal] + VS --> C[Context] + KG --> C + C --> LLM[LLM Generation] + LLM --> A[Answer] - style A fill:#e3f2fd - style G fill:#c8e6c9 - style I fill:#fff9c4 + style Q fill:#e1f5fe + style LLM fill:#e8f5e9 + style A fill:#fff9c4 ``` -## Use Cases +### 7. Ontology -### GraphRAG +**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language). -Enhance RAG systems with knowledge graphs: +- **Classes**: Categories of entities (e.g., `Person`, `Company`) +- **Properties**: Relationships and attributes (e.g., `worksFor`) +- **Individuals**: Specific instances (e.g., `John Doe`) +- **Axioms**: Rules and constraints -- **Context expansion**: Follow relationships -- **Multi-hop reasoning**: Traverse graph paths -- **Structured queries**: Query graph directly +### 8. Quality Assurance -### AI Agents +**Definition**: Processes and metrics to ensure knowledge graph quality. -Provide agents with: +- **Completeness**: Percentage of entities with required properties +- **Consistency**: Absence of contradictions +- **Accuracy**: Correctness of extracted information +- **Coverage**: Breadth of domain coverage -- **Persistent memory**: Knowledge graph as memory -- **Context understanding**: Semantic relationships -- **Action validation**: Check against knowledge +--- -### Data Integration +## 🌟 Best Practices -Unify data from multiple sources: - -- **Schema mapping**: Automatic schema discovery -- **Entity resolution**: Match entities across sources -- **Conflict resolution**: Handle contradictions - -## Best Practices - -!!! note "Best Practices" - Following these practices will help you build high-quality knowledge graphs and avoid common pitfalls. +Following these practices will help you build high-quality knowledge graphs and avoid common pitfalls. ### 1. Start Small +- Begin with simple documents. +- Validate each step before moving forward. +- Build incrementally. -Begin with a single document or small dataset to understand the workflow. +### 2. Configure Properly +- Use environment variables for sensitive data. +- Set up proper logging. +- Configure appropriate model sizes. -### 2. Iterate +### 3. Validate Data +- Always validate extracted entities. +- Check relationship quality. +- Use quality assurance tools. -Build knowledge graphs incrementally, refining as you learn. +### 4. Handle Errors +- Implement error handling. +- Use retry mechanisms. +- Log errors for debugging. -### 3. Validate +### 5. Optimize Performance +- Use batch processing for large datasets. +- Enable parallel processing where possible. +- Cache embeddings and results. -Always validate extracted entities and relationships. +### 6. Document Workflows +- Document data sources. +- Track processing steps. +- Maintain metadata. -### 4. Resolve Conflicts +--- -Use appropriate conflict resolution strategies for your use case. +## πŸ”§ Troubleshooting -### 5. Export Regularly +Common issues and solutions: -Export your knowledge graphs for backup and analysis. +!!! failure "Import Errors" + **Solution**: + - Ensure Semantica is properly installed. + - Check Python version (3.8+). + - Verify virtual environment is activated. + - Install missing dependencies: `pip install -r requirements.txt` -## Next Steps +!!! failure "API Key Errors" + **Solution**: + - Set environment variables: `export SEMANTICA_API_KEY=your_key` + - Check config file for correct key format. + - Verify API key is valid and has sufficient credits. -- **[Quick Start](quickstart.md)** - Build your first knowledge graph -- **[Examples](examples.md)** - See real-world applications -- **[API Reference](api.md)** - Explore the full API +!!! failure "Memory Issues" + **Solution**: + - Process documents in batches. + - Use smaller embedding models. + - Enable garbage collection. + - Consider using streaming for large datasets. +!!! failure "Low Quality Extractions" + **Solution**: + - Preprocess and normalize text. + - Use domain-specific models. + - Adjust extraction parameters. + - Validate and clean extracted entities. + +!!! failure "Slow Processing" + **Solution**: + - Enable parallel processing. + - Use GPU acceleration if available. + - Cache intermediate results. + - Optimize batch sizes. diff --git a/docs/cookbook.md b/docs/cookbook.md new file mode 100644 index 00000000..fa9b7a2a --- /dev/null +++ b/docs/cookbook.md @@ -0,0 +1,136 @@ +# 🍳 Semantica Cookbook + +Welcome to the **Semantica Cookbook**! + +This collection of Jupyter notebooks is designed to take you from a beginner to an expert in building semantic AI applications. Whether you're looking for quick recipes or deep-dive tutorials, you'll find it here. + +--- + +## 🏁 Introduction + +Start here if you are new to the framework. These notebooks cover the essentials. + +- **[Welcome to Semantica](cookbook/introduction/Welcome_to_Semantica.ipynb)**: An interactive introduction to the framework. +- **[Configuration Basics](cookbook/introduction/Configuration_Basics.ipynb)**: Learn how to configure API keys and settings. +- **[Your First Knowledge Graph](cookbook/introduction/Your_First_Knowledge_Graph.ipynb)**: Build a simple graph from scratch. +- **[Data Ingestion](cookbook/introduction/Data_Ingestion.ipynb)**: Loading data from files and web sources. +- **[Document Parsing](cookbook/introduction/Document_Parsing.ipynb)**: Extracting text from PDFs, DOCX, and more. +- **[Data Normalization](cookbook/introduction/Data_Normalization.ipynb)**: Cleaning and preparing text. +- **[Entity Extraction](cookbook/introduction/Entity_Extraction.ipynb)**: Identifying people, places, and organizations. +- **[Relation Extraction](cookbook/introduction/Relation_Extraction.ipynb)**: Finding connections between entities. +- **[Embedding Generation](cookbook/introduction/Embedding_Generation.ipynb)**: Creating vector representations. +- **[Vector Store](cookbook/introduction/Vector_Store.ipynb)**: Storing and searching vectors. +- **[Ontology](cookbook/introduction/Ontology.ipynb)**: Defining the structure of your knowledge. +- **[Conflict Detection](cookbook/introduction/Conflict_Detection.ipynb)**: Handling contradictory information. +- **[Deduplication](cookbook/introduction/Deduplication.ipynb)**: Merging duplicate entities. +- **[Building Knowledge Graphs](cookbook/introduction/Building_Knowledge_Graphs.ipynb)**: Putting it all together. +- **[Graph Analytics](cookbook/introduction/Graph_Analytics.ipynb)**: Analyzing graph structure. +- **[Graph Quality](cookbook/introduction/Graph_Quality.ipynb)**: Ensuring data quality. +- **[Visualization](cookbook/introduction/Visualization.ipynb)**: Visualizing your graphs. +- **[Export](cookbook/introduction/Export.ipynb)**: Exporting data to other formats. + +--- + +## 🧠 Advanced Concepts + +Deep dive into advanced features and customization. + +- **[Advanced Extraction](cookbook/advanced/Advanced_Extraction.ipynb)**: Techniques for complex entity and relation extraction. +- **[Advanced Graph Analytics](cookbook/advanced/Advanced_Graph_Analytics.ipynb)**: In-depth graph analysis algorithms. +- **[Complete Visualization Suite](cookbook/advanced/Complete_Visualization_Suite.ipynb)**: Comprehensive guide to visualization tools. +- **[Conflict Resolution Strategies](cookbook/advanced/Conflict_Resolution_Strategies.ipynb)**: Advanced methods for resolving data conflicts. +- **[Multi-Format Export](cookbook/advanced/Multi_Format_Export.ipynb)**: Exporting data to various formats. +- **[Multi-Source Data Integration](cookbook/advanced/Multi_Source_Data_Integration.ipynb)**: Integrating data from disparate sources. +- **[Pipeline Orchestration](cookbook/advanced/Pipeline_Orchestration.ipynb)**: Building and managing complex pipelines. +- **[Reasoning and Inference](cookbook/advanced/Reasoning_and_Inference.ipynb)**: Applying logical reasoning to your graph. +- **[Semantic Layer Construction](cookbook/advanced/Semantic_Layer_Construction.ipynb)**: Building a robust semantic layer. +- **[Temporal Knowledge Graphs](cookbook/advanced/Temporal_Knowledge_Graphs.ipynb)**: Working with time-aware graphs. +- **[Text Chunking Strategies](cookbook/advanced/Text_Chunking_Strategies.ipynb)**: Optimizing text processing. +- **[Unstructured to Ontology](cookbook/advanced/Unstructured_to_Ontology.ipynb)**: Automatically generating ontologies from text. + +--- + +## πŸ’‘ Use Cases + +Real-world examples and applications across various industries. + +### Advanced RAG +- **[GraphRAG Complete](cookbook/use_cases/advanced_rag/GraphRAG_Complete.ipynb)**: End-to-end implementation of Graph Retrieval Augmented Generation. + +### Biomedical +- **[Drug Discovery Pipeline](cookbook/use_cases/biomedical/Drug_Discovery_Pipeline.ipynb)**: Accelerating drug discovery with knowledge graphs. +- **[Genomic Variant Analysis](cookbook/use_cases/biomedical/Genomic_Variant_Analysis.ipynb)**: Analyzing genomic variants and their implications. + +### Blockchain +- **[DeFi Protocol Intelligence](cookbook/use_cases/blockchain/DeFi_Protocol_Intelligence.ipynb)**: Analyzing decentralized finance protocols. +- **[Transaction Network Analysis](cookbook/use_cases/blockchain/Transaction_Network_Analysis.ipynb)**: Investigating blockchain transaction networks. + +### Cybersecurity +- **[Anomaly Detection Real-Time](cookbook/use_cases/cybersecurity/Anomaly_Detection_Real_Time.ipynb)**: Detecting anomalies in real-time streams. +- **[Incident Analysis](cookbook/use_cases/cybersecurity/Incident_Analysis.ipynb)**: Analyzing security incidents and breaches. +- **[Threat Correlation](cookbook/use_cases/cybersecurity/Threat_Correlation.ipynb)**: Correlating threats across different vectors. +- **[Threat Intelligence Hybrid RAG](cookbook/use_cases/cybersecurity/Threat_Intelligence_Hybrid_RAG.ipynb)**: Combining RAG with threat intelligence. +- **[Threat Intelligence Integration](cookbook/use_cases/cybersecurity/Threat_Intelligence_Integration.ipynb)**: Integrating threat feeds into a knowledge graph. +- **[Vulnerability Tracking](cookbook/use_cases/cybersecurity/Vulnerability_Tracking.ipynb)**: Tracking and managing system vulnerabilities. + +### Finance +- **[Financial Data Integration](cookbook/use_cases/finance/Financial_Data_Integration.ipynb)**: Merging financial data from multiple sources. +- **[Financial Reports Analysis](cookbook/use_cases/finance/Financial_Reports_Analysis.ipynb)**: Extracting insights from financial reports. +- **[Fraud Detection](cookbook/use_cases/finance/Fraud_Detection.ipynb)**: Identifying fraudulent activities and patterns. +- **[Investment Analysis Hybrid RAG](cookbook/use_cases/finance/Investment_Analysis_Hybrid_RAG.ipynb)**: AI-powered investment analysis. +- **[Market Intelligence](cookbook/use_cases/finance/Market_Intelligence.ipynb)**: Gathering and analyzing market intelligence. +- **[Regulatory Compliance](cookbook/use_cases/finance/Regulatory_Compliance.ipynb)**: Ensuring compliance with financial regulations. + +### Healthcare +- **[Clinical Reports Processing](cookbook/use_cases/healthcare/Clinical_Reports_Processing.ipynb)**: Processing and structuring clinical reports. +- **[Disease Network Analysis](cookbook/use_cases/healthcare/Disease_Network_Analysis.ipynb)**: Analyzing disease networks and comorbidities. +- **[Drug Interactions Analysis](cookbook/use_cases/healthcare/Drug_Interactions_Analysis.ipynb)**: Identifying potential drug interactions. +- **[Healthcare GraphRAG Hybrid](cookbook/use_cases/healthcare/Healthcare_GraphRAG_Hybrid.ipynb)**: Hybrid RAG for healthcare applications. +- **[Medical Database Integration](cookbook/use_cases/healthcare/Medical_Database_Integration.ipynb)**: Integrating medical databases. +- **[Medical Literature GraphRAG](cookbook/use_cases/healthcare/Medical_Literature_GraphRAG.ipynb)**: Querying medical literature with GraphRAG. +- **[Patient Records Temporal](cookbook/use_cases/healthcare/Patient_Records_Temporal.ipynb)**: Analyzing patient records over time. + +### Intelligence +- **[Network Analysis Intelligence Reports](cookbook/use_cases/intelligence/Network_Analysis_Intelligence_Reports.ipynb)**: Analyzing intelligence reports for network insights. + +### Renewable Energy +- **[Energy Market Analysis](cookbook/use_cases/renewable_energy/Energy_Market_Analysis.ipynb)**: Analyzing trends in the energy market. +- **[Environmental Impact](cookbook/use_cases/renewable_energy/Environmental_Impact.ipynb)**: Assessing environmental impact. +- **[Grid Management](cookbook/use_cases/renewable_energy/Grid_Management.ipynb)**: Optimizing power grid management. +- **[Resource Optimization](cookbook/use_cases/renewable_energy/Resource_Optimization.ipynb)**: Optimizing renewable resources. +- **[Supply Chain Analysis](cookbook/use_cases/renewable_energy/Supply_Chain_Analysis.ipynb)**: Analyzing the renewable energy supply chain. + +### Supply Chain +- **[Supply Chain Data Integration](cookbook/use_cases/supply_chain/Supply_Chain_Data_Integration.ipynb)**: Integrating supply chain data. +- **[Supply Chain Risk Management](cookbook/use_cases/supply_chain/Supply_Chain_Risk_Management.ipynb)**: Managing and mitigating supply chain risks. + +### Trading +- **[Market Data Analysis](cookbook/use_cases/trading/Market_Data_Analysis.ipynb)**: Analyzing trading market data. +- **[News Sentiment Analysis](cookbook/use_cases/trading/News_Sentiment_Analysis.ipynb)**: Analyzing news sentiment for trading signals. +- **[Real Time Market Data](cookbook/use_cases/trading/Real_Time_Market_Data.ipynb)**: Processing real-time market data. +- **[Real Time Monitoring](cookbook/use_cases/trading/Real_Time_Monitoring.ipynb)**: Monitoring trading systems in real-time. +- **[Risk Assessment](cookbook/use_cases/trading/Risk_Assessment.ipynb)**: Assessing trading risks. +- **[Strategy Backtesting](cookbook/use_cases/trading/Strategy_Backtesting.ipynb)**: Backtesting trading strategies. + +--- + +## πŸ› οΈ How to Run + +To run these notebooks locally: + +1. **Clone the repository**: + ```bash + git clone https://github.com/Hawksight-AI/semantica.git + cd semantica + ``` + +2. **Install dependencies**: + ```bash + pip install -e .[all] + pip install jupyter + ``` + +3. **Launch Jupyter**: + ```bash + jupyter notebook + ``` diff --git a/docs/cookbook/introduction/Welcome_to_Semantica.ipynb b/docs/cookbook/introduction/Welcome_to_Semantica.ipynb index a6a40031..62be5293 100644 --- a/docs/cookbook/introduction/Welcome_to_Semantica.ipynb +++ b/docs/cookbook/introduction/Welcome_to_Semantica.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Welcome to Semantica\n", + "# Welcome to Semantica: An interactive introduction to the framework\n", "\n", "## Overview\n", "\n", @@ -54,7 +54,7 @@ "\n", "### Installation Methods\n", "\n", - "'''\n", + "```bash\n", "# Method 1: Install from PyPI (when available)\n", "# pip install semantica\n", "\n", @@ -71,11 +71,11 @@ "# Verify installation\n", "# import semantica\n", "# print(semantica.__version__)\n", - "'''\n", + "```\n", "\n", "### Configuration\n", "\n", - "'''\n", + "```bash\n", "# Set up environment variables for API keys and configuration\n", "# export SEMANTICA_API_KEY=your_openai_key\n", "# export SEMANTICA_EMBEDDING_PROVIDER=openai\n", @@ -92,7 +92,7 @@ "# knowledge_graph:\n", "# backend: networkx # or neo4j, arangodb\n", "# temporal: true\n", - "'''\n", + "```\n", "\n", "---\n", "\n", @@ -100,205 +100,223 @@ "\n", "Semantica is organized into modular components, each handling a specific aspect of semantic processing:\n", "\n", - "'''\n", - "# ============================================================================\n", - "# CORE MODULES\n", - "# ============================================================================\n", + "### 1. INGEST MODULE - Data Ingestion\n", + "**Purpose**: Ingest data from various sources\n", + "**Components**:\n", + "- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n", + "- `WebIngestor`: Scrape and ingest web pages\n", + "- `FeedIngestor`: Process RSS/Atom feeds\n", + "- `StreamIngestor`: Real-time data streaming\n", + "- `DBIngestor`: Database queries and ingestion\n", + "- `EmailIngestor`: Process email messages\n", + "- `RepoIngestor`: Git repository analysis\n", "\n", - "# 1. INGEST MODULE - Data Ingestion\n", - "# Purpose: Ingest data from various sources\n", - "# Components:\n", - "# - FileIngestor: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n", - "# - WebIngestor: Scrape and ingest web pages\n", - "# - FeedIngestor: Process RSS/Atom feeds\n", - "# - StreamIngestor: Real-time data streaming\n", - "# - DBIngestor: Database queries and ingestion\n", - "# - EmailIngestor: Process email messages\n", - "# - RepoIngestor: Git repository analysis\n", - "#\n", - "# Example:\n", - "# from semantica.ingest import FileIngestor, WebIngestor\n", - "# file_ingestor = FileIngestor()\n", - "# web_ingestor = WebIngestor()\n", - "# documents = file_ingestor.ingest(\"data/\")\n", - "# web_docs = web_ingestor.ingest(\"https://example.com\")\n", + "**Example**:\n", + "```python\n", + "from semantica.ingest import FileIngestor, WebIngestor\n", + "file_ingestor = FileIngestor()\n", + "web_ingestor = WebIngestor()\n", + "documents = file_ingestor.ingest(\"data/\")\n", + "web_docs = web_ingestor.ingest(\"https://example.com\")\n", + "```\n", "\n", - "# 2. PARSE MODULE - Document Parsing\n", - "# Purpose: Parse and extract content from various formats\n", - "# Components:\n", - "# - DocumentParser: Main parser orchestrator\n", - "# - PDFParser: Extract text, tables, images from PDFs\n", - "# - DOCXParser: Parse Word documents\n", - "# - HTMLParser: Extract content from HTML\n", - "# - JSONParser: Parse structured JSON data\n", - "# - ExcelParser: Process spreadsheets\n", - "# - ImageParser: OCR and image analysis\n", - "# - CodeParser: Parse source code files\n", - "#\n", - "# Example:\n", - "# from semantica.parse import DocumentParser\n", - "# parser = DocumentParser()\n", - "# parsed_docs = parser.parse(documents)\n", + "### 2. PARSE MODULE - Document Parsing\n", + "**Purpose**: Parse and extract content from various formats\n", + "**Components**:\n", + "- `DocumentParser`: Main parser orchestrator\n", + "- `PDFParser`: Extract text, tables, images from PDFs\n", + "- `DOCXParser`: Parse Word documents\n", + "- `HTMLParser`: Extract content from HTML\n", + "- `JSONParser`: Parse structured JSON data\n", + "- `ExcelParser`: Process spreadsheets\n", + "- `ImageParser`: OCR and image analysis\n", + "- `CodeParser`: Parse source code files\n", "\n", - "# 3. NORMALIZE MODULE - Text Normalization\n", - "# Purpose: Clean and normalize text for processing\n", - "# Components:\n", - "# - TextNormalizer: Main normalization orchestrator\n", - "# - TextCleaner: Remove noise, fix encoding\n", - "# - DataCleaner: Clean structured data\n", - "# - EntityNormalizer: Normalize entity names\n", - "# - DateNormalizer: Standardize date formats\n", - "# - NumberNormalizer: Normalize numeric values\n", - "# - LanguageDetector: Detect document language\n", - "# - EncodingHandler: Handle character encoding\n", - "#\n", - "# Example:\n", - "# from semantica.normalize import TextNormalizer\n", - "# normalizer = TextNormalizer()\n", - "# normalized = normalizer.normalize(parsed_docs)\n", + "**Example**:\n", + "```python\n", + "from semantica.parse import DocumentParser\n", + "parser = DocumentParser()\n", + "parsed_docs = parser.parse(documents)\n", + "```\n", "\n", - "# 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n", - "# Purpose: Extract entities, relationships, and semantic information\n", - "# Components:\n", - "# - NERExtractor: Named Entity Recognition\n", - "# - RelationExtractor: Extract relationships between entities\n", - "# - SemanticAnalyzer: Deep semantic analysis\n", - "# - SemanticNetworkExtractor: Extract semantic networks\n", - "#\n", - "# Example:\n", - "# from semantica.semantic_extract import NERExtractor, RelationExtractor\n", - "# extractor = NERExtractor()\n", - "# entities = extractor.extract(normalized_docs)\n", - "# relation_extractor = RelationExtractor()\n", - "# relationships = relation_extractor.extract(normalized_docs, entities)\n", + "### 3. NORMALIZE MODULE - Text Normalization\n", + "**Purpose**: Clean and normalize text for processing\n", + "**Components**:\n", + "- `TextNormalizer`: Main normalization orchestrator\n", + "- `TextCleaner`: Remove noise, fix encoding\n", + "- `DataCleaner`: Clean structured data\n", + "- `EntityNormalizer`: Normalize entity names\n", + "- `DateNormalizer`: Standardize date formats\n", + "- `NumberNormalizer`: Normalize numeric values\n", + "- `LanguageDetector`: Detect document language\n", + "- `EncodingHandler`: Handle character encoding\n", "\n", - "# 5. KG MODULE - Knowledge Graph Construction\n", - "# Purpose: Build and manage knowledge graphs\n", - "# Components:\n", - "# - GraphBuilder: Construct knowledge graphs from entities/relationships\n", - "# - GraphAnalyzer: Analyze graph structure and properties\n", - "# - GraphValidator: Validate graph quality and consistency\n", - "# - EntityResolver: Resolve entity conflicts and duplicates\n", - "# - ConflictDetector: Detect conflicting information\n", - "# - CentralityCalculator: Calculate node importance metrics\n", - "# - CommunityDetector: Detect communities in graphs\n", - "# - ConnectivityAnalyzer: Analyze graph connectivity\n", - "# - TemporalQuery: Query temporal knowledge graphs\n", - "# - Deduplicator: Remove duplicate entities/relationships\n", - "#\n", - "# Example:\n", - "# from semantica.kg import GraphBuilder, GraphAnalyzer\n", - "# builder = GraphBuilder()\n", - "# kg = builder.build(entities, relationships)\n", - "# analyzer = GraphAnalyzer()\n", - "# metrics = analyzer.analyze(kg)\n", + "**Example**:\n", + "```python\n", + "from semantica.normalize import TextNormalizer\n", + "normalizer = TextNormalizer()\n", + "normalized = normalizer.normalize(parsed_docs)\n", + "```\n", "\n", - "# 6. EMBEDDINGS MODULE - Embedding Generation\n", - "# Purpose: Generate vector embeddings for various data types\n", - "# Components:\n", - "# - EmbeddingGenerator: Main embedding orchestrator\n", - "# - TextEmbedder: Generate text embeddings\n", - "# - ImageEmbedder: Generate image embeddings\n", - "# - AudioEmbedder: Generate audio embeddings\n", - "# - MultimodalEmbedder: Combine multiple modalities\n", - "# - EmbeddingOptimizer: Optimize embedding quality\n", - "# - ProviderAdapters: Support for OpenAI, Cohere, etc.\n", - "#\n", - "# Example:\n", - "# from semantica.embeddings import EmbeddingGenerator\n", - "# generator = EmbeddingGenerator()\n", - "# embeddings = generator.generate(documents)\n", + "### 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n", + "**Purpose**: Extract entities, relationships, and semantic information\n", + "**Components**:\n", + "- `NERExtractor`: Named Entity Recognition\n", + "- `RelationExtractor`: Extract relationships between entities\n", + "- `SemanticAnalyzer`: Deep semantic analysis\n", + "- `SemanticNetworkExtractor`: Extract semantic networks\n", "\n", - "# 7. VECTOR_STORE MODULE - Vector Database Operations\n", - "# Purpose: Store and search vector embeddings\n", - "# Components:\n", - "# - VectorStore: Main vector store interface\n", - "# - FAISSAdapter: FAISS integration\n", - "# - HybridSearch: Combine vector and keyword search\n", - "# - VectorRetriever: Retrieve relevant vectors\n", - "#\n", - "# Example:\n", - "# from semantica.vector_store import VectorStore, HybridSearch\n", - "# vector_store = VectorStore()\n", - "# vector_store.store(embeddings, documents, metadata)\n", - "# hybrid_search = HybridSearch(vector_store)\n", - "# results = hybrid_search.search(query, top_k=10)\n", + "**Example**:\n", + "```python\n", + "from semantica.semantic_extract import NERExtractor, RelationExtractor\n", + "extractor = NERExtractor()\n", + "entities = extractor.extract(normalized_docs)\n", + "relation_extractor = RelationExtractor()\n", + "relationships = relation_extractor.extract(normalized_docs, entities)\n", + "```\n", "\n", - "# 8. REASONING MODULE - Inference and Reasoning\n", - "# Purpose: Perform logical inference and reasoning\n", - "# Components:\n", - "# - InferenceEngine: Main inference orchestrator\n", - "# - RuleManager: Manage inference rules\n", - "# - DeductiveReasoner: Deductive reasoning\n", - "# - AbductiveReasoner: Abductive reasoning\n", - "# - ExplanationGenerator: Generate explanations for inferences\n", - "# - RETEEngine: RETE algorithm for rule matching\n", - "#\n", - "# Example:\n", - "# from semantica.reasoning import InferenceEngine, RuleManager\n", - "# inference_engine = InferenceEngine()\n", - "# rule_manager = RuleManager()\n", - "# new_facts = inference_engine.forward_chain(kg, rule_manager)\n", + "### 5. KG MODULE - Knowledge Graph Construction\n", + "**Purpose**: Build and manage knowledge graphs\n", + "**Components**:\n", + "- `GraphBuilder`: Construct knowledge graphs from entities/relationships\n", + "- `GraphAnalyzer`: Analyze graph structure and properties\n", + "- `GraphValidator`: Validate graph quality and consistency\n", + "- `EntityResolver`: Resolve entity conflicts and duplicates\n", + "- `ConflictDetector`: Detect conflicting information\n", + "- `CentralityCalculator`: Calculate node importance metrics\n", + "- `CommunityDetector`: Detect communities in graphs\n", + "- `ConnectivityAnalyzer`: Analyze graph connectivity\n", + "- `TemporalQuery`: Query temporal knowledge graphs\n", + "- `Deduplicator`: Remove duplicate entities/relationships\n", "\n", - "# 9. ONTOLOGY MODULE - Ontology Generation\n", - "# Purpose: Generate and manage ontologies\n", - "# Components:\n", - "# - OntologyGenerator: Generate ontologies from knowledge graphs\n", - "# - OntologyValidator: Validate ontology structure\n", - "# - OWLGenerator: Generate OWL format ontologies\n", - "# - PropertyGenerator: Generate ontology properties\n", - "# - ClassInferrer: Infer ontology classes\n", - "#\n", - "# Example:\n", - "# from semantica.ontology import OntologyGenerator\n", - "# generator = OntologyGenerator()\n", - "# ontology = generator.generate_from_graph(kg)\n", + "**Example**:\n", + "```python\n", + "from semantica.kg import GraphBuilder, GraphAnalyzer\n", + "builder = GraphBuilder()\n", + "kg = builder.build(entities, relationships)\n", + "analyzer = GraphAnalyzer()\n", + "metrics = analyzer.analyze(kg)\n", + "```\n", "\n", - "# 10. EXPORT MODULE - Data Export\n", - "# Purpose: Export data in various formats\n", - "# Components:\n", - "# - JSONExporter: Export to JSON\n", - "# - RDFExporter: Export to RDF/XML\n", - "# - CSVExporter: Export to CSV\n", - "# - GraphExporter: Export to graph formats (GraphML, GEXF)\n", - "# - OWLExporter: Export to OWL\n", - "# - VectorExporter: Export vectors\n", - "#\n", - "# Example:\n", - "# from semantica.export import JSONExporter, RDFExporter\n", - "# json_exporter = JSONExporter()\n", - "# json_exporter.export(kg, \"output.json\")\n", + "### 6. EMBEDDINGS MODULE - Embedding Generation\n", + "**Purpose**: Generate vector embeddings for various data types\n", + "**Components**:\n", + "- `EmbeddingGenerator`: Main embedding orchestrator\n", + "- `TextEmbedder`: Generate text embeddings\n", + "- `ImageEmbedder`: Generate image embeddings\n", + "- `AudioEmbedder`: Generate audio embeddings\n", + "- `MultimodalEmbedder`: Combine multiple modalities\n", + "- `EmbeddingOptimizer`: Optimize embedding quality\n", + "- `ProviderAdapters`: Support for OpenAI, Cohere, etc.\n", "\n", - "# 11. VISUALIZATION MODULE - Graph Visualization\n", - "# Purpose: Visualize knowledge graphs and analytics\n", - "# Components:\n", - "# - KGVisualizer: Visualize knowledge graphs\n", - "# - EmbeddingVisualizer: Visualize embeddings (t-SNE, PCA, UMAP)\n", - "# - QualityVisualizer: Visualize quality metrics\n", - "# - AnalyticsVisualizer: Visualize graph analytics\n", - "# - TemporalVisualizer: Visualize temporal data\n", - "#\n", - "# Example:\n", - "# from semantica.visualization import KGVisualizer\n", - "# visualizer = KGVisualizer()\n", - "# visualizer.visualize(kg)\n", + "**Example**:\n", + "```python\n", + "from semantica.embeddings import EmbeddingGenerator\n", + "generator = EmbeddingGenerator()\n", + "embeddings = generator.generate(documents)\n", + "```\n", "\n", - "# 12. PIPELINE MODULE - Pipeline Orchestration\n", - "# Purpose: Build and execute processing pipelines\n", - "# Components:\n", - "# - PipelineBuilder: Build complex pipelines\n", - "# - ExecutionEngine: Execute pipelines\n", - "# - FailureHandler: Handle pipeline failures\n", - "# - ParallelismManager: Enable parallel processing\n", - "# - ResourceScheduler: Schedule resources\n", - "#\n", - "# Example:\n", - "# from semantica.pipeline import PipelineBuilder\n", - "# builder = PipelineBuilder()\n", - "# pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n", - "# .add_step(\"parse\", DocumentParser()) \\\\\n", - "# .build()\n", - "'''\n", + "### 7. VECTOR_STORE MODULE - Vector Database Operations\n", + "**Purpose**: Store and search vector embeddings\n", + "**Components**:\n", + "- `VectorStore`: Main vector store interface\n", + "- `FAISSAdapter`: FAISS integration\n", + "- `HybridSearch`: Combine vector and keyword search\n", + "- `VectorRetriever`: Retrieve relevant vectors\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.vector_store import VectorStore, HybridSearch\n", + "vector_store = VectorStore()\n", + "vector_store.store(embeddings, documents, metadata)\n", + "hybrid_search = HybridSearch(vector_store)\n", + "results = hybrid_search.search(query, top_k=10)\n", + "```\n", + "\n", + "### 8. REASONING MODULE - Inference and Reasoning\n", + "**Purpose**: Perform logical inference and reasoning\n", + "**Components**:\n", + "- `InferenceEngine`: Main inference orchestrator\n", + "- `RuleManager`: Manage inference rules\n", + "- `DeductiveReasoner`: Deductive reasoning\n", + "- `AbductiveReasoner`: Abductive reasoning\n", + "- `ExplanationGenerator`: Generate explanations for inferences\n", + "- `RETEEngine`: RETE algorithm for rule matching\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.reasoning import InferenceEngine, RuleManager\n", + "inference_engine = InferenceEngine()\n", + "rule_manager = RuleManager()\n", + "new_facts = inference_engine.forward_chain(kg, rule_manager)\n", + "```\n", + "\n", + "### 9. ONTOLOGY MODULE - Ontology Generation\n", + "**Purpose**: Generate and manage ontologies\n", + "**Components**:\n", + "- `OntologyGenerator`: Generate ontologies from knowledge graphs\n", + "- `OntologyValidator`: Validate ontology structure\n", + "- `OWLGenerator`: Generate OWL format ontologies\n", + "- `PropertyGenerator`: Generate ontology properties\n", + "- `ClassInferrer`: Infer ontology classes\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.ontology import OntologyGenerator\n", + "generator = OntologyGenerator()\n", + "ontology = generator.generate_from_graph(kg)\n", + "```\n", + "\n", + "### 10. EXPORT MODULE - Data Export\n", + "**Purpose**: Export data in various formats\n", + "**Components**:\n", + "- `JSONExporter`: Export to JSON\n", + "- `RDFExporter`: Export to RDF/XML\n", + "- `CSVExporter`: Export to CSV\n", + "- `GraphExporter`: Export to graph formats (GraphML, GEXF)\n", + "- `OWLExporter`: Export to OWL\n", + "- `VectorExporter`: Export vectors\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.export import JSONExporter, RDFExporter\n", + "json_exporter = JSONExporter()\n", + "json_exporter.export(kg, \"output.json\")\n", + "```\n", + "\n", + "### 11. VISUALIZATION MODULE - Graph Visualization\n", + "**Purpose**: Visualize knowledge graphs and analytics\n", + "**Components**:\n", + "- `KGVisualizer`: Visualize knowledge graphs\n", + "- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP)\n", + "- `QualityVisualizer`: Visualize quality metrics\n", + "- `AnalyticsVisualizer`: Visualize graph analytics\n", + "- `TemporalVisualizer`: Visualize temporal data\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.visualization import KGVisualizer\n", + "visualizer = KGVisualizer()\n", + "visualizer.visualize(kg)\n", + "```\n", + "\n", + "### 12. PIPELINE MODULE - Pipeline Orchestration\n", + "**Purpose**: Build and execute processing pipelines\n", + "**Components**:\n", + "- `PipelineBuilder`: Build complex pipelines\n", + "- `ExecutionEngine`: Execute pipelines\n", + "- `FailureHandler`: Handle pipeline failures\n", + "- `ParallelismManager`: Enable parallel processing\n", + "- `ResourceScheduler`: Schedule resources\n", + "\n", + "**Example**:\n", + "```python\n", + "from semantica.pipeline import PipelineBuilder\n", + "builder = PipelineBuilder()\n", + "pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n", + " .add_step(\"parse\", DocumentParser()) \\\\\n", + " .build()\n", + "```\n", "\n", "---\n", "\n", @@ -306,183 +324,166 @@ "\n", "Understanding these concepts is crucial for working with Semantica:\n", "\n", - "'''\n", - "# ============================================================================\n", - "# CORE CONCEPTS\n", - "# ============================================================================\n", + "### 1. KNOWLEDGE GRAPHS\n", + "**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes.\n", "\n", - "# 1. KNOWLEDGE GRAPHS\n", - "# Definition: A knowledge graph is a structured representation of entities\n", - "# (nodes) and their relationships (edges) with properties and\n", - "# attributes.\n", - "#\n", - "# Structure:\n", - "# - Nodes: Represent entities (people, places, concepts, events)\n", - "# - Edges: Represent relationships (works_for, located_in, causes)\n", - "# - Properties: Attributes of entities and relationships\n", - "# - Metadata: Additional information (sources, timestamps, confidence)\n", - "#\n", - "# Example:\n", - "# Entity: \"John Doe\" (Person)\n", - "# Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n", - "# Properties: {start_date: \"2020-01-01\", role: \"Engineer\"}\n", - "#\n", - "# Benefits:\n", - "# - Structured representation of unstructured data\n", - "# - Enables complex queries and reasoning\n", - "# - Supports temporal tracking\n", - "# - Facilitates knowledge discovery\n", + "**Structure**:\n", + "- **Nodes**: Represent entities (people, places, concepts, events)\n", + "- **Edges**: Represent relationships (works_for, located_in, causes)\n", + "- **Properties**: Attributes of entities and relationships\n", + "- **Metadata**: Additional information (sources, timestamps, confidence)\n", "\n", - "# 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n", - "# Definition: The process of identifying and classifying named entities\n", - "# in text into predefined categories.\n", - "#\n", - "# Entity Types:\n", - "# - Person: Names of people\n", - "# - Organization: Companies, institutions\n", - "# - Location: Places, geographic entities\n", - "# - Date/Time: Temporal expressions\n", - "# - Money: Monetary values\n", - "# - Product: Products and services\n", - "# - Event: Events and occurrences\n", - "# - Custom: Domain-specific entities\n", - "#\n", - "# Example:\n", - "# Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n", - "# Entities:\n", - "# - \"Apple Inc.\" -> Organization\n", - "# - \"Steve Jobs\" -> Person\n", - "# - \"Cupertino, California\" -> Location\n", - "#\n", - "# Methods:\n", - "# - Rule-based: Pattern matching\n", - "# - Machine Learning: Trained models (spaCy, transformers)\n", - "# - LLM-based: Using large language models\n", + "**Example**:\n", + "- Entity: \"John Doe\" (Person)\n", + "- Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n", + "- Properties: `{start_date: \"2020-01-01\", role: \"Engineer\"}`\n", "\n", - "# 3. RELATIONSHIP EXTRACTION\n", - "# Definition: Identifying and extracting relationships between entities\n", - "# in text.\n", - "#\n", - "# Relationship Types:\n", - "# - Semantic: \"works_for\", \"located_in\", \"causes\"\n", - "# - Temporal: \"before\", \"after\", \"during\"\n", - "# - Causal: \"causes\", \"results_in\", \"prevents\"\n", - "# - Hierarchical: \"part_of\", \"subclass_of\", \"instance_of\"\n", - "#\n", - "# Example:\n", - "# Text: \"John works for Acme Corp in New York.\"\n", - "# Relationships:\n", - "# - (John, works_for, Acme Corp)\n", - "# - (Acme Corp, located_in, New York)\n", - "#\n", - "# Methods:\n", - "# - Pattern matching\n", - "# - Dependency parsing\n", - "# - Machine learning models\n", - "# - LLM-based extraction\n", + "**Benefits**:\n", + "- Structured representation of unstructured data\n", + "- Enables complex queries and reasoning\n", + "- Supports temporal tracking\n", + "- Facilitates knowledge discovery\n", "\n", - "# 4. EMBEDDINGS\n", - "# Definition: Dense vector representations of text, images, or other data\n", - "# that capture semantic meaning in a continuous vector space.\n", - "#\n", - "# Properties:\n", - "# - Similar entities have similar embeddings (close in vector space)\n", - "# - Enable semantic search and similarity calculations\n", - "# - Fixed or variable dimensions (typically 128-4096)\n", - "#\n", - "# Example:\n", - "# Text: \"machine learning\"\n", - "# Embedding: [0.123, -0.456, 0.789, ..., 0.234] (vector of 1536 dimensions)\n", - "#\n", - "# Use Cases:\n", - "# - Semantic search\n", - "# - Clustering and classification\n", - "# - Recommendation systems\n", - "# - Anomaly detection\n", + "### 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n", + "**Definition**: The process of identifying and classifying named entities in text into predefined categories.\n", "\n", - "# 5. TEMPORAL GRAPHS\n", - "# Definition: Knowledge graphs that track changes over time, allowing\n", - "# queries about the state of the graph at specific time points.\n", - "#\n", - "# Features:\n", - "# - Timestamps on entities and relationships\n", - "# - Version history\n", - "# - Time-point queries\n", - "# - Temporal pattern detection\n", - "#\n", - "# Example:\n", - "# Entity: \"Company X\"\n", - "# Relationship: (Company X, has_CEO, Person Y)\n", - "# Temporal: valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"\n", - "#\n", - "# Use Cases:\n", - "# - Tracking organizational changes\n", - "# - Monitoring system evolution\n", - "# - Analyzing trends over time\n", - "# - Historical analysis\n", + "**Entity Types**:\n", + "- **Person**: Names of people\n", + "- **Organization**: Companies, institutions\n", + "- **Location**: Places, geographic entities\n", + "- **Date/Time**: Temporal expressions\n", + "- **Money**: Monetary values\n", + "- **Product**: Products and services\n", + "- **Event**: Events and occurrences\n", + "- **Custom**: Domain-specific entities\n", "\n", - "# 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n", - "# Definition: An advanced RAG approach that combines vector search with\n", - "# knowledge graph traversal to provide more accurate and\n", - "# contextually relevant information to LLMs.\n", - "#\n", - "# Components:\n", - "# - Vector Store: For semantic similarity search\n", - "# - Knowledge Graph: For structured relationship traversal\n", - "# - Hybrid Search: Combines both approaches\n", - "# - LLM Integration: Uses retrieved context for generation\n", - "#\n", - "# Advantages over Traditional RAG:\n", - "# - Better handling of complex queries\n", - "# - Relationship-aware retrieval\n", - "# - Reduced hallucinations\n", - "# - More accurate answers\n", - "#\n", - "# Example Workflow:\n", - "# 1. Query: \"Who worked with John at Acme Corp?\"\n", - "# 2. Vector search finds relevant documents\n", - "# 3. Knowledge graph traversal finds relationships\n", - "# 4. Combined context sent to LLM\n", - "# 5. LLM generates accurate answer using both sources\n", + "**Example**:\n", + "Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n", + "Entities:\n", + "- \"Apple Inc.\" -> Organization\n", + "- \"Steve Jobs\" -> Person\n", + "- \"Cupertino, California\" -> Location\n", "\n", - "# 7. ONTOLOGY\n", - "# Definition: A formal specification of concepts, relationships, and\n", - "# constraints in a domain, typically expressed in OWL (Web\n", - "# Ontology Language).\n", - "#\n", - "# Components:\n", - "# - Classes: Categories of entities\n", - "# - Properties: Relationships and attributes\n", - "# - Individuals: Specific instances\n", - "# - Axioms: Rules and constraints\n", - "#\n", - "# Example:\n", - "# Class: Person\n", - "# SubClass: Employee, Customer\n", - "# Property: worksFor (domain: Person, range: Organization)\n", - "#\n", - "# Use Cases:\n", - "# - Standardize domain knowledge\n", - "# - Enable reasoning\n", - "# - Facilitate data integration\n", - "# - Support semantic web\n", + "**Methods**:\n", + "- Rule-based: Pattern matching\n", + "- Machine Learning: Trained models (spaCy, transformers)\n", + "- LLM-based: Using large language models\n", "\n", - "# 8. QUALITY ASSURANCE\n", - "# Definition: Processes and metrics to ensure knowledge graph quality,\n", - "# including completeness, consistency, and accuracy.\n", - "#\n", - "# Metrics:\n", - "# - Completeness: Percentage of entities with required properties\n", - "# - Consistency: Absence of contradictions\n", - "# - Accuracy: Correctness of extracted information\n", - "# - Coverage: Breadth of domain coverage\n", - "#\n", - "# Methods:\n", - "# - Validation rules\n", - "# - Automated quality checks\n", - "# - Conflict detection\n", - "# - Source verification\n", - "'''\n", + "### 3. RELATIONSHIP EXTRACTION\n", + "**Definition**: Identifying and extracting relationships between entities in text.\n", + "\n", + "**Relationship Types**:\n", + "- **Semantic**: \"works_for\", \"located_in\", \"causes\"\n", + "- **Temporal**: \"before\", \"after\", \"during\"\n", + "- **Causal**: \"causes\", \"results_in\", \"prevents\"\n", + "- **Hierarchical**: \"part_of\", \"subclass_of\", \"instance_of\"\n", + "\n", + "**Example**:\n", + "Text: \"John works for Acme Corp in New York.\"\n", + "Relationships:\n", + "- (John, works_for, Acme Corp)\n", + "- (Acme Corp, located_in, New York)\n", + "\n", + "**Methods**:\n", + "- Pattern matching\n", + "- Dependency parsing\n", + "- Machine learning models\n", + "- LLM-based extraction\n", + "\n", + "### 4. EMBEDDINGS\n", + "**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space.\n", + "\n", + "**Properties**:\n", + "- Similar entities have similar embeddings (close in vector space)\n", + "- Enable semantic search and similarity calculations\n", + "- Fixed or variable dimensions (typically 128-4096)\n", + "\n", + "**Example**:\n", + "Text: \"machine learning\"\n", + "Embedding: `[0.123, -0.456, 0.789, ..., 0.234]` (vector of 1536 dimensions)\n", + "\n", + "**Use Cases**:\n", + "- Semantic search\n", + "- Clustering and classification\n", + "- Recommendation systems\n", + "- Anomaly detection\n", + "\n", + "### 5. TEMPORAL GRAPHS\n", + "**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points.\n", + "\n", + "**Features**:\n", + "- Timestamps on entities and relationships\n", + "- Version history\n", + "- Time-point queries\n", + "- Temporal pattern detection\n", + "\n", + "**Example**:\n", + "- Entity: \"Company X\"\n", + "- Relationship: (Company X, has_CEO, Person Y)\n", + "- Temporal: `valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"`\n", + "\n", + "**Use Cases**:\n", + "- Tracking organizational changes\n", + "- Monitoring system evolution\n", + "- Analyzing trends over time\n", + "- Historical analysis\n", + "\n", + "### 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n", + "**Definition**: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs.\n", + "\n", + "**Components**:\n", + "- **Vector Store**: For semantic similarity search\n", + "- **Knowledge Graph**: For structured relationship traversal\n", + "- **Hybrid Search**: Combines both approaches\n", + "- **LLM Integration**: Uses retrieved context for generation\n", + "\n", + "**Advantages over Traditional RAG**:\n", + "- Better handling of complex queries\n", + "- Relationship-aware retrieval\n", + "- Reduced hallucinations\n", + "- More accurate answers\n", + "\n", + "**Example Workflow**:\n", + "1. Query: \"Who worked with John at Acme Corp?\"\n", + "2. Vector search finds relevant documents\n", + "3. Knowledge graph traversal finds relationships\n", + "4. Combined context sent to LLM\n", + "5. LLM generates accurate answer using both sources\n", + "\n", + "### 7. ONTOLOGY\n", + "**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language).\n", + "\n", + "**Components**:\n", + "- **Classes**: Categories of entities\n", + "- **Properties**: Relationships and attributes\n", + "- **Individuals**: Specific instances\n", + "- **Axioms**: Rules and constraints\n", + "\n", + "**Example**:\n", + "- Class: Person\n", + "- SubClass: Employee, Customer\n", + "- Property: worksFor (domain: Person, range: Organization)\n", + "\n", + "**Use Cases**:\n", + "- Standardize domain knowledge\n", + "- Enable reasoning\n", + "- Facilitate data integration\n", + "- Support semantic web\n", + "\n", + "### 8. QUALITY ASSURANCE\n", + "**Definition**: Processes and metrics to ensure knowledge graph quality, including completeness, consistency, and accuracy.\n", + "\n", + "**Metrics**:\n", + "- **Completeness**: Percentage of entities with required properties\n", + "- **Consistency**: Absence of contradictions\n", + "- **Accuracy**: Correctness of extracted information\n", + "- **Coverage**: Breadth of domain coverage\n", + "\n", + "**Methods**:\n", + "- Validation rules\n", + "- Automated quality checks\n", + "- Conflict detection\n", + "- Source verification\n", "\n", "---\n", "\n", @@ -510,41 +511,35 @@ "\n", "## Best Practices\n", "\n", - "'''\n", - "# ============================================================================\n", - "# BEST PRACTICES\n", - "# ============================================================================\n", + "### 1. START SMALL\n", + "- Begin with simple documents\n", + "- Validate each step before moving forward\n", + "- Build incrementally\n", "\n", - "# 1. START SMALL\n", - "# - Begin with simple documents\n", - "# - Validate each step before moving forward\n", - "# - Build incrementally\n", + "### 2. CONFIGURE PROPERLY\n", + "- Use environment variables for sensitive data\n", + "- Set up proper logging\n", + "- Configure appropriate model sizes\n", "\n", - "# 2. CONFIGURE PROPERLY\n", - "# - Use environment variables for sensitive data\n", - "# - Set up proper logging\n", - "# - Configure appropriate model sizes\n", + "### 3. VALIDATE DATA\n", + "- Always validate extracted entities\n", + "- Check relationship quality\n", + "- Use quality assurance tools\n", "\n", - "# 3. VALIDATE DATA\n", - "# - Always validate extracted entities\n", - "# - Check relationship quality\n", - "# - Use quality assurance tools\n", + "### 4. HANDLE ERRORS\n", + "- Implement error handling\n", + "- Use retry mechanisms\n", + "- Log errors for debugging\n", "\n", - "# 4. HANDLE ERRORS\n", - "# - Implement error handling\n", - "# - Use retry mechanisms\n", - "# - Log errors for debugging\n", + "### 5. OPTIMIZE PERFORMANCE\n", + "- Use batch processing for large datasets\n", + "- Enable parallel processing where possible\n", + "- Cache embeddings and results\n", "\n", - "# 5. OPTIMIZE PERFORMANCE\n", - "# - Use batch processing for large datasets\n", - "# - Enable parallel processing where possible\n", - "# - Cache embeddings and results\n", - "\n", - "# 6. DOCUMENT YOUR WORKFLOWS\n", - "# - Document data sources\n", - "# - Track processing steps\n", - "# - Maintain metadata\n", - "'''\n", + "### 6. DOCUMENT YOUR WORKFLOWS\n", + "- Document data sources\n", + "- Track processing steps\n", + "- Maintain metadata\n", "\n", "---\n", "\n", @@ -552,45 +547,39 @@ "\n", "Common issues and solutions:\n", "\n", - "'''\n", - "# ============================================================================\n", - "# TROUBLESHOOTING\n", - "# ============================================================================\n", + "### Issue 1: Import Errors\n", + "**Solution**:\n", + "- Ensure Semantica is properly installed\n", + "- Check Python version (3.8+)\n", + "- Verify virtual environment is activated\n", + "- Install missing dependencies: `pip install -r requirements.txt`\n", "\n", - "# Issue 1: Import Errors\n", - "# Solution:\n", - "# - Ensure Semantica is properly installed\n", - "# - Check Python version (3.8+)\n", - "# - Verify virtual environment is activated\n", - "# - Install missing dependencies: pip install -r requirements.txt\n", + "### Issue 2: API Key Errors\n", + "**Solution**:\n", + "- Set environment variables: `export SEMANTICA_API_KEY=your_key`\n", + "- Check config file for correct key format\n", + "- Verify API key is valid and has sufficient credits\n", "\n", - "# Issue 2: API Key Errors\n", - "# Solution:\n", - "# - Set environment variables: export SEMANTICA_API_KEY=your_key\n", - "# - Check config file for correct key format\n", - "# - Verify API key is valid and has sufficient credits\n", + "### Issue 3: Memory Issues\n", + "**Solution**:\n", + "- Process documents in batches\n", + "- Use smaller embedding models\n", + "- Enable garbage collection\n", + "- Consider using streaming for large datasets\n", "\n", - "# Issue 3: Memory Issues\n", - "# Solution:\n", - "# - Process documents in batches\n", - "# - Use smaller embedding models\n", - "# - Enable garbage collection\n", - "# - Consider using streaming for large datasets\n", + "### Issue 4: Low Quality Extractions\n", + "**Solution**:\n", + "- Preprocess and normalize text\n", + "- Use domain-specific models\n", + "- Adjust extraction parameters\n", + "- Validate and clean extracted entities\n", "\n", - "# Issue 4: Low Quality Extractions\n", - "# Solution:\n", - "# - Preprocess and normalize text\n", - "# - Use domain-specific models\n", - "# - Adjust extraction parameters\n", - "# - Validate and clean extracted entities\n", - "\n", - "# Issue 5: Slow Processing\n", - "# Solution:\n", - "# - Enable parallel processing\n", - "# - Use GPU acceleration if available\n", - "# - Cache intermediate results\n", - "# - Optimize batch sizes\n", - "'''\n" + "### Issue 5: Slow Processing\n", + "**Solution**:\n", + "- Enable parallel processing\n", + "- Use GPU acceleration if available\n", + "- Cache intermediate results\n", + "- Optimize batch sizes\n" ] }, { @@ -859,38 +848,6 @@ "print(\" generator = EmbeddingGenerator()\")\n", "print(\" embeddings = generator.generate(documents)\")\n" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\n" - ] } ], "metadata": { @@ -900,4 +857,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/docs/cookbook/introduction/Your_First_Knowledge_Graph.ipynb b/docs/cookbook/introduction/Your_First_Knowledge_Graph.ipynb index 5d79366a..61e23295 100644 --- a/docs/cookbook/introduction/Your_First_Knowledge_Graph.ipynb +++ b/docs/cookbook/introduction/Your_First_Knowledge_Graph.ipynb @@ -4,37 +4,40 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Your First Knowledge Graph\n", + "# πŸš€ Your First Knowledge Graph\n", "\n", "## Overview\n", "\n", "This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n", "\n", - "### Learning Objectives\n", + "> [!TIP]\n", + "> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n", "\n", - "- Understand the basic workflow: **File β†’ Parse β†’ Extract β†’ Graph**\n", - "- Learn how to ingest documents using `FileIngestor`\n", - "- Parse documents using `DocumentParser`\n", - "- Extract entities using NER extractors\n", - "- Build a knowledge graph using `GraphBuilder`\n", - "- Visualize and analyze the resulting graph\n", + "### 🎯 Learning Objectives\n", + "\n", + "- **Understand the Workflow**: Learn the `File β†’ Parse β†’ Extract β†’ Graph` pipeline\n", + "- **Ingest Data**: Load documents using `FileIngestor`\n", + "- **Parse Content**: Extract text using `DocumentParser`\n", + "- **Extract Knowledge**: Identify entities using `NERExtractor`\n", + "- **Build Graph**: Construct a graph using `GraphBuilder`\n", + "- **Visualize**: See your graph come to life with `KGVisualizer`\n", "\n", "---\n", "\n", - "## Simple End-to-End Workflow\n", + "## πŸ”„ Simple End-to-End Workflow\n", "\n", "The complete workflow consists of four main steps:\n", "\n", - "1. **Ingest** - Load data from files or other sources\n", - "2. **Parse** - Extract and structure content from documents\n", - "3. **Extract** - Identify entities and relationships\n", - "4. **Build Graph** - Construct the knowledge graph\n", + "1. **πŸ“₯ Ingest** - Load data from files or other sources\n", + "2. **πŸ“„ Parse** - Extract and structure content from documents\n", + "3. **⛏️ Extract** - Identify entities and relationships\n", + "4. **πŸ•ΈοΈ Build Graph** - Construct the knowledge graph\n", "\n", "Each step is demonstrated in the code cells below.\n", "\n", "---\n", "\n", - "## Step 1: Ingest a File\n", + "## πŸ“‚ Step 1: Ingest a File\n", "\n", "In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n" ] @@ -48,8 +51,10 @@ "from semantica.ingest import FileIngestor\n", "from pathlib import Path\n", "\n", + "# Initialize the ingestor\n", "ingestor = FileIngestor()\n", "\n", + "# Create a sample document for demonstration\n", "sample_text = \"\"\"\n", "Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n", "The company is headquartered in Cupertino, California.\n", @@ -64,6 +69,7 @@ "print(f\"File: {sample_file}\")\n", "print(f\"Content length: {len(sample_text)} characters\")\n", "\n", + "# Ingest the file\n", "try:\n", " file_object = ingestor.ingest_file(sample_file, read_content=True)\n", " print(f\"\\nβœ“ File ingested successfully!\")\n", @@ -78,7 +84,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 2: Parse the Document\n", + "## πŸ“„ Step 2: Parse the Document\n", "\n", "After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n" ] @@ -94,12 +100,14 @@ "parser = DocumentParser()\n", "\n", "try:\n", + " # Parse the document to extract text\n", " if 'file_object' in locals():\n", " parsed_content = parser.parse_document(str(sample_file))\n", " print(\"βœ“ Document parsed successfully!\")\n", " print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n", " print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n", " else:\n", + " # Fallback if ingestion failed\n", " parsed_content = parser.parse_document(str(sample_file))\n", " print(\"βœ“ Document parsed successfully!\")\n", " print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n", @@ -113,9 +121,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 3: Extract Entities\n", + "## ⛏️ Step 3: Extract Entities\n", "\n", - "Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n" + "Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n", + "\n", + "> [!NOTE]\n", + "> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n" ] }, { @@ -133,6 +144,7 @@ " print(\"Extracting entities from text...\")\n", " print(f\"\\nText: {parsed_content[:100]}...\")\n", " \n", + " # Simulated extraction results\n", " expected_entities = [\n", " {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n", " {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n", @@ -156,7 +168,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 4: Build the Knowledge Graph\n", + "## πŸ•ΈοΈ Step 4: Build the Knowledge Graph\n", "\n", "Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n" ] @@ -172,6 +184,7 @@ "\n", "builder = GraphBuilder()\n", "\n", + "# Prepare data for graph construction\n", "entities_data = [\n", " {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n", " for i, entity in enumerate(expected_entities)\n", @@ -187,6 +200,7 @@ "]\n", "\n", "try:\n", + " # Build the graph using NetworkX\n", " kg = nx.DiGraph()\n", " \n", " for entity in entities_data:\n", @@ -221,7 +235,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 5: Visualize and Analyze\n", + "## πŸ“Š Step 5: Visualize and Analyze\n", "\n", "Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n" ] @@ -268,6 +282,7 @@ "except Exception as e:\n", " print(f\"βœ— Error visualizing graph: {e}\")\n", "\n", + "# Cleanup\n", "try:\n", " if sample_file.exists():\n", " sample_file.unlink()\n", @@ -284,4 +299,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/docs/cookbook/use_cases/finance/Financial_Data_Integration.ipynb b/docs/cookbook/use_cases/finance/Financial_Data_Integration.ipynb index d3836cce..7300151c 100644 --- a/docs/cookbook/use_cases/finance/Financial_Data_Integration.ipynb +++ b/docs/cookbook/use_cases/finance/Financial_Data_Integration.ipynb @@ -4,32 +4,33 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Financial Data Integration Pipeline\n", + "# πŸ“ˆ Financial Data Integration Pipeline\n", "\n", "## Overview\n", "\n", "This notebook demonstrates how to integrate Python/FastMCP MCP servers as data sources for financial data ingestion. Connect to financial data MCP servers via URL, ingest market data, stock prices, and financial metrics, then build a knowledge graph for financial analysis.\n", "\n", - "**IMPORTANT**: This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n", + "> [!IMPORTANT]\n", + "> This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n", "\n", - "### Modules Used (20+)\n", + "### 🧩 Modules Used (20+)\n", "\n", - "- **Ingestion**: MCPIngestor, ingest_mcp, WebIngestor, FileIngestor\n", - "- **Parsing**: MCPParser, JSONParser, StructuredDataParser\n", - "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n", - "- **KG**: GraphBuilder, TemporalGraphQuery, GraphAnalyzer\n", - "- **Analytics**: CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n", - "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n", - "- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n", - "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n", + "- **Ingestion**: `MCPIngestor`, `ingest_mcp`, `WebIngestor`, `FileIngestor`\n", + "- **Parsing**: `MCPParser`, `JSONParser`, `StructuredDataParser`\n", + "- **Extraction**: `NERExtractor`, `RelationExtractor`, `EventDetector`, `SemanticAnalyzer`\n", + "- **KG**: `GraphBuilder`, `TemporalGraphQuery`, `GraphAnalyzer`\n", + "- **Analytics**: `CentralityCalculator`, `CommunityDetector`, `ConnectivityAnalyzer`\n", + "- **Reasoning**: `InferenceEngine`, `RuleManager`, `ExplanationGenerator`\n", + "- **Export**: `JSONExporter`, `CSVExporter`, `RDFExporter`, `ReportGenerator`\n", + "- **Visualization**: `KGVisualizer`, `TemporalVisualizer`, `AnalyticsVisualizer`\n", "\n", - "### Pipeline\n", + "### πŸ”„ Pipeline\n", "\n", "**Connect to Financial MCP Server β†’ Ingest Market Data via MCP β†’ Parse MCP Responses β†’ Extract Financial Entities β†’ Build Financial KG β†’ Analyze Trends β†’ Generate Reports β†’ Visualize**\n", "\n", "---\n", "\n", - "## Step 1: Connect to Financial Data MCP Server\n", + "## πŸ”Œ Step 1: Connect to Financial Data MCP Server\n", "\n", "Connect to a Python/FastMCP MCP server that provides financial data via URL. The MCP server can expose resources (datasets, market data) and tools (queries, calculations).\n" ] @@ -92,7 +93,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 2: Ingest Financial Data from MCP Server\n", + "## πŸ“₯ Step 2: Ingest Financial Data from MCP Server\n", "\n", "Ingest financial data using both resource-based and tool-based methods from the MCP server.\n" ] @@ -220,7 +221,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 3: Parse MCP Data\n", + "## πŸ“„ Step 3: Parse MCP Data\n", "\n", "Parse the data received from MCP server responses (JSON, structured data).\n" ] @@ -266,7 +267,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 4: Extract Financial Entities and Relationships\n", + "## ⛏️ Step 4: Extract Financial Entities and Relationships\n", "\n", "Extract financial entities (companies, stocks, sectors) and relationships from MCP data.\n" ] @@ -363,7 +364,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 5: Build Financial Knowledge Graph\n", + "## πŸ•ΈοΈ Step 5: Build Financial Knowledge Graph\n", "\n", "Build a knowledge graph from the extracted financial entities and relationships.\n" ] @@ -404,7 +405,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 6: Analyze Financial Trends\n", + "## πŸ“Š Step 6: Analyze Financial Trends\n", "\n", "Analyze financial trends using temporal queries and pattern detection.\n" ] @@ -462,7 +463,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 7: Export and Visualize\n", + "## πŸ“€ Step 7: Export and Visualize\n", "\n", "Export the financial knowledge graph and generate visualizations.\n" ] @@ -535,4 +536,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/docs/css/custom.css b/docs/css/custom.css index 709642c5..faedbde9 100644 --- a/docs/css/custom.css +++ b/docs/css/custom.css @@ -150,4 +150,105 @@ html { [data-md-color-scheme="slate"] ::-webkit-scrollbar-thumb { background-color: rgba(255, 255, 255, 0.2); +} + +/* + ========================================================================== + Version Selector + ========================================================================== +*/ +.version-scroll-container { + display: flex; + align-items: center; + margin-left: 1.5rem; + /* Increased spacing */ + overflow-x: auto; + white-space: nowrap; + max-width: 300px; + padding: 4px 0; + scrollbar-width: none; + -ms-overflow-style: none; + height: 100%; + /* Match header height context */ +} + +.version-scroll-container::-webkit-scrollbar { + display: none; +} + +.version-list { + display: flex; + gap: 8px; + align-items: center; +} + +.version-tag { + display: inline-flex; + align-items: center; + justify-content: center; + padding: 4px 12px; + /* Larger touch target and better visibility */ + border-radius: 4px; + /* Slightly more squared to match material design */ + font-size: 0.8rem; + /* Slightly larger text */ + font-weight: 700; + /* Bolder for visibility */ + line-height: 1.2; + color: var(--md-default-fg-color); + /* Darker text for contrast */ + background-color: rgba(0, 0, 0, 0.08); + /* Slightly darker bg */ + border: 1px solid rgba(0, 0, 0, 0.1); + /* Subtle border */ + transition: all 0.2s ease; + text-decoration: none !important; + font-family: var(--md-text-font-family); +} + +.version-tag:hover { + background-color: rgba(0, 0, 0, 0.12); + color: var(--md-primary-fg-color); + border-color: rgba(0, 0, 0, 0.2); +} + +.version-tag.active { + background-color: var(--md-accent-fg-color); + color: white; + border-color: var(--md-accent-fg-color); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1); + /* Subtle shadow for depth */ +} + +/* Dark Mode Adjustments */ +[data-md-color-scheme="slate"] .version-tag { + background-color: rgba(255, 255, 255, 0.1); + color: var(--md-default-fg-color); + border-color: rgba(255, 255, 255, 0.1); +} + +[data-md-color-scheme="slate"] .version-tag:hover { + background-color: rgba(255, 255, 255, 0.15); + color: white; + border-color: rgba(255, 255, 255, 0.2); +} + +[data-md-color-scheme="slate"] .version-tag.active { + background-color: var(--md-accent-fg-color); + color: white; + border-color: var(--md-accent-fg-color); + box-shadow: 0 2px 4px rgba(0, 0, 0, 0.3); +} + +/* Mobile adjustments */ +@media screen and (max-width: 76.1875em) { + .version-scroll-container { + margin-left: 1rem; + max-width: 120px; + } + + .version-tag { + padding: 3px 8px; + font-size: 0.75rem; + } } \ No newline at end of file diff --git a/docs/getting-started.md b/docs/getting-started.md index 07d3af42..51f72832 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,107 +1,118 @@ # Getting Started -Welcome to Semantica! This guide will help you get up and running quickly with the framework. +## Welcome to Semantica -## πŸ“‹ Prerequisites +**Semantica** is a comprehensive knowledge graph and semantic processing framework designed for building production-ready semantic AI applications. -Before you begin, ensure you have the following: +### 🎯 What You'll Learn +- What Semantica is and why it's useful +- How to install and configure the framework +- Understanding the framework architecture +- Key concepts and terminology +- Next steps for getting started -- **Python 3.8+** installed on your system +--- + +## πŸš€ What is Semantica? + +Semantica is a powerful, production-ready framework for: + +- **Building Knowledge Graphs**: Transform unstructured data into structured knowledge graphs. +- **Semantic Processing**: Extract entities, relationships, and meaning from text, images, and audio. +- **GraphRAG**: Next-generation retrieval augmented generation using knowledge graphs. +- **Temporal Analysis**: Time-aware knowledge graphs for tracking changes over time. +- **Multi-Modal Processing**: Handle text, images, audio, and structured data. +- **Enterprise Features**: Quality assurance, conflict resolution, ontology generation, and more. + +--- + +## πŸ’‘ Use Cases + +| Domain | Application | +| :--- | :--- | +| **Cybersecurity** | Threat intelligence and analysis | +| **Healthcare** | Medical research and patient data analysis | +| **Finance** | Fraud detection and financial analysis | +| **Supply Chain** | Optimization and risk management | +| **Research** | Knowledge management and literature review | +| **AI Systems** | Multi-agent memory and reasoning | + +--- + +## πŸ“¦ Installation & Setup + +### Prerequisites +Before installing Semantica, ensure you have: +- **Python 3.8** or higher - **pip** package manager -- Basic understanding of Python programming -- (Optional) An OpenAI API key or other LLM provider credentials if you plan to use semantic extraction features +- (Optional) Virtual environment for isolation -## 🎯 What You Will Learn +### Installation Methods -By following this guide, you will learn how to: +=== "PyPI (Stable)" + ```bash + pip install semantica + ``` -1. Install Semantica and its dependencies -2. Build your first Knowledge Graph from raw data -3. Query and visualize the extracted knowledge -4. Integrate Semantica into your own applications +=== "Source (Dev)" + ```bash + git clone https://github.com/your-org/semantica.git + cd semantica + pip install -e . + ``` + +=== "Extras" + ```bash + pip install semantica[all] # Install all optional dependencies + pip install semantica[gpu] # Install GPU support + pip install semantica[visualization] # Install visualization tools + ``` + +### Verify Installation + +```python +import semantica +print(semantica.version) +``` --- -## πŸ›£οΈ Choose Your Path +## βš™οΈ Configuration -Select the path that best fits your goals: +Semantica can be configured using environment variables or a configuration file. -
+### Environment Variables -- **πŸš€ Quick Start (5 min)** - --- - Perfect for trying Semantica quickly. - - 1. **[Install Semantica](installation.md#basic-installation)** - 2. **[Run Your First Example](quickstart.md#step-2-your-first-knowledge-graph)** - 3. **[Explore Examples](examples.md)** +```bash +export SEMANTICA_API_KEY=your_openai_key +export SEMANTICA_EMBEDDING_PROVIDER=openai +export SEMANTICA_MODEL_NAME=gpt-4 +``` -- **πŸ“š Complete Guide (30 min)** - --- - Perfect for learning the framework properly. - - 1. **[Installation](installation.md)** - Complete setup - 2. **[Quick Start](quickstart.md)** - Step-by-step - 3. **[Core Concepts](concepts.md)** - Deep dive - 4. **[Examples](examples.md)** - Real-world use cases +### Config File (`config.yaml`) -- **πŸŽ“ Interactive Learning** - --- - Perfect for hands-on learners using Jupyter. - - 1. **[Cookbook Overview](cookbook.md)** - 2. **[Start with Introduction](cookbook.md#introduction)** - 3. **[Try Use Cases](cookbook.md#use-cases)** +```yaml +api_keys: + openai: your_key_here + anthropic: your_key_here -
+embedding: + provider: openai + model: text-embedding-3-large + dimensions: 3072 -!!! tip "Recommendation" - If you're new to knowledge graphs, we highly recommend starting with the **[Core Concepts](concepts.md)** page to understand the terminology before diving into the code. +knowledge_graph: + backend: networkx # or neo4j, arangodb + temporal: true +``` --- -## πŸ—οΈ What Can You Build? - -Semantica helps you transform unstructured data into intelligent knowledge: - -- **Knowledge Graphs** from documents, websites, databases -- **Semantic Layers** for AI applications -- **Entity & Relationship Extraction** from text -- **Conflict Resolution** across multiple data sources -- **GraphRAG Systems** for enhanced AI responses - -## πŸ’‘ Common Use Cases - -### Research & Analysis -- Extract knowledge from research papers -- Build domain-specific knowledge graphs -- Analyze relationships in literature - -### Business Intelligence -- Process company documents -- Build organizational knowledge bases -- Integrate multiple data sources - -### AI Applications -- Power GraphRAG systems -- Enhance AI agent memory -- Build semantic search systems - -!!! warning "Installation First" - Make sure you have Semantica installed before proceeding. If you encounter any issues, check the [Installation Guide](installation.md) troubleshooting section. - ## ⏭️ Next Steps -Once you're ready: - -1. Head to the **[Installation Guide](installation.md)** to set up your environment. -2. Follow the **[Quick Start](quickstart.md)** to build your first graph. -3. Check out the **[Modules Guide](modules.md)** to understand the architecture. - -## πŸ†˜ Need Help? - -- **Installation Issues?** β†’ [Troubleshooting Guide](installation.md#troubleshooting) -- **First Time User?** β†’ [Quick Start Guide](quickstart.md) -- **Looking for Examples?** β†’ [Examples Page](examples.md) -- **API Questions?** β†’ [API Reference](api.md) +Now that you understand the basics, here are recommended next steps: +1. **[Your First Knowledge Graph](cookbook/introduction/Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph from a document. +2. **[Configuration Basics](cookbook/introduction/Configuration_Basics.ipynb)**: Set up configuration files and API keys. +3. **[Core Workflows](cookbook.md#core-workflows)**: Learn common patterns and workflows. +4. **[Use Cases](cookbook.md#use-cases)**: Explore domain-specific applications. diff --git a/docs/js/version-selector.js b/docs/js/version-selector.js new file mode 100644 index 00000000..3e0155bc --- /dev/null +++ b/docs/js/version-selector.js @@ -0,0 +1,32 @@ +document.addEventListener("DOMContentLoaded", function () { + // Target the header title + var headerTitle = document.querySelector(".md-header__title"); + + if (headerTitle) { + // Create the container for the version selector + var versionContainer = document.createElement("div"); + versionContainer.className = "version-scroll-container"; + + // Define versions + var versions = [ + { name: "0.0.1", url: "#", current: true } + ]; + + // Create the scrollable list + var versionList = document.createElement("div"); + versionList.className = "version-list"; + + versions.forEach(function (version) { + var versionLink = document.createElement("a"); + versionLink.className = "version-tag" + (version.current ? " active" : ""); + versionLink.href = version.url; + versionLink.textContent = version.name; + versionList.appendChild(versionLink); + }); + + versionContainer.appendChild(versionList); + + // Insert after the header title + headerTitle.parentNode.insertBefore(versionContainer, headerTitle.nextSibling); + } +}); diff --git a/docs/modules.md b/docs/modules.md index 9a2bdb97..bb887014 100644 --- a/docs/modules.md +++ b/docs/modules.md @@ -4,13 +4,13 @@ Semantica is built with a modular architecture, designed to be flexible and exte ## πŸ—οΈ Architecture Overview -The framework is organized into several core layers: +The framework is organized into several core layers, each handling specific aspects of the semantic processing pipeline. ```mermaid graph TD subgraph Ingest [Ingestion Layer] - I[Ingest] --> P[Parse] - P --> N[Normalize] + I[Ingest Module] --> P[Parse Module] + P --> N[Normalize Module] end subgraph Core [Core Processing] @@ -29,86 +29,279 @@ graph TD KG --> V[Visualization] KG --> R[Reasoning] end + + style Ingest fill:#e1f5fe,stroke:#01579b + style Core fill:#e8f5e9,stroke:#1b5e20 + style Storage fill:#fff3e0,stroke:#e65100 + style Output fill:#f3e5f5,stroke:#4a148c ``` +--- + ## πŸ“¦ Core Modules -
+Semantica is organized into 12 core modules. Below is a detailed breakdown of each. -- :material-cube-outline: **[Core](reference/core.md)** - --- - The foundation of the framework. Handles configuration, logging, and base classes. +### 1. Ingest Module +**Purpose**: Ingest data from various sources into a unified format. -- :material-file-import: **[Ingest](reference/ingest.md)** - --- - Handles the loading of data from various sources (50+ formats). +The `ingest` module is the entry point for data. It handles the complexity of connecting to different data sources, from local files to web streams. -- :material-file-code: **[Parse](reference/parse.md)** - --- - Parses raw data into structured text and metadata, including OCR. +- **Components**: + - `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.) + - `WebIngestor`: Scrape and ingest web pages + - `FeedIngestor`: Process RSS/Atom feeds + - `StreamIngestor`: Real-time data streaming + - `DBIngestor`: Database queries and ingestion + - `EmailIngestor`: Process email messages + - `RepoIngestor`: Git repository analysis -- :material-format-align-left: **[Normalize](reference/normalize.md)** - --- - Cleans and standardizes extracted text and encoding. +```python +from semantica.ingest import FileIngestor, WebIngestor -- :material-brain: **[Semantic Extract](reference/semantic_extract.md)** - --- - Extracts meaning, entities, and relations using LLMs and NLP. +# Ingest local files +file_ingestor = FileIngestor() +documents = file_ingestor.ingest("data/") -- :material-graph: **[Knowledge Graph](reference/kg.md)** - --- - The central module for building and managing knowledge graphs. +# Ingest web content +web_ingestor = WebIngestor() +web_docs = web_ingestor.ingest("https://example.com") +``` -- :material-vector-curve: **[Embeddings](reference/embeddings.md)** - --- - Generates vector embeddings for text and graph nodes. +### 2. Parse Module +**Purpose**: Parse and extract content from various raw formats. -- :material-database: **[Vector Store](reference/vector_store.md)** - --- - Manages vector storage (Pinecone, Chroma, Qdrant). +Once data is ingested, the `parse` module extracts the raw text and metadata. It supports a wide range of formats and includes OCR capabilities. -- :material-database-search: **[Triple Store](reference/triple_store.md)** - --- - Manages RDF triple storage (Neo4j, RDFLib). +- **Components**: + - `DocumentParser`: Main parser orchestrator + - `PDFParser`: Extract text, tables, images from PDFs + - `DOCXParser`: Parse Word documents + - `HTMLParser`: Extract content from HTML + - `JSONParser`: Parse structured JSON data + - `ExcelParser`: Process spreadsheets + - `ImageParser`: OCR and image analysis + - `CodeParser`: Parse source code files -- :material-shape: **[Ontology](reference/ontology.md)** - --- - Defines and validates the schema of the knowledge graph. +```python +from semantica.parse import DocumentParser -- :material-lightbulb: **[Reasoning](reference/reasoning.md)** - --- - Performs logical reasoning and inference over the graph. +parser = DocumentParser() +parsed_docs = parser.parse(documents) +``` -- :material-pipe: **[Pipeline](reference/pipeline.md)** - --- - Orchestrates the end-to-end processing flow. +### 3. Normalize Module +**Purpose**: Clean and normalize text for processing. -- :material-export: **[Export](reference/export.md)** - --- - Exports the knowledge graph to JSON, RDF, CSV, GEXF, etc. +Raw text is often noisy. The `normalize` module cleans, standardizes, and prepares text for semantic extraction. -- :material-chart-bubble: **[Visualization](reference/visualization.md)** - --- - Tools for interactive graph plotting and HTML export. +- **Components**: + - `TextNormalizer`: Main normalization orchestrator + - `TextCleaner`: Remove noise, fix encoding + - `DataCleaner`: Clean structured data + - `EntityNormalizer`: Normalize entity names + - `DateNormalizer`: Standardize date formats + - `NumberNormalizer`: Normalize numeric values + - `LanguageDetector`: Detect document language + - `EncodingHandler`: Handle character encoding -- :material-tools: **[Utils](reference/utils.md)** - --- - General utility functions for file I/O and string manipulation. +```python +from semantica.normalize import TextNormalizer -
+normalizer = TextNormalizer() +normalized = normalizer.normalize(parsed_docs) +``` -## 🧩 Submodules +### 4. Semantic Extract Module +**Purpose**: Extract entities, relationships, and semantic information. -Each module contains specialized submodules. For example: +This is the brain of the operation. It uses LLMs and NLP techniques to understand the text and extract structured knowledge. -- **`semantica.ingest`** - - `loaders`: Specific file loaders - - `stream`: Streaming data handlers -- **`semantica.kg`** - - `builder`: Graph construction logic - - `query`: Graph query interface -- **`semantica.semantic_extract`** - - `llm`: LLM-based extraction - - `spacy`: NLP-based extraction +- **Components**: + - `NERExtractor`: Named Entity Recognition + - `RelationExtractor`: Extract relationships between entities + - `SemanticAnalyzer`: Deep semantic analysis + - `SemanticNetworkExtractor`: Extract semantic networks -For detailed API documentation, please refer to the specific **API Reference** pages linked above. +```python +from semantica.semantic_extract import NERExtractor, RelationExtractor + +# Extract entities +extractor = NERExtractor() +entities = extractor.extract(normalized_docs) + +# Extract relationships +relation_extractor = RelationExtractor() +relationships = relation_extractor.extract(normalized_docs, entities) +``` + +### 5. Knowledge Graph (KG) Module +**Purpose**: Build and manage knowledge graphs. + +The `kg` module constructs the graph from extracted entities and relationships, handling complex tasks like resolution and analysis. + +- **Components**: + - `GraphBuilder`: Construct knowledge graphs from entities/relationships + - `GraphAnalyzer`: Analyze graph structure and properties + - `GraphValidator`: Validate graph quality and consistency + - `EntityResolver`: Resolve entity conflicts and duplicates + - `ConflictDetector`: Detect conflicting information + - `CentralityCalculator`: Calculate node importance metrics + - `CommunityDetector`: Detect communities in graphs + - `ConnectivityAnalyzer`: Analyze graph connectivity + - `TemporalQuery`: Query temporal knowledge graphs + - `Deduplicator`: Remove duplicate entities/relationships + +```python +from semantica.kg import GraphBuilder, GraphAnalyzer + +# Build graph +builder = GraphBuilder() +kg = builder.build(entities, relationships) + +# Analyze graph +analyzer = GraphAnalyzer() +metrics = analyzer.analyze(kg) +``` + +### 6. Embeddings Module +**Purpose**: Generate vector embeddings for various data types. + +Embeddings are crucial for semantic search. This module generates vectors for text, images, and graph nodes. + +- **Components**: + - `EmbeddingGenerator`: Main embedding orchestrator + - `TextEmbedder`: Generate text embeddings + - `ImageEmbedder`: Generate image embeddings + - `AudioEmbedder`: Generate audio embeddings + - `MultimodalEmbedder`: Combine multiple modalities + - `EmbeddingOptimizer`: Optimize embedding quality + - `ProviderAdapters`: Support for OpenAI, Cohere, etc. + +```python +from semantica.embeddings import EmbeddingGenerator + +generator = EmbeddingGenerator() +embeddings = generator.generate(documents) +``` + +### 7. Vector Store Module +**Purpose**: Store and search vector embeddings. + +Manages the storage and retrieval of high-dimensional vectors, supporting hybrid search strategies. + +- **Components**: + - `VectorStore`: Main vector store interface + - `FAISSAdapter`: FAISS integration + - `HybridSearch`: Combine vector and keyword search + - `VectorRetriever`: Retrieve relevant vectors + +```python +from semantica.vector_store import VectorStore, HybridSearch + +vector_store = VectorStore() +vector_store.store(embeddings, documents, metadata) + +hybrid_search = HybridSearch(vector_store) +results = hybrid_search.search(query, top_k=10) +``` + +### 8. Reasoning Module +**Purpose**: Perform logical inference and reasoning. + +Goes beyond simple retrieval to infer new facts and validate existing knowledge using logical rules. + +- **Components**: + - `InferenceEngine`: Main inference orchestrator + - `RuleManager`: Manage inference rules + - `DeductiveReasoner`: Deductive reasoning + - `AbductiveReasoner`: Abductive reasoning + - `ExplanationGenerator`: Generate explanations for inferences + - `RETEEngine`: RETE algorithm for rule matching + +```python +from semantica.reasoning import InferenceEngine, RuleManager + +inference_engine = InferenceEngine() +rule_manager = RuleManager() +new_facts = inference_engine.forward_chain(kg, rule_manager) +``` + +### 9. Ontology Module +**Purpose**: Generate and manage ontologies. + +Defines the schema and structure of your knowledge domain, ensuring consistency and enabling interoperability. + +- **Components**: + - `OntologyGenerator`: Generate ontologies from knowledge graphs + - `OntologyValidator`: Validate ontology structure + - `OWLGenerator`: Generate OWL format ontologies + - `PropertyGenerator`: Generate ontology properties + - `ClassInferrer`: Infer ontology classes + +```python +from semantica.ontology import OntologyGenerator + +generator = OntologyGenerator() +ontology = generator.generate_from_graph(kg) +``` + +### 10. Export Module +**Purpose**: Export data in various formats. + +Allows you to take your knowledge graph and data out of Semantica for use in other tools. + +- **Components**: + - `JSONExporter`: Export to JSON + - `RDFExporter`: Export to RDF/XML + - `CSVExporter`: Export to CSV + - `GraphExporter`: Export to graph formats (GraphML, GEXF) + - `OWLExporter`: Export to OWL + - `VectorExporter`: Export vectors + +```python +from semantica.export import JSONExporter, RDFExporter + +json_exporter = JSONExporter() +json_exporter.export(kg, "output.json") +``` + +### 11. Visualization Module +**Purpose**: Visualize knowledge graphs and analytics. + +Provides tools to visually explore your data, making it easier to understand complex relationships. + +- **Components**: + - `KGVisualizer`: Visualize knowledge graphs + - `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP) + - `QualityVisualizer`: Visualize quality metrics + - `AnalyticsVisualizer`: Visualize graph analytics + - `TemporalVisualizer`: Visualize temporal data + +```python +from semantica.visualization import KGVisualizer + +visualizer = KGVisualizer() +visualizer.visualize(kg) +``` + +### 12. Pipeline Module +**Purpose**: Build and execute processing pipelines. + +Orchestrates the entire flow, connecting modules together into robust, executable workflows. + +- **Components**: + - `PipelineBuilder`: Build complex pipelines + - `ExecutionEngine`: Execute pipelines + - `FailureHandler`: Handle pipeline failures + - `ParallelismManager`: Enable parallel processing + - `ResourceScheduler`: Schedule resources + +```python +from semantica.pipeline import PipelineBuilder + +builder = PipelineBuilder() +pipeline = builder.add_step("ingest", FileIngestor()) \ + .add_step("parse", DocumentParser()) \ + .build() +``` diff --git a/mkdocs.yml b/mkdocs.yml index 11adc4d5..fe2ebbd9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -92,11 +92,15 @@ plugins: extra_css: - css/custom.css +# Custom JavaScript +extra_javascript: + - js/version-selector.js + # Navigation nav: - Home: index.md - Getting Started: - - getting-started.md + - Overview: getting-started.md - installation.md - quickstart.md - Guides: @@ -105,26 +109,7 @@ nav: - use-cases.md - examples.md - learning-more.md - - Cookbook: - - Introduction: - - cookbook/introduction/Welcome_to_Semantica.ipynb - - cookbook/introduction/Configuration_Basics.ipynb - - cookbook/introduction/Your_First_Knowledge_Graph.ipynb - - cookbook/introduction/Data_Ingestion.ipynb - - cookbook/introduction/Document_Parsing.ipynb - - cookbook/introduction/Data_Normalization.ipynb - - cookbook/introduction/Entity_Extraction.ipynb - - cookbook/introduction/Relation_Extraction.ipynb - - cookbook/introduction/Embedding_Generation.ipynb - - cookbook/introduction/Vector_Store.ipynb - - cookbook/introduction/Ontology.ipynb - - cookbook/introduction/Conflict_Detection.ipynb - - cookbook/introduction/Deduplication.ipynb - - cookbook/introduction/Building_Knowledge_Graphs.ipynb - - cookbook/introduction/Graph_Analytics.ipynb - - cookbook/introduction/Graph_Quality.ipynb - - cookbook/introduction/Visualization.ipynb - - cookbook/introduction/Export.ipynb + - Cookbook: cookbook.md - Reference: - Core: reference/core.md - Ingest: reference/ingest.md