mengzhx/semantica
1
0
Fork 0
mirror of https://github.com/semantica-agi/semantica.git synced 2026-08-29 04:26:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: enhance documentation structure and notebook formatting

Browse Source 
- Update mkdocs.yml navigation to use new cookbook index
- Create comprehensive docs/cookbook.md index
- Refactor Welcome_to_Semantica.ipynb to use Markdown cells
- Enhance markdown formatting in Your_First_Knowledge_Graph.ipynb and Financial_Data_Integration.ipynb
- Update custom.css and version-selector.js for better styling
- Populate modules.md, concepts.md, and getting-started.md with detailed content
This commit is contained in:
KaifAhmad1
2025-11-23 14:17:36 +05:30
parent 08650a4219
commit 89a64a483b
13 changed files with 1709 additions and 1272 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cookbook/introduction/Welcome_to_Semantica.ipynb
+416 -459
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Welcome to Semantica\n",
"# Welcome to Semantica: An interactive introduction to the framework\n",
"\n",
"## Overview\n",
"\n",
@@ -54,7 +54,7 @@
"\n",
"### Installation Methods\n",
"\n",
"'''\n",
"```bash\n",
"# Method 1: Install from PyPI (when available)\n",
"# pip install semantica\n",
"\n",
@@ -71,11 +71,11 @@
"# Verify installation\n",
"# import semantica\n",
"# print(semantica.__version__)\n",
"'''\n",
"```\n",
"\n",
"### Configuration\n",
"\n",
"'''\n",
"```bash\n",
"# Set up environment variables for API keys and configuration\n",
"# export SEMANTICA_API_KEY=your_openai_key\n",
"# export SEMANTICA_EMBEDDING_PROVIDER=openai\n",
@@ -92,7 +92,7 @@
"# knowledge_graph:\n",
"# backend: networkx # or neo4j, arangodb\n",
"# temporal: true\n",
"'''\n",
"```\n",
"\n",
"---\n",
"\n",
@@ -100,205 +100,223 @@
"\n",
"Semantica is organized into modular components, each handling a specific aspect of semantic processing:\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# CORE MODULES\n",
"# ============================================================================\n",
"### 1. INGEST MODULE - Data Ingestion\n",
"**Purpose**: Ingest data from various sources\n",
"**Components**:\n",
"- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n",
"- `WebIngestor`: Scrape and ingest web pages\n",
"- `FeedIngestor`: Process RSS/Atom feeds\n",
"- `StreamIngestor`: Real-time data streaming\n",
"- `DBIngestor`: Database queries and ingestion\n",
"- `EmailIngestor`: Process email messages\n",
"- `RepoIngestor`: Git repository analysis\n",
"\n",
"# 1. INGEST MODULE - Data Ingestion\n",
"# Purpose: Ingest data from various sources\n",
"# Components:\n",
"# - FileIngestor: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n",
"# - WebIngestor: Scrape and ingest web pages\n",
"# - FeedIngestor: Process RSS/Atom feeds\n",
"# - StreamIngestor: Real-time data streaming\n",
"# - DBIngestor: Database queries and ingestion\n",
"# - EmailIngestor: Process email messages\n",
"# - RepoIngestor: Git repository analysis\n",
"#\n",
"# Example:\n",
"# from semantica.ingest import FileIngestor, WebIngestor\n",
"# file_ingestor = FileIngestor()\n",
"# web_ingestor = WebIngestor()\n",
"# documents = file_ingestor.ingest(\"data/\")\n",
"# web_docs = web_ingestor.ingest(\"https://example.com\")\n",
"**Example**:\n",
"```python\n",
"from semantica.ingest import FileIngestor, WebIngestor\n",
"file_ingestor = FileIngestor()\n",
"web_ingestor = WebIngestor()\n",
"documents = file_ingestor.ingest(\"data/\")\n",
"web_docs = web_ingestor.ingest(\"https://example.com\")\n",
"```\n",
"\n",
"# 2. PARSE MODULE - Document Parsing\n",
"# Purpose: Parse and extract content from various formats\n",
"# Components:\n",
"# - DocumentParser: Main parser orchestrator\n",
"# - PDFParser: Extract text, tables, images from PDFs\n",
"# - DOCXParser: Parse Word documents\n",
"# - HTMLParser: Extract content from HTML\n",
"# - JSONParser: Parse structured JSON data\n",
"# - ExcelParser: Process spreadsheets\n",
"# - ImageParser: OCR and image analysis\n",
"# - CodeParser: Parse source code files\n",
"#\n",
"# Example:\n",
"# from semantica.parse import DocumentParser\n",
"# parser = DocumentParser()\n",
"# parsed_docs = parser.parse(documents)\n",
"### 2. PARSE MODULE - Document Parsing\n",
"**Purpose**: Parse and extract content from various formats\n",
"**Components**:\n",
"- `DocumentParser`: Main parser orchestrator\n",
"- `PDFParser`: Extract text, tables, images from PDFs\n",
"- `DOCXParser`: Parse Word documents\n",
"- `HTMLParser`: Extract content from HTML\n",
"- `JSONParser`: Parse structured JSON data\n",
"- `ExcelParser`: Process spreadsheets\n",
"- `ImageParser`: OCR and image analysis\n",
"- `CodeParser`: Parse source code files\n",
"\n",
"# 3. NORMALIZE MODULE - Text Normalization\n",
"# Purpose: Clean and normalize text for processing\n",
"# Components:\n",
"# - TextNormalizer: Main normalization orchestrator\n",
"# - TextCleaner: Remove noise, fix encoding\n",
"# - DataCleaner: Clean structured data\n",
"# - EntityNormalizer: Normalize entity names\n",
"# - DateNormalizer: Standardize date formats\n",
"# - NumberNormalizer: Normalize numeric values\n",
"# - LanguageDetector: Detect document language\n",
"# - EncodingHandler: Handle character encoding\n",
"#\n",
"# Example:\n",
"# from semantica.normalize import TextNormalizer\n",
"# normalizer = TextNormalizer()\n",
"# normalized = normalizer.normalize(parsed_docs)\n",
"**Example**:\n",
"```python\n",
"from semantica.parse import DocumentParser\n",
"parser = DocumentParser()\n",
"parsed_docs = parser.parse(documents)\n",
"```\n",
"\n",
"# 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n",
"# Purpose: Extract entities, relationships, and semantic information\n",
"# Components:\n",
"# - NERExtractor: Named Entity Recognition\n",
"# - RelationExtractor: Extract relationships between entities\n",
"# - SemanticAnalyzer: Deep semantic analysis\n",
"# - SemanticNetworkExtractor: Extract semantic networks\n",
"#\n",
"# Example:\n",
"# from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
"# extractor = NERExtractor()\n",
"# entities = extractor.extract(normalized_docs)\n",
"# relation_extractor = RelationExtractor()\n",
"# relationships = relation_extractor.extract(normalized_docs, entities)\n",
"### 3. NORMALIZE MODULE - Text Normalization\n",
"**Purpose**: Clean and normalize text for processing\n",
"**Components**:\n",
"- `TextNormalizer`: Main normalization orchestrator\n",
"- `TextCleaner`: Remove noise, fix encoding\n",
"- `DataCleaner`: Clean structured data\n",
"- `EntityNormalizer`: Normalize entity names\n",
"- `DateNormalizer`: Standardize date formats\n",
"- `NumberNormalizer`: Normalize numeric values\n",
"- `LanguageDetector`: Detect document language\n",
"- `EncodingHandler`: Handle character encoding\n",
"\n",
"# 5. KG MODULE - Knowledge Graph Construction\n",
"# Purpose: Build and manage knowledge graphs\n",
"# Components:\n",
"# - GraphBuilder: Construct knowledge graphs from entities/relationships\n",
"# - GraphAnalyzer: Analyze graph structure and properties\n",
"# - GraphValidator: Validate graph quality and consistency\n",
"# - EntityResolver: Resolve entity conflicts and duplicates\n",
"# - ConflictDetector: Detect conflicting information\n",
"# - CentralityCalculator: Calculate node importance metrics\n",
"# - CommunityDetector: Detect communities in graphs\n",
"# - ConnectivityAnalyzer: Analyze graph connectivity\n",
"# - TemporalQuery: Query temporal knowledge graphs\n",
"# - Deduplicator: Remove duplicate entities/relationships\n",
"#\n",
"# Example:\n",
"# from semantica.kg import GraphBuilder, GraphAnalyzer\n",
"# builder = GraphBuilder()\n",
"# kg = builder.build(entities, relationships)\n",
"# analyzer = GraphAnalyzer()\n",
"# metrics = analyzer.analyze(kg)\n",
"**Example**:\n",
"```python\n",
"from semantica.normalize import TextNormalizer\n",
"normalizer = TextNormalizer()\n",
"normalized = normalizer.normalize(parsed_docs)\n",
"```\n",
"\n",
"# 6. EMBEDDINGS MODULE - Embedding Generation\n",
"# Purpose: Generate vector embeddings for various data types\n",
"# Components:\n",
"# - EmbeddingGenerator: Main embedding orchestrator\n",
"# - TextEmbedder: Generate text embeddings\n",
"# - ImageEmbedder: Generate image embeddings\n",
"# - AudioEmbedder: Generate audio embeddings\n",
"# - MultimodalEmbedder: Combine multiple modalities\n",
"# - EmbeddingOptimizer: Optimize embedding quality\n",
"# - ProviderAdapters: Support for OpenAI, Cohere, etc.\n",
"#\n",
"# Example:\n",
"# from semantica.embeddings import EmbeddingGenerator\n",
"# generator = EmbeddingGenerator()\n",
"# embeddings = generator.generate(documents)\n",
"### 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n",
"**Purpose**: Extract entities, relationships, and semantic information\n",
"**Components**:\n",
"- `NERExtractor`: Named Entity Recognition\n",
"- `RelationExtractor`: Extract relationships between entities\n",
"- `SemanticAnalyzer`: Deep semantic analysis\n",
"- `SemanticNetworkExtractor`: Extract semantic networks\n",
"\n",
"# 7. VECTOR_STORE MODULE - Vector Database Operations\n",
"# Purpose: Store and search vector embeddings\n",
"# Components:\n",
"# - VectorStore: Main vector store interface\n",
"# - FAISSAdapter: FAISS integration\n",
"# - HybridSearch: Combine vector and keyword search\n",
"# - VectorRetriever: Retrieve relevant vectors\n",
"#\n",
"# Example:\n",
"# from semantica.vector_store import VectorStore, HybridSearch\n",
"# vector_store = VectorStore()\n",
"# vector_store.store(embeddings, documents, metadata)\n",
"# hybrid_search = HybridSearch(vector_store)\n",
"# results = hybrid_search.search(query, top_k=10)\n",
"**Example**:\n",
"```python\n",
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
"extractor = NERExtractor()\n",
"entities = extractor.extract(normalized_docs)\n",
"relation_extractor = RelationExtractor()\n",
"relationships = relation_extractor.extract(normalized_docs, entities)\n",
"```\n",
"\n",
"# 8. REASONING MODULE - Inference and Reasoning\n",
"# Purpose: Perform logical inference and reasoning\n",
"# Components:\n",
"# - InferenceEngine: Main inference orchestrator\n",
"# - RuleManager: Manage inference rules\n",
"# - DeductiveReasoner: Deductive reasoning\n",
"# - AbductiveReasoner: Abductive reasoning\n",
"# - ExplanationGenerator: Generate explanations for inferences\n",
"# - RETEEngine: RETE algorithm for rule matching\n",
"#\n",
"# Example:\n",
"# from semantica.reasoning import InferenceEngine, RuleManager\n",
"# inference_engine = InferenceEngine()\n",
"# rule_manager = RuleManager()\n",
"# new_facts = inference_engine.forward_chain(kg, rule_manager)\n",
"### 5. KG MODULE - Knowledge Graph Construction\n",
"**Purpose**: Build and manage knowledge graphs\n",
"**Components**:\n",
"- `GraphBuilder`: Construct knowledge graphs from entities/relationships\n",
"- `GraphAnalyzer`: Analyze graph structure and properties\n",
"- `GraphValidator`: Validate graph quality and consistency\n",
"- `EntityResolver`: Resolve entity conflicts and duplicates\n",
"- `ConflictDetector`: Detect conflicting information\n",
"- `CentralityCalculator`: Calculate node importance metrics\n",
"- `CommunityDetector`: Detect communities in graphs\n",
"- `ConnectivityAnalyzer`: Analyze graph connectivity\n",
"- `TemporalQuery`: Query temporal knowledge graphs\n",
"- `Deduplicator`: Remove duplicate entities/relationships\n",
"\n",
"# 9. ONTOLOGY MODULE - Ontology Generation\n",
"# Purpose: Generate and manage ontologies\n",
"# Components:\n",
"# - OntologyGenerator: Generate ontologies from knowledge graphs\n",
"# - OntologyValidator: Validate ontology structure\n",
"# - OWLGenerator: Generate OWL format ontologies\n",
"# - PropertyGenerator: Generate ontology properties\n",
"# - ClassInferrer: Infer ontology classes\n",
"#\n",
"# Example:\n",
"# from semantica.ontology import OntologyGenerator\n",
"# generator = OntologyGenerator()\n",
"# ontology = generator.generate_from_graph(kg)\n",
"**Example**:\n",
"```python\n",
"from semantica.kg import GraphBuilder, GraphAnalyzer\n",
"builder = GraphBuilder()\n",
"kg = builder.build(entities, relationships)\n",
"analyzer = GraphAnalyzer()\n",
"metrics = analyzer.analyze(kg)\n",
"```\n",
"\n",
"# 10. EXPORT MODULE - Data Export\n",
"# Purpose: Export data in various formats\n",
"# Components:\n",
"# - JSONExporter: Export to JSON\n",
"# - RDFExporter: Export to RDF/XML\n",
"# - CSVExporter: Export to CSV\n",
"# - GraphExporter: Export to graph formats (GraphML, GEXF)\n",
"# - OWLExporter: Export to OWL\n",
"# - VectorExporter: Export vectors\n",
"#\n",
"# Example:\n",
"# from semantica.export import JSONExporter, RDFExporter\n",
"# json_exporter = JSONExporter()\n",
"# json_exporter.export(kg, \"output.json\")\n",
"### 6. EMBEDDINGS MODULE - Embedding Generation\n",
"**Purpose**: Generate vector embeddings for various data types\n",
"**Components**:\n",
"- `EmbeddingGenerator`: Main embedding orchestrator\n",
"- `TextEmbedder`: Generate text embeddings\n",
"- `ImageEmbedder`: Generate image embeddings\n",
"- `AudioEmbedder`: Generate audio embeddings\n",
"- `MultimodalEmbedder`: Combine multiple modalities\n",
"- `EmbeddingOptimizer`: Optimize embedding quality\n",
"- `ProviderAdapters`: Support for OpenAI, Cohere, etc.\n",
"\n",
"# 11. VISUALIZATION MODULE - Graph Visualization\n",
"# Purpose: Visualize knowledge graphs and analytics\n",
"# Components:\n",
"# - KGVisualizer: Visualize knowledge graphs\n",
"# - EmbeddingVisualizer: Visualize embeddings (t-SNE, PCA, UMAP)\n",
"# - QualityVisualizer: Visualize quality metrics\n",
"# - AnalyticsVisualizer: Visualize graph analytics\n",
"# - TemporalVisualizer: Visualize temporal data\n",
"#\n",
"# Example:\n",
"# from semantica.visualization import KGVisualizer\n",
"# visualizer = KGVisualizer()\n",
"# visualizer.visualize(kg)\n",
"**Example**:\n",
"```python\n",
"from semantica.embeddings import EmbeddingGenerator\n",
"generator = EmbeddingGenerator()\n",
"embeddings = generator.generate(documents)\n",
"```\n",
"\n",
"# 12. PIPELINE MODULE - Pipeline Orchestration\n",
"# Purpose: Build and execute processing pipelines\n",
"# Components:\n",
"# - PipelineBuilder: Build complex pipelines\n",
"# - ExecutionEngine: Execute pipelines\n",
"# - FailureHandler: Handle pipeline failures\n",
"# - ParallelismManager: Enable parallel processing\n",
"# - ResourceScheduler: Schedule resources\n",
"#\n",
"# Example:\n",
"# from semantica.pipeline import PipelineBuilder\n",
"# builder = PipelineBuilder()\n",
"# pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n",
"# .add_step(\"parse\", DocumentParser()) \\\\\n",
"# .build()\n",
"'''\n",
"### 7. VECTOR_STORE MODULE - Vector Database Operations\n",
"**Purpose**: Store and search vector embeddings\n",
"**Components**:\n",
"- `VectorStore`: Main vector store interface\n",
"- `FAISSAdapter`: FAISS integration\n",
"- `HybridSearch`: Combine vector and keyword search\n",
"- `VectorRetriever`: Retrieve relevant vectors\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.vector_store import VectorStore, HybridSearch\n",
"vector_store = VectorStore()\n",
"vector_store.store(embeddings, documents, metadata)\n",
"hybrid_search = HybridSearch(vector_store)\n",
"results = hybrid_search.search(query, top_k=10)\n",
"```\n",
"\n",
"### 8. REASONING MODULE - Inference and Reasoning\n",
"**Purpose**: Perform logical inference and reasoning\n",
"**Components**:\n",
"- `InferenceEngine`: Main inference orchestrator\n",
"- `RuleManager`: Manage inference rules\n",
"- `DeductiveReasoner`: Deductive reasoning\n",
"- `AbductiveReasoner`: Abductive reasoning\n",
"- `ExplanationGenerator`: Generate explanations for inferences\n",
"- `RETEEngine`: RETE algorithm for rule matching\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.reasoning import InferenceEngine, RuleManager\n",
"inference_engine = InferenceEngine()\n",
"rule_manager = RuleManager()\n",
"new_facts = inference_engine.forward_chain(kg, rule_manager)\n",
"```\n",
"\n",
"### 9. ONTOLOGY MODULE - Ontology Generation\n",
"**Purpose**: Generate and manage ontologies\n",
"**Components**:\n",
"- `OntologyGenerator`: Generate ontologies from knowledge graphs\n",
"- `OntologyValidator`: Validate ontology structure\n",
"- `OWLGenerator`: Generate OWL format ontologies\n",
"- `PropertyGenerator`: Generate ontology properties\n",
"- `ClassInferrer`: Infer ontology classes\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.ontology import OntologyGenerator\n",
"generator = OntologyGenerator()\n",
"ontology = generator.generate_from_graph(kg)\n",
"```\n",
"\n",
"### 10. EXPORT MODULE - Data Export\n",
"**Purpose**: Export data in various formats\n",
"**Components**:\n",
"- `JSONExporter`: Export to JSON\n",
"- `RDFExporter`: Export to RDF/XML\n",
"- `CSVExporter`: Export to CSV\n",
"- `GraphExporter`: Export to graph formats (GraphML, GEXF)\n",
"- `OWLExporter`: Export to OWL\n",
"- `VectorExporter`: Export vectors\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.export import JSONExporter, RDFExporter\n",
"json_exporter = JSONExporter()\n",
"json_exporter.export(kg, \"output.json\")\n",
"```\n",
"\n",
"### 11. VISUALIZATION MODULE - Graph Visualization\n",
"**Purpose**: Visualize knowledge graphs and analytics\n",
"**Components**:\n",
"- `KGVisualizer`: Visualize knowledge graphs\n",
"- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP)\n",
"- `QualityVisualizer`: Visualize quality metrics\n",
"- `AnalyticsVisualizer`: Visualize graph analytics\n",
"- `TemporalVisualizer`: Visualize temporal data\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.visualization import KGVisualizer\n",
"visualizer = KGVisualizer()\n",
"visualizer.visualize(kg)\n",
"```\n",
"\n",
"### 12. PIPELINE MODULE - Pipeline Orchestration\n",
"**Purpose**: Build and execute processing pipelines\n",
"**Components**:\n",
"- `PipelineBuilder`: Build complex pipelines\n",
"- `ExecutionEngine`: Execute pipelines\n",
"- `FailureHandler`: Handle pipeline failures\n",
"- `ParallelismManager`: Enable parallel processing\n",
"- `ResourceScheduler`: Schedule resources\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.pipeline import PipelineBuilder\n",
"builder = PipelineBuilder()\n",
"pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n",
" .add_step(\"parse\", DocumentParser()) \\\\\n",
" .build()\n",
"```\n",
"\n",
"---\n",
"\n",
@@ -306,183 +324,166 @@
"\n",
"Understanding these concepts is crucial for working with Semantica:\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# CORE CONCEPTS\n",
"# ============================================================================\n",
"### 1. KNOWLEDGE GRAPHS\n",
"**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes.\n",
"\n",
"# 1. KNOWLEDGE GRAPHS\n",
"# Definition: A knowledge graph is a structured representation of entities\n",
"# (nodes) and their relationships (edges) with properties and\n",
"# attributes.\n",
"#\n",
"# Structure:\n",
"# - Nodes: Represent entities (people, places, concepts, events)\n",
"# - Edges: Represent relationships (works_for, located_in, causes)\n",
"# - Properties: Attributes of entities and relationships\n",
"# - Metadata: Additional information (sources, timestamps, confidence)\n",
"#\n",
"# Example:\n",
"# Entity: \"John Doe\" (Person)\n",
"# Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n",
"# Properties: {start_date: \"2020-01-01\", role: \"Engineer\"}\n",
"#\n",
"# Benefits:\n",
"# - Structured representation of unstructured data\n",
"# - Enables complex queries and reasoning\n",
"# - Supports temporal tracking\n",
"# - Facilitates knowledge discovery\n",
"**Structure**:\n",
"- **Nodes**: Represent entities (people, places, concepts, events)\n",
"- **Edges**: Represent relationships (works_for, located_in, causes)\n",
"- **Properties**: Attributes of entities and relationships\n",
"- **Metadata**: Additional information (sources, timestamps, confidence)\n",
"\n",
"# 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n",
"# Definition: The process of identifying and classifying named entities\n",
"# in text into predefined categories.\n",
"#\n",
"# Entity Types:\n",
"# - Person: Names of people\n",
"# - Organization: Companies, institutions\n",
"# - Location: Places, geographic entities\n",
"# - Date/Time: Temporal expressions\n",
"# - Money: Monetary values\n",
"# - Product: Products and services\n",
"# - Event: Events and occurrences\n",
"# - Custom: Domain-specific entities\n",
"#\n",
"# Example:\n",
"# Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n",
"# Entities:\n",
"# - \"Apple Inc.\" -> Organization\n",
"# - \"Steve Jobs\" -> Person\n",
"# - \"Cupertino, California\" -> Location\n",
"#\n",
"# Methods:\n",
"# - Rule-based: Pattern matching\n",
"# - Machine Learning: Trained models (spaCy, transformers)\n",
"# - LLM-based: Using large language models\n",
"**Example**:\n",
"- Entity: \"John Doe\" (Person)\n",
"- Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n",
"- Properties: `{start_date: \"2020-01-01\", role: \"Engineer\"}`\n",
"\n",
"# 3. RELATIONSHIP EXTRACTION\n",
"# Definition: Identifying and extracting relationships between entities\n",
"# in text.\n",
"#\n",
"# Relationship Types:\n",
"# - Semantic: \"works_for\", \"located_in\", \"causes\"\n",
"# - Temporal: \"before\", \"after\", \"during\"\n",
"# - Causal: \"causes\", \"results_in\", \"prevents\"\n",
"# - Hierarchical: \"part_of\", \"subclass_of\", \"instance_of\"\n",
"#\n",
"# Example:\n",
"# Text: \"John works for Acme Corp in New York.\"\n",
"# Relationships:\n",
"# - (John, works_for, Acme Corp)\n",
"# - (Acme Corp, located_in, New York)\n",
"#\n",
"# Methods:\n",
"# - Pattern matching\n",
"# - Dependency parsing\n",
"# - Machine learning models\n",
"# - LLM-based extraction\n",
"**Benefits**:\n",
"- Structured representation of unstructured data\n",
"- Enables complex queries and reasoning\n",
"- Supports temporal tracking\n",
"- Facilitates knowledge discovery\n",
"\n",
"# 4. EMBEDDINGS\n",
"# Definition: Dense vector representations of text, images, or other data\n",
"# that capture semantic meaning in a continuous vector space.\n",
"#\n",
"# Properties:\n",
"# - Similar entities have similar embeddings (close in vector space)\n",
"# - Enable semantic search and similarity calculations\n",
"# - Fixed or variable dimensions (typically 128-4096)\n",
"#\n",
"# Example:\n",
"# Text: \"machine learning\"\n",
"# Embedding: [0.123, -0.456, 0.789, ..., 0.234] (vector of 1536 dimensions)\n",
"#\n",
"# Use Cases:\n",
"# - Semantic search\n",
"# - Clustering and classification\n",
"# - Recommendation systems\n",
"# - Anomaly detection\n",
"### 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n",
"**Definition**: The process of identifying and classifying named entities in text into predefined categories.\n",
"\n",
"# 5. TEMPORAL GRAPHS\n",
"# Definition: Knowledge graphs that track changes over time, allowing\n",
"# queries about the state of the graph at specific time points.\n",
"#\n",
"# Features:\n",
"# - Timestamps on entities and relationships\n",
"# - Version history\n",
"# - Time-point queries\n",
"# - Temporal pattern detection\n",
"#\n",
"# Example:\n",
"# Entity: \"Company X\"\n",
"# Relationship: (Company X, has_CEO, Person Y)\n",
"# Temporal: valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"\n",
"#\n",
"# Use Cases:\n",
"# - Tracking organizational changes\n",
"# - Monitoring system evolution\n",
"# - Analyzing trends over time\n",
"# - Historical analysis\n",
"**Entity Types**:\n",
"- **Person**: Names of people\n",
"- **Organization**: Companies, institutions\n",
"- **Location**: Places, geographic entities\n",
"- **Date/Time**: Temporal expressions\n",
"- **Money**: Monetary values\n",
"- **Product**: Products and services\n",
"- **Event**: Events and occurrences\n",
"- **Custom**: Domain-specific entities\n",
"\n",
"# 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n",
"# Definition: An advanced RAG approach that combines vector search with\n",
"# knowledge graph traversal to provide more accurate and\n",
"# contextually relevant information to LLMs.\n",
"#\n",
"# Components:\n",
"# - Vector Store: For semantic similarity search\n",
"# - Knowledge Graph: For structured relationship traversal\n",
"# - Hybrid Search: Combines both approaches\n",
"# - LLM Integration: Uses retrieved context for generation\n",
"#\n",
"# Advantages over Traditional RAG:\n",
"# - Better handling of complex queries\n",
"# - Relationship-aware retrieval\n",
"# - Reduced hallucinations\n",
"# - More accurate answers\n",
"#\n",
"# Example Workflow:\n",
"# 1. Query: \"Who worked with John at Acme Corp?\"\n",
"# 2. Vector search finds relevant documents\n",
"# 3. Knowledge graph traversal finds relationships\n",
"# 4. Combined context sent to LLM\n",
"# 5. LLM generates accurate answer using both sources\n",
"**Example**:\n",
"Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n",
"Entities:\n",
"- \"Apple Inc.\" -> Organization\n",
"- \"Steve Jobs\" -> Person\n",
"- \"Cupertino, California\" -> Location\n",
"\n",
"# 7. ONTOLOGY\n",
"# Definition: A formal specification of concepts, relationships, and\n",
"# constraints in a domain, typically expressed in OWL (Web\n",
"# Ontology Language).\n",
"#\n",
"# Components:\n",
"# - Classes: Categories of entities\n",
"# - Properties: Relationships and attributes\n",
"# - Individuals: Specific instances\n",
"# - Axioms: Rules and constraints\n",
"#\n",
"# Example:\n",
"# Class: Person\n",
"# SubClass: Employee, Customer\n",
"# Property: worksFor (domain: Person, range: Organization)\n",
"#\n",
"# Use Cases:\n",
"# - Standardize domain knowledge\n",
"# - Enable reasoning\n",
"# - Facilitate data integration\n",
"# - Support semantic web\n",
"**Methods**:\n",
"- Rule-based: Pattern matching\n",
"- Machine Learning: Trained models (spaCy, transformers)\n",
"- LLM-based: Using large language models\n",
"\n",
"# 8. QUALITY ASSURANCE\n",
"# Definition: Processes and metrics to ensure knowledge graph quality,\n",
"# including completeness, consistency, and accuracy.\n",
"#\n",
"# Metrics:\n",
"# - Completeness: Percentage of entities with required properties\n",
"# - Consistency: Absence of contradictions\n",
"# - Accuracy: Correctness of extracted information\n",
"# - Coverage: Breadth of domain coverage\n",
"#\n",
"# Methods:\n",
"# - Validation rules\n",
"# - Automated quality checks\n",
"# - Conflict detection\n",
"# - Source verification\n",
"'''\n",
"### 3. RELATIONSHIP EXTRACTION\n",
"**Definition**: Identifying and extracting relationships between entities in text.\n",
"\n",
"**Relationship Types**:\n",
"- **Semantic**: \"works_for\", \"located_in\", \"causes\"\n",
"- **Temporal**: \"before\", \"after\", \"during\"\n",
"- **Causal**: \"causes\", \"results_in\", \"prevents\"\n",
"- **Hierarchical**: \"part_of\", \"subclass_of\", \"instance_of\"\n",
"\n",
"**Example**:\n",
"Text: \"John works for Acme Corp in New York.\"\n",
"Relationships:\n",
"- (John, works_for, Acme Corp)\n",
"- (Acme Corp, located_in, New York)\n",
"\n",
"**Methods**:\n",
"- Pattern matching\n",
"- Dependency parsing\n",
"- Machine learning models\n",
"- LLM-based extraction\n",
"\n",
"### 4. EMBEDDINGS\n",
"**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space.\n",
"\n",
"**Properties**:\n",
"- Similar entities have similar embeddings (close in vector space)\n",
"- Enable semantic search and similarity calculations\n",
"- Fixed or variable dimensions (typically 128-4096)\n",
"\n",
"**Example**:\n",
"Text: \"machine learning\"\n",
"Embedding: `[0.123, -0.456, 0.789, ..., 0.234]` (vector of 1536 dimensions)\n",
"\n",
"**Use Cases**:\n",
"- Semantic search\n",
"- Clustering and classification\n",
"- Recommendation systems\n",
"- Anomaly detection\n",
"\n",
"### 5. TEMPORAL GRAPHS\n",
"**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points.\n",
"\n",
"**Features**:\n",
"- Timestamps on entities and relationships\n",
"- Version history\n",
"- Time-point queries\n",
"- Temporal pattern detection\n",
"\n",
"**Example**:\n",
"- Entity: \"Company X\"\n",
"- Relationship: (Company X, has_CEO, Person Y)\n",
"- Temporal: `valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"`\n",
"\n",
"**Use Cases**:\n",
"- Tracking organizational changes\n",
"- Monitoring system evolution\n",
"- Analyzing trends over time\n",
"- Historical analysis\n",
"\n",
"### 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n",
"**Definition**: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs.\n",
"\n",
"**Components**:\n",
"- **Vector Store**: For semantic similarity search\n",
"- **Knowledge Graph**: For structured relationship traversal\n",
"- **Hybrid Search**: Combines both approaches\n",
"- **LLM Integration**: Uses retrieved context for generation\n",
"\n",
"**Advantages over Traditional RAG**:\n",
"- Better handling of complex queries\n",
"- Relationship-aware retrieval\n",
"- Reduced hallucinations\n",
"- More accurate answers\n",
"\n",
"**Example Workflow**:\n",
"1. Query: \"Who worked with John at Acme Corp?\"\n",
"2. Vector search finds relevant documents\n",
"3. Knowledge graph traversal finds relationships\n",
"4. Combined context sent to LLM\n",
"5. LLM generates accurate answer using both sources\n",
"\n",
"### 7. ONTOLOGY\n",
"**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language).\n",
"\n",
"**Components**:\n",
"- **Classes**: Categories of entities\n",
"- **Properties**: Relationships and attributes\n",
"- **Individuals**: Specific instances\n",
"- **Axioms**: Rules and constraints\n",
"\n",
"**Example**:\n",
"- Class: Person\n",
"- SubClass: Employee, Customer\n",
"- Property: worksFor (domain: Person, range: Organization)\n",
"\n",
"**Use Cases**:\n",
"- Standardize domain knowledge\n",
"- Enable reasoning\n",
"- Facilitate data integration\n",
"- Support semantic web\n",
"\n",
"### 8. QUALITY ASSURANCE\n",
"**Definition**: Processes and metrics to ensure knowledge graph quality, including completeness, consistency, and accuracy.\n",
"\n",
"**Metrics**:\n",
"- **Completeness**: Percentage of entities with required properties\n",
"- **Consistency**: Absence of contradictions\n",
"- **Accuracy**: Correctness of extracted information\n",
"- **Coverage**: Breadth of domain coverage\n",
"\n",
"**Methods**:\n",
"- Validation rules\n",
"- Automated quality checks\n",
"- Conflict detection\n",
"- Source verification\n",
"\n",
"---\n",
"\n",
@@ -510,41 +511,35 @@
"\n",
"## Best Practices\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# BEST PRACTICES\n",
"# ============================================================================\n",
"### 1. START SMALL\n",
"- Begin with simple documents\n",
"- Validate each step before moving forward\n",
"- Build incrementally\n",
"\n",
"# 1. START SMALL\n",
"# - Begin with simple documents\n",
"# - Validate each step before moving forward\n",
"# - Build incrementally\n",
"### 2. CONFIGURE PROPERLY\n",
"- Use environment variables for sensitive data\n",
"- Set up proper logging\n",
"- Configure appropriate model sizes\n",
"\n",
"# 2. CONFIGURE PROPERLY\n",
"# - Use environment variables for sensitive data\n",
"# - Set up proper logging\n",
"# - Configure appropriate model sizes\n",
"### 3. VALIDATE DATA\n",
"- Always validate extracted entities\n",
"- Check relationship quality\n",
"- Use quality assurance tools\n",
"\n",
"# 3. VALIDATE DATA\n",
"# - Always validate extracted entities\n",
"# - Check relationship quality\n",
"# - Use quality assurance tools\n",
"### 4. HANDLE ERRORS\n",
"- Implement error handling\n",
"- Use retry mechanisms\n",
"- Log errors for debugging\n",
"\n",
"# 4. HANDLE ERRORS\n",
"# - Implement error handling\n",
"# - Use retry mechanisms\n",
"# - Log errors for debugging\n",
"### 5. OPTIMIZE PERFORMANCE\n",
"- Use batch processing for large datasets\n",
"- Enable parallel processing where possible\n",
"- Cache embeddings and results\n",
"\n",
"# 5. OPTIMIZE PERFORMANCE\n",
"# - Use batch processing for large datasets\n",
"# - Enable parallel processing where possible\n",
"# - Cache embeddings and results\n",
"\n",
"# 6. DOCUMENT YOUR WORKFLOWS\n",
"# - Document data sources\n",
"# - Track processing steps\n",
"# - Maintain metadata\n",
"'''\n",
"### 6. DOCUMENT YOUR WORKFLOWS\n",
"- Document data sources\n",
"- Track processing steps\n",
"- Maintain metadata\n",
"\n",
"---\n",
"\n",
@@ -552,45 +547,39 @@
"\n",
"Common issues and solutions:\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# TROUBLESHOOTING\n",
"# ============================================================================\n",
"### Issue 1: Import Errors\n",
"**Solution**:\n",
"- Ensure Semantica is properly installed\n",
"- Check Python version (3.8+)\n",
"- Verify virtual environment is activated\n",
"- Install missing dependencies: `pip install -r requirements.txt`\n",
"\n",
"# Issue 1: Import Errors\n",
"# Solution:\n",
"# - Ensure Semantica is properly installed\n",
"# - Check Python version (3.8+)\n",
"# - Verify virtual environment is activated\n",
"# - Install missing dependencies: pip install -r requirements.txt\n",
"### Issue 2: API Key Errors\n",
"**Solution**:\n",
"- Set environment variables: `export SEMANTICA_API_KEY=your_key`\n",
"- Check config file for correct key format\n",
"- Verify API key is valid and has sufficient credits\n",
"\n",
"# Issue 2: API Key Errors\n",
"# Solution:\n",
"# - Set environment variables: export SEMANTICA_API_KEY=your_key\n",
"# - Check config file for correct key format\n",
"# - Verify API key is valid and has sufficient credits\n",
"### Issue 3: Memory Issues\n",
"**Solution**:\n",
"- Process documents in batches\n",
"- Use smaller embedding models\n",
"- Enable garbage collection\n",
"- Consider using streaming for large datasets\n",
"\n",
"# Issue 3: Memory Issues\n",
"# Solution:\n",
"# - Process documents in batches\n",
"# - Use smaller embedding models\n",
"# - Enable garbage collection\n",
"# - Consider using streaming for large datasets\n",
"### Issue 4: Low Quality Extractions\n",
"**Solution**:\n",
"- Preprocess and normalize text\n",
"- Use domain-specific models\n",
"- Adjust extraction parameters\n",
"- Validate and clean extracted entities\n",
"\n",
"# Issue 4: Low Quality Extractions\n",
"# Solution:\n",
"# - Preprocess and normalize text\n",
"# - Use domain-specific models\n",
"# - Adjust extraction parameters\n",
"# - Validate and clean extracted entities\n",
"\n",
"# Issue 5: Slow Processing\n",
"# Solution:\n",
"# - Enable parallel processing\n",
"# - Use GPU acceleration if available\n",
"# - Cache intermediate results\n",
"# - Optimize batch sizes\n",
"'''\n"
"### Issue 5: Slow Processing\n",
"**Solution**:\n",
"- Enable parallel processing\n",
"- Use GPU acceleration if available\n",
"- Cache intermediate results\n",
"- Optimize batch sizes\n"
]
},
{
@@ -859,38 +848,6 @@
"print(\" generator = EmbeddingGenerator()\")\n",
"print(\" embeddings = generator.generate(documents)\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n"
]
}
],
"metadata": {
@@ -900,4 +857,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
cookbook/introduction/Your_First_Knowledge_Graph.ipynb
+35 -20
View File
@@ -4,37 +4,40 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Your First Knowledge Graph\n",
"# 🚀 Your First Knowledge Graph\n",
"\n",
"## Overview\n",
"\n",
"This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n",
"\n",
"### Learning Objectives\n",
"> [!TIP]\n",
"> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n",
"\n",
"- Understand the basic workflow: **File → Parse → Extract → Graph**\n",
"- Learn how to ingest documents using `FileIngestor`\n",
"- Parse documents using `DocumentParser`\n",
"- Extract entities using NER extractors\n",
"- Build a knowledge graph using `GraphBuilder`\n",
"- Visualize and analyze the resulting graph\n",
"### 🎯 Learning Objectives\n",
"\n",
"- **Understand the Workflow**: Learn the `File → Parse → Extract → Graph` pipeline\n",
"- **Ingest Data**: Load documents using `FileIngestor`\n",
"- **Parse Content**: Extract text using `DocumentParser`\n",
"- **Extract Knowledge**: Identify entities using `NERExtractor`\n",
"- **Build Graph**: Construct a graph using `GraphBuilder`\n",
"- **Visualize**: See your graph come to life with `KGVisualizer`\n",
"\n",
"---\n",
"\n",
"## Simple End-to-End Workflow\n",
"## 🔄 Simple End-to-End Workflow\n",
"\n",
"The complete workflow consists of four main steps:\n",
"\n",
"1. **Ingest** - Load data from files or other sources\n",
"2. **Parse** - Extract and structure content from documents\n",
"3. **Extract** - Identify entities and relationships\n",
"4. **Build Graph** - Construct the knowledge graph\n",
"1. **📥 Ingest** - Load data from files or other sources\n",
"2. **📄 Parse** - Extract and structure content from documents\n",
"3. **⛏️ Extract** - Identify entities and relationships\n",
"4. **🕸️ Build Graph** - Construct the knowledge graph\n",
"\n",
"Each step is demonstrated in the code cells below.\n",
"\n",
"---\n",
"\n",
"## Step 1: Ingest a File\n",
"## 📂 Step 1: Ingest a File\n",
"\n",
"In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n"
]
@@ -48,8 +51,10 @@
"from semantica.ingest import FileIngestor\n",
"from pathlib import Path\n",
"\n",
"# Initialize the ingestor\n",
"ingestor = FileIngestor()\n",
"\n",
"# Create a sample document for demonstration\n",
"sample_text = \"\"\"\n",
"Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
"The company is headquartered in Cupertino, California.\n",
@@ -64,6 +69,7 @@
"print(f\"File: {sample_file}\")\n",
"print(f\"Content length: {len(sample_text)} characters\")\n",
"\n",
"# Ingest the file\n",
"try:\n",
" file_object = ingestor.ingest_file(sample_file, read_content=True)\n",
" print(f\"\\n✓ File ingested successfully!\")\n",
@@ -78,7 +84,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Parse the Document\n",
"## 📄 Step 2: Parse the Document\n",
"\n",
"After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n"
]
@@ -94,12 +100,14 @@
"parser = DocumentParser()\n",
"\n",
"try:\n",
" # Parse the document to extract text\n",
" if 'file_object' in locals():\n",
" parsed_content = parser.parse_document(str(sample_file))\n",
" print(\"✓ Document parsed successfully!\")\n",
" print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n",
" print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n",
" else:\n",
" # Fallback if ingestion failed\n",
" parsed_content = parser.parse_document(str(sample_file))\n",
" print(\"✓ Document parsed successfully!\")\n",
" print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n",
@@ -113,9 +121,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Extract Entities\n",
"## ⛏️ Step 3: Extract Entities\n",
"\n",
"Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n"
"Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n",
"\n",
"> [!NOTE]\n",
"> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n"
]
},
{
@@ -133,6 +144,7 @@
" print(\"Extracting entities from text...\")\n",
" print(f\"\\nText: {parsed_content[:100]}...\")\n",
" \n",
" # Simulated extraction results\n",
" expected_entities = [\n",
" {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n",
" {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n",
@@ -156,7 +168,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Build the Knowledge Graph\n",
"## 🕸️ Step 4: Build the Knowledge Graph\n",
"\n",
"Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n"
]
@@ -172,6 +184,7 @@
"\n",
"builder = GraphBuilder()\n",
"\n",
"# Prepare data for graph construction\n",
"entities_data = [\n",
" {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n",
" for i, entity in enumerate(expected_entities)\n",
@@ -187,6 +200,7 @@
"]\n",
"\n",
"try:\n",
" # Build the graph using NetworkX\n",
" kg = nx.DiGraph()\n",
" \n",
" for entity in entities_data:\n",
@@ -221,7 +235,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Visualize and Analyze\n",
"## 📊 Step 5: Visualize and Analyze\n",
"\n",
"Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n"
]
@@ -268,6 +282,7 @@
"except Exception as e:\n",
" print(f\"✗ Error visualizing graph: {e}\")\n",
"\n",
"# Cleanup\n",
"try:\n",
" if sample_file.exists():\n",
" sample_file.unlink()\n",
@@ -284,4 +299,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
cookbook/use_cases/finance/Financial_Data_Integration.ipynb
+21 -20
View File
@@ -4,32 +4,33 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Financial Data Integration Pipeline\n",
"# 📈 Financial Data Integration Pipeline\n",
"\n",
"## Overview\n",
"\n",
"This notebook demonstrates how to integrate Python/FastMCP MCP servers as data sources for financial data ingestion. Connect to financial data MCP servers via URL, ingest market data, stock prices, and financial metrics, then build a knowledge graph for financial analysis.\n",
"\n",
"**IMPORTANT**: This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n",
"> [!IMPORTANT]\n",
"> This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n",
"\n",
"### Modules Used (20+)\n",
"### 🧩 Modules Used (20+)\n",
"\n",
"- **Ingestion**: MCPIngestor, ingest_mcp, WebIngestor, FileIngestor\n",
"- **Parsing**: MCPParser, JSONParser, StructuredDataParser\n",
"- **Extraction**: NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
"- **KG**: GraphBuilder, TemporalGraphQuery, GraphAnalyzer\n",
"- **Analytics**: CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
"- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
"- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
"- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
"- **Ingestion**: `MCPIngestor`, `ingest_mcp`, `WebIngestor`, `FileIngestor`\n",
"- **Parsing**: `MCPParser`, `JSONParser`, `StructuredDataParser`\n",
"- **Extraction**: `NERExtractor`, `RelationExtractor`, `EventDetector`, `SemanticAnalyzer`\n",
"- **KG**: `GraphBuilder`, `TemporalGraphQuery`, `GraphAnalyzer`\n",
"- **Analytics**: `CentralityCalculator`, `CommunityDetector`, `ConnectivityAnalyzer`\n",
"- **Reasoning**: `InferenceEngine`, `RuleManager`, `ExplanationGenerator`\n",
"- **Export**: `JSONExporter`, `CSVExporter`, `RDFExporter`, `ReportGenerator`\n",
"- **Visualization**: `KGVisualizer`, `TemporalVisualizer`, `AnalyticsVisualizer`\n",
"\n",
"### Pipeline\n",
"### 🔄 Pipeline\n",
"\n",
"**Connect to Financial MCP Server → Ingest Market Data via MCP → Parse MCP Responses → Extract Financial Entities → Build Financial KG → Analyze Trends → Generate Reports → Visualize**\n",
"\n",
"---\n",
"\n",
"## Step 1: Connect to Financial Data MCP Server\n",
"## 🔌 Step 1: Connect to Financial Data MCP Server\n",
"\n",
"Connect to a Python/FastMCP MCP server that provides financial data via URL. The MCP server can expose resources (datasets, market data) and tools (queries, calculations).\n"
]
@@ -92,7 +93,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Ingest Financial Data from MCP Server\n",
"## 📥 Step 2: Ingest Financial Data from MCP Server\n",
"\n",
"Ingest financial data using both resource-based and tool-based methods from the MCP server.\n"
]
@@ -220,7 +221,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Parse MCP Data\n",
"## 📄 Step 3: Parse MCP Data\n",
"\n",
"Parse the data received from MCP server responses (JSON, structured data).\n"
]
@@ -266,7 +267,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Extract Financial Entities and Relationships\n",
"## ⛏️ Step 4: Extract Financial Entities and Relationships\n",
"\n",
"Extract financial entities (companies, stocks, sectors) and relationships from MCP data.\n"
]
@@ -363,7 +364,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Build Financial Knowledge Graph\n",
"## 🕸️ Step 5: Build Financial Knowledge Graph\n",
"\n",
"Build a knowledge graph from the extracted financial entities and relationships.\n"
]
@@ -404,7 +405,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: Analyze Financial Trends\n",
"## 📊 Step 6: Analyze Financial Trends\n",
"\n",
"Analyze financial trends using temporal queries and pattern detection.\n"
]
@@ -462,7 +463,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 7: Export and Visualize\n",
"## 📤 Step 7: Export and Visualize\n",
"\n",
"Export the financial knowledge graph and generate visualizations.\n"
]
@@ -535,4 +536,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
docs/concepts.md
+142 -109
View File
@@ -1,20 +1,23 @@
# Core Concepts
Understand the fundamental concepts behind Semantica.
Understand the fundamental concepts behind Semantica. This guide covers the theoretical foundations, key components, and best practices for building semantic applications.
!!! note "Conceptual Foundation"
This guide covers the theoretical foundations of Semantica. For hands-on examples, see the [Quickstart Guide](quickstart.md) or [Examples](examples.md).
## 🧠 Core Concepts
## What is a Knowledge Graph?
### 1. Knowledge Graphs
A knowledge graph is a structured representation of information where:
**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes.
!!! tip "Visual Understanding"
The diagram below shows how entities (nodes) are connected by relationships (edges) to form a knowledge graph. This structure enables powerful querying and reasoning capabilities.
- **Nodes**: Represent entities (people, places, concepts, events)
- **Edges**: Represent relationships (works_for, located_in, causes)
- **Properties**: Attributes of entities and relationships
- **Metadata**: Additional information (sources, timestamps, confidence)
- **Entities** are the nodes (people, places, concepts, etc.)
- **Relationships** are the edges connecting entities
- **Properties** describe attributes of entities
**Benefits**:
- Structured representation of unstructured data
- Enables complex queries and reasoning
- Supports temporal tracking
- Facilitates knowledge discovery
```mermaid
graph LR
@@ -22,149 +25,179 @@ graph LR
A -->|located_in| C[Cupertino<br/>Location]
C -->|in_state| D[California<br/>Location]
style A fill:#e3f2fd
style B fill:#fff3e0
style C fill:#f3e5f5
style D fill:#f3e5f5
style A fill:#e3f2fd,stroke:#1565c0
style B fill:#fff3e0,stroke:#ef6c00
style C fill:#f3e5f5,stroke:#7b1fa2
style D fill:#f3e5f5,stroke:#7b1fa2
```
## Semantic Layer
### 2. Entity Extraction (NER)
A semantic layer provides:
**Definition**: The process of identifying and classifying named entities in text into predefined categories.
- **Structured meaning** from unstructured data
- **Contextual relationships** between concepts
- **Queryable knowledge** for AI systems
- **Quality-assured data** with conflict resolution
| Entity Type | Description | Example |
| :--- | :--- | :--- |
| **Person** | Names of people | Steve Jobs, Elon Musk |
| **Organization** | Companies, institutions | Apple Inc., NASA |
| **Location** | Places, geographic entities | Cupertino, Mars |
| **Date/Time** | Temporal expressions | 1976, next Monday |
| **Money** | Monetary values | $100 million |
| **Event** | Events and occurrences | WWDC 2024 |
## Key Components
### 1. Data Ingestion
Import data from various sources:
- Documents (PDF, DOCX, HTML)
- Databases
- APIs and web content
- Structured data (JSON, CSV)
### 2. Entity Extraction
Identify and extract:
- **Named Entities**: People, organizations, locations
- **Concepts**: Ideas, topics, themes
- **Events**: Actions, occurrences
- **Relations**: Connections between entities
**Methods**:
- **Rule-based**: Pattern matching (Regex)
- **Machine Learning**: Trained models (spaCy, transformers)
- **LLM-based**: Using large language models (GPT-4, Claude)
### 3. Relationship Extraction
Discover relationships:
**Definition**: Identifying and extracting relationships between entities in text.
- **Explicit**: Directly stated in text
- **Implicit**: Inferred from context
- **Temporal**: Time-based relationships
- **Causal**: Cause-and-effect connections
- **Semantic**: `works_for`, `located_in`, `causes`
- **Temporal**: `before`, `after`, `during`
- **Causal**: `causes`, `results_in`, `prevents`
- **Hierarchical**: `part_of`, `subclass_of`, `instance_of`
### 4. Knowledge Graph Construction
### 4. Embeddings
Build structured graphs:
**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space.
- **Node creation**: Entities as nodes
- **Edge creation**: Relationships as edges
- **Property assignment**: Attributes and metadata
- **Graph validation**: Quality checks
- **Properties**:
- Similar entities have similar embeddings (close in vector space).
- Enable semantic search and similarity calculations.
- Fixed dimensions (typically 128-4096).
### 5. Conflict Resolution
**Example**:
```python
Text: "machine learning"
Embedding: [0.123, -0.456, 0.789, ..., 0.234]
# (vector of 1536 dimensions)
```
Handle conflicting information:
### 5. Temporal Graphs
- **Multiple sources**: Same entity, different facts
- **Resolution strategies**: Voting, credibility, recency
- **Quality assurance**: Validation and verification
**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points.
### 6. Embedding Generation
- **Features**:
- Timestamps on entities and relationships
- Version history
- Time-point queries
- Temporal pattern detection
Create vector representations:
### 6. GraphRAG
- **Text embeddings**: Semantic text vectors
- **Graph embeddings**: Node and edge vectors
- **Multimodal**: Text, image, audio embeddings
**Definition**: An advanced RAG (Retrieval Augmented Generation) approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs.
## Workflow
Typical Semantica workflow:
**Advantages over Traditional RAG**:
- Better handling of complex queries
- Relationship-aware retrieval
- Reduced hallucinations
- More accurate answers
```mermaid
flowchart TD
A[Data Source] --> B[Ingestion]
B --> C[Parsing]
C --> D[Extraction<br/>Entities & Relationships]
D --> E[Normalization]
E --> F[Conflict Resolution]
F --> G[Knowledge Graph]
G --> H[Embeddings]
H --> I[Export]
Q[User Query] --> VS[Vector Search]
Q --> KG[Graph Traversal]
VS --> C[Context]
KG --> C
C --> LLM[LLM Generation]
LLM --> A[Answer]
style A fill:#e3f2fd
style G fill:#c8e6c9
style I fill:#fff9c4
style Q fill:#e1f5fe
style LLM fill:#e8f5e9
style A fill:#fff9c4
```
## Use Cases
### 7. Ontology
### GraphRAG
**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language).
Enhance RAG systems with knowledge graphs:
- **Classes**: Categories of entities (e.g., `Person`, `Company`)
- **Properties**: Relationships and attributes (e.g., `worksFor`)
- **Individuals**: Specific instances (e.g., `John Doe`)
- **Axioms**: Rules and constraints
- **Context expansion**: Follow relationships
- **Multi-hop reasoning**: Traverse graph paths
- **Structured queries**: Query graph directly
### 8. Quality Assurance
### AI Agents
**Definition**: Processes and metrics to ensure knowledge graph quality.
Provide agents with:
- **Completeness**: Percentage of entities with required properties
- **Consistency**: Absence of contradictions
- **Accuracy**: Correctness of extracted information
- **Coverage**: Breadth of domain coverage
- **Persistent memory**: Knowledge graph as memory
- **Context understanding**: Semantic relationships
- **Action validation**: Check against knowledge
---
### Data Integration
## 🌟 Best Practices
Unify data from multiple sources:
- **Schema mapping**: Automatic schema discovery
- **Entity resolution**: Match entities across sources
- **Conflict resolution**: Handle contradictions
## Best Practices
!!! note "Best Practices"
Following these practices will help you build high-quality knowledge graphs and avoid common pitfalls.
Following these practices will help you build high-quality knowledge graphs and avoid common pitfalls.
### 1. Start Small
- Begin with simple documents.
- Validate each step before moving forward.
- Build incrementally.
Begin with a single document or small dataset to understand the workflow.
### 2. Configure Properly
- Use environment variables for sensitive data.
- Set up proper logging.
- Configure appropriate model sizes.
### 2. Iterate
### 3. Validate Data
- Always validate extracted entities.
- Check relationship quality.
- Use quality assurance tools.
Build knowledge graphs incrementally, refining as you learn.
### 4. Handle Errors
- Implement error handling.
- Use retry mechanisms.
- Log errors for debugging.
### 3. Validate
### 5. Optimize Performance
- Use batch processing for large datasets.
- Enable parallel processing where possible.
- Cache embeddings and results.
Always validate extracted entities and relationships.
### 6. Document Workflows
- Document data sources.
- Track processing steps.
- Maintain metadata.
### 4. Resolve Conflicts
---
Use appropriate conflict resolution strategies for your use case.
## 🔧 Troubleshooting
### 5. Export Regularly
Common issues and solutions:
Export your knowledge graphs for backup and analysis.
!!! failure "Import Errors"
**Solution**:
- Ensure Semantica is properly installed.
- Check Python version (3.8+).
- Verify virtual environment is activated.
- Install missing dependencies: `pip install -r requirements.txt`
## Next Steps
!!! failure "API Key Errors"
**Solution**:
- Set environment variables: `export SEMANTICA_API_KEY=your_key`
- Check config file for correct key format.
- Verify API key is valid and has sufficient credits.
- **[Quick Start](quickstart.md)** - Build your first knowledge graph
- **[Examples](examples.md)** - See real-world applications
- **[API Reference](api.md)** - Explore the full API
!!! failure "Memory Issues"
**Solution**:
- Process documents in batches.
- Use smaller embedding models.
- Enable garbage collection.
- Consider using streaming for large datasets.
!!! failure "Low Quality Extractions"
**Solution**:
- Preprocess and normalize text.
- Use domain-specific models.
- Adjust extraction parameters.
- Validate and clean extracted entities.
!!! failure "Slow Processing"
**Solution**:
- Enable parallel processing.
- Use GPU acceleration if available.
- Cache intermediate results.
- Optimize batch sizes.
docs/cookbook.md
+136
View File
@@ -0,0 +1,136 @@
# 🍳 Semantica Cookbook
Welcome to the **Semantica Cookbook**!
This collection of Jupyter notebooks is designed to take you from a beginner to an expert in building semantic AI applications. Whether you're looking for quick recipes or deep-dive tutorials, you'll find it here.
---
## 🏁 Introduction
Start here if you are new to the framework. These notebooks cover the essentials.
- **[Welcome to Semantica](cookbook/introduction/Welcome_to_Semantica.ipynb)**: An interactive introduction to the framework.
- **[Configuration Basics](cookbook/introduction/Configuration_Basics.ipynb)**: Learn how to configure API keys and settings.
- **[Your First Knowledge Graph](cookbook/introduction/Your_First_Knowledge_Graph.ipynb)**: Build a simple graph from scratch.
- **[Data Ingestion](cookbook/introduction/Data_Ingestion.ipynb)**: Loading data from files and web sources.
- **[Document Parsing](cookbook/introduction/Document_Parsing.ipynb)**: Extracting text from PDFs, DOCX, and more.
- **[Data Normalization](cookbook/introduction/Data_Normalization.ipynb)**: Cleaning and preparing text.
- **[Entity Extraction](cookbook/introduction/Entity_Extraction.ipynb)**: Identifying people, places, and organizations.
- **[Relation Extraction](cookbook/introduction/Relation_Extraction.ipynb)**: Finding connections between entities.
- **[Embedding Generation](cookbook/introduction/Embedding_Generation.ipynb)**: Creating vector representations.
- **[Vector Store](cookbook/introduction/Vector_Store.ipynb)**: Storing and searching vectors.
- **[Ontology](cookbook/introduction/Ontology.ipynb)**: Defining the structure of your knowledge.
- **[Conflict Detection](cookbook/introduction/Conflict_Detection.ipynb)**: Handling contradictory information.
- **[Deduplication](cookbook/introduction/Deduplication.ipynb)**: Merging duplicate entities.
- **[Building Knowledge Graphs](cookbook/introduction/Building_Knowledge_Graphs.ipynb)**: Putting it all together.
- **[Graph Analytics](cookbook/introduction/Graph_Analytics.ipynb)**: Analyzing graph structure.
- **[Graph Quality](cookbook/introduction/Graph_Quality.ipynb)**: Ensuring data quality.
- **[Visualization](cookbook/introduction/Visualization.ipynb)**: Visualizing your graphs.
- **[Export](cookbook/introduction/Export.ipynb)**: Exporting data to other formats.
---
## 🧠 Advanced Concepts
Deep dive into advanced features and customization.
- **[Advanced Extraction](cookbook/advanced/Advanced_Extraction.ipynb)**: Techniques for complex entity and relation extraction.
- **[Advanced Graph Analytics](cookbook/advanced/Advanced_Graph_Analytics.ipynb)**: In-depth graph analysis algorithms.
- **[Complete Visualization Suite](cookbook/advanced/Complete_Visualization_Suite.ipynb)**: Comprehensive guide to visualization tools.
- **[Conflict Resolution Strategies](cookbook/advanced/Conflict_Resolution_Strategies.ipynb)**: Advanced methods for resolving data conflicts.
- **[Multi-Format Export](cookbook/advanced/Multi_Format_Export.ipynb)**: Exporting data to various formats.
- **[Multi-Source Data Integration](cookbook/advanced/Multi_Source_Data_Integration.ipynb)**: Integrating data from disparate sources.
- **[Pipeline Orchestration](cookbook/advanced/Pipeline_Orchestration.ipynb)**: Building and managing complex pipelines.
- **[Reasoning and Inference](cookbook/advanced/Reasoning_and_Inference.ipynb)**: Applying logical reasoning to your graph.
- **[Semantic Layer Construction](cookbook/advanced/Semantic_Layer_Construction.ipynb)**: Building a robust semantic layer.
- **[Temporal Knowledge Graphs](cookbook/advanced/Temporal_Knowledge_Graphs.ipynb)**: Working with time-aware graphs.
- **[Text Chunking Strategies](cookbook/advanced/Text_Chunking_Strategies.ipynb)**: Optimizing text processing.
- **[Unstructured to Ontology](cookbook/advanced/Unstructured_to_Ontology.ipynb)**: Automatically generating ontologies from text.
---
## 💡 Use Cases
Real-world examples and applications across various industries.
### Advanced RAG
- **[GraphRAG Complete](cookbook/use_cases/advanced_rag/GraphRAG_Complete.ipynb)**: End-to-end implementation of Graph Retrieval Augmented Generation.
### Biomedical
- **[Drug Discovery Pipeline](cookbook/use_cases/biomedical/Drug_Discovery_Pipeline.ipynb)**: Accelerating drug discovery with knowledge graphs.
- **[Genomic Variant Analysis](cookbook/use_cases/biomedical/Genomic_Variant_Analysis.ipynb)**: Analyzing genomic variants and their implications.
### Blockchain
- **[DeFi Protocol Intelligence](cookbook/use_cases/blockchain/DeFi_Protocol_Intelligence.ipynb)**: Analyzing decentralized finance protocols.
- **[Transaction Network Analysis](cookbook/use_cases/blockchain/Transaction_Network_Analysis.ipynb)**: Investigating blockchain transaction networks.
### Cybersecurity
- **[Anomaly Detection Real-Time](cookbook/use_cases/cybersecurity/Anomaly_Detection_Real_Time.ipynb)**: Detecting anomalies in real-time streams.
- **[Incident Analysis](cookbook/use_cases/cybersecurity/Incident_Analysis.ipynb)**: Analyzing security incidents and breaches.
- **[Threat Correlation](cookbook/use_cases/cybersecurity/Threat_Correlation.ipynb)**: Correlating threats across different vectors.
- **[Threat Intelligence Hybrid RAG](cookbook/use_cases/cybersecurity/Threat_Intelligence_Hybrid_RAG.ipynb)**: Combining RAG with threat intelligence.
- **[Threat Intelligence Integration](cookbook/use_cases/cybersecurity/Threat_Intelligence_Integration.ipynb)**: Integrating threat feeds into a knowledge graph.
- **[Vulnerability Tracking](cookbook/use_cases/cybersecurity/Vulnerability_Tracking.ipynb)**: Tracking and managing system vulnerabilities.
### Finance
- **[Financial Data Integration](cookbook/use_cases/finance/Financial_Data_Integration.ipynb)**: Merging financial data from multiple sources.
- **[Financial Reports Analysis](cookbook/use_cases/finance/Financial_Reports_Analysis.ipynb)**: Extracting insights from financial reports.
- **[Fraud Detection](cookbook/use_cases/finance/Fraud_Detection.ipynb)**: Identifying fraudulent activities and patterns.
- **[Investment Analysis Hybrid RAG](cookbook/use_cases/finance/Investment_Analysis_Hybrid_RAG.ipynb)**: AI-powered investment analysis.
- **[Market Intelligence](cookbook/use_cases/finance/Market_Intelligence.ipynb)**: Gathering and analyzing market intelligence.
- **[Regulatory Compliance](cookbook/use_cases/finance/Regulatory_Compliance.ipynb)**: Ensuring compliance with financial regulations.
### Healthcare
- **[Clinical Reports Processing](cookbook/use_cases/healthcare/Clinical_Reports_Processing.ipynb)**: Processing and structuring clinical reports.
- **[Disease Network Analysis](cookbook/use_cases/healthcare/Disease_Network_Analysis.ipynb)**: Analyzing disease networks and comorbidities.
- **[Drug Interactions Analysis](cookbook/use_cases/healthcare/Drug_Interactions_Analysis.ipynb)**: Identifying potential drug interactions.
- **[Healthcare GraphRAG Hybrid](cookbook/use_cases/healthcare/Healthcare_GraphRAG_Hybrid.ipynb)**: Hybrid RAG for healthcare applications.
- **[Medical Database Integration](cookbook/use_cases/healthcare/Medical_Database_Integration.ipynb)**: Integrating medical databases.
- **[Medical Literature GraphRAG](cookbook/use_cases/healthcare/Medical_Literature_GraphRAG.ipynb)**: Querying medical literature with GraphRAG.
- **[Patient Records Temporal](cookbook/use_cases/healthcare/Patient_Records_Temporal.ipynb)**: Analyzing patient records over time.
### Intelligence
- **[Network Analysis Intelligence Reports](cookbook/use_cases/intelligence/Network_Analysis_Intelligence_Reports.ipynb)**: Analyzing intelligence reports for network insights.
### Renewable Energy
- **[Energy Market Analysis](cookbook/use_cases/renewable_energy/Energy_Market_Analysis.ipynb)**: Analyzing trends in the energy market.
- **[Environmental Impact](cookbook/use_cases/renewable_energy/Environmental_Impact.ipynb)**: Assessing environmental impact.
- **[Grid Management](cookbook/use_cases/renewable_energy/Grid_Management.ipynb)**: Optimizing power grid management.
- **[Resource Optimization](cookbook/use_cases/renewable_energy/Resource_Optimization.ipynb)**: Optimizing renewable resources.
- **[Supply Chain Analysis](cookbook/use_cases/renewable_energy/Supply_Chain_Analysis.ipynb)**: Analyzing the renewable energy supply chain.
### Supply Chain
- **[Supply Chain Data Integration](cookbook/use_cases/supply_chain/Supply_Chain_Data_Integration.ipynb)**: Integrating supply chain data.
- **[Supply Chain Risk Management](cookbook/use_cases/supply_chain/Supply_Chain_Risk_Management.ipynb)**: Managing and mitigating supply chain risks.
### Trading
- **[Market Data Analysis](cookbook/use_cases/trading/Market_Data_Analysis.ipynb)**: Analyzing trading market data.
- **[News Sentiment Analysis](cookbook/use_cases/trading/News_Sentiment_Analysis.ipynb)**: Analyzing news sentiment for trading signals.
- **[Real Time Market Data](cookbook/use_cases/trading/Real_Time_Market_Data.ipynb)**: Processing real-time market data.
- **[Real Time Monitoring](cookbook/use_cases/trading/Real_Time_Monitoring.ipynb)**: Monitoring trading systems in real-time.
- **[Risk Assessment](cookbook/use_cases/trading/Risk_Assessment.ipynb)**: Assessing trading risks.
- **[Strategy Backtesting](cookbook/use_cases/trading/Strategy_Backtesting.ipynb)**: Backtesting trading strategies.
---
## 🛠️ How to Run
To run these notebooks locally:
1. **Clone the repository**:
```bash
git clone https://github.com/Hawksight-AI/semantica.git
cd semantica
```
2. **Install dependencies**:
```bash
pip install -e .[all]
pip install jupyter
```
3. **Launch Jupyter**:
```bash
jupyter notebook
```
docs/cookbook/introduction/Welcome_to_Semantica.ipynb
+416 -459
View File
@@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Welcome to Semantica\n",
"# Welcome to Semantica: An interactive introduction to the framework\n",
"\n",
"## Overview\n",
"\n",
@@ -54,7 +54,7 @@
"\n",
"### Installation Methods\n",
"\n",
"'''\n",
"```bash\n",
"# Method 1: Install from PyPI (when available)\n",
"# pip install semantica\n",
"\n",
@@ -71,11 +71,11 @@
"# Verify installation\n",
"# import semantica\n",
"# print(semantica.__version__)\n",
"'''\n",
"```\n",
"\n",
"### Configuration\n",
"\n",
"'''\n",
"```bash\n",
"# Set up environment variables for API keys and configuration\n",
"# export SEMANTICA_API_KEY=your_openai_key\n",
"# export SEMANTICA_EMBEDDING_PROVIDER=openai\n",
@@ -92,7 +92,7 @@
"# knowledge_graph:\n",
"# backend: networkx # or neo4j, arangodb\n",
"# temporal: true\n",
"'''\n",
"```\n",
"\n",
"---\n",
"\n",
@@ -100,205 +100,223 @@
"\n",
"Semantica is organized into modular components, each handling a specific aspect of semantic processing:\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# CORE MODULES\n",
"# ============================================================================\n",
"### 1. INGEST MODULE - Data Ingestion\n",
"**Purpose**: Ingest data from various sources\n",
"**Components**:\n",
"- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n",
"- `WebIngestor`: Scrape and ingest web pages\n",
"- `FeedIngestor`: Process RSS/Atom feeds\n",
"- `StreamIngestor`: Real-time data streaming\n",
"- `DBIngestor`: Database queries and ingestion\n",
"- `EmailIngestor`: Process email messages\n",
"- `RepoIngestor`: Git repository analysis\n",
"\n",
"# 1. INGEST MODULE - Data Ingestion\n",
"# Purpose: Ingest data from various sources\n",
"# Components:\n",
"# - FileIngestor: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)\n",
"# - WebIngestor: Scrape and ingest web pages\n",
"# - FeedIngestor: Process RSS/Atom feeds\n",
"# - StreamIngestor: Real-time data streaming\n",
"# - DBIngestor: Database queries and ingestion\n",
"# - EmailIngestor: Process email messages\n",
"# - RepoIngestor: Git repository analysis\n",
"#\n",
"# Example:\n",
"# from semantica.ingest import FileIngestor, WebIngestor\n",
"# file_ingestor = FileIngestor()\n",
"# web_ingestor = WebIngestor()\n",
"# documents = file_ingestor.ingest(\"data/\")\n",
"# web_docs = web_ingestor.ingest(\"https://example.com\")\n",
"**Example**:\n",
"```python\n",
"from semantica.ingest import FileIngestor, WebIngestor\n",
"file_ingestor = FileIngestor()\n",
"web_ingestor = WebIngestor()\n",
"documents = file_ingestor.ingest(\"data/\")\n",
"web_docs = web_ingestor.ingest(\"https://example.com\")\n",
"```\n",
"\n",
"# 2. PARSE MODULE - Document Parsing\n",
"# Purpose: Parse and extract content from various formats\n",
"# Components:\n",
"# - DocumentParser: Main parser orchestrator\n",
"# - PDFParser: Extract text, tables, images from PDFs\n",
"# - DOCXParser: Parse Word documents\n",
"# - HTMLParser: Extract content from HTML\n",
"# - JSONParser: Parse structured JSON data\n",
"# - ExcelParser: Process spreadsheets\n",
"# - ImageParser: OCR and image analysis\n",
"# - CodeParser: Parse source code files\n",
"#\n",
"# Example:\n",
"# from semantica.parse import DocumentParser\n",
"# parser = DocumentParser()\n",
"# parsed_docs = parser.parse(documents)\n",
"### 2. PARSE MODULE - Document Parsing\n",
"**Purpose**: Parse and extract content from various formats\n",
"**Components**:\n",
"- `DocumentParser`: Main parser orchestrator\n",
"- `PDFParser`: Extract text, tables, images from PDFs\n",
"- `DOCXParser`: Parse Word documents\n",
"- `HTMLParser`: Extract content from HTML\n",
"- `JSONParser`: Parse structured JSON data\n",
"- `ExcelParser`: Process spreadsheets\n",
"- `ImageParser`: OCR and image analysis\n",
"- `CodeParser`: Parse source code files\n",
"\n",
"# 3. NORMALIZE MODULE - Text Normalization\n",
"# Purpose: Clean and normalize text for processing\n",
"# Components:\n",
"# - TextNormalizer: Main normalization orchestrator\n",
"# - TextCleaner: Remove noise, fix encoding\n",
"# - DataCleaner: Clean structured data\n",
"# - EntityNormalizer: Normalize entity names\n",
"# - DateNormalizer: Standardize date formats\n",
"# - NumberNormalizer: Normalize numeric values\n",
"# - LanguageDetector: Detect document language\n",
"# - EncodingHandler: Handle character encoding\n",
"#\n",
"# Example:\n",
"# from semantica.normalize import TextNormalizer\n",
"# normalizer = TextNormalizer()\n",
"# normalized = normalizer.normalize(parsed_docs)\n",
"**Example**:\n",
"```python\n",
"from semantica.parse import DocumentParser\n",
"parser = DocumentParser()\n",
"parsed_docs = parser.parse(documents)\n",
"```\n",
"\n",
"# 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n",
"# Purpose: Extract entities, relationships, and semantic information\n",
"# Components:\n",
"# - NERExtractor: Named Entity Recognition\n",
"# - RelationExtractor: Extract relationships between entities\n",
"# - SemanticAnalyzer: Deep semantic analysis\n",
"# - SemanticNetworkExtractor: Extract semantic networks\n",
"#\n",
"# Example:\n",
"# from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
"# extractor = NERExtractor()\n",
"# entities = extractor.extract(normalized_docs)\n",
"# relation_extractor = RelationExtractor()\n",
"# relationships = relation_extractor.extract(normalized_docs, entities)\n",
"### 3. NORMALIZE MODULE - Text Normalization\n",
"**Purpose**: Clean and normalize text for processing\n",
"**Components**:\n",
"- `TextNormalizer`: Main normalization orchestrator\n",
"- `TextCleaner`: Remove noise, fix encoding\n",
"- `DataCleaner`: Clean structured data\n",
"- `EntityNormalizer`: Normalize entity names\n",
"- `DateNormalizer`: Standardize date formats\n",
"- `NumberNormalizer`: Normalize numeric values\n",
"- `LanguageDetector`: Detect document language\n",
"- `EncodingHandler`: Handle character encoding\n",
"\n",
"# 5. KG MODULE - Knowledge Graph Construction\n",
"# Purpose: Build and manage knowledge graphs\n",
"# Components:\n",
"# - GraphBuilder: Construct knowledge graphs from entities/relationships\n",
"# - GraphAnalyzer: Analyze graph structure and properties\n",
"# - GraphValidator: Validate graph quality and consistency\n",
"# - EntityResolver: Resolve entity conflicts and duplicates\n",
"# - ConflictDetector: Detect conflicting information\n",
"# - CentralityCalculator: Calculate node importance metrics\n",
"# - CommunityDetector: Detect communities in graphs\n",
"# - ConnectivityAnalyzer: Analyze graph connectivity\n",
"# - TemporalQuery: Query temporal knowledge graphs\n",
"# - Deduplicator: Remove duplicate entities/relationships\n",
"#\n",
"# Example:\n",
"# from semantica.kg import GraphBuilder, GraphAnalyzer\n",
"# builder = GraphBuilder()\n",
"# kg = builder.build(entities, relationships)\n",
"# analyzer = GraphAnalyzer()\n",
"# metrics = analyzer.analyze(kg)\n",
"**Example**:\n",
"```python\n",
"from semantica.normalize import TextNormalizer\n",
"normalizer = TextNormalizer()\n",
"normalized = normalizer.normalize(parsed_docs)\n",
"```\n",
"\n",
"# 6. EMBEDDINGS MODULE - Embedding Generation\n",
"# Purpose: Generate vector embeddings for various data types\n",
"# Components:\n",
"# - EmbeddingGenerator: Main embedding orchestrator\n",
"# - TextEmbedder: Generate text embeddings\n",
"# - ImageEmbedder: Generate image embeddings\n",
"# - AudioEmbedder: Generate audio embeddings\n",
"# - MultimodalEmbedder: Combine multiple modalities\n",
"# - EmbeddingOptimizer: Optimize embedding quality\n",
"# - ProviderAdapters: Support for OpenAI, Cohere, etc.\n",
"#\n",
"# Example:\n",
"# from semantica.embeddings import EmbeddingGenerator\n",
"# generator = EmbeddingGenerator()\n",
"# embeddings = generator.generate(documents)\n",
"### 4. SEMANTIC_EXTRACT MODULE - Entity & Relationship Extraction\n",
"**Purpose**: Extract entities, relationships, and semantic information\n",
"**Components**:\n",
"- `NERExtractor`: Named Entity Recognition\n",
"- `RelationExtractor`: Extract relationships between entities\n",
"- `SemanticAnalyzer`: Deep semantic analysis\n",
"- `SemanticNetworkExtractor`: Extract semantic networks\n",
"\n",
"# 7. VECTOR_STORE MODULE - Vector Database Operations\n",
"# Purpose: Store and search vector embeddings\n",
"# Components:\n",
"# - VectorStore: Main vector store interface\n",
"# - FAISSAdapter: FAISS integration\n",
"# - HybridSearch: Combine vector and keyword search\n",
"# - VectorRetriever: Retrieve relevant vectors\n",
"#\n",
"# Example:\n",
"# from semantica.vector_store import VectorStore, HybridSearch\n",
"# vector_store = VectorStore()\n",
"# vector_store.store(embeddings, documents, metadata)\n",
"# hybrid_search = HybridSearch(vector_store)\n",
"# results = hybrid_search.search(query, top_k=10)\n",
"**Example**:\n",
"```python\n",
"from semantica.semantic_extract import NERExtractor, RelationExtractor\n",
"extractor = NERExtractor()\n",
"entities = extractor.extract(normalized_docs)\n",
"relation_extractor = RelationExtractor()\n",
"relationships = relation_extractor.extract(normalized_docs, entities)\n",
"```\n",
"\n",
"# 8. REASONING MODULE - Inference and Reasoning\n",
"# Purpose: Perform logical inference and reasoning\n",
"# Components:\n",
"# - InferenceEngine: Main inference orchestrator\n",
"# - RuleManager: Manage inference rules\n",
"# - DeductiveReasoner: Deductive reasoning\n",
"# - AbductiveReasoner: Abductive reasoning\n",
"# - ExplanationGenerator: Generate explanations for inferences\n",
"# - RETEEngine: RETE algorithm for rule matching\n",
"#\n",
"# Example:\n",
"# from semantica.reasoning import InferenceEngine, RuleManager\n",
"# inference_engine = InferenceEngine()\n",
"# rule_manager = RuleManager()\n",
"# new_facts = inference_engine.forward_chain(kg, rule_manager)\n",
"### 5. KG MODULE - Knowledge Graph Construction\n",
"**Purpose**: Build and manage knowledge graphs\n",
"**Components**:\n",
"- `GraphBuilder`: Construct knowledge graphs from entities/relationships\n",
"- `GraphAnalyzer`: Analyze graph structure and properties\n",
"- `GraphValidator`: Validate graph quality and consistency\n",
"- `EntityResolver`: Resolve entity conflicts and duplicates\n",
"- `ConflictDetector`: Detect conflicting information\n",
"- `CentralityCalculator`: Calculate node importance metrics\n",
"- `CommunityDetector`: Detect communities in graphs\n",
"- `ConnectivityAnalyzer`: Analyze graph connectivity\n",
"- `TemporalQuery`: Query temporal knowledge graphs\n",
"- `Deduplicator`: Remove duplicate entities/relationships\n",
"\n",
"# 9. ONTOLOGY MODULE - Ontology Generation\n",
"# Purpose: Generate and manage ontologies\n",
"# Components:\n",
"# - OntologyGenerator: Generate ontologies from knowledge graphs\n",
"# - OntologyValidator: Validate ontology structure\n",
"# - OWLGenerator: Generate OWL format ontologies\n",
"# - PropertyGenerator: Generate ontology properties\n",
"# - ClassInferrer: Infer ontology classes\n",
"#\n",
"# Example:\n",
"# from semantica.ontology import OntologyGenerator\n",
"# generator = OntologyGenerator()\n",
"# ontology = generator.generate_from_graph(kg)\n",
"**Example**:\n",
"```python\n",
"from semantica.kg import GraphBuilder, GraphAnalyzer\n",
"builder = GraphBuilder()\n",
"kg = builder.build(entities, relationships)\n",
"analyzer = GraphAnalyzer()\n",
"metrics = analyzer.analyze(kg)\n",
"```\n",
"\n",
"# 10. EXPORT MODULE - Data Export\n",
"# Purpose: Export data in various formats\n",
"# Components:\n",
"# - JSONExporter: Export to JSON\n",
"# - RDFExporter: Export to RDF/XML\n",
"# - CSVExporter: Export to CSV\n",
"# - GraphExporter: Export to graph formats (GraphML, GEXF)\n",
"# - OWLExporter: Export to OWL\n",
"# - VectorExporter: Export vectors\n",
"#\n",
"# Example:\n",
"# from semantica.export import JSONExporter, RDFExporter\n",
"# json_exporter = JSONExporter()\n",
"# json_exporter.export(kg, \"output.json\")\n",
"### 6. EMBEDDINGS MODULE - Embedding Generation\n",
"**Purpose**: Generate vector embeddings for various data types\n",
"**Components**:\n",
"- `EmbeddingGenerator`: Main embedding orchestrator\n",
"- `TextEmbedder`: Generate text embeddings\n",
"- `ImageEmbedder`: Generate image embeddings\n",
"- `AudioEmbedder`: Generate audio embeddings\n",
"- `MultimodalEmbedder`: Combine multiple modalities\n",
"- `EmbeddingOptimizer`: Optimize embedding quality\n",
"- `ProviderAdapters`: Support for OpenAI, Cohere, etc.\n",
"\n",
"# 11. VISUALIZATION MODULE - Graph Visualization\n",
"# Purpose: Visualize knowledge graphs and analytics\n",
"# Components:\n",
"# - KGVisualizer: Visualize knowledge graphs\n",
"# - EmbeddingVisualizer: Visualize embeddings (t-SNE, PCA, UMAP)\n",
"# - QualityVisualizer: Visualize quality metrics\n",
"# - AnalyticsVisualizer: Visualize graph analytics\n",
"# - TemporalVisualizer: Visualize temporal data\n",
"#\n",
"# Example:\n",
"# from semantica.visualization import KGVisualizer\n",
"# visualizer = KGVisualizer()\n",
"# visualizer.visualize(kg)\n",
"**Example**:\n",
"```python\n",
"from semantica.embeddings import EmbeddingGenerator\n",
"generator = EmbeddingGenerator()\n",
"embeddings = generator.generate(documents)\n",
"```\n",
"\n",
"# 12. PIPELINE MODULE - Pipeline Orchestration\n",
"# Purpose: Build and execute processing pipelines\n",
"# Components:\n",
"# - PipelineBuilder: Build complex pipelines\n",
"# - ExecutionEngine: Execute pipelines\n",
"# - FailureHandler: Handle pipeline failures\n",
"# - ParallelismManager: Enable parallel processing\n",
"# - ResourceScheduler: Schedule resources\n",
"#\n",
"# Example:\n",
"# from semantica.pipeline import PipelineBuilder\n",
"# builder = PipelineBuilder()\n",
"# pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n",
"# .add_step(\"parse\", DocumentParser()) \\\\\n",
"# .build()\n",
"'''\n",
"### 7. VECTOR_STORE MODULE - Vector Database Operations\n",
"**Purpose**: Store and search vector embeddings\n",
"**Components**:\n",
"- `VectorStore`: Main vector store interface\n",
"- `FAISSAdapter`: FAISS integration\n",
"- `HybridSearch`: Combine vector and keyword search\n",
"- `VectorRetriever`: Retrieve relevant vectors\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.vector_store import VectorStore, HybridSearch\n",
"vector_store = VectorStore()\n",
"vector_store.store(embeddings, documents, metadata)\n",
"hybrid_search = HybridSearch(vector_store)\n",
"results = hybrid_search.search(query, top_k=10)\n",
"```\n",
"\n",
"### 8. REASONING MODULE - Inference and Reasoning\n",
"**Purpose**: Perform logical inference and reasoning\n",
"**Components**:\n",
"- `InferenceEngine`: Main inference orchestrator\n",
"- `RuleManager`: Manage inference rules\n",
"- `DeductiveReasoner`: Deductive reasoning\n",
"- `AbductiveReasoner`: Abductive reasoning\n",
"- `ExplanationGenerator`: Generate explanations for inferences\n",
"- `RETEEngine`: RETE algorithm for rule matching\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.reasoning import InferenceEngine, RuleManager\n",
"inference_engine = InferenceEngine()\n",
"rule_manager = RuleManager()\n",
"new_facts = inference_engine.forward_chain(kg, rule_manager)\n",
"```\n",
"\n",
"### 9. ONTOLOGY MODULE - Ontology Generation\n",
"**Purpose**: Generate and manage ontologies\n",
"**Components**:\n",
"- `OntologyGenerator`: Generate ontologies from knowledge graphs\n",
"- `OntologyValidator`: Validate ontology structure\n",
"- `OWLGenerator`: Generate OWL format ontologies\n",
"- `PropertyGenerator`: Generate ontology properties\n",
"- `ClassInferrer`: Infer ontology classes\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.ontology import OntologyGenerator\n",
"generator = OntologyGenerator()\n",
"ontology = generator.generate_from_graph(kg)\n",
"```\n",
"\n",
"### 10. EXPORT MODULE - Data Export\n",
"**Purpose**: Export data in various formats\n",
"**Components**:\n",
"- `JSONExporter`: Export to JSON\n",
"- `RDFExporter`: Export to RDF/XML\n",
"- `CSVExporter`: Export to CSV\n",
"- `GraphExporter`: Export to graph formats (GraphML, GEXF)\n",
"- `OWLExporter`: Export to OWL\n",
"- `VectorExporter`: Export vectors\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.export import JSONExporter, RDFExporter\n",
"json_exporter = JSONExporter()\n",
"json_exporter.export(kg, \"output.json\")\n",
"```\n",
"\n",
"### 11. VISUALIZATION MODULE - Graph Visualization\n",
"**Purpose**: Visualize knowledge graphs and analytics\n",
"**Components**:\n",
"- `KGVisualizer`: Visualize knowledge graphs\n",
"- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP)\n",
"- `QualityVisualizer`: Visualize quality metrics\n",
"- `AnalyticsVisualizer`: Visualize graph analytics\n",
"- `TemporalVisualizer`: Visualize temporal data\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.visualization import KGVisualizer\n",
"visualizer = KGVisualizer()\n",
"visualizer.visualize(kg)\n",
"```\n",
"\n",
"### 12. PIPELINE MODULE - Pipeline Orchestration\n",
"**Purpose**: Build and execute processing pipelines\n",
"**Components**:\n",
"- `PipelineBuilder`: Build complex pipelines\n",
"- `ExecutionEngine`: Execute pipelines\n",
"- `FailureHandler`: Handle pipeline failures\n",
"- `ParallelismManager`: Enable parallel processing\n",
"- `ResourceScheduler`: Schedule resources\n",
"\n",
"**Example**:\n",
"```python\n",
"from semantica.pipeline import PipelineBuilder\n",
"builder = PipelineBuilder()\n",
"pipeline = builder.add_step(\"ingest\", FileIngestor()) \\\\\n",
" .add_step(\"parse\", DocumentParser()) \\\\\n",
" .build()\n",
"```\n",
"\n",
"---\n",
"\n",
@@ -306,183 +324,166 @@
"\n",
"Understanding these concepts is crucial for working with Semantica:\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# CORE CONCEPTS\n",
"# ============================================================================\n",
"### 1. KNOWLEDGE GRAPHS\n",
"**Definition**: A knowledge graph is a structured representation of entities (nodes) and their relationships (edges) with properties and attributes.\n",
"\n",
"# 1. KNOWLEDGE GRAPHS\n",
"# Definition: A knowledge graph is a structured representation of entities\n",
"# (nodes) and their relationships (edges) with properties and\n",
"# attributes.\n",
"#\n",
"# Structure:\n",
"# - Nodes: Represent entities (people, places, concepts, events)\n",
"# - Edges: Represent relationships (works_for, located_in, causes)\n",
"# - Properties: Attributes of entities and relationships\n",
"# - Metadata: Additional information (sources, timestamps, confidence)\n",
"#\n",
"# Example:\n",
"# Entity: \"John Doe\" (Person)\n",
"# Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n",
"# Properties: {start_date: \"2020-01-01\", role: \"Engineer\"}\n",
"#\n",
"# Benefits:\n",
"# - Structured representation of unstructured data\n",
"# - Enables complex queries and reasoning\n",
"# - Supports temporal tracking\n",
"# - Facilitates knowledge discovery\n",
"**Structure**:\n",
"- **Nodes**: Represent entities (people, places, concepts, events)\n",
"- **Edges**: Represent relationships (works_for, located_in, causes)\n",
"- **Properties**: Attributes of entities and relationships\n",
"- **Metadata**: Additional information (sources, timestamps, confidence)\n",
"\n",
"# 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n",
"# Definition: The process of identifying and classifying named entities\n",
"# in text into predefined categories.\n",
"#\n",
"# Entity Types:\n",
"# - Person: Names of people\n",
"# - Organization: Companies, institutions\n",
"# - Location: Places, geographic entities\n",
"# - Date/Time: Temporal expressions\n",
"# - Money: Monetary values\n",
"# - Product: Products and services\n",
"# - Event: Events and occurrences\n",
"# - Custom: Domain-specific entities\n",
"#\n",
"# Example:\n",
"# Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n",
"# Entities:\n",
"# - \"Apple Inc.\" -> Organization\n",
"# - \"Steve Jobs\" -> Person\n",
"# - \"Cupertino, California\" -> Location\n",
"#\n",
"# Methods:\n",
"# - Rule-based: Pattern matching\n",
"# - Machine Learning: Trained models (spaCy, transformers)\n",
"# - LLM-based: Using large language models\n",
"**Example**:\n",
"- Entity: \"John Doe\" (Person)\n",
"- Relationship: \"works_for\" -> \"Acme Corp\" (Organization)\n",
"- Properties: `{start_date: \"2020-01-01\", role: \"Engineer\"}`\n",
"\n",
"# 3. RELATIONSHIP EXTRACTION\n",
"# Definition: Identifying and extracting relationships between entities\n",
"# in text.\n",
"#\n",
"# Relationship Types:\n",
"# - Semantic: \"works_for\", \"located_in\", \"causes\"\n",
"# - Temporal: \"before\", \"after\", \"during\"\n",
"# - Causal: \"causes\", \"results_in\", \"prevents\"\n",
"# - Hierarchical: \"part_of\", \"subclass_of\", \"instance_of\"\n",
"#\n",
"# Example:\n",
"# Text: \"John works for Acme Corp in New York.\"\n",
"# Relationships:\n",
"# - (John, works_for, Acme Corp)\n",
"# - (Acme Corp, located_in, New York)\n",
"#\n",
"# Methods:\n",
"# - Pattern matching\n",
"# - Dependency parsing\n",
"# - Machine learning models\n",
"# - LLM-based extraction\n",
"**Benefits**:\n",
"- Structured representation of unstructured data\n",
"- Enables complex queries and reasoning\n",
"- Supports temporal tracking\n",
"- Facilitates knowledge discovery\n",
"\n",
"# 4. EMBEDDINGS\n",
"# Definition: Dense vector representations of text, images, or other data\n",
"# that capture semantic meaning in a continuous vector space.\n",
"#\n",
"# Properties:\n",
"# - Similar entities have similar embeddings (close in vector space)\n",
"# - Enable semantic search and similarity calculations\n",
"# - Fixed or variable dimensions (typically 128-4096)\n",
"#\n",
"# Example:\n",
"# Text: \"machine learning\"\n",
"# Embedding: [0.123, -0.456, 0.789, ..., 0.234] (vector of 1536 dimensions)\n",
"#\n",
"# Use Cases:\n",
"# - Semantic search\n",
"# - Clustering and classification\n",
"# - Recommendation systems\n",
"# - Anomaly detection\n",
"### 2. ENTITY EXTRACTION (NER - Named Entity Recognition)\n",
"**Definition**: The process of identifying and classifying named entities in text into predefined categories.\n",
"\n",
"# 5. TEMPORAL GRAPHS\n",
"# Definition: Knowledge graphs that track changes over time, allowing\n",
"# queries about the state of the graph at specific time points.\n",
"#\n",
"# Features:\n",
"# - Timestamps on entities and relationships\n",
"# - Version history\n",
"# - Time-point queries\n",
"# - Temporal pattern detection\n",
"#\n",
"# Example:\n",
"# Entity: \"Company X\"\n",
"# Relationship: (Company X, has_CEO, Person Y)\n",
"# Temporal: valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"\n",
"#\n",
"# Use Cases:\n",
"# - Tracking organizational changes\n",
"# - Monitoring system evolution\n",
"# - Analyzing trends over time\n",
"# - Historical analysis\n",
"**Entity Types**:\n",
"- **Person**: Names of people\n",
"- **Organization**: Companies, institutions\n",
"- **Location**: Places, geographic entities\n",
"- **Date/Time**: Temporal expressions\n",
"- **Money**: Monetary values\n",
"- **Product**: Products and services\n",
"- **Event**: Events and occurrences\n",
"- **Custom**: Domain-specific entities\n",
"\n",
"# 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n",
"# Definition: An advanced RAG approach that combines vector search with\n",
"# knowledge graph traversal to provide more accurate and\n",
"# contextually relevant information to LLMs.\n",
"#\n",
"# Components:\n",
"# - Vector Store: For semantic similarity search\n",
"# - Knowledge Graph: For structured relationship traversal\n",
"# - Hybrid Search: Combines both approaches\n",
"# - LLM Integration: Uses retrieved context for generation\n",
"#\n",
"# Advantages over Traditional RAG:\n",
"# - Better handling of complex queries\n",
"# - Relationship-aware retrieval\n",
"# - Reduced hallucinations\n",
"# - More accurate answers\n",
"#\n",
"# Example Workflow:\n",
"# 1. Query: \"Who worked with John at Acme Corp?\"\n",
"# 2. Vector search finds relevant documents\n",
"# 3. Knowledge graph traversal finds relationships\n",
"# 4. Combined context sent to LLM\n",
"# 5. LLM generates accurate answer using both sources\n",
"**Example**:\n",
"Text: \"Apple Inc. was founded by Steve Jobs in Cupertino, California.\"\n",
"Entities:\n",
"- \"Apple Inc.\" -> Organization\n",
"- \"Steve Jobs\" -> Person\n",
"- \"Cupertino, California\" -> Location\n",
"\n",
"# 7. ONTOLOGY\n",
"# Definition: A formal specification of concepts, relationships, and\n",
"# constraints in a domain, typically expressed in OWL (Web\n",
"# Ontology Language).\n",
"#\n",
"# Components:\n",
"# - Classes: Categories of entities\n",
"# - Properties: Relationships and attributes\n",
"# - Individuals: Specific instances\n",
"# - Axioms: Rules and constraints\n",
"#\n",
"# Example:\n",
"# Class: Person\n",
"# SubClass: Employee, Customer\n",
"# Property: worksFor (domain: Person, range: Organization)\n",
"#\n",
"# Use Cases:\n",
"# - Standardize domain knowledge\n",
"# - Enable reasoning\n",
"# - Facilitate data integration\n",
"# - Support semantic web\n",
"**Methods**:\n",
"- Rule-based: Pattern matching\n",
"- Machine Learning: Trained models (spaCy, transformers)\n",
"- LLM-based: Using large language models\n",
"\n",
"# 8. QUALITY ASSURANCE\n",
"# Definition: Processes and metrics to ensure knowledge graph quality,\n",
"# including completeness, consistency, and accuracy.\n",
"#\n",
"# Metrics:\n",
"# - Completeness: Percentage of entities with required properties\n",
"# - Consistency: Absence of contradictions\n",
"# - Accuracy: Correctness of extracted information\n",
"# - Coverage: Breadth of domain coverage\n",
"#\n",
"# Methods:\n",
"# - Validation rules\n",
"# - Automated quality checks\n",
"# - Conflict detection\n",
"# - Source verification\n",
"'''\n",
"### 3. RELATIONSHIP EXTRACTION\n",
"**Definition**: Identifying and extracting relationships between entities in text.\n",
"\n",
"**Relationship Types**:\n",
"- **Semantic**: \"works_for\", \"located_in\", \"causes\"\n",
"- **Temporal**: \"before\", \"after\", \"during\"\n",
"- **Causal**: \"causes\", \"results_in\", \"prevents\"\n",
"- **Hierarchical**: \"part_of\", \"subclass_of\", \"instance_of\"\n",
"\n",
"**Example**:\n",
"Text: \"John works for Acme Corp in New York.\"\n",
"Relationships:\n",
"- (John, works_for, Acme Corp)\n",
"- (Acme Corp, located_in, New York)\n",
"\n",
"**Methods**:\n",
"- Pattern matching\n",
"- Dependency parsing\n",
"- Machine learning models\n",
"- LLM-based extraction\n",
"\n",
"### 4. EMBEDDINGS\n",
"**Definition**: Dense vector representations of text, images, or other data that capture semantic meaning in a continuous vector space.\n",
"\n",
"**Properties**:\n",
"- Similar entities have similar embeddings (close in vector space)\n",
"- Enable semantic search and similarity calculations\n",
"- Fixed or variable dimensions (typically 128-4096)\n",
"\n",
"**Example**:\n",
"Text: \"machine learning\"\n",
"Embedding: `[0.123, -0.456, 0.789, ..., 0.234]` (vector of 1536 dimensions)\n",
"\n",
"**Use Cases**:\n",
"- Semantic search\n",
"- Clustering and classification\n",
"- Recommendation systems\n",
"- Anomaly detection\n",
"\n",
"### 5. TEMPORAL GRAPHS\n",
"**Definition**: Knowledge graphs that track changes over time, allowing queries about the state of the graph at specific time points.\n",
"\n",
"**Features**:\n",
"- Timestamps on entities and relationships\n",
"- Version history\n",
"- Time-point queries\n",
"- Temporal pattern detection\n",
"\n",
"**Example**:\n",
"- Entity: \"Company X\"\n",
"- Relationship: (Company X, has_CEO, Person Y)\n",
"- Temporal: `valid_from=\"2020-01-01\", valid_to=\"2023-12-31\"`\n",
"\n",
"**Use Cases**:\n",
"- Tracking organizational changes\n",
"- Monitoring system evolution\n",
"- Analyzing trends over time\n",
"- Historical analysis\n",
"\n",
"### 6. GraphRAG (Graph-based Retrieval Augmented Generation)\n",
"**Definition**: An advanced RAG approach that combines vector search with knowledge graph traversal to provide more accurate and contextually relevant information to LLMs.\n",
"\n",
"**Components**:\n",
"- **Vector Store**: For semantic similarity search\n",
"- **Knowledge Graph**: For structured relationship traversal\n",
"- **Hybrid Search**: Combines both approaches\n",
"- **LLM Integration**: Uses retrieved context for generation\n",
"\n",
"**Advantages over Traditional RAG**:\n",
"- Better handling of complex queries\n",
"- Relationship-aware retrieval\n",
"- Reduced hallucinations\n",
"- More accurate answers\n",
"\n",
"**Example Workflow**:\n",
"1. Query: \"Who worked with John at Acme Corp?\"\n",
"2. Vector search finds relevant documents\n",
"3. Knowledge graph traversal finds relationships\n",
"4. Combined context sent to LLM\n",
"5. LLM generates accurate answer using both sources\n",
"\n",
"### 7. ONTOLOGY\n",
"**Definition**: A formal specification of concepts, relationships, and constraints in a domain, typically expressed in OWL (Web Ontology Language).\n",
"\n",
"**Components**:\n",
"- **Classes**: Categories of entities\n",
"- **Properties**: Relationships and attributes\n",
"- **Individuals**: Specific instances\n",
"- **Axioms**: Rules and constraints\n",
"\n",
"**Example**:\n",
"- Class: Person\n",
"- SubClass: Employee, Customer\n",
"- Property: worksFor (domain: Person, range: Organization)\n",
"\n",
"**Use Cases**:\n",
"- Standardize domain knowledge\n",
"- Enable reasoning\n",
"- Facilitate data integration\n",
"- Support semantic web\n",
"\n",
"### 8. QUALITY ASSURANCE\n",
"**Definition**: Processes and metrics to ensure knowledge graph quality, including completeness, consistency, and accuracy.\n",
"\n",
"**Metrics**:\n",
"- **Completeness**: Percentage of entities with required properties\n",
"- **Consistency**: Absence of contradictions\n",
"- **Accuracy**: Correctness of extracted information\n",
"- **Coverage**: Breadth of domain coverage\n",
"\n",
"**Methods**:\n",
"- Validation rules\n",
"- Automated quality checks\n",
"- Conflict detection\n",
"- Source verification\n",
"\n",
"---\n",
"\n",
@@ -510,41 +511,35 @@
"\n",
"## Best Practices\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# BEST PRACTICES\n",
"# ============================================================================\n",
"### 1. START SMALL\n",
"- Begin with simple documents\n",
"- Validate each step before moving forward\n",
"- Build incrementally\n",
"\n",
"# 1. START SMALL\n",
"# - Begin with simple documents\n",
"# - Validate each step before moving forward\n",
"# - Build incrementally\n",
"### 2. CONFIGURE PROPERLY\n",
"- Use environment variables for sensitive data\n",
"- Set up proper logging\n",
"- Configure appropriate model sizes\n",
"\n",
"# 2. CONFIGURE PROPERLY\n",
"# - Use environment variables for sensitive data\n",
"# - Set up proper logging\n",
"# - Configure appropriate model sizes\n",
"### 3. VALIDATE DATA\n",
"- Always validate extracted entities\n",
"- Check relationship quality\n",
"- Use quality assurance tools\n",
"\n",
"# 3. VALIDATE DATA\n",
"# - Always validate extracted entities\n",
"# - Check relationship quality\n",
"# - Use quality assurance tools\n",
"### 4. HANDLE ERRORS\n",
"- Implement error handling\n",
"- Use retry mechanisms\n",
"- Log errors for debugging\n",
"\n",
"# 4. HANDLE ERRORS\n",
"# - Implement error handling\n",
"# - Use retry mechanisms\n",
"# - Log errors for debugging\n",
"### 5. OPTIMIZE PERFORMANCE\n",
"- Use batch processing for large datasets\n",
"- Enable parallel processing where possible\n",
"- Cache embeddings and results\n",
"\n",
"# 5. OPTIMIZE PERFORMANCE\n",
"# - Use batch processing for large datasets\n",
"# - Enable parallel processing where possible\n",
"# - Cache embeddings and results\n",
"\n",
"# 6. DOCUMENT YOUR WORKFLOWS\n",
"# - Document data sources\n",
"# - Track processing steps\n",
"# - Maintain metadata\n",
"'''\n",
"### 6. DOCUMENT YOUR WORKFLOWS\n",
"- Document data sources\n",
"- Track processing steps\n",
"- Maintain metadata\n",
"\n",
"---\n",
"\n",
@@ -552,45 +547,39 @@
"\n",
"Common issues and solutions:\n",
"\n",
"'''\n",
"# ============================================================================\n",
"# TROUBLESHOOTING\n",
"# ============================================================================\n",
"### Issue 1: Import Errors\n",
"**Solution**:\n",
"- Ensure Semantica is properly installed\n",
"- Check Python version (3.8+)\n",
"- Verify virtual environment is activated\n",
"- Install missing dependencies: `pip install -r requirements.txt`\n",
"\n",
"# Issue 1: Import Errors\n",
"# Solution:\n",
"# - Ensure Semantica is properly installed\n",
"# - Check Python version (3.8+)\n",
"# - Verify virtual environment is activated\n",
"# - Install missing dependencies: pip install -r requirements.txt\n",
"### Issue 2: API Key Errors\n",
"**Solution**:\n",
"- Set environment variables: `export SEMANTICA_API_KEY=your_key`\n",
"- Check config file for correct key format\n",
"- Verify API key is valid and has sufficient credits\n",
"\n",
"# Issue 2: API Key Errors\n",
"# Solution:\n",
"# - Set environment variables: export SEMANTICA_API_KEY=your_key\n",
"# - Check config file for correct key format\n",
"# - Verify API key is valid and has sufficient credits\n",
"### Issue 3: Memory Issues\n",
"**Solution**:\n",
"- Process documents in batches\n",
"- Use smaller embedding models\n",
"- Enable garbage collection\n",
"- Consider using streaming for large datasets\n",
"\n",
"# Issue 3: Memory Issues\n",
"# Solution:\n",
"# - Process documents in batches\n",
"# - Use smaller embedding models\n",
"# - Enable garbage collection\n",
"# - Consider using streaming for large datasets\n",
"### Issue 4: Low Quality Extractions\n",
"**Solution**:\n",
"- Preprocess and normalize text\n",
"- Use domain-specific models\n",
"- Adjust extraction parameters\n",
"- Validate and clean extracted entities\n",
"\n",
"# Issue 4: Low Quality Extractions\n",
"# Solution:\n",
"# - Preprocess and normalize text\n",
"# - Use domain-specific models\n",
"# - Adjust extraction parameters\n",
"# - Validate and clean extracted entities\n",
"\n",
"# Issue 5: Slow Processing\n",
"# Solution:\n",
"# - Enable parallel processing\n",
"# - Use GPU acceleration if available\n",
"# - Cache intermediate results\n",
"# - Optimize batch sizes\n",
"'''\n"
"### Issue 5: Slow Processing\n",
"**Solution**:\n",
"- Enable parallel processing\n",
"- Use GPU acceleration if available\n",
"- Cache intermediate results\n",
"- Optimize batch sizes\n"
]
},
{
@@ -859,38 +848,6 @@
"print(\" generator = EmbeddingGenerator()\")\n",
"print(\" embeddings = generator.generate(documents)\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n"
]
}
],
"metadata": {
@@ -900,4 +857,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
docs/cookbook/introduction/Your_First_Knowledge_Graph.ipynb
+35 -20
View File
@@ -4,37 +4,40 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Your First Knowledge Graph\n",
"# 🚀 Your First Knowledge Graph\n",
"\n",
"## Overview\n",
"\n",
"This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n",
"\n",
"### Learning Objectives\n",
"> [!TIP]\n",
"> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n",
"\n",
"- Understand the basic workflow: **File → Parse → Extract → Graph**\n",
"- Learn how to ingest documents using `FileIngestor`\n",
"- Parse documents using `DocumentParser`\n",
"- Extract entities using NER extractors\n",
"- Build a knowledge graph using `GraphBuilder`\n",
"- Visualize and analyze the resulting graph\n",
"### 🎯 Learning Objectives\n",
"\n",
"- **Understand the Workflow**: Learn the `File → Parse → Extract → Graph` pipeline\n",
"- **Ingest Data**: Load documents using `FileIngestor`\n",
"- **Parse Content**: Extract text using `DocumentParser`\n",
"- **Extract Knowledge**: Identify entities using `NERExtractor`\n",
"- **Build Graph**: Construct a graph using `GraphBuilder`\n",
"- **Visualize**: See your graph come to life with `KGVisualizer`\n",
"\n",
"---\n",
"\n",
"## Simple End-to-End Workflow\n",
"## 🔄 Simple End-to-End Workflow\n",
"\n",
"The complete workflow consists of four main steps:\n",
"\n",
"1. **Ingest** - Load data from files or other sources\n",
"2. **Parse** - Extract and structure content from documents\n",
"3. **Extract** - Identify entities and relationships\n",
"4. **Build Graph** - Construct the knowledge graph\n",
"1. **📥 Ingest** - Load data from files or other sources\n",
"2. **📄 Parse** - Extract and structure content from documents\n",
"3. **⛏️ Extract** - Identify entities and relationships\n",
"4. **🕸️ Build Graph** - Construct the knowledge graph\n",
"\n",
"Each step is demonstrated in the code cells below.\n",
"\n",
"---\n",
"\n",
"## Step 1: Ingest a File\n",
"## 📂 Step 1: Ingest a File\n",
"\n",
"In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n"
]
@@ -48,8 +51,10 @@
"from semantica.ingest import FileIngestor\n",
"from pathlib import Path\n",
"\n",
"# Initialize the ingestor\n",
"ingestor = FileIngestor()\n",
"\n",
"# Create a sample document for demonstration\n",
"sample_text = \"\"\"\n",
"Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
"The company is headquartered in Cupertino, California.\n",
@@ -64,6 +69,7 @@
"print(f\"File: {sample_file}\")\n",
"print(f\"Content length: {len(sample_text)} characters\")\n",
"\n",
"# Ingest the file\n",
"try:\n",
" file_object = ingestor.ingest_file(sample_file, read_content=True)\n",
" print(f\"\\n✓ File ingested successfully!\")\n",
@@ -78,7 +84,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Parse the Document\n",
"## 📄 Step 2: Parse the Document\n",
"\n",
"After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n"
]
@@ -94,12 +100,14 @@
"parser = DocumentParser()\n",
"\n",
"try:\n",
" # Parse the document to extract text\n",
" if 'file_object' in locals():\n",
" parsed_content = parser.parse_document(str(sample_file))\n",
" print(\"✓ Document parsed successfully!\")\n",
" print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n",
" print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n",
" else:\n",
" # Fallback if ingestion failed\n",
" parsed_content = parser.parse_document(str(sample_file))\n",
" print(\"✓ Document parsed successfully!\")\n",
" print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n",
@@ -113,9 +121,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Extract Entities\n",
"## ⛏️ Step 3: Extract Entities\n",
"\n",
"Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n"
"Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n",
"\n",
"> [!NOTE]\n",
"> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n"
]
},
{
@@ -133,6 +144,7 @@
" print(\"Extracting entities from text...\")\n",
" print(f\"\\nText: {parsed_content[:100]}...\")\n",
" \n",
" # Simulated extraction results\n",
" expected_entities = [\n",
" {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n",
" {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n",
@@ -156,7 +168,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Build the Knowledge Graph\n",
"## 🕸️ Step 4: Build the Knowledge Graph\n",
"\n",
"Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n"
]
@@ -172,6 +184,7 @@
"\n",
"builder = GraphBuilder()\n",
"\n",
"# Prepare data for graph construction\n",
"entities_data = [\n",
" {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n",
" for i, entity in enumerate(expected_entities)\n",
@@ -187,6 +200,7 @@
"]\n",
"\n",
"try:\n",
" # Build the graph using NetworkX\n",
" kg = nx.DiGraph()\n",
" \n",
" for entity in entities_data:\n",
@@ -221,7 +235,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Visualize and Analyze\n",
"## 📊 Step 5: Visualize and Analyze\n",
"\n",
"Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n"
]
@@ -268,6 +282,7 @@
"except Exception as e:\n",
" print(f\"✗ Error visualizing graph: {e}\")\n",
"\n",
"# Cleanup\n",
"try:\n",
" if sample_file.exists():\n",
" sample_file.unlink()\n",
@@ -284,4 +299,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
docs/cookbook/use_cases/finance/Financial_Data_Integration.ipynb
+21 -20
View File
@@ -4,32 +4,33 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Financial Data Integration Pipeline\n",
"# 📈 Financial Data Integration Pipeline\n",
"\n",
"## Overview\n",
"\n",
"This notebook demonstrates how to integrate Python/FastMCP MCP servers as data sources for financial data ingestion. Connect to financial data MCP servers via URL, ingest market data, stock prices, and financial metrics, then build a knowledge graph for financial analysis.\n",
"\n",
"**IMPORTANT**: This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n",
"> [!IMPORTANT]\n",
"> This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n",
"\n",
"### Modules Used (20+)\n",
"### 🧩 Modules Used (20+)\n",
"\n",
"- **Ingestion**: MCPIngestor, ingest_mcp, WebIngestor, FileIngestor\n",
"- **Parsing**: MCPParser, JSONParser, StructuredDataParser\n",
"- **Extraction**: NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
"- **KG**: GraphBuilder, TemporalGraphQuery, GraphAnalyzer\n",
"- **Analytics**: CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
"- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
"- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
"- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
"- **Ingestion**: `MCPIngestor`, `ingest_mcp`, `WebIngestor`, `FileIngestor`\n",
"- **Parsing**: `MCPParser`, `JSONParser`, `StructuredDataParser`\n",
"- **Extraction**: `NERExtractor`, `RelationExtractor`, `EventDetector`, `SemanticAnalyzer`\n",
"- **KG**: `GraphBuilder`, `TemporalGraphQuery`, `GraphAnalyzer`\n",
"- **Analytics**: `CentralityCalculator`, `CommunityDetector`, `ConnectivityAnalyzer`\n",
"- **Reasoning**: `InferenceEngine`, `RuleManager`, `ExplanationGenerator`\n",
"- **Export**: `JSONExporter`, `CSVExporter`, `RDFExporter`, `ReportGenerator`\n",
"- **Visualization**: `KGVisualizer`, `TemporalVisualizer`, `AnalyticsVisualizer`\n",
"\n",
"### Pipeline\n",
"### 🔄 Pipeline\n",
"\n",
"**Connect to Financial MCP Server → Ingest Market Data via MCP → Parse MCP Responses → Extract Financial Entities → Build Financial KG → Analyze Trends → Generate Reports → Visualize**\n",
"\n",
"---\n",
"\n",
"## Step 1: Connect to Financial Data MCP Server\n",
"## 🔌 Step 1: Connect to Financial Data MCP Server\n",
"\n",
"Connect to a Python/FastMCP MCP server that provides financial data via URL. The MCP server can expose resources (datasets, market data) and tools (queries, calculations).\n"
]
@@ -92,7 +93,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2: Ingest Financial Data from MCP Server\n",
"## 📥 Step 2: Ingest Financial Data from MCP Server\n",
"\n",
"Ingest financial data using both resource-based and tool-based methods from the MCP server.\n"
]
@@ -220,7 +221,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 3: Parse MCP Data\n",
"## 📄 Step 3: Parse MCP Data\n",
"\n",
"Parse the data received from MCP server responses (JSON, structured data).\n"
]
@@ -266,7 +267,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Extract Financial Entities and Relationships\n",
"## ⛏️ Step 4: Extract Financial Entities and Relationships\n",
"\n",
"Extract financial entities (companies, stocks, sectors) and relationships from MCP data.\n"
]
@@ -363,7 +364,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Build Financial Knowledge Graph\n",
"## 🕸️ Step 5: Build Financial Knowledge Graph\n",
"\n",
"Build a knowledge graph from the extracted financial entities and relationships.\n"
]
@@ -404,7 +405,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: Analyze Financial Trends\n",
"## 📊 Step 6: Analyze Financial Trends\n",
"\n",
"Analyze financial trends using temporal queries and pattern detection.\n"
]
@@ -462,7 +463,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 7: Export and Visualize\n",
"## 📤 Step 7: Export and Visualize\n",
"\n",
"Export the financial knowledge graph and generate visualizations.\n"
]
@@ -535,4 +536,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
docs/css/custom.css
+101
View File
@@ -150,4 +150,105 @@ html {
[data-md-color-scheme="slate"] ::-webkit-scrollbar-thumb {
background-color: rgba(255, 255, 255, 0.2);
}
/*
==========================================================================
Version Selector
==========================================================================
*/
.version-scroll-container {
display: flex;
align-items: center;
margin-left: 1.5rem;
/* Increased spacing */
overflow-x: auto;
white-space: nowrap;
max-width: 300px;
padding: 4px 0;
scrollbar-width: none;
-ms-overflow-style: none;
height: 100%;
/* Match header height context */
}
.version-scroll-container::-webkit-scrollbar {
display: none;
}
.version-list {
display: flex;
gap: 8px;
align-items: center;
}
.version-tag {
display: inline-flex;
align-items: center;
justify-content: center;
padding: 4px 12px;
/* Larger touch target and better visibility */
border-radius: 4px;
/* Slightly more squared to match material design */
font-size: 0.8rem;
/* Slightly larger text */
font-weight: 700;
/* Bolder for visibility */
line-height: 1.2;
color: var(--md-default-fg-color);
/* Darker text for contrast */
background-color: rgba(0, 0, 0, 0.08);
/* Slightly darker bg */
border: 1px solid rgba(0, 0, 0, 0.1);
/* Subtle border */
transition: all 0.2s ease;
text-decoration: none !important;
font-family: var(--md-text-font-family);
}
.version-tag:hover {
background-color: rgba(0, 0, 0, 0.12);
color: var(--md-primary-fg-color);
border-color: rgba(0, 0, 0, 0.2);
}
.version-tag.active {
background-color: var(--md-accent-fg-color);
color: white;
border-color: var(--md-accent-fg-color);
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
/* Subtle shadow for depth */
}
/* Dark Mode Adjustments */
[data-md-color-scheme="slate"] .version-tag {
background-color: rgba(255, 255, 255, 0.1);
color: var(--md-default-fg-color);
border-color: rgba(255, 255, 255, 0.1);
}
[data-md-color-scheme="slate"] .version-tag:hover {
background-color: rgba(255, 255, 255, 0.15);
color: white;
border-color: rgba(255, 255, 255, 0.2);
}
[data-md-color-scheme="slate"] .version-tag.active {
background-color: var(--md-accent-fg-color);
color: white;
border-color: var(--md-accent-fg-color);
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.3);
}
/* Mobile adjustments */
@media screen and (max-width: 76.1875em) {
.version-scroll-container {
margin-left: 1rem;
max-width: 120px;
}
.version-tag {
padding: 3px 8px;
font-size: 0.75rem;
}
}
docs/getting-started.md
+93 -82
View File
@@ -1,107 +1,118 @@
# Getting Started
Welcome to Semantica! This guide will help you get up and running quickly with the framework.
## Welcome to Semantica
## 📋 Prerequisites
**Semantica** is a comprehensive knowledge graph and semantic processing framework designed for building production-ready semantic AI applications.
Before you begin, ensure you have the following:
### 🎯 What You'll Learn
- What Semantica is and why it's useful
- How to install and configure the framework
- Understanding the framework architecture
- Key concepts and terminology
- Next steps for getting started
- **Python 3.8+** installed on your system
---
## 🚀 What is Semantica?
Semantica is a powerful, production-ready framework for:
- **Building Knowledge Graphs**: Transform unstructured data into structured knowledge graphs.
- **Semantic Processing**: Extract entities, relationships, and meaning from text, images, and audio.
- **GraphRAG**: Next-generation retrieval augmented generation using knowledge graphs.
- **Temporal Analysis**: Time-aware knowledge graphs for tracking changes over time.
- **Multi-Modal Processing**: Handle text, images, audio, and structured data.
- **Enterprise Features**: Quality assurance, conflict resolution, ontology generation, and more.
---
## 💡 Use Cases
| Domain | Application |
| :--- | :--- |
| **Cybersecurity** | Threat intelligence and analysis |
| **Healthcare** | Medical research and patient data analysis |
| **Finance** | Fraud detection and financial analysis |
| **Supply Chain** | Optimization and risk management |
| **Research** | Knowledge management and literature review |
| **AI Systems** | Multi-agent memory and reasoning |
---
## 📦 Installation & Setup
### Prerequisites
Before installing Semantica, ensure you have:
- **Python 3.8** or higher
- **pip** package manager
- Basic understanding of Python programming
- (Optional) An OpenAI API key or other LLM provider credentials if you plan to use semantic extraction features
- (Optional) Virtual environment for isolation
## 🎯 What You Will Learn
### Installation Methods
By following this guide, you will learn how to:
=== "PyPI (Stable)"
```bash
pip install semantica
```
1. Install Semantica and its dependencies
2. Build your first Knowledge Graph from raw data
3. Query and visualize the extracted knowledge
4. Integrate Semantica into your own applications
=== "Source (Dev)"
```bash
git clone https://github.com/your-org/semantica.git
cd semantica
pip install -e .
```
=== "Extras"
```bash
pip install semantica[all] # Install all optional dependencies
pip install semantica[gpu] # Install GPU support
pip install semantica[visualization] # Install visualization tools
```
### Verify Installation
```python
import semantica
print(semantica.version)
```
---
## 🛣️ Choose Your Path
## ⚙️ Configuration
Select the path that best fits your goals:
Semantica can be configured using environment variables or a configuration file.
<div class="grid cards" markdown>
### Environment Variables
- **🚀 Quick Start (5 min)**
---
Perfect for trying Semantica quickly.
1. **[Install Semantica](installation.md#basic-installation)**
2. **[Run Your First Example](quickstart.md#step-2-your-first-knowledge-graph)**
3. **[Explore Examples](examples.md)**
```bash
export SEMANTICA_API_KEY=your_openai_key
export SEMANTICA_EMBEDDING_PROVIDER=openai
export SEMANTICA_MODEL_NAME=gpt-4
```
- **📚 Complete Guide (30 min)**
---
Perfect for learning the framework properly.
1. **[Installation](installation.md)** - Complete setup
2. **[Quick Start](quickstart.md)** - Step-by-step
3. **[Core Concepts](concepts.md)** - Deep dive
4. **[Examples](examples.md)** - Real-world use cases
### Config File (`config.yaml`)
- **🎓 Interactive Learning**
---
Perfect for hands-on learners using Jupyter.
1. **[Cookbook Overview](cookbook.md)**
2. **[Start with Introduction](cookbook.md#introduction)**
3. **[Try Use Cases](cookbook.md#use-cases)**
```yaml
api_keys:
openai: your_key_here
anthropic: your_key_here
</div>
embedding:
provider: openai
model: text-embedding-3-large
dimensions: 3072
!!! tip "Recommendation"
If you're new to knowledge graphs, we highly recommend starting with the **[Core Concepts](concepts.md)** page to understand the terminology before diving into the code.
knowledge_graph:
backend: networkx # or neo4j, arangodb
temporal: true
```
---
## 🏗️ What Can You Build?
Semantica helps you transform unstructured data into intelligent knowledge:
- **Knowledge Graphs** from documents, websites, databases
- **Semantic Layers** for AI applications
- **Entity & Relationship Extraction** from text
- **Conflict Resolution** across multiple data sources
- **GraphRAG Systems** for enhanced AI responses
## 💡 Common Use Cases
### Research & Analysis
- Extract knowledge from research papers
- Build domain-specific knowledge graphs
- Analyze relationships in literature
### Business Intelligence
- Process company documents
- Build organizational knowledge bases
- Integrate multiple data sources
### AI Applications
- Power GraphRAG systems
- Enhance AI agent memory
- Build semantic search systems
!!! warning "Installation First"
Make sure you have Semantica installed before proceeding. If you encounter any issues, check the [Installation Guide](installation.md) troubleshooting section.
## ⏭️ Next Steps
Once you're ready:
1. Head to the **[Installation Guide](installation.md)** to set up your environment.
2. Follow the **[Quick Start](quickstart.md)** to build your first graph.
3. Check out the **[Modules Guide](modules.md)** to understand the architecture.
## 🆘 Need Help?
- **Installation Issues?** → [Troubleshooting Guide](installation.md#troubleshooting)
- **First Time User?** → [Quick Start Guide](quickstart.md)
- **Looking for Examples?** → [Examples Page](examples.md)
- **API Questions?** → [API Reference](api.md)
Now that you understand the basics, here are recommended next steps:
1. **[Your First Knowledge Graph](cookbook/introduction/Your_First_Knowledge_Graph.ipynb)**: Build your first knowledge graph from a document.
2. **[Configuration Basics](cookbook/introduction/Configuration_Basics.ipynb)**: Set up configuration files and API keys.
3. **[Core Workflows](cookbook.md#core-workflows)**: Learn common patterns and workflows.
4. **[Use Cases](cookbook.md#use-cases)**: Explore domain-specific applications.
docs/js/version-selector.js
+32
View File
@@ -0,0 +1,32 @@
document.addEventListener("DOMContentLoaded", function () {
// Target the header title
var headerTitle = document.querySelector(".md-header__title");
if (headerTitle) {
// Create the container for the version selector
var versionContainer = document.createElement("div");
versionContainer.className = "version-scroll-container";
// Define versions
var versions = [
{ name: "0.0.1", url: "#", current: true }
];
// Create the scrollable list
var versionList = document.createElement("div");
versionList.className = "version-list";
versions.forEach(function (version) {
var versionLink = document.createElement("a");
versionLink.className = "version-tag" + (version.current ? " active" : "");
versionLink.href = version.url;
versionLink.textContent = version.name;
versionList.appendChild(versionLink);
});
versionContainer.appendChild(versionList);
// Insert after the header title
headerTitle.parentNode.insertBefore(versionContainer, headerTitle.nextSibling);
}
});
docs/modules.md
+255 -62
View File
@@ -4,13 +4,13 @@ Semantica is built with a modular architecture, designed to be flexible and exte
## 🏗️ Architecture Overview
The framework is organized into several core layers:
The framework is organized into several core layers, each handling specific aspects of the semantic processing pipeline.
```mermaid
graph TD
subgraph Ingest [Ingestion Layer]
I[Ingest] --> P[Parse]
P --> N[Normalize]
I[Ingest Module] --> P[Parse Module]
P --> N[Normalize Module]
end
subgraph Core [Core Processing]
@@ -29,86 +29,279 @@ graph TD
KG --> V[Visualization]
KG --> R[Reasoning]
end
style Ingest fill:#e1f5fe,stroke:#01579b
style Core fill:#e8f5e9,stroke:#1b5e20
style Storage fill:#fff3e0,stroke:#e65100
style Output fill:#f3e5f5,stroke:#4a148c
```
---
## 📦 Core Modules
<div class="grid cards" markdown>
Semantica is organized into 12 core modules. Below is a detailed breakdown of each.
- :material-cube-outline: **[Core](reference/core.md)**
---
The foundation of the framework. Handles configuration, logging, and base classes.
### 1. Ingest Module
**Purpose**: Ingest data from various sources into a unified format.
- :material-file-import: **[Ingest](reference/ingest.md)**
---
Handles the loading of data from various sources (50+ formats).
The `ingest` module is the entry point for data. It handles the complexity of connecting to different data sources, from local files to web streams.
- :material-file-code: **[Parse](reference/parse.md)**
---
Parses raw data into structured text and metadata, including OCR.
- **Components**:
- `FileIngestor`: Read files (PDF, DOCX, HTML, JSON, CSV, etc.)
- `WebIngestor`: Scrape and ingest web pages
- `FeedIngestor`: Process RSS/Atom feeds
- `StreamIngestor`: Real-time data streaming
- `DBIngestor`: Database queries and ingestion
- `EmailIngestor`: Process email messages
- `RepoIngestor`: Git repository analysis
- :material-format-align-left: **[Normalize](reference/normalize.md)**
---
Cleans and standardizes extracted text and encoding.
```python
from semantica.ingest import FileIngestor, WebIngestor
- :material-brain: **[Semantic Extract](reference/semantic_extract.md)**
---
Extracts meaning, entities, and relations using LLMs and NLP.
# Ingest local files
file_ingestor = FileIngestor()
documents = file_ingestor.ingest("data/")
- :material-graph: **[Knowledge Graph](reference/kg.md)**
---
The central module for building and managing knowledge graphs.
# Ingest web content
web_ingestor = WebIngestor()
web_docs = web_ingestor.ingest("https://example.com")
```
- :material-vector-curve: **[Embeddings](reference/embeddings.md)**
---
Generates vector embeddings for text and graph nodes.
### 2. Parse Module
**Purpose**: Parse and extract content from various raw formats.
- :material-database: **[Vector Store](reference/vector_store.md)**
---
Manages vector storage (Pinecone, Chroma, Qdrant).
Once data is ingested, the `parse` module extracts the raw text and metadata. It supports a wide range of formats and includes OCR capabilities.
- :material-database-search: **[Triple Store](reference/triple_store.md)**
---
Manages RDF triple storage (Neo4j, RDFLib).
- **Components**:
- `DocumentParser`: Main parser orchestrator
- `PDFParser`: Extract text, tables, images from PDFs
- `DOCXParser`: Parse Word documents
- `HTMLParser`: Extract content from HTML
- `JSONParser`: Parse structured JSON data
- `ExcelParser`: Process spreadsheets
- `ImageParser`: OCR and image analysis
- `CodeParser`: Parse source code files
- :material-shape: **[Ontology](reference/ontology.md)**
---
Defines and validates the schema of the knowledge graph.
```python
from semantica.parse import DocumentParser
- :material-lightbulb: **[Reasoning](reference/reasoning.md)**
---
Performs logical reasoning and inference over the graph.
parser = DocumentParser()
parsed_docs = parser.parse(documents)
```
- :material-pipe: **[Pipeline](reference/pipeline.md)**
---
Orchestrates the end-to-end processing flow.
### 3. Normalize Module
**Purpose**: Clean and normalize text for processing.
- :material-export: **[Export](reference/export.md)**
---
Exports the knowledge graph to JSON, RDF, CSV, GEXF, etc.
Raw text is often noisy. The `normalize` module cleans, standardizes, and prepares text for semantic extraction.
- :material-chart-bubble: **[Visualization](reference/visualization.md)**
---
Tools for interactive graph plotting and HTML export.
- **Components**:
- `TextNormalizer`: Main normalization orchestrator
- `TextCleaner`: Remove noise, fix encoding
- `DataCleaner`: Clean structured data
- `EntityNormalizer`: Normalize entity names
- `DateNormalizer`: Standardize date formats
- `NumberNormalizer`: Normalize numeric values
- `LanguageDetector`: Detect document language
- `EncodingHandler`: Handle character encoding
- :material-tools: **[Utils](reference/utils.md)**
---
General utility functions for file I/O and string manipulation.
```python
from semantica.normalize import TextNormalizer
</div>
normalizer = TextNormalizer()
normalized = normalizer.normalize(parsed_docs)
```
## 🧩 Submodules
### 4. Semantic Extract Module
**Purpose**: Extract entities, relationships, and semantic information.
Each module contains specialized submodules. For example:
This is the brain of the operation. It uses LLMs and NLP techniques to understand the text and extract structured knowledge.
- **`semantica.ingest`**
- `loaders`: Specific file loaders
- `stream`: Streaming data handlers
- **`semantica.kg`**
- `builder`: Graph construction logic
- `query`: Graph query interface
- **`semantica.semantic_extract`**
- `llm`: LLM-based extraction
- `spacy`: NLP-based extraction
- **Components**:
- `NERExtractor`: Named Entity Recognition
- `RelationExtractor`: Extract relationships between entities
- `SemanticAnalyzer`: Deep semantic analysis
- `SemanticNetworkExtractor`: Extract semantic networks
For detailed API documentation, please refer to the specific **API Reference** pages linked above.
```python
from semantica.semantic_extract import NERExtractor, RelationExtractor
# Extract entities
extractor = NERExtractor()
entities = extractor.extract(normalized_docs)
# Extract relationships
relation_extractor = RelationExtractor()
relationships = relation_extractor.extract(normalized_docs, entities)
```
### 5. Knowledge Graph (KG) Module
**Purpose**: Build and manage knowledge graphs.
The `kg` module constructs the graph from extracted entities and relationships, handling complex tasks like resolution and analysis.
- **Components**:
- `GraphBuilder`: Construct knowledge graphs from entities/relationships
- `GraphAnalyzer`: Analyze graph structure and properties
- `GraphValidator`: Validate graph quality and consistency
- `EntityResolver`: Resolve entity conflicts and duplicates
- `ConflictDetector`: Detect conflicting information
- `CentralityCalculator`: Calculate node importance metrics
- `CommunityDetector`: Detect communities in graphs
- `ConnectivityAnalyzer`: Analyze graph connectivity
- `TemporalQuery`: Query temporal knowledge graphs
- `Deduplicator`: Remove duplicate entities/relationships
```python
from semantica.kg import GraphBuilder, GraphAnalyzer
# Build graph
builder = GraphBuilder()
kg = builder.build(entities, relationships)
# Analyze graph
analyzer = GraphAnalyzer()
metrics = analyzer.analyze(kg)
```
### 6. Embeddings Module
**Purpose**: Generate vector embeddings for various data types.
Embeddings are crucial for semantic search. This module generates vectors for text, images, and graph nodes.
- **Components**:
- `EmbeddingGenerator`: Main embedding orchestrator
- `TextEmbedder`: Generate text embeddings
- `ImageEmbedder`: Generate image embeddings
- `AudioEmbedder`: Generate audio embeddings
- `MultimodalEmbedder`: Combine multiple modalities
- `EmbeddingOptimizer`: Optimize embedding quality
- `ProviderAdapters`: Support for OpenAI, Cohere, etc.
```python
from semantica.embeddings import EmbeddingGenerator
generator = EmbeddingGenerator()
embeddings = generator.generate(documents)
```
### 7. Vector Store Module
**Purpose**: Store and search vector embeddings.
Manages the storage and retrieval of high-dimensional vectors, supporting hybrid search strategies.
- **Components**:
- `VectorStore`: Main vector store interface
- `FAISSAdapter`: FAISS integration
- `HybridSearch`: Combine vector and keyword search
- `VectorRetriever`: Retrieve relevant vectors
```python
from semantica.vector_store import VectorStore, HybridSearch
vector_store = VectorStore()
vector_store.store(embeddings, documents, metadata)
hybrid_search = HybridSearch(vector_store)
results = hybrid_search.search(query, top_k=10)
```
### 8. Reasoning Module
**Purpose**: Perform logical inference and reasoning.
Goes beyond simple retrieval to infer new facts and validate existing knowledge using logical rules.
- **Components**:
- `InferenceEngine`: Main inference orchestrator
- `RuleManager`: Manage inference rules
- `DeductiveReasoner`: Deductive reasoning
- `AbductiveReasoner`: Abductive reasoning
- `ExplanationGenerator`: Generate explanations for inferences
- `RETEEngine`: RETE algorithm for rule matching
```python
from semantica.reasoning import InferenceEngine, RuleManager
inference_engine = InferenceEngine()
rule_manager = RuleManager()
new_facts = inference_engine.forward_chain(kg, rule_manager)
```
### 9. Ontology Module
**Purpose**: Generate and manage ontologies.
Defines the schema and structure of your knowledge domain, ensuring consistency and enabling interoperability.
- **Components**:
- `OntologyGenerator`: Generate ontologies from knowledge graphs
- `OntologyValidator`: Validate ontology structure
- `OWLGenerator`: Generate OWL format ontologies
- `PropertyGenerator`: Generate ontology properties
- `ClassInferrer`: Infer ontology classes
```python
from semantica.ontology import OntologyGenerator
generator = OntologyGenerator()
ontology = generator.generate_from_graph(kg)
```
### 10. Export Module
**Purpose**: Export data in various formats.
Allows you to take your knowledge graph and data out of Semantica for use in other tools.
- **Components**:
- `JSONExporter`: Export to JSON
- `RDFExporter`: Export to RDF/XML
- `CSVExporter`: Export to CSV
- `GraphExporter`: Export to graph formats (GraphML, GEXF)
- `OWLExporter`: Export to OWL
- `VectorExporter`: Export vectors
```python
from semantica.export import JSONExporter, RDFExporter
json_exporter = JSONExporter()
json_exporter.export(kg, "output.json")
```
### 11. Visualization Module
**Purpose**: Visualize knowledge graphs and analytics.
Provides tools to visually explore your data, making it easier to understand complex relationships.
- **Components**:
- `KGVisualizer`: Visualize knowledge graphs
- `EmbeddingVisualizer`: Visualize embeddings (t-SNE, PCA, UMAP)
- `QualityVisualizer`: Visualize quality metrics
- `AnalyticsVisualizer`: Visualize graph analytics
- `TemporalVisualizer`: Visualize temporal data
```python
from semantica.visualization import KGVisualizer
visualizer = KGVisualizer()
visualizer.visualize(kg)
```
### 12. Pipeline Module
**Purpose**: Build and execute processing pipelines.
Orchestrates the entire flow, connecting modules together into robust, executable workflows.
- **Components**:
- `PipelineBuilder`: Build complex pipelines
- `ExecutionEngine`: Execute pipelines
- `FailureHandler`: Handle pipeline failures
- `ParallelismManager`: Enable parallel processing
- `ResourceScheduler`: Schedule resources
```python
from semantica.pipeline import PipelineBuilder
builder = PipelineBuilder()
pipeline = builder.add_step("ingest", FileIngestor()) \
.add_step("parse", DocumentParser()) \
.build()
```
mkdocs.yml
+6 -21
View File
@@ -92,11 +92,15 @@ plugins:
extra_css:
- css/custom.css
# Custom JavaScript
extra_javascript:
- js/version-selector.js
# Navigation
nav:
- Home: index.md
- Getting Started:
- getting-started.md
- Overview: getting-started.md
- installation.md
- quickstart.md
- Guides:
@@ -105,26 +109,7 @@ nav:
- use-cases.md
- examples.md
- learning-more.md
- Cookbook:
- Introduction:
- cookbook/introduction/Welcome_to_Semantica.ipynb
- cookbook/introduction/Configuration_Basics.ipynb
- cookbook/introduction/Your_First_Knowledge_Graph.ipynb
- cookbook/introduction/Data_Ingestion.ipynb
- cookbook/introduction/Document_Parsing.ipynb
- cookbook/introduction/Data_Normalization.ipynb
- cookbook/introduction/Entity_Extraction.ipynb
- cookbook/introduction/Relation_Extraction.ipynb
- cookbook/introduction/Embedding_Generation.ipynb
- cookbook/introduction/Vector_Store.ipynb
- cookbook/introduction/Ontology.ipynb
- cookbook/introduction/Conflict_Detection.ipynb
- cookbook/introduction/Deduplication.ipynb
- cookbook/introduction/Building_Knowledge_Graphs.ipynb
- cookbook/introduction/Graph_Analytics.ipynb
- cookbook/introduction/Graph_Quality.ipynb
- cookbook/introduction/Visualization.ipynb
- cookbook/introduction/Export.ipynb
- Cookbook: cookbook.md
- Reference:
- Core: reference/core.md
- Ingest: reference/ingest.md