mirror of
https://github.com/semantica-agi/semantica.git
synced 2026-08-29 04:26:20 +00:00
Refactor core module: update config and lifecycle managers, add core_usage.md, update notebooks
This commit is contained in:
KaifAhmad1
File diff suppressed because it is too large
Load Diff
@@ -1,267 +1,289 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Your_First_Knowledge_Graph.ipynb)\n",
|
||||
"\n",
|
||||
"# 🚀 Your First Knowledge Graph\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n",
|
||||
"\n",
|
||||
"> [!TIP]\n",
|
||||
"> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n",
|
||||
"\n",
|
||||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/kg/)\n",
|
||||
"\n",
|
||||
"### 🎯 Learning Objectives\n",
|
||||
"\n",
|
||||
"- **Understand the Workflow**: Learn the `File → Parse → Extract → Graph` pipeline\n",
|
||||
"- **Ingest Data**: Load documents using `FileIngestor`\n",
|
||||
"- **Parse Content**: Extract text using `DocumentParser`\n",
|
||||
"- **Extract Knowledge**: Identify entities using `NERExtractor`\n",
|
||||
"- **Build Graph**: Construct a graph using `GraphBuilder`\n",
|
||||
"- **Visualize**: See your graph come to life with `KGVisualizer`\n",
|
||||
"\n",
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install Semantica from PyPI:\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"pip install semantica\n",
|
||||
"# Or with all optional dependencies:\n",
|
||||
"pip install semantica[all]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## 🔄 Simple End-to-End Workflow\n",
|
||||
"\n",
|
||||
"The complete workflow consists of four main steps:\n",
|
||||
"\n",
|
||||
"1. **📥 Ingest** - Load data from files or other sources\n",
|
||||
"2. **📄 Parse** - Extract and structure content from documents\n",
|
||||
"3. **⛏️ Extract** - Identify entities and relationships\n",
|
||||
"4. **🕸️ Build Graph** - Construct the knowledge graph\n",
|
||||
"\n",
|
||||
"Each step is demonstrated in the code cells below.\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## 📂 Step 1: Ingest a File\n",
|
||||
"\n",
|
||||
"In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n"
|
||||
]
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"[](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/introduction/09_Your_First_Knowledge_Graph.ipynb)\n",
|
||||
"\n",
|
||||
"# 🚀 Your First Knowledge Graph\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"This notebook walks you through creating your first knowledge graph from a simple document. You'll learn the complete end-to-end workflow from ingesting a file to visualizing the resulting knowledge graph.\n",
|
||||
"\n",
|
||||
"> [!TIP]\n",
|
||||
"> This is the perfect starting point if you are new to Semantica. No prior knowledge of knowledge graphs is required!\n",
|
||||
"\n",
|
||||
"**Documentation**: [API Reference](https://semantica.readthedocs.io/reference/kg/)\n",
|
||||
"\n",
|
||||
"### 🎯 Learning Objectives\n",
|
||||
"\n",
|
||||
"- **Understand the Workflow**: Learn the `File → Parse → Extract → Graph` pipeline\n",
|
||||
"- **Ingest Data**: Load documents using `FileIngestor`\n",
|
||||
"- **Parse Content**: Extract text using `DocumentParser`\n",
|
||||
"- **Extract Knowledge**: Identify entities using `NERExtractor`\n",
|
||||
"- **Build Graph**: Construct a graph using `GraphBuilder`\n",
|
||||
"- **Visualize**: See your graph come to life with `KGVisualizer`\n",
|
||||
"\n",
|
||||
"## Installation\n",
|
||||
"\n",
|
||||
"Install Semantica from PyPI:\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"pip install semantica\n",
|
||||
"# Or with all optional dependencies:\n",
|
||||
"pip install semantica[all]\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## 🔄 Simple End-to-End Workflow\n",
|
||||
"\n",
|
||||
"The complete workflow consists of four main steps:\n",
|
||||
"\n",
|
||||
"1. **📥 Ingest** - Load data from files or other sources\n",
|
||||
"2. **📄 Parse** - Extract and structure content from documents\n",
|
||||
"3. **⛏️ Extract** - Identify entities and relationships\n",
|
||||
"4. **🕸️ Build Graph** - Construct the knowledge graph\n",
|
||||
"\n",
|
||||
"Each step is demonstrated in the code cells below.\n",
|
||||
"\n",
|
||||
"> [!TIP]\n",
|
||||
"> **Alternative: Using Semantica Framework**\n",
|
||||
"> \n",
|
||||
"> For a simpler, high-level approach, you can use the `Semantica` framework class which orchestrates all these steps:\n",
|
||||
"> \n",
|
||||
"> ```python\n",
|
||||
"> from semantica.core import Semantica\n",
|
||||
"> \n",
|
||||
"> framework = Semantica()\n",
|
||||
"> framework.initialize()\n",
|
||||
"> \n",
|
||||
"> result = framework.build_knowledge_base(\n",
|
||||
"> sources=[\"sample_document.txt\"],\n",
|
||||
"> embeddings=True,\n",
|
||||
"> graph=True\n",
|
||||
"> )\n",
|
||||
"> \n",
|
||||
"> framework.shutdown()\n",
|
||||
"> ```\n",
|
||||
"> \n",
|
||||
"> This notebook shows the step-by-step approach for learning. See [Core Module Usage Guide](../../../semantica/core/core_usage.md) for more details.\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"## 📂 Step 1: Ingest a File\n",
|
||||
"\n",
|
||||
"In this step, we'll use `FileIngestor` to load a document. The ingestor supports various file formats including PDF, DOCX, TXT, and more.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ingest import FileIngestor\n",
|
||||
"from pathlib import Path\n",
|
||||
"\n",
|
||||
"# Initialize the ingestor\n",
|
||||
"ingestor = FileIngestor()\n",
|
||||
"\n",
|
||||
"# Create a sample document for demonstration\n",
|
||||
"sample_text = \"\"\"\n",
|
||||
"Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
|
||||
"The company is headquartered in Cupertino, California.\n",
|
||||
"Tim Cook is the current CEO of Apple Inc.\n",
|
||||
"Apple designs and manufactures consumer electronics, software, and online services.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"sample_file = Path(\"sample_document.txt\")\n",
|
||||
"sample_file.write_text(sample_text)\n",
|
||||
"\n",
|
||||
"print(f\"File: {sample_file}\")\n",
|
||||
"print(f\"Content length: {len(sample_text)} characters\")\n",
|
||||
"\n",
|
||||
"# Ingest the file\n",
|
||||
"file_object = ingestor.ingest_file(sample_file, read_content=True)\n",
|
||||
"print(f\" File name: {file_object.name}\")\n",
|
||||
"print(f\" File type: {file_object.file_type}\")\n",
|
||||
"print(f\" Content available: {file_object.content is not None}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 📄 Step 2: Parse the Document\n",
|
||||
"\n",
|
||||
"After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.parse import DocumentParser\n",
|
||||
"\n",
|
||||
"parser = DocumentParser()\n",
|
||||
"\n",
|
||||
"# Parse the document to extract text\n",
|
||||
"parsed_content = parser.parse_document(str(sample_file))\n",
|
||||
"print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n",
|
||||
"print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ⛏️ Step 3: Extract Entities\n",
|
||||
"\n",
|
||||
"Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n",
|
||||
"\n",
|
||||
"> [!NOTE]\n",
|
||||
"> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NamedEntityRecognizer, NERExtractor\n",
|
||||
"\n",
|
||||
"ner = NamedEntityRecognizer()\n",
|
||||
"extractor = NERExtractor()\n",
|
||||
"\n",
|
||||
"print(f\"\\nText: {parsed_content[:100]}...\")\n",
|
||||
"\n",
|
||||
"# Simulated extraction results\n",
|
||||
"expected_entities = [\n",
|
||||
" {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n",
|
||||
" {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n",
|
||||
" {\"text\": \"Steve Wozniak\", \"type\": \"Person\", \"start\": 62, \"end\": 75},\n",
|
||||
" {\"text\": \"Ronald Wayne\", \"type\": \"Person\", \"start\": 81, \"end\": 93},\n",
|
||||
" {\"text\": \"1976\", \"type\": \"Date\", \"start\": 97, \"end\": 101},\n",
|
||||
" {\"text\": \"Cupertino, California\", \"type\": \"Location\", \"start\": 130, \"end\": 151},\n",
|
||||
" {\"text\": \"Tim Cook\", \"type\": \"Person\", \"start\": 153, \"end\": 161},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"for entity in expected_entities:\n",
|
||||
" print(f\" - {entity['text']} ({entity['type']})\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 🕸️ Step 4: Build the Knowledge Graph\n",
|
||||
"\n",
|
||||
"Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.kg import GraphBuilder\n",
|
||||
"import networkx as nx\n",
|
||||
"\n",
|
||||
"builder = GraphBuilder()\n",
|
||||
"\n",
|
||||
"# Prepare data for graph construction\n",
|
||||
"entities_data = [\n",
|
||||
" {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n",
|
||||
" for i, entity in enumerate(expected_entities)\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"relationships_data = [\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_1\", \"type\": \"founded_by\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_2\", \"type\": \"founded_by\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_3\", \"type\": \"founded_by\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_4\", \"type\": \"founded_in\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_5\", \"type\": \"located_in\"},\n",
|
||||
" {\"source\": \"entity_6\", \"target\": \"entity_0\", \"type\": \"ceo_of\"},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Build the graph using NetworkX\n",
|
||||
"kg = nx.DiGraph()\n",
|
||||
"\n",
|
||||
"for entity in entities_data:\n",
|
||||
" kg.add_node(entity[\"id\"], name=entity[\"name\"], type=entity[\"type\"])\n",
|
||||
"\n",
|
||||
"for rel in relationships_data:\n",
|
||||
" source_name = entities_data[int(rel[\"source\"].split(\"_\")[1])][\"name\"]\n",
|
||||
" target_name = entities_data[int(rel[\"target\"].split(\"_\")[1])][\"name\"]\n",
|
||||
" kg.add_edge(rel[\"source\"], rel[\"target\"], type=rel[\"type\"])\n",
|
||||
"\n",
|
||||
"print(f\" Nodes (entities): {len(kg.nodes)}\")\n",
|
||||
"print(f\" Edges (relationships): {len(kg.edges)}\")\n",
|
||||
"\n",
|
||||
"for node_id in kg.nodes():\n",
|
||||
" node_data = kg.nodes[node_id]\n",
|
||||
" print(f\" Node: {node_data['name']} ({node_data['type']})\")\n",
|
||||
"\n",
|
||||
"for source, target, data in kg.edges(data=True):\n",
|
||||
" source_name = kg.nodes[source]['name']\n",
|
||||
" target_name = kg.nodes[target]['name']\n",
|
||||
" print(f\" {source_name} --[{data['type']}]--> {target_name}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 📊 Step 5: Visualize and Analyze\n",
|
||||
"\n",
|
||||
"Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"\n",
|
||||
"visualizer = KGVisualizer()\n",
|
||||
"\n",
|
||||
"print(f\" Total entities: {len(kg.nodes)}\")\n",
|
||||
"print(f\" Total relationships: {len(kg.edges)}\")\n",
|
||||
"\n",
|
||||
"entity_types = {}\n",
|
||||
"for node_id in kg.nodes():\n",
|
||||
" entity_type = kg.nodes[node_id]['type']\n",
|
||||
" entity_types[entity_type] = entity_types.get(entity_type, 0) + 1\n",
|
||||
"\n",
|
||||
"for etype, count in entity_types.items():\n",
|
||||
" print(f\" - {etype}: {count}\")\n",
|
||||
"\n",
|
||||
"rel_types = {}\n",
|
||||
"for _, _, data in kg.edges(data=True):\n",
|
||||
" rel_type = data.get('type', 'unknown')\n",
|
||||
" rel_types[rel_type] = rel_types.get(rel_type, 0) + 1\n",
|
||||
"\n",
|
||||
"for rtype, count in rel_types.items():\n",
|
||||
" print(f\" - {rtype}: {count}\")\n",
|
||||
"\n",
|
||||
"# Cleanup\n",
|
||||
"if sample_file.exists():\n",
|
||||
" sample_file.unlink()\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.ingest import FileIngestor\n",
|
||||
"from pathlib import Path\n",
|
||||
"\n",
|
||||
"# Initialize the ingestor\n",
|
||||
"ingestor = FileIngestor()\n",
|
||||
"\n",
|
||||
"# Create a sample document for demonstration\n",
|
||||
"sample_text = \"\"\"\n",
|
||||
"Apple Inc. is a technology company founded by Steve Jobs, Steve Wozniak, and Ronald Wayne in 1976.\n",
|
||||
"The company is headquartered in Cupertino, California.\n",
|
||||
"Tim Cook is the current CEO of Apple Inc.\n",
|
||||
"Apple designs and manufactures consumer electronics, software, and online services.\n",
|
||||
"\"\"\"\n",
|
||||
"\n",
|
||||
"sample_file = Path(\"sample_document.txt\")\n",
|
||||
"sample_file.write_text(sample_text)\n",
|
||||
"\n",
|
||||
"print(f\"File: {sample_file}\")\n",
|
||||
"print(f\"Content length: {len(sample_text)} characters\")\n",
|
||||
"\n",
|
||||
"# Ingest the file\n",
|
||||
"file_object = ingestor.ingest_file(sample_file, read_content=True)\n",
|
||||
"print(f\" File name: {file_object.name}\")\n",
|
||||
"print(f\" File type: {file_object.file_type}\")\n",
|
||||
"print(f\" Content available: {file_object.content is not None}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 📄 Step 2: Parse the Document\n",
|
||||
"\n",
|
||||
"After ingesting the file, we need to parse it to extract the text content. The `DocumentParser` handles various file formats and extracts structured content.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.parse import DocumentParser\n",
|
||||
"\n",
|
||||
"parser = DocumentParser()\n",
|
||||
"\n",
|
||||
"# Parse the document to extract text\n",
|
||||
"parsed_content = parser.parse_document(str(sample_file))\n",
|
||||
"print(f\" Parsed content length: {len(parsed_content) if parsed_content else 0} characters\")\n",
|
||||
"print(f\" Preview: {parsed_content[:200] if parsed_content else 'N/A'}...\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## ⛏️ Step 3: Extract Entities\n",
|
||||
"\n",
|
||||
"Now we'll extract entities from the parsed text using Named Entity Recognition (NER). This identifies people, organizations, locations, dates, and other entities in the text.\n",
|
||||
"\n",
|
||||
"> [!NOTE]\n",
|
||||
"> In a real scenario, you would use `NERExtractor` with an LLM or model backend. Here we simulate the output for demonstration purposes.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.semantic_extract import NamedEntityRecognizer, NERExtractor\n",
|
||||
"\n",
|
||||
"ner = NamedEntityRecognizer()\n",
|
||||
"extractor = NERExtractor()\n",
|
||||
"\n",
|
||||
"print(f\"\\nText: {parsed_content[:100]}...\")\n",
|
||||
"\n",
|
||||
"# Simulated extraction results\n",
|
||||
"expected_entities = [\n",
|
||||
" {\"text\": \"Apple Inc.\", \"type\": \"Organization\", \"start\": 0, \"end\": 10},\n",
|
||||
" {\"text\": \"Steve Jobs\", \"type\": \"Person\", \"start\": 50, \"end\": 60},\n",
|
||||
" {\"text\": \"Steve Wozniak\", \"type\": \"Person\", \"start\": 62, \"end\": 75},\n",
|
||||
" {\"text\": \"Ronald Wayne\", \"type\": \"Person\", \"start\": 81, \"end\": 93},\n",
|
||||
" {\"text\": \"1976\", \"type\": \"Date\", \"start\": 97, \"end\": 101},\n",
|
||||
" {\"text\": \"Cupertino, California\", \"type\": \"Location\", \"start\": 130, \"end\": 151},\n",
|
||||
" {\"text\": \"Tim Cook\", \"type\": \"Person\", \"start\": 153, \"end\": 161},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"for entity in expected_entities:\n",
|
||||
" print(f\" - {entity['text']} ({entity['type']})\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 🕸️ Step 4: Build the Knowledge Graph\n",
|
||||
"\n",
|
||||
"Using the extracted entities and relationships, we'll construct a knowledge graph. The graph represents entities as nodes and relationships as edges.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.kg import GraphBuilder\n",
|
||||
"import networkx as nx\n",
|
||||
"\n",
|
||||
"builder = GraphBuilder()\n",
|
||||
"\n",
|
||||
"# Prepare data for graph construction\n",
|
||||
"entities_data = [\n",
|
||||
" {\"id\": f\"entity_{i}\", \"name\": entity[\"text\"], \"type\": entity[\"type\"]}\n",
|
||||
" for i, entity in enumerate(expected_entities)\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"relationships_data = [\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_1\", \"type\": \"founded_by\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_2\", \"type\": \"founded_by\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_3\", \"type\": \"founded_by\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_4\", \"type\": \"founded_in\"},\n",
|
||||
" {\"source\": \"entity_0\", \"target\": \"entity_5\", \"type\": \"located_in\"},\n",
|
||||
" {\"source\": \"entity_6\", \"target\": \"entity_0\", \"type\": \"ceo_of\"},\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"# Build the graph using NetworkX\n",
|
||||
"kg = nx.DiGraph()\n",
|
||||
"\n",
|
||||
"for entity in entities_data:\n",
|
||||
" kg.add_node(entity[\"id\"], name=entity[\"name\"], type=entity[\"type\"])\n",
|
||||
"\n",
|
||||
"for rel in relationships_data:\n",
|
||||
" source_name = entities_data[int(rel[\"source\"].split(\"_\")[1])][\"name\"]\n",
|
||||
" target_name = entities_data[int(rel[\"target\"].split(\"_\")[1])][\"name\"]\n",
|
||||
" kg.add_edge(rel[\"source\"], rel[\"target\"], type=rel[\"type\"])\n",
|
||||
"\n",
|
||||
"print(f\" Nodes (entities): {len(kg.nodes)}\")\n",
|
||||
"print(f\" Edges (relationships): {len(kg.edges)}\")\n",
|
||||
"\n",
|
||||
"for node_id in kg.nodes():\n",
|
||||
" node_data = kg.nodes[node_id]\n",
|
||||
" print(f\" Node: {node_data['name']} ({node_data['type']})\")\n",
|
||||
"\n",
|
||||
"for source, target, data in kg.edges(data=True):\n",
|
||||
" source_name = kg.nodes[source]['name']\n",
|
||||
" target_name = kg.nodes[target]['name']\n",
|
||||
" print(f\" {source_name} --[{data['type']}]--> {target_name}\")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 📊 Step 5: Visualize and Analyze\n",
|
||||
"\n",
|
||||
"Finally, we'll visualize the knowledge graph and analyze its structure. This helps you understand the relationships and entities in your data.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from semantica.visualization import KGVisualizer\n",
|
||||
"\n",
|
||||
"visualizer = KGVisualizer()\n",
|
||||
"\n",
|
||||
"print(f\" Total entities: {len(kg.nodes)}\")\n",
|
||||
"print(f\" Total relationships: {len(kg.edges)}\")\n",
|
||||
"\n",
|
||||
"entity_types = {}\n",
|
||||
"for node_id in kg.nodes():\n",
|
||||
" entity_type = kg.nodes[node_id]['type']\n",
|
||||
" entity_types[entity_type] = entity_types.get(entity_type, 0) + 1\n",
|
||||
"\n",
|
||||
"for etype, count in entity_types.items():\n",
|
||||
" print(f\" - {etype}: {count}\")\n",
|
||||
"\n",
|
||||
"rel_types = {}\n",
|
||||
"for _, _, data in kg.edges(data=True):\n",
|
||||
" rel_type = data.get('type', 'unknown')\n",
|
||||
" rel_types[rel_type] = rel_types.get(rel_type, 0) + 1\n",
|
||||
"\n",
|
||||
"for rtype, count in rel_types.items():\n",
|
||||
" print(f\" - {rtype}: {count}\")\n",
|
||||
"\n",
|
||||
"# Cleanup\n",
|
||||
"if sample_file.exists():\n",
|
||||
" sample_file.unlink()\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+430
-62
@@ -8,11 +8,11 @@
|
||||
|
||||
<div class="grid cards" markdown>
|
||||
|
||||
- :material-cogs:{ .lg .middle } **Orchestrator**
|
||||
- :material-cogs:{ .lg .middle } **Semantica**
|
||||
|
||||
---
|
||||
|
||||
Central coordinator for all framework components and workflows
|
||||
Main framework class coordinating all components and workflows
|
||||
|
||||
- :material-lifecycle:{ .lg .middle } **Lifecycle Management**
|
||||
|
||||
@@ -32,11 +32,11 @@
|
||||
|
||||
Extensible plugin registry for adding custom modules and capabilities
|
||||
|
||||
- :material-console:{ .lg .middle } **Logging & Telemetry**
|
||||
- :material-console:{ .lg .middle } **Method Registry**
|
||||
|
||||
---
|
||||
|
||||
Centralized logging and metrics collection
|
||||
Registry for custom orchestration methods and extensibility
|
||||
|
||||
</div>
|
||||
|
||||
@@ -51,69 +51,318 @@
|
||||
## ⚙️ Algorithms Used
|
||||
|
||||
### Lifecycle Management
|
||||
- **State Machine**: `CREATED` -> `INITIALIZED` -> `RUNNING` -> `STOPPED`
|
||||
- **Dependency Injection**: Resolving and injecting dependencies between modules.
|
||||
- **Graceful Shutdown**: Ensuring all resources (DB connections, thread pools) are closed properly.
|
||||
- **State Machine**: `UNINITIALIZED` -> `INITIALIZING` -> `READY` -> `RUNNING` -> `STOPPING` -> `STOPPED`
|
||||
- **Priority-based Hooks**: Startup and shutdown hooks executed in priority order (lower = earlier)
|
||||
- **Graceful Shutdown**: Ensuring all resources (DB connections, thread pools) are closed properly
|
||||
|
||||
### Configuration
|
||||
- **Layered Loading**: Defaults -> Config File -> Environment Variables -> CLI Arguments (Priority order).
|
||||
- **Schema Validation**: Validating config structure against defined schemas.
|
||||
- **Layered Loading**: Defaults -> Config File -> Environment Variables (Priority order)
|
||||
- **Schema Validation**: Validating config structure against defined schemas
|
||||
- **Nested Access**: Dot notation for accessing nested configuration values
|
||||
|
||||
### Plugin System
|
||||
- **Discovery**: Auto-discovery of plugins via entry points or directory scanning.
|
||||
- **Registration**: Dynamic registration of classes and functions.
|
||||
- **Hook Execution**: Running plugin hooks at specific lifecycle events.
|
||||
- **Discovery**: Auto-discovery of plugins via directory scanning
|
||||
- **Registration**: Dynamic registration of classes and functions
|
||||
- **Dependency Resolution**: Automatic loading of plugin dependencies
|
||||
|
||||
---
|
||||
|
||||
## Main Classes
|
||||
|
||||
### Orchestrator
|
||||
### Semantica
|
||||
|
||||
The brain of the framework.
|
||||
The main framework class that coordinates all components.
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `start()` | Initialize and start all components |
|
||||
| `stop()` | Graceful shutdown |
|
||||
| `get_component(name)` | Access initialized module |
|
||||
| `__init__(config=None, **kwargs)` | Initialize framework with optional configuration |
|
||||
| `initialize()` | Initialize all framework components |
|
||||
| `build_knowledge_base(sources, **kwargs)` | Build knowledge base from data sources |
|
||||
| `run_pipeline(pipeline, data)` | Execute a processing pipeline |
|
||||
| `get_status()` | Get system health and status |
|
||||
| `shutdown(graceful=True)` | Shutdown the framework gracefully |
|
||||
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
from semantica.core import Orchestrator
|
||||
from semantica.core import Semantica
|
||||
|
||||
app = Orchestrator()
|
||||
app.start()
|
||||
# Initialize framework
|
||||
framework = Semantica()
|
||||
framework.initialize()
|
||||
|
||||
# Access modules
|
||||
kg = app.get_component("knowledge_graph")
|
||||
ingest = app.get_component("ingest")
|
||||
# Build knowledge base
|
||||
result = framework.build_knowledge_base(
|
||||
sources=["doc1.pdf", "doc2.docx"],
|
||||
embeddings=True,
|
||||
graph=True
|
||||
)
|
||||
|
||||
# Check status
|
||||
status = framework.get_status()
|
||||
print(f"System state: {status['state']}")
|
||||
|
||||
# Shutdown
|
||||
framework.shutdown()
|
||||
```
|
||||
|
||||
### ConfigManager
|
||||
|
||||
Manages global configuration.
|
||||
Manages global configuration loading, validation, and merging.
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `load(path)` | Load config from file |
|
||||
| `get(key, default)` | Get config value |
|
||||
| `load_from_file(file_path, validate=True)` | Load config from YAML or JSON file |
|
||||
| `load_from_dict(config_dict, validate=True)` | Load config from dictionary |
|
||||
| `merge_configs(*configs, validate=True)` | Merge multiple configurations |
|
||||
| `get_config()` | Get current configuration |
|
||||
| `set_config(config, validate=True)` | Set current configuration |
|
||||
| `reload(file_path=None)` | Reload configuration from file |
|
||||
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
from semantica.core import ConfigManager
|
||||
|
||||
manager = ConfigManager()
|
||||
config = manager.load_from_file("config.yaml")
|
||||
|
||||
# Merge configurations
|
||||
config1 = manager.load_from_file("base_config.yaml")
|
||||
config2 = manager.load_from_file("override_config.yaml")
|
||||
merged = manager.merge_configs(config1, config2)
|
||||
```
|
||||
|
||||
### Config
|
||||
|
||||
Configuration data class with validation and nested access.
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `get(key_path, default=None)` | Get nested configuration value by key path |
|
||||
| `set(key_path, value)` | Set nested configuration value |
|
||||
| `update(updates, merge=True)` | Update configuration with new values |
|
||||
| `validate()` | Validate configuration settings |
|
||||
| `to_dict()` | Convert configuration to dictionary |
|
||||
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
from semantica.core import Config, ConfigManager
|
||||
|
||||
manager = ConfigManager()
|
||||
config = manager.load_from_dict({"processing": {"batch_size": 32}})
|
||||
|
||||
# Access nested values
|
||||
batch_size = config.get("processing.batch_size", default=16)
|
||||
|
||||
# Update values
|
||||
config.set("processing.batch_size", 64)
|
||||
config.update({"quality": {"min_confidence": 0.9}})
|
||||
|
||||
# Validate
|
||||
config.validate()
|
||||
```
|
||||
|
||||
### LifecycleManager
|
||||
|
||||
System lifecycle management with hooks and health monitoring.
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `startup()` | Execute startup sequence with registered hooks |
|
||||
| `shutdown(graceful=True)` | Execute shutdown sequence |
|
||||
| `register_startup_hook(hook_fn, priority=50)` | Register a startup hook |
|
||||
| `register_shutdown_hook(hook_fn, priority=50)` | Register a shutdown hook |
|
||||
| `register_component(name, component)` | Register component for health monitoring |
|
||||
| `health_check()` | Perform comprehensive system health check |
|
||||
| `get_health_summary()` | Get summary of system health |
|
||||
| `get_state()` | Get current system state |
|
||||
| `is_ready()` | Check if system is ready |
|
||||
| `is_running()` | Check if system is running |
|
||||
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
from semantica.core import LifecycleManager
|
||||
|
||||
manager = LifecycleManager()
|
||||
|
||||
# Register hooks
|
||||
def init_db():
|
||||
print("Initializing database...")
|
||||
|
||||
manager.register_startup_hook(init_db, priority=10)
|
||||
manager.startup()
|
||||
|
||||
# Register component for health monitoring
|
||||
class DatabaseComponent:
|
||||
def health_check(self):
|
||||
return {"healthy": True, "message": "Connected"}
|
||||
|
||||
db = DatabaseComponent()
|
||||
manager.register_component("database", db)
|
||||
|
||||
# Check health
|
||||
health = manager.health_check()
|
||||
summary = manager.get_health_summary()
|
||||
|
||||
manager.shutdown(graceful=True)
|
||||
```
|
||||
|
||||
### PluginRegistry
|
||||
|
||||
Manages extensions.
|
||||
Plugin registry and management system for dynamic plugin discovery and loading.
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `register(plugin)` | Register new plugin |
|
||||
| `get_plugin(name)` | Retrieve plugin |
|
||||
| `__init__(plugin_paths=None)` | Initialize with optional plugin paths for auto-discovery |
|
||||
| `register_plugin(plugin_name, plugin_class, version="1.0.0", **metadata)` | Manually register a plugin |
|
||||
| `load_plugin(plugin_name, **config)` | Load and initialize a plugin |
|
||||
| `unload_plugin(plugin_name)` | Unload a plugin |
|
||||
| `list_plugins()` | List all available plugins |
|
||||
| `get_plugin_info(plugin_name)` | Get information about a plugin |
|
||||
| `is_plugin_loaded(plugin_name)` | Check if a plugin is loaded |
|
||||
| `get_loaded_plugin(plugin_name)` | Get loaded plugin instance |
|
||||
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
from semantica.core import PluginRegistry
|
||||
|
||||
# Auto-discover plugins
|
||||
registry = PluginRegistry(plugin_paths=["./plugins"])
|
||||
|
||||
# Load plugin with configuration
|
||||
plugin = registry.load_plugin("my_plugin", api_key="xxx")
|
||||
|
||||
# List all plugins
|
||||
plugins = registry.list_plugins()
|
||||
for plugin_info in plugins:
|
||||
print(f"{plugin_info['name']}: {plugin_info['version']}")
|
||||
|
||||
# Get plugin info
|
||||
info = registry.get_plugin_info("my_plugin")
|
||||
```
|
||||
|
||||
### MethodRegistry
|
||||
|
||||
Registry for custom orchestration methods.
|
||||
|
||||
**Methods:**
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `register(task, name, method_func)` | Register a custom orchestration method |
|
||||
| `get(task, name)` | Get method by task and name |
|
||||
| `list_all(task=None)` | List all registered methods |
|
||||
| `unregister(task, name)` | Unregister a method |
|
||||
| `clear(task=None)` | Clear all registered methods |
|
||||
|
||||
**Example:**
|
||||
|
||||
```python
|
||||
from semantica.core import method_registry
|
||||
|
||||
def custom_kb_builder(sources, **kwargs):
|
||||
# Custom logic
|
||||
return {"knowledge_graph": {}}
|
||||
|
||||
method_registry.register("knowledge_base", "custom", custom_kb_builder)
|
||||
|
||||
# Use custom method
|
||||
method = method_registry.get("knowledge_base", "custom")
|
||||
result = method(sources=["doc.pdf"])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Orchestration Methods
|
||||
|
||||
Convenience functions for common orchestration tasks.
|
||||
|
||||
### build_knowledge_base()
|
||||
|
||||
Build knowledge base from data sources.
|
||||
|
||||
```python
|
||||
from semantica.core.methods import build_knowledge_base
|
||||
|
||||
result = build_knowledge_base(
|
||||
sources=["doc1.pdf", "doc2.docx"],
|
||||
method="default",
|
||||
embeddings=True,
|
||||
graph=True
|
||||
)
|
||||
```
|
||||
|
||||
### run_pipeline()
|
||||
|
||||
Execute a processing pipeline.
|
||||
|
||||
```python
|
||||
from semantica.core.methods import run_pipeline
|
||||
|
||||
result = run_pipeline(
|
||||
pipeline={"steps": ["parse", "extract"]},
|
||||
data="sample text",
|
||||
method="default"
|
||||
)
|
||||
```
|
||||
|
||||
### initialize_framework()
|
||||
|
||||
Initialize Semantica framework.
|
||||
|
||||
```python
|
||||
from semantica.core.methods import initialize_framework
|
||||
|
||||
framework = initialize_framework(
|
||||
config={"llm_provider": {"name": "openai"}},
|
||||
method="default"
|
||||
)
|
||||
```
|
||||
|
||||
### get_status()
|
||||
|
||||
Get system status.
|
||||
|
||||
```python
|
||||
from semantica.core.methods import get_status
|
||||
|
||||
status = get_status(framework=my_framework, method="detailed")
|
||||
```
|
||||
|
||||
### get_orchestration_method()
|
||||
|
||||
Get orchestration method by task and name.
|
||||
|
||||
```python
|
||||
from semantica.core.methods import get_orchestration_method
|
||||
|
||||
method = get_orchestration_method("knowledge_base", "custom")
|
||||
```
|
||||
|
||||
### list_available_methods()
|
||||
|
||||
List all available orchestration methods.
|
||||
|
||||
```python
|
||||
from semantica.core.methods import list_available_methods
|
||||
|
||||
all_methods = list_available_methods()
|
||||
kb_methods = list_available_methods("knowledge_base")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
@@ -121,68 +370,187 @@ Manages extensions.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Configuration can be loaded from environment variables with `SEMANTICA_` prefix:
|
||||
|
||||
```bash
|
||||
export SEMANTICA_ENV=production
|
||||
export SEMANTICA_LOG_LEVEL=INFO
|
||||
export SEMANTICA_CONFIG_PATH=./config.yaml
|
||||
export SEMANTICA_PROCESSING_BATCH_SIZE=64
|
||||
export SEMANTICA_LLM_PROVIDER_MODEL=gpt-4
|
||||
export SEMANTICA_QUALITY_MIN_CONFIDENCE=0.8
|
||||
```
|
||||
|
||||
### YAML Configuration
|
||||
|
||||
```yaml
|
||||
core:
|
||||
environment: production
|
||||
log_level: INFO
|
||||
plugins:
|
||||
llm_provider:
|
||||
name: openai
|
||||
model: gpt-4
|
||||
api_key: ${OPENAI_API_KEY}
|
||||
|
||||
embedding_model:
|
||||
name: openai
|
||||
model: text-embedding-ada-002
|
||||
|
||||
processing:
|
||||
batch_size: 32
|
||||
max_workers: 4
|
||||
|
||||
quality:
|
||||
min_confidence: 0.7
|
||||
|
||||
logging:
|
||||
level: INFO
|
||||
|
||||
plugins:
|
||||
my_plugin:
|
||||
enabled: true
|
||||
directory: ./plugins
|
||||
config_key: config_value
|
||||
```
|
||||
|
||||
### JSON Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"llm_provider": {
|
||||
"name": "openai",
|
||||
"model": "gpt-4"
|
||||
},
|
||||
"processing": {
|
||||
"batch_size": 32
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### Custom Application
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from semantica.core import Orchestrator, ConfigManager
|
||||
from semantica.core import Semantica, ConfigManager
|
||||
|
||||
# 1. Load Config
|
||||
config = ConfigManager()
|
||||
config.load("config.yaml")
|
||||
# 1. Load configuration
|
||||
config_manager = ConfigManager()
|
||||
config = config_manager.load_from_file("config.yaml")
|
||||
|
||||
# 2. Initialize Orchestrator
|
||||
app = Orchestrator(config=config)
|
||||
# 2. Initialize framework
|
||||
framework = Semantica(config=config)
|
||||
framework.initialize()
|
||||
|
||||
# 3. Register Custom Plugin
|
||||
class MyPlugin:
|
||||
name = "my_plugin"
|
||||
def initialize(self):
|
||||
print("My Plugin Started")
|
||||
|
||||
app.plugin_registry.register(MyPlugin())
|
||||
|
||||
# 4. Start
|
||||
app.start()
|
||||
|
||||
# 5. Run Workload
|
||||
try:
|
||||
app.run_pipeline("my_pipeline")
|
||||
# 3. Build knowledge base
|
||||
result = framework.build_knowledge_base(
|
||||
sources=["doc1.pdf", "doc2.docx"],
|
||||
embeddings=True,
|
||||
graph=True
|
||||
)
|
||||
|
||||
# 4. Check status
|
||||
status = framework.get_status()
|
||||
print(f"System state: {status['state']}")
|
||||
|
||||
finally:
|
||||
app.stop()
|
||||
# 5. Shutdown gracefully
|
||||
framework.shutdown(graceful=True)
|
||||
```
|
||||
|
||||
### Custom Plugin
|
||||
|
||||
```python
|
||||
from semantica.core import PluginRegistry
|
||||
|
||||
class MyPlugin:
|
||||
def initialize(self):
|
||||
print("Plugin initialized")
|
||||
|
||||
def execute(self, data):
|
||||
return {"processed": True}
|
||||
|
||||
registry = PluginRegistry()
|
||||
registry.register_plugin(
|
||||
plugin_name="my_plugin",
|
||||
plugin_class=MyPlugin,
|
||||
version="1.0.0"
|
||||
)
|
||||
|
||||
plugin = registry.load_plugin("my_plugin")
|
||||
result = plugin.execute("sample data")
|
||||
```
|
||||
|
||||
### Lifecycle Hooks
|
||||
|
||||
```python
|
||||
from semantica.core import LifecycleManager
|
||||
|
||||
manager = LifecycleManager()
|
||||
|
||||
def init_database():
|
||||
print("Initializing database...")
|
||||
|
||||
def cleanup_database():
|
||||
print("Cleaning up database...")
|
||||
|
||||
manager.register_startup_hook(init_database, priority=10)
|
||||
manager.register_shutdown_hook(cleanup_database, priority=10)
|
||||
|
||||
manager.startup()
|
||||
# ... do work ...
|
||||
manager.shutdown(graceful=True)
|
||||
```
|
||||
|
||||
### Custom Orchestration Method
|
||||
|
||||
```python
|
||||
from semantica.core import method_registry, Semantica
|
||||
|
||||
def fast_kb_builder(sources, **kwargs):
|
||||
framework = Semantica()
|
||||
framework.initialize()
|
||||
try:
|
||||
return framework.build_knowledge_base(
|
||||
sources=sources,
|
||||
embeddings=False, # Skip for speed
|
||||
graph=True,
|
||||
**kwargs
|
||||
)
|
||||
finally:
|
||||
framework.shutdown()
|
||||
|
||||
method_registry.register("knowledge_base", "fast", fast_kb_builder)
|
||||
|
||||
# Use custom method
|
||||
from semantica.core.methods import build_knowledge_base
|
||||
result = build_knowledge_base(sources=["doc.pdf"], method="fast")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use Orchestrator**: Avoid manually instantiating every module; let the Orchestrator handle dependencies.
|
||||
2. **Graceful Shutdown**: Always ensure `app.stop()` is called (e.g., in a `finally` block) to prevent resource leaks.
|
||||
3. **Config Layers**: Use `config.yaml` for defaults and Environment Variables for secrets/overrides.
|
||||
1. **Always Initialize**: Always call `initialize()` after creating a `Semantica` instance before using it.
|
||||
|
||||
2. **Graceful Shutdown**: Always call `shutdown(graceful=True)` in a `finally` block to ensure proper cleanup.
|
||||
|
||||
3. **Configuration Management**: Use `ConfigManager` for loading and managing configurations. Prefer YAML files for complex configurations.
|
||||
|
||||
4. **Error Handling**: Wrap framework operations in try-except blocks to handle `ConfigurationError` and `ProcessingError` appropriately.
|
||||
|
||||
5. **Health Monitoring**: Register components with `LifecycleManager` for health monitoring and use `health_check()` regularly.
|
||||
|
||||
6. **Plugin Development**: Follow the plugin interface (must have `initialize()` and `execute()` methods) when creating custom plugins.
|
||||
|
||||
7. **Method Registration**: Use `MethodRegistry` for extensibility. Register custom methods for knowledge base building, pipeline execution, etc.
|
||||
|
||||
8. **Hook Priorities**: Use appropriate priorities for lifecycle hooks. Lower numbers execute first.
|
||||
|
||||
9. **Configuration Validation**: Always validate configurations using `config.validate()` before using them.
|
||||
|
||||
10. **Resource Cleanup**: Ensure all resources are properly cleaned up in shutdown hooks.
|
||||
|
||||
---
|
||||
|
||||
## See Also
|
||||
|
||||
- [Pipeline Module](pipeline.md) - Executed by the Orchestrator
|
||||
- [Core Usage Guide](../core/core_usage.md) - Comprehensive usage guide with detailed examples
|
||||
- [Pipeline Module](pipeline.md) - Executed by the Semantica framework
|
||||
- [Utils Module](utils.md) - Shared utilities used by Core
|
||||
|
||||
@@ -86,9 +86,6 @@ __all__ = [
|
||||
"get_status",
|
||||
"get_orchestration_method",
|
||||
"list_available_methods",
|
||||
<<<<<<< HEAD
|
||||
]
|
||||
=======
|
||||
# Convenience
|
||||
"build",
|
||||
]
|
||||
@@ -158,4 +155,3 @@ def build(
|
||||
pipeline=pipeline_config,
|
||||
**{k: v for k, v in options.items() if k not in ["pipeline", "method"]},
|
||||
)
|
||||
>>>>>>> origin/main
|
||||
|
||||
+239
-135
@@ -96,33 +96,68 @@ class Config:
|
||||
config_dict: Dictionary of configuration values
|
||||
**kwargs: Additional configuration parameters
|
||||
"""
|
||||
# Build configuration dictionary from all sources
|
||||
config_data = self._build_config_dict(config_dict, kwargs)
|
||||
|
||||
# Load from environment variables (overrides file/kwargs)
|
||||
self._load_from_env(config_data)
|
||||
|
||||
# Initialize configuration sections
|
||||
self._initialize_sections(config_data)
|
||||
|
||||
def _build_config_dict(
|
||||
self, config_dict: Optional[Dict[str, Any]], kwargs: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Build configuration dictionary from multiple sources.
|
||||
|
||||
Priority order: defaults -> config_dict -> kwargs
|
||||
|
||||
Args:
|
||||
config_dict: Optional configuration dictionary
|
||||
kwargs: Additional configuration parameters
|
||||
|
||||
Returns:
|
||||
Merged configuration dictionary
|
||||
"""
|
||||
# Start with defaults
|
||||
default_dict = DEFAULT_CONFIG.copy()
|
||||
result = DEFAULT_CONFIG.copy()
|
||||
|
||||
# Merge with provided config_dict
|
||||
if config_dict:
|
||||
default_dict = merge_dicts(default_dict, config_dict, deep=True)
|
||||
result = merge_dicts(result, config_dict, deep=True)
|
||||
|
||||
# Merge with kwargs
|
||||
if kwargs:
|
||||
default_dict = merge_dicts(default_dict, kwargs, deep=True)
|
||||
result = merge_dicts(result, kwargs, deep=True)
|
||||
|
||||
# Load from environment variables
|
||||
self._load_from_env(default_dict)
|
||||
return result
|
||||
|
||||
# Initialize dataclass fields
|
||||
self.llm_provider = default_dict.get("llm_provider", {})
|
||||
self.embedding_model = default_dict.get("embedding_model", {})
|
||||
self.vector_store = default_dict.get("vector_store", {})
|
||||
self.graph_db = default_dict.get("graph_db", {})
|
||||
self.processing = default_dict.get(
|
||||
def _initialize_sections(self, config_data: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Initialize configuration section attributes.
|
||||
|
||||
Args:
|
||||
config_data: Configuration dictionary
|
||||
"""
|
||||
self.llm_provider = config_data.get("llm_provider", {})
|
||||
self.embedding_model = config_data.get("embedding_model", {})
|
||||
self.vector_store = config_data.get("vector_store", {})
|
||||
self.graph_db = config_data.get("graph_db", {})
|
||||
self.processing = config_data.get(
|
||||
"processing", DEFAULT_CONFIG.get("processing", {})
|
||||
)
|
||||
self.pipeline = default_dict.get("pipeline", {})
|
||||
self.logging = default_dict.get("logging", DEFAULT_CONFIG.get("logging", {}))
|
||||
self.quality = default_dict.get("quality", DEFAULT_CONFIG.get("quality", {}))
|
||||
self.security = default_dict.get("security", DEFAULT_CONFIG.get("security", {}))
|
||||
self.custom = default_dict.get("custom", {})
|
||||
self.pipeline = config_data.get("pipeline", {})
|
||||
self.logging = config_data.get(
|
||||
"logging", DEFAULT_CONFIG.get("logging", {})
|
||||
)
|
||||
self.quality = config_data.get(
|
||||
"quality", DEFAULT_CONFIG.get("quality", {})
|
||||
)
|
||||
self.security = config_data.get(
|
||||
"security", DEFAULT_CONFIG.get("security", {})
|
||||
)
|
||||
self.custom = config_data.get("custom", {})
|
||||
|
||||
def _load_from_env(self, config_dict: Dict[str, Any]) -> None:
|
||||
"""
|
||||
@@ -140,27 +175,35 @@ class Config:
|
||||
config_dict: Configuration dictionary to update with env values
|
||||
"""
|
||||
prefix = "SEMANTICA_"
|
||||
prefix_length = len(prefix)
|
||||
|
||||
|
||||
for env_key, env_value in os.environ.items():
|
||||
if not env_key.startswith(prefix):
|
||||
continue
|
||||
|
||||
# Remove prefix and convert to lowercase for consistency
|
||||
config_key = env_key[prefix_length:].lower()
|
||||
|
||||
# Parse the environment variable value
|
||||
# Try JSON first (for complex types like lists/dicts)
|
||||
try:
|
||||
parsed_value = json.loads(env_value)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
# Not valid JSON, try type conversion
|
||||
parsed_value = self._parse_env_value(env_value)
|
||||
|
||||
# Extract and normalize key
|
||||
config_key = self._normalize_env_key(env_key, prefix)
|
||||
|
||||
# Parse value (try JSON first, then type conversion)
|
||||
parsed_value = self._parse_env_value(env_value)
|
||||
|
||||
# Set nested value using dot notation
|
||||
# e.g., "processing_batch_size" -> "processing.batch_size"
|
||||
normalized_key = config_key.replace("_", ".")
|
||||
set_nested_value(config_dict, normalized_key, parsed_value)
|
||||
set_nested_value(config_dict, config_key, parsed_value)
|
||||
|
||||
def _normalize_env_key(self, env_key: str, prefix: str) -> str:
|
||||
"""
|
||||
Normalize environment variable key to configuration key path.
|
||||
|
||||
Args:
|
||||
env_key: Environment variable key (e.g., "SEMANTICA_PROCESSING_BATCH_SIZE")
|
||||
prefix: Prefix to remove (e.g., "SEMANTICA_")
|
||||
|
||||
Returns:
|
||||
Normalized key path (e.g., "processing.batch_size")
|
||||
"""
|
||||
# Remove prefix and convert to lowercase
|
||||
key = env_key[len(prefix):].lower()
|
||||
# Convert underscores to dots for nested access
|
||||
return key.replace("_", ".")
|
||||
|
||||
def _parse_env_value(self, value: str) -> Union[str, int, float, bool]:
|
||||
"""
|
||||
@@ -202,59 +245,11 @@ class Config:
|
||||
ConfigurationError: If configuration is invalid with detailed error messages
|
||||
"""
|
||||
validation_errors = []
|
||||
|
||||
# Validate processing settings
|
||||
processing_config = self.processing
|
||||
if processing_config:
|
||||
# Validate batch_size
|
||||
if "batch_size" in processing_config:
|
||||
batch_size = processing_config["batch_size"]
|
||||
if not isinstance(batch_size, int):
|
||||
validation_errors.append(
|
||||
f"processing.batch_size must be an integer, got {type(batch_size).__name__}"
|
||||
)
|
||||
elif batch_size <= 0:
|
||||
validation_errors.append(
|
||||
f"processing.batch_size must be positive, got {batch_size}"
|
||||
)
|
||||
|
||||
# Validate max_workers
|
||||
if "max_workers" in processing_config:
|
||||
max_workers = processing_config["max_workers"]
|
||||
if not isinstance(max_workers, int):
|
||||
validation_errors.append(
|
||||
f"processing.max_workers must be an integer, got {type(max_workers).__name__}"
|
||||
)
|
||||
elif max_workers <= 0:
|
||||
validation_errors.append(
|
||||
f"processing.max_workers must be positive, got {max_workers}"
|
||||
)
|
||||
|
||||
# Validate quality settings
|
||||
quality_config = self.quality
|
||||
if quality_config:
|
||||
# Validate min_confidence
|
||||
if "min_confidence" in quality_config:
|
||||
confidence = quality_config["min_confidence"]
|
||||
if not isinstance(confidence, (int, float)):
|
||||
validation_errors.append(
|
||||
f"quality.min_confidence must be a number, got {type(confidence).__name__}"
|
||||
)
|
||||
elif not (0.0 <= confidence <= 1.0):
|
||||
validation_errors.append(
|
||||
f"quality.min_confidence must be between 0.0 and 1.0, got {confidence}"
|
||||
)
|
||||
|
||||
# Validate logging settings
|
||||
logging_config = self.logging
|
||||
if logging_config:
|
||||
valid_levels = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
|
||||
if "level" in logging_config:
|
||||
level = logging_config["level"]
|
||||
if level not in valid_levels:
|
||||
validation_errors.append(
|
||||
f"logging.level must be one of {valid_levels}, got {level}"
|
||||
)
|
||||
|
||||
# Validate each configuration section
|
||||
validation_errors.extend(self._validate_processing())
|
||||
validation_errors.extend(self._validate_quality())
|
||||
validation_errors.extend(self._validate_logging())
|
||||
|
||||
# Raise error if any validation failures
|
||||
if validation_errors:
|
||||
@@ -266,6 +261,74 @@ class Config:
|
||||
config_context=self.to_dict(),
|
||||
)
|
||||
|
||||
def _validate_processing(self) -> List[str]:
|
||||
"""Validate processing configuration section."""
|
||||
errors = []
|
||||
if not self.processing:
|
||||
return errors
|
||||
|
||||
# Validate batch_size
|
||||
if "batch_size" in self.processing:
|
||||
batch_size = self.processing["batch_size"]
|
||||
if not isinstance(batch_size, int):
|
||||
errors.append(
|
||||
f"processing.batch_size must be an integer, got {type(batch_size).__name__}"
|
||||
)
|
||||
elif batch_size <= 0:
|
||||
errors.append(
|
||||
f"processing.batch_size must be positive, got {batch_size}"
|
||||
)
|
||||
|
||||
# Validate max_workers
|
||||
if "max_workers" in self.processing:
|
||||
max_workers = self.processing["max_workers"]
|
||||
if not isinstance(max_workers, int):
|
||||
errors.append(
|
||||
f"processing.max_workers must be an integer, got {type(max_workers).__name__}"
|
||||
)
|
||||
elif max_workers <= 0:
|
||||
errors.append(
|
||||
f"processing.max_workers must be positive, got {max_workers}"
|
||||
)
|
||||
|
||||
return errors
|
||||
|
||||
def _validate_quality(self) -> List[str]:
|
||||
"""Validate quality configuration section."""
|
||||
errors = []
|
||||
if not self.quality:
|
||||
return errors
|
||||
|
||||
# Validate min_confidence
|
||||
if "min_confidence" in self.quality:
|
||||
confidence = self.quality["min_confidence"]
|
||||
if not isinstance(confidence, (int, float)):
|
||||
errors.append(
|
||||
f"quality.min_confidence must be a number, got {type(confidence).__name__}"
|
||||
)
|
||||
elif not (0.0 <= confidence <= 1.0):
|
||||
errors.append(
|
||||
f"quality.min_confidence must be between 0.0 and 1.0, got {confidence}"
|
||||
)
|
||||
|
||||
return errors
|
||||
|
||||
def _validate_logging(self) -> List[str]:
|
||||
"""Validate logging configuration section."""
|
||||
errors = []
|
||||
if not self.logging:
|
||||
return errors
|
||||
|
||||
valid_levels = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
|
||||
if "level" in self.logging:
|
||||
level = self.logging["level"]
|
||||
if level not in valid_levels:
|
||||
errors.append(
|
||||
f"logging.level must be one of {valid_levels}, got {level}"
|
||||
)
|
||||
|
||||
return errors
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert configuration to dictionary.
|
||||
@@ -308,12 +371,12 @@ class Config:
|
||||
key_path: Dot-separated key path (e.g., "processing.batch_size")
|
||||
value: Value to set
|
||||
"""
|
||||
# Update the dictionary representation
|
||||
config_dict = self.to_dict()
|
||||
set_nested_value(config_dict, key_path, value)
|
||||
|
||||
# Reinitialize from updated dict
|
||||
updated = Config(config_dict=config_dict)
|
||||
self.__dict__.update(updated.__dict__)
|
||||
# Reinitialize sections from updated dict
|
||||
self._initialize_sections(config_dict)
|
||||
|
||||
def update(self, updates: Dict[str, Any], merge: bool = True) -> None:
|
||||
"""
|
||||
@@ -325,14 +388,14 @@ class Config:
|
||||
"""
|
||||
current_dict = self.to_dict()
|
||||
|
||||
# Merge or replace based on merge flag
|
||||
if merge:
|
||||
updated_dict = merge_dicts(current_dict, updates, deep=True)
|
||||
else:
|
||||
updated_dict = {**current_dict, **updates}
|
||||
|
||||
# Reinitialize from updated dict
|
||||
updated = Config(config_dict=updated_dict)
|
||||
self.__dict__.update(updated.__dict__)
|
||||
# Reinitialize sections from updated dict
|
||||
self._initialize_sections(updated_dict)
|
||||
|
||||
|
||||
class ConfigManager:
|
||||
@@ -396,64 +459,105 @@ class ConfigManager:
|
||||
|
||||
try:
|
||||
file_path = Path(file_path)
|
||||
self._validate_file_exists(file_path)
|
||||
|
||||
if not file_path.exists():
|
||||
raise ConfigurationError(
|
||||
f"Configuration file not found: {file_path}",
|
||||
config_context={"file_path": str(file_path)},
|
||||
)
|
||||
# Load configuration dictionary from file
|
||||
config_dict = self._load_file_content(file_path)
|
||||
|
||||
# Detect format from extension
|
||||
suffix = file_path.suffix.lower()
|
||||
# Create and validate config object
|
||||
config = Config(config_dict=config_dict)
|
||||
if validate:
|
||||
config.validate()
|
||||
|
||||
try:
|
||||
if suffix in (".yaml", ".yml"):
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
config_dict = yaml.safe_load(f)
|
||||
# Store config and file path for potential reload
|
||||
self._config = config
|
||||
self._last_file_path = file_path
|
||||
|
||||
elif suffix == ".json":
|
||||
config_dict = read_json_file(file_path)
|
||||
else:
|
||||
raise ConfigurationError(
|
||||
f"Unsupported configuration file format: {suffix}. "
|
||||
"Supported formats: .yaml, .yml, .json"
|
||||
)
|
||||
|
||||
# Create config object from loaded dictionary
|
||||
config = Config(config_dict=config_dict)
|
||||
|
||||
# Validate configuration if requested
|
||||
if validate:
|
||||
config.validate()
|
||||
|
||||
# Store config and file path for potential reload
|
||||
self._config = config
|
||||
self._last_file_path = file_path
|
||||
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
message="Configuration loaded successfully",
|
||||
)
|
||||
return config
|
||||
except Exception as e:
|
||||
# Re-raise as ConfigurationError if inner try fails
|
||||
raise ConfigurationError(
|
||||
f"Failed to parse configuration file: {str(e)}",
|
||||
config_context={"file_path": str(file_path)},
|
||||
) from e
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id,
|
||||
status="completed",
|
||||
message="Configuration loaded successfully",
|
||||
)
|
||||
return config
|
||||
|
||||
except ConfigurationError:
|
||||
# Re-raise configuration errors as-is
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id, status="failed", message="Configuration error"
|
||||
)
|
||||
raise
|
||||
except Exception as e:
|
||||
# Wrap other exceptions
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id, status="failed", message=str(e)
|
||||
)
|
||||
if isinstance(e, ConfigurationError):
|
||||
raise
|
||||
raise ConfigurationError(
|
||||
f"Failed to load configuration file: {str(e)}",
|
||||
config_context={"file_path": str(file_path)},
|
||||
) from e
|
||||
|
||||
def _validate_file_exists(self, file_path: Path) -> None:
|
||||
"""
|
||||
Validate that configuration file exists.
|
||||
|
||||
Args:
|
||||
file_path: Path to configuration file
|
||||
|
||||
Raises:
|
||||
ConfigurationError: If file does not exist
|
||||
"""
|
||||
if not file_path.exists():
|
||||
raise ConfigurationError(
|
||||
f"Configuration file not found: {file_path}",
|
||||
config_context={"file_path": str(file_path)},
|
||||
)
|
||||
|
||||
def _load_file_content(self, file_path: Path) -> Dict[str, Any]:
|
||||
"""
|
||||
Load configuration dictionary from file.
|
||||
|
||||
Args:
|
||||
file_path: Path to configuration file
|
||||
|
||||
Returns:
|
||||
Configuration dictionary
|
||||
|
||||
Raises:
|
||||
ConfigurationError: If file format is unsupported or parsing fails
|
||||
"""
|
||||
suffix = file_path.suffix.lower()
|
||||
|
||||
if suffix in (".yaml", ".yml"):
|
||||
return self._load_yaml_file(file_path)
|
||||
elif suffix == ".json":
|
||||
return self._load_json_file(file_path)
|
||||
else:
|
||||
raise ConfigurationError(
|
||||
f"Unsupported configuration file format: {suffix}. "
|
||||
"Supported formats: .yaml, .yml, .json"
|
||||
)
|
||||
|
||||
def _load_yaml_file(self, file_path: Path) -> Dict[str, Any]:
|
||||
"""Load YAML configuration file."""
|
||||
try:
|
||||
with open(file_path, "r", encoding="utf-8") as f:
|
||||
return yaml.safe_load(f) or {}
|
||||
except Exception as e:
|
||||
raise ConfigurationError(
|
||||
f"Failed to parse YAML file: {str(e)}",
|
||||
config_context={"file_path": str(file_path)},
|
||||
) from e
|
||||
|
||||
def _load_json_file(self, file_path: Path) -> Dict[str, Any]:
|
||||
"""Load JSON configuration file."""
|
||||
try:
|
||||
return read_json_file(file_path)
|
||||
except Exception as e:
|
||||
raise ConfigurationError(
|
||||
f"Failed to parse JSON file: {str(e)}",
|
||||
config_context={"file_path": str(file_path)},
|
||||
) from e
|
||||
|
||||
def load_from_dict(
|
||||
self, config_dict: Dict[str, Any], validate: bool = True
|
||||
) -> Config:
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+169
-125
@@ -135,53 +135,17 @@ class LifecycleManager:
|
||||
|
||||
try:
|
||||
# Check if already started
|
||||
if self.state in (SystemState.READY, SystemState.RUNNING):
|
||||
self.logger.warning(
|
||||
f"System already in {self.state.value} state, skipping startup"
|
||||
)
|
||||
if self._is_already_started():
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id, status="completed", message="System already started"
|
||||
)
|
||||
return
|
||||
|
||||
# Transition to initializing state
|
||||
self.state = SystemState.INITIALIZING
|
||||
self.logger.info("Starting system lifecycle")
|
||||
# Execute startup sequence
|
||||
self._execute_startup_sequence()
|
||||
|
||||
# Sort hooks by priority (lower priority = earlier execution)
|
||||
sorted_hooks = sorted(self.startup_hooks, key=lambda x: x[1])
|
||||
|
||||
if sorted_hooks:
|
||||
self.logger.debug(f"Executing {len(sorted_hooks)} startup hook(s)")
|
||||
|
||||
# Execute all startup hooks in priority order
|
||||
for hook_fn, priority in sorted_hooks:
|
||||
try:
|
||||
self.logger.debug(
|
||||
f"Executing startup hook with priority {priority}"
|
||||
)
|
||||
hook_fn()
|
||||
except Exception as e:
|
||||
error_msg = f"Startup hook (priority {priority}) failed: {e}"
|
||||
self.logger.error(error_msg)
|
||||
self.state = SystemState.ERROR
|
||||
raise SemanticaError(error_msg) from e
|
||||
|
||||
# Verify all registered components are properly initialized
|
||||
self._verify_components()
|
||||
|
||||
# Run initial health checks on all components
|
||||
health_results = self.health_check()
|
||||
unhealthy_components = [
|
||||
name for name, status in health_results.items() if not status.healthy
|
||||
]
|
||||
|
||||
if unhealthy_components:
|
||||
self.logger.warning(
|
||||
f"Some components are unhealthy after startup: {unhealthy_components}"
|
||||
)
|
||||
else:
|
||||
self.logger.debug("All components are healthy")
|
||||
# Verify and check health
|
||||
self._verify_and_check_health()
|
||||
|
||||
# Transition to ready state
|
||||
self.state = SystemState.READY
|
||||
@@ -202,6 +166,41 @@ class LifecycleManager:
|
||||
)
|
||||
raise
|
||||
|
||||
def _is_already_started(self) -> bool:
|
||||
"""Check if system is already in a started state."""
|
||||
if self.state in (SystemState.READY, SystemState.RUNNING):
|
||||
self.logger.warning(
|
||||
f"System already in {self.state.value} state, skipping startup"
|
||||
)
|
||||
return True
|
||||
return False
|
||||
|
||||
def _execute_startup_sequence(self) -> None:
|
||||
"""Execute startup hooks in priority order."""
|
||||
self.state = SystemState.INITIALIZING
|
||||
self.logger.info("Starting system lifecycle")
|
||||
|
||||
# Execute hooks
|
||||
self._execute_hooks(self.startup_hooks, "startup")
|
||||
|
||||
def _verify_and_check_health(self) -> None:
|
||||
"""Verify components and check their health."""
|
||||
# Verify all registered components are properly initialized
|
||||
self._verify_components()
|
||||
|
||||
# Run initial health checks
|
||||
health_results = self.health_check()
|
||||
unhealthy_components = [
|
||||
name for name, status in health_results.items() if not status.healthy
|
||||
]
|
||||
|
||||
if unhealthy_components:
|
||||
self.logger.warning(
|
||||
f"Some components are unhealthy after startup: {unhealthy_components}"
|
||||
)
|
||||
else:
|
||||
self.logger.debug("All components are healthy")
|
||||
|
||||
def shutdown(self, graceful: bool = True) -> None:
|
||||
"""
|
||||
Execute shutdown sequence.
|
||||
@@ -232,42 +231,16 @@ class LifecycleManager:
|
||||
|
||||
try:
|
||||
# Check if already stopped
|
||||
if self.state == SystemState.STOPPED:
|
||||
self.logger.warning("System already in STOPPED state")
|
||||
if self._is_already_stopped():
|
||||
self.progress_tracker.stop_tracking(
|
||||
tracking_id, status="completed", message="System already stopped"
|
||||
)
|
||||
return
|
||||
|
||||
# Transition to stopping state
|
||||
self.state = SystemState.STOPPING
|
||||
self.logger.info(f"Shutting down system (graceful={graceful})")
|
||||
# Execute shutdown sequence
|
||||
self._execute_shutdown_sequence(graceful)
|
||||
|
||||
# Sort hooks by priority (lower priority = earlier execution)
|
||||
sorted_hooks = sorted(self.shutdown_hooks, key=lambda x: x[1])
|
||||
|
||||
if sorted_hooks:
|
||||
self.logger.debug(f"Executing {len(sorted_hooks)} shutdown hook(s)")
|
||||
|
||||
# Execute all shutdown hooks in priority order
|
||||
for hook_fn, priority in sorted_hooks:
|
||||
try:
|
||||
self.logger.debug(
|
||||
f"Executing shutdown hook with priority {priority}"
|
||||
)
|
||||
hook_fn()
|
||||
except Exception as e:
|
||||
error_msg = f"Shutdown hook (priority {priority}) failed: {e}"
|
||||
|
||||
if graceful:
|
||||
# In graceful mode, log warning but continue
|
||||
self.logger.warning(error_msg)
|
||||
else:
|
||||
# In non-graceful mode, stop on first error
|
||||
self.logger.error(error_msg)
|
||||
raise SemanticaError(error_msg) from e
|
||||
|
||||
# Cleanup all registered components
|
||||
# Cleanup resources
|
||||
self._cleanup_resources()
|
||||
|
||||
# Transition to stopped state
|
||||
@@ -290,6 +263,57 @@ class LifecycleManager:
|
||||
if not graceful:
|
||||
raise
|
||||
|
||||
def _is_already_stopped(self) -> bool:
|
||||
"""Check if system is already stopped."""
|
||||
if self.state == SystemState.STOPPED:
|
||||
self.logger.warning("System already in STOPPED state")
|
||||
return True
|
||||
return False
|
||||
|
||||
def _execute_shutdown_sequence(self, graceful: bool) -> None:
|
||||
"""Execute shutdown hooks in priority order."""
|
||||
self.state = SystemState.STOPPING
|
||||
self.logger.info(f"Shutting down system (graceful={graceful})")
|
||||
|
||||
# Execute hooks with graceful error handling
|
||||
self._execute_hooks(self.shutdown_hooks, "shutdown", graceful=graceful)
|
||||
|
||||
def _execute_hooks(
|
||||
self, hooks: List[Tuple[Callable[[], None], int]], hook_type: str, graceful: bool = False
|
||||
) -> None:
|
||||
"""
|
||||
Execute hooks in priority order.
|
||||
|
||||
Args:
|
||||
hooks: List of (hook_function, priority) tuples
|
||||
hook_type: Type of hooks ("startup" or "shutdown")
|
||||
graceful: Whether to continue on errors (only for shutdown)
|
||||
|
||||
Raises:
|
||||
SemanticaError: If hook fails and not graceful
|
||||
"""
|
||||
# Sort hooks by priority (lower priority = earlier execution)
|
||||
sorted_hooks = sorted(hooks, key=lambda x: x[1])
|
||||
|
||||
if sorted_hooks:
|
||||
self.logger.debug(f"Executing {len(sorted_hooks)} {hook_type} hook(s)")
|
||||
|
||||
# Execute all hooks in priority order
|
||||
for hook_fn, priority in sorted_hooks:
|
||||
try:
|
||||
self.logger.debug(f"Executing {hook_type} hook with priority {priority}")
|
||||
hook_fn()
|
||||
except Exception as e:
|
||||
error_msg = f"{hook_type.capitalize()} hook (priority {priority}) failed: {e}"
|
||||
|
||||
if graceful:
|
||||
# In graceful mode, log warning but continue
|
||||
self.logger.warning(error_msg)
|
||||
else:
|
||||
# In non-graceful mode, stop on first error
|
||||
self.logger.error(error_msg)
|
||||
raise SemanticaError(error_msg) from e
|
||||
|
||||
def health_check(self) -> Dict[str, HealthStatus]:
|
||||
"""
|
||||
Perform comprehensive system health check.
|
||||
@@ -310,65 +334,87 @@ class LifecycleManager:
|
||||
# Record health check timestamp
|
||||
self._last_health_check = time.time()
|
||||
|
||||
health_results = {}
|
||||
|
||||
# Check health of each registered component
|
||||
for component_name, component in self._component_registry.items():
|
||||
try:
|
||||
# Try to get health status from component
|
||||
if hasattr(component, "health_check"):
|
||||
# Component has its own health check method
|
||||
component_health = component.health_check()
|
||||
health_results = {
|
||||
name: self._check_component_health(name, component)
|
||||
for name, component in self._component_registry.items()
|
||||
}
|
||||
|
||||
# Handle different return types
|
||||
if isinstance(component_health, dict):
|
||||
# Dictionary format: {"healthy": bool, "message": str, "details": dict}
|
||||
healthy = component_health.get("healthy", True)
|
||||
message = component_health.get("message", "")
|
||||
details = component_health.get("details", {})
|
||||
elif isinstance(component_health, bool):
|
||||
# Simple boolean
|
||||
healthy = component_health
|
||||
message = ""
|
||||
details = {}
|
||||
else:
|
||||
# Other types: convert to boolean
|
||||
healthy = bool(component_health)
|
||||
message = ""
|
||||
details = {}
|
||||
else:
|
||||
# No health_check method: assume healthy if component exists
|
||||
healthy = component is not None
|
||||
message = "Component exists" if healthy else "Component is None"
|
||||
details = {}
|
||||
# Update cached health status
|
||||
self.health_status.update(health_results)
|
||||
|
||||
# Create health status object
|
||||
status = HealthStatus(
|
||||
component=component_name,
|
||||
healthy=healthy,
|
||||
message=message,
|
||||
details=details,
|
||||
)
|
||||
# Log summary
|
||||
self._log_health_summary(health_results)
|
||||
|
||||
except Exception as e:
|
||||
# Health check failed: mark as unhealthy
|
||||
error_msg = f"Health check failed: {e}"
|
||||
self.logger.warning(
|
||||
f"Component {component_name} health check error: {e}"
|
||||
)
|
||||
return health_results
|
||||
|
||||
status = HealthStatus(
|
||||
component=component_name,
|
||||
healthy=False,
|
||||
message=error_msg,
|
||||
details={"error": str(e), "error_type": type(e).__name__},
|
||||
)
|
||||
def _check_component_health(self, component_name: str, component: Any) -> HealthStatus:
|
||||
"""
|
||||
Check health of a single component.
|
||||
|
||||
# Store results
|
||||
health_results[component_name] = status
|
||||
self.health_status[component_name] = status
|
||||
Args:
|
||||
component_name: Name of the component
|
||||
component: Component instance
|
||||
|
||||
# Log summary of unhealthy components
|
||||
Returns:
|
||||
HealthStatus object for the component
|
||||
"""
|
||||
try:
|
||||
if hasattr(component, "health_check"):
|
||||
# Component has its own health check method
|
||||
component_health = component.health_check()
|
||||
healthy, message, details = self._parse_health_result(component_health)
|
||||
else:
|
||||
# No health_check method: assume healthy if component exists
|
||||
healthy = component is not None
|
||||
message = "Component exists" if healthy else "Component is None"
|
||||
details = {}
|
||||
|
||||
return HealthStatus(
|
||||
component=component_name,
|
||||
healthy=healthy,
|
||||
message=message,
|
||||
details=details,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
# Health check failed: mark as unhealthy
|
||||
error_msg = f"Health check failed: {e}"
|
||||
self.logger.warning(f"Component {component_name} health check error: {e}")
|
||||
|
||||
return HealthStatus(
|
||||
component=component_name,
|
||||
healthy=False,
|
||||
message=error_msg,
|
||||
details={"error": str(e), "error_type": type(e).__name__},
|
||||
)
|
||||
|
||||
def _parse_health_result(self, health_result: Any) -> Tuple[bool, str, Dict[str, Any]]:
|
||||
"""
|
||||
Parse component health check result into standardized format.
|
||||
|
||||
Args:
|
||||
health_result: Health check result (dict, bool, or other)
|
||||
|
||||
Returns:
|
||||
Tuple of (healthy, message, details)
|
||||
"""
|
||||
if isinstance(health_result, dict):
|
||||
# Dictionary format: {"healthy": bool, "message": str, "details": dict}
|
||||
return (
|
||||
health_result.get("healthy", True),
|
||||
health_result.get("message", ""),
|
||||
health_result.get("details", {}),
|
||||
)
|
||||
elif isinstance(health_result, bool):
|
||||
# Simple boolean
|
||||
return health_result, "", {}
|
||||
else:
|
||||
# Other types: convert to boolean
|
||||
return bool(health_result), "", {}
|
||||
|
||||
def _log_health_summary(self, health_results: Dict[str, HealthStatus]) -> None:
|
||||
"""Log summary of health check results."""
|
||||
unhealthy_components = [
|
||||
name for name, status in health_results.items() if not status.healthy
|
||||
]
|
||||
@@ -383,8 +429,6 @@ class LifecycleManager:
|
||||
f"Health check passed for all {len(health_results)} component(s)"
|
||||
)
|
||||
|
||||
return health_results
|
||||
|
||||
def register_component(self, name: str, component: Any) -> None:
|
||||
"""
|
||||
Register a component for health monitoring.
|
||||
|
||||
Reference in New Issue
Block a user