{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/03_Grid_Management.ipynb)\n",
    "\n",
    "# Smart Grid Management Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete smart grid management pipeline: stream grid data from multiple sources (grid sensors, SCADA systems, databases), build temporal grid knowledge graph, monitor grid health in real-time, detect anomalies, and predict failures.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, RepoIngestor, EmailIngestor, MCPIngestor\n",
    "- **Parsing**: JSONParser, StructuredDataParser, CSVParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "- **KG**: GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer\n",
    "- **Analytics**: CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, AutomatedFixer\n",
    "- **Export**: JSONExporter, CSVExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Stream Grid Data → Parse → Extract Entities → Build Temporal Grid KG → Monitor Grid Health → Detect Anomalies → Predict Failures → Generate Alerts → Visualize**\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Stream Grid Data from Multiple Sources\n",
    "\n",
    "Stream grid data from grid sensors, SCADA systems, and databases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import StreamIngestor, FileIngestor, DBIngestor, FeedIngestor\n",
    "from semantica.parse import JSONParser, StructuredDataParser, CSVParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, TripleExtractor\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer\n",
    "from semantica.kg import CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    
    "from semantica.export import JSONExporter, CSVExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "import time\n",
    "from datetime import datetime, timedelta\n",
    "from collections import deque\n",
    "\n",
    "stream_ingestor = StreamIngestor()\n",
    "file_ingestor = FileIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "json_parser = JSONParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "csv_parser = CSVParser()\n",
    "\n",
    "# Real streaming sources for grid data\n",
    "stream_sources = [\n",
    "    {\n",
    "        \"type\": \"kafka\",\n",
    "        \"topic\": \"grid_sensors\",\n",
    "        \"bootstrap_servers\": [\"localhost:9092\"],\n",
    "        \"consumer_config\": {\"group_id\": \"grid_monitor\"}\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"rabbitmq\",\n",
    "        \"queue\": \"scada_events\",\n",
    "        \"connection_url\": \"amqp://user:password@localhost:5672/\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Real grid monitoring APIs\n",
    "grid_apis = [\n",
    "    \"https://api.eia.gov/v2/electricity/operating-generator-capacity/data/\",  # EIA Grid Capacity\n",
    "    \"https://www.energy.gov/data\"  # US Energy Department Grid Data\n",
    "]\n",
    "\n",
    "# Real database connection for grid data\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/grid_db\"\n",
    "db_query = \"SELECT node_id, voltage, current, power, frequency, status, timestamp FROM grid_nodes WHERE timestamp > NOW() - INTERVAL '1 hour' ORDER BY timestamp DESC LIMIT 1000\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample real-time grid data (simulating SCADA system)\n",
    "grid_stream_file = os.path.join(temp_dir, \"grid_stream.json\")\n",
    "grid_stream = [\n",
    "    {\n",
    "        \"node_id\": \"GRID-001\",\n",
    "        \"voltage\": 230.0,\n",
    "        \"current\": 100.0,\n",
    "        \"power\": 23000.0,\n",
    "        \"frequency\": 60.0,\n",
    "        \"status\": \"normal\",\n",
    "        \"timestamp\": (datetime.now() - timedelta(minutes=5)).isoformat(),\n",
    "        \"location\": \"Substation A\"\n",
    "    },\n",
    "    {\n",
    "        \"node_id\": \"GRID-002\",\n",
    "        \"voltage\": 225.0,\n",
    "        \"current\": 95.0,\n",
    "        \"power\": 21375.0,\n",
    "        \"frequency\": 59.8,\n",
    "        \"status\": \"warning\",\n",
    "        \"timestamp\": (datetime.now() - timedelta(minutes=4)).isoformat(),\n",
    "        \"location\": \"Substation B\"\n",
    "    },\n",
    "    {\n",
    "        \"node_id\": \"GRID-003\",\n",
    "        \"voltage\": 235.0,\n",
    "        \"current\": 105.0,\n",
    "        \"power\": 24675.0,\n",
    "        \"frequency\": 60.2,\n",
    "        \"status\": \"normal\",\n",
    "        \"timestamp\": (datetime.now() - timedelta(minutes=3)).isoformat(),\n",
    "        \"location\": \"Substation C\"\n",
    "    },\n",
    "    {\n",
    "        \"node_id\": \"GRID-004\",\n",
    "        \"voltage\": 200.0,\n",
    "        \"current\": 80.0,\n",
    "        \"power\": 16000.0,\n",
    "        \"frequency\": 58.5,\n",
    "        \"status\": \"critical\",\n",
    "        \"timestamp\": (datetime.now() - timedelta(minutes=2)).isoformat(),\n",
    "        \"location\": \"Substation D\"\n",
    "    }\n",
    "]\n",
    "\n",
    "with open(grid_stream_file, 'w') as f:\n",
    "    json.dump(grid_stream, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(grid_stream_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_json(grid_stream_file)\n",
    "\n",
    "print(f\"\\n📊 Grid Data Ingestion Summary:\")\n",
    "print(f\"  Grid stream files: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  Streaming sources: {len(stream_sources)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Grid Entities and Build Temporal Grid Knowledge Graph\n",
    "\n",
    "Extract grid entities and build temporal grid knowledge graph for real-time monitoring.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "triple_extractor = TripleExtractor()\n",
    "\n",
    "grid_entities = []\n",
    "grid_relationships = []\n",
    "\n",
    "# Extract from grid stream data\n",
    "if parsed_data and parsed_data.data:\n",
    "    for node in parsed_data.data if isinstance(parsed_data.data, list) else [parsed_data.data]:\n",
    "        if isinstance(node, dict):\n",
    "            node_id = node.get(\"node_id\", \"\")\n",
    "            location = node.get(\"location\", \"\")\n",
    "            \n",
    "            grid_entities.append({\n",
    "                \"id\": node_id,\n",
    "                \"type\": \"Grid_Node\",\n",
    "                \"name\": node_id,\n",
    "                \"properties\": {\n",
    "                    \"voltage\": node.get(\"voltage\", 0),\n",
    "                    \"current\": node.get(\"current\", 0),\n",
    "                    \"power\": node.get(\"power\", 0),\n",
    "                    \"frequency\": node.get(\"frequency\", 0),\n",
    "                    \"status\": node.get(\"status\", \"\"),\n",
    "                    \"location\": location,\n",
    "                    \"timestamp\": node.get(\"timestamp\", \"\")\n",
    "                }\n",
    "            })\n",
    "            \n",
    "            grid_entities.append({\n",
    "                \"id\": location,\n",
    "                \"type\": \"Location\",\n",
    "                \"name\": location,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            \n",
    "            grid_relationships.append({\n",
    "                \"source\": node_id,\n",
    "                \"target\": location,\n",
    "                \"type\": \"located_at\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            \n",
    "            # Status relationships\n",
    "            if node.get(\"status\") != \"normal\":\n",
    "                grid_entities.append({\n",
    "                    \"id\": f\"{node_id}_status\",\n",
    "                    \"type\": \"Grid_Status\",\n",
    "                    \"name\": node.get(\"status\", \"\"),\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "                grid_relationships.append({\n",
    "                    \"source\": node_id,\n",
    "                    \"target\": f\"{node_id}_status\",\n",
    "                    \"type\": \"has_status\",\n",
    "                    \"properties\": {\"timestamp\": node.get(\"timestamp\", \"\")}\n",
    "                })\n",
    "\n",
    "builder = GraphBuilder()\n",
    "temporal_query = TemporalGraphQuery()\n",
    "temporal_pattern_detector = TemporalPatternDetector()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "\n",
    "grid_kg = builder.build(grid_entities, grid_relationships)\n",
    "\n",
    "metrics = graph_analyzer.compute_metrics(grid_kg)\n",
    "\n",
    "print(f\"Extracted {len(grid_entities)} grid entities\")\n",
    "print(f\"Extracted {len(grid_relationships)} relationships\")\n",
    "print(f\"Built temporal grid knowledge graph with {len(grid_kg.get('entities', []))} entities\")\n",
    "print(f\"Graph density: {metrics.get('density', 0):.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Monitor Grid Health in Real-Time\n",
    "\n",
    "Monitor grid health using temporal queries and pattern detection.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "centrality_calculator = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "current_time = datetime.now().isoformat()\n",
    "start_time = (datetime.now() - timedelta(minutes=10)).isoformat()\n",
    "\n",
    "# Query current grid status\n",
    "current_grid_status = temporal_query.query_time_range(\n",
    "    graph=grid_kg,\n",
    "    query=\"Find current grid status\",\n",
    "    start_time=start_time,\n",
    "    end_time=current_time\n",
    ")\n",
    "\n",
    "# Monitor grid health metrics\n",
    "grid_health_metrics = []\n",
    "if parsed_data and parsed_data.data:\n",
    "    for node in parsed_data.data if isinstance(parsed_data.data, list) else [parsed_data.data]:\n",
    "        if isinstance(node, dict):\n",
    "            voltage = node.get(\"voltage\", 0)\n",
    "            frequency = node.get(\"frequency\", 0)\n",
    "            status = node.get(\"status\", \"\")\n",
    "            \n",
    "            health_score = 100\n",
    "            issues = []\n",
    "            \n",
    "            # Voltage check (normal: 220-240V)\n",
    "            if voltage < 210 or voltage > 250:\n",
    "                health_score -= 20\n",
    "                issues.append(\"Voltage out of range\")\n",
    "            \n",
    "            # Frequency check (normal: 59.5-60.5 Hz)\n",
    "            if frequency < 59.0 or frequency > 61.0:\n",
    "                health_score -= 20\n",
    "                issues.append(\"Frequency out of range\")\n",
    "            \n",
    "            if status == \"critical\":\n",
    "                health_score -= 30\n",
    "                issues.append(\"Critical status\")\n",
    "            elif status == \"warning\":\n",
    "                health_score -= 10\n",
    "                issues.append(\"Warning status\")\n",
    "            \n",
    "            grid_health_metrics.append({\n",
    "                \"node_id\": node.get(\"node_id\", \"\"),\n",
    "                \"health_score\": health_score,\n",
    "                \"status\": status,\n",
    "                \"issues\": issues,\n",
    "                \"timestamp\": node.get(\"timestamp\", \"\")\n",
    "            })\n",
    "\n",
    "# Detect temporal patterns\n",
    "temporal_patterns = temporal_pattern_detector.detect_temporal_patterns(\n",
    "    grid_kg,\n",
    "    pattern_type=\"anomaly\",\n",
    "    min_frequency=1\n",
    ")\n",
    "\n",
    "centrality_result = centrality_calculator.calculate_degree_centrality(grid_kg)\n",
    "centrality_scores = centrality_result.get('centrality', {})\n",
    "communities = community_detector.detect_communities(grid_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(grid_kg)\n",
    "\n",
    "print(f\"Monitoring {len(current_grid_status.get('entities', []))} grid nodes\")\n",
    "print(f\"Grid health metrics tracked: {len(grid_health_metrics)}\")\n",
    "print(f\"Nodes with issues: {len([m for m in grid_health_metrics if m.get('health_score') < 80])}\")\n",
    "print(f\"Temporal patterns: {len(temporal_patterns)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Detect Anomalies and Predict Failures\n",
    "\n",
    "Detect grid anomalies and predict potential failures.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Grid failure prediction rules\n",
    "inference_engine.add_rule(\"IF voltage < 200 AND frequency < 59 THEN potential_failure\")\n",
    "inference_engine.add_rule(\"IF status is critical AND health_score < 50 THEN immediate_action_required\")\n",
    "inference_engine.add_rule(\"IF multiple nodes show same issue THEN systemic_problem\")\n",
    "\n",
    "# Detect anomalies and predict failures\n",
    "anomalies = []\n",
    "failure_predictions = []\n",
    "alerts = []\n",
    "\n",
    "for health_metric in grid_health_metrics:\n",
    "    if health_metric.get(\"health_score\", 100) < 80:\n",
    "        anomaly = {\n",
    "            \"node_id\": health_metric.get(\"node_id\", \"\"),\n",
    "            \"severity\": \"high\" if health_metric.get(\"health_score\", 100) < 50 else \"medium\",\n",
    "            \"health_score\": health_metric.get(\"health_score\", 100),\n",
    "            \"issues\": health_metric.get(\"issues\", []),\n",
    "            \"timestamp\": health_metric.get(\"timestamp\", \"\")\n",
    "        }\n",
    "        anomalies.append(anomaly)\n",
    "        \n",
    "        if health_metric.get(\"health_score\", 100) < 50:\n",
    "            failure_predictions.append({\n",
    "                \"node_id\": health_metric.get(\"node_id\", \"\"),\n",
    "                \"predicted_failure\": True,\n",
    "                \"confidence\": 0.8,\n",
    "                \"reasons\": health_metric.get(\"issues\", []),\n",
    "                \"timestamp\": health_metric.get(\"timestamp\", \"\")\n",
    "            })\n",
    "            \n",
    "            alert = {\n",
    "                \"alert_id\": f\"alert_{health_metric.get('node_id', '')}_{int(time.time())}\",\n",
    "                \"type\": \"grid_failure_prediction\",\n",
    "                \"severity\": \"critical\",\n",
    "                \"node\": health_metric.get(\"node_id\", \"\"),\n",
    "                \"message\": f\"Potential failure predicted: {', '.join(health_metric.get('issues', []))}\",\n",
    "                \"timestamp\": health_metric.get(\"timestamp\", \"\")\n",
    "            }\n",
    "            alerts.append(alert)\n",
    "        \n",
    "        inference_engine.add_fact({\n",
    "            \"node_id\": health_metric.get(\"node_id\", \"\"),\n",
    "            \"health_score\": health_metric.get(\"health_score\", 100),\n",
    "            \"status\": health_metric.get(\"status\", \"\")\n",
    "        })\n",
    "\n",
    "predicted_failures = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Detected {len(anomalies)} grid anomalies\")\n",
    "print(f\"Predicted {len(failure_predictions)} potential failures\")\n",
    "print(f\"Generated {len(alerts)} alerts\")\n",
    "print(f\"Inferred {len(predicted_failures)} failure patterns\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Reports and Visualize\n",
    "\n",
    "Generate grid management reports and visualize results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(grid_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(grid_kg, os.path.join(temp_dir, \"grid_kg.json\"))\n",
    "csv_exporter.export_entities(grid_entities, os.path.join(temp_dir, \"grid_entities.csv\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Grid management detected {len(anomalies)} anomalies and predicted {len(failure_predictions)} potential failures\",\n",
    "    \"nodes_monitored\": len([e for e in grid_entities if e.get(\"type\") == \"Grid_Node\"]),\n",
    "    \"anomalies\": len(anomalies),\n",
    "    \"failure_predictions\": len(failure_predictions),\n",
    "    \"alerts\": len(alerts),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(grid_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(grid_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(grid_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Real-time grid monitoring active\")\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Stream Grid Data → Parse → Extract → Build Temporal Grid KG → Monitor Health → Detect Anomalies → Predict Failures → Alerts → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
