{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)\n",
    "\n",
    "# Energy Market Analysis - Temporal KGs & Trend Prediction\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **energy market analysis** using Semantica with focus on **temporal knowledge graphs**, **trend prediction**, and **market entity extraction**. The pipeline analyzes pricing trends and market movements using temporal market knowledge graphs to predict energy market trends and forecast pricing.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Temporal Knowledge Graphs**: Builds temporal KGs to track energy market trends over time\n",
    "- **Trend Prediction**: Uses temporal analysis and reasoning to predict market movements\n",
    "- **Market Entity Extraction**: Extracts energy market entities (Market, Price, Region, Trend, Forecast, EnergyType)\n",
    "- **Temporal Pattern Detection**: Identifies patterns in energy pricing and market trends\n",
    "- **Seed Data Integration**: Uses market foundation data for entity resolution\n",
    "- **Forecasting**: Emphasizes reasoning-based market forecasting\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Understand how to build temporal knowledge graphs for market analysis\n",
    "- Learn to detect temporal patterns in energy pricing data\n",
    "- Master trend prediction using reasoning and pattern detection\n",
    "- Explore temporal graph queries for market trend analysis\n",
    "- Practice market entity extraction and relationship mapping\n",
    "- Analyze energy market trends and forecasting\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Seed Data Loading]\n",
    "    A --> C[Document Parsing]\n",
    "    B --> D[Text Processing]\n",
    "    C --> D\n",
    "    D --> E[Entity Extraction]\n",
    "    E --> F[Relationship Extraction]\n",
    "    F --> G[Deduplication]\n",
    "    G --> H[Temporal KG Construction]\n",
    "    H --> I[Embedding Generation]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Temporal Pattern Detection]\n",
    "    H --> L[Temporal Queries]\n",
    "```\n",
    "\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n"
     ]
    }
   ],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Configuration & Setup\n",
    "\n",
    "Configure API keys and set up constants for the energy market analysis pipeline, including temporal granularity for trend tracking.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 200\n",
    "TEMPORAL_GRANULARITY = \"day\"  # For market trend tracking\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Data Ingestion\n",
    "\n",
    "Ingest energy market data from multiple sources including RSS feeds, web APIs, and local files.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting from 15 RSS feeds...\n",
      "🧠 Semantica is ingesting: File: energy_market.txt 🔄📥 (0.0s) | 🧠 Semantica is ingesting: 500 Server Error: Internal Server Error for url: https://www.energycentral.com/rss ❌📥 (19.5s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.energycentral.com/rss: 500 Server Error: Internal Server Error for url: https://www.energycentral.com/rss\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [1/15] Failed to load https://www.energycentral.com/rss: Failed to fetch feed: 500 Server Error: Internal S\n",
      "🧠 Semantica is ingesting: 500 Server Error: Internal Server Error for url: https://www.energycentral.com/rss ❌📥 (19.5s) | 🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.renewableenergyworld.com/rss ❌📥 (0.6s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.renewableenergyworld.com/rss: 403 Client Error: Forbidden for url: https://www.renewableenergyworld.com/rss\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [2/15] Failed to load https://www.renewableenergyworld.com/rss: Failed to fetch feed: 403 Client Error: Forbidden \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to parse feed: not well-formed (invalid token): line 16, column 71\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.renewableenergyworld.com/rss ❌📥 (0.6s) | 🧠 Semantica is ingesting: Failed to parse feed: not well-formed (invalid token): line 16, column 71 ❌📥 (1.9s)  [3/15] Failed to load https://www.greentechmedia.com/rss: Failed to parse feed: not well-formed (invalid tok\n",
      "🧠 Semantica is ingesting: Failed to parse feed: not well-formed (invalid token): line 16, column 71 ❌📥 (1.9s) | 🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.utilitydive.com/rss ❌📥 (1.6s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.utilitydive.com/rss: 404 Client Error: Not Found for url: https://www.utilitydive.com/rss\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [4/15] Failed to load https://www.utilitydive.com/rss: Failed to fetch feed: 404 Client Error: Not Found \n",
      "🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.utilitydive.com/rss ❌📥 (1.6s) | 🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.power-eng.com/rss ❌📥 (0.8s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.power-eng.com/rss: 403 Client Error: Forbidden for url: https://www.power-eng.com/rss\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [5/15] Failed to load https://www.power-eng.com/rss: Failed to fetch feed: 403 Client Error: Forbidden \n",
      "🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.power-eng.com/rss ❌📥 (0.8s) | 🧠 Semantica is ingesting: Ingested 50 items |███████████████| 100.0% ✅📥  [6/15] Loaded 50 documents from https://www.energy-storage.news/rss\n",
      "🧠 Semantica is ingesting: Ingested 50 items |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.pv-magazine.com/rss ❌📥 (0.8s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.pv-magazine.com/rss: 403 Client Error: Forbidden for url: https://www.pv-magazine.com/rss\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [7/15] Failed to load https://www.pv-magazine.com/rss: Failed to fetch feed: 403 Client Error: Forbidden \n",
      "🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.pv-magazine.com/rss ❌📥 (0.8s) | 🧠 Semantica is ingesting: Ingested 20 items |███████████████| 100.0% ✅📥  [8/15] Loaded 20 documents from https://www.windpowermonthly.com/rss\n",
      "🧠 Semantica is ingesting: Ingested 20 items |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.rechargenews.com/rss?zephr_sso_ott=JBlrTY ❌📥 (2.9s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.rechargenews.com/rss: 404 Client Error: Not Found for url: https://www.rechargenews.com/rss?zephr_sso_ott=JBlrTY\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [9/15] Failed to load https://www.rechargenews.com/rss: Failed to fetch feed: 404 Client Error: Not Found \n",
      "🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.rechargenews.com/rss?zephr_sso_ott=JBlrTY ❌📥 (2.9s) | 🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.energystoragejournal.com/rss ❌📥 (3.7s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.energystoragejournal.com/rss: 403 Client Error: Forbidden for url: https://www.energystoragejournal.com/rss\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [10/15] Failed to load https://www.energystoragejournal.com/rss: Failed to fetch feed: 403 Client Error: Forbidden \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to parse feed: mismatched tag: line 17, column 6\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.energystoragejournal.com/rss ❌📥 (3.7s) | 🧠 Semantica is ingesting: Failed to parse feed: mismatched tag: line 17, column 6 ❌📥 (1.0s)  [11/15] Failed to load https://feeds.feedburner.com/EnergyBiz: Failed to parse feed: mismatched tag: line 17, col\n",
      "🧠 Semantica is ingesting: Failed to parse feed: mismatched tag: line 17, column 6 ❌📥 (1.0s) | 🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.energy.gov/rss-feeds ❌📥 (2.9s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.energy.gov/rss-feeds: 404 Client Error: Not Found for url: https://www.energy.gov/rss-feeds\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [12/15] Failed to load https://www.energy.gov/rss-feeds: Failed to fetch feed: 404 Client Error: Not Found \n",
      "🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.energy.gov/rss-feeds ❌📥 (2.9s) | 🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.iea.org/rss ❌📥 (1.8s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to fetch feed https://www.iea.org/rss: 404 Client Error: Not Found for url: https://www.iea.org/rss\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  [13/15] Failed to load https://www.iea.org/rss: Failed to fetch feed: 404 Client Error: Not Found \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Failed to parse feed: syntax error: line 4, column 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is ingesting: 404 Client Error: Not Found for url: https://www.iea.org/rss ❌📥 (1.8s) | 🧠 Semantica is ingesting: Failed to parse feed: syntax error: line 4, column 0 ❌📥 (1.2s)  [14/15] Failed to load https://www.irena.org/rss: Failed to parse feed: syntax error: line 4, column\n",
      "🧠 Semantica is ingesting: Failed to parse feed: syntax error: line 4, column 0 ❌📥 (1.2s) | 🧠 Semantica is ingesting: Ingested 73 items |███████████████| 100.0% ✅📥  [15/15] Loaded 73 documents from https://www.cleanenergywire.org/rss\n",
      "\n",
      "Ingesting from 7 web sources...\n",
      "🧠 Semantica is ingesting: Ingested 73 items |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: Ingested https://www.eia.gov/todayinenergy/ (200) |███████████████| 100.0% ✅📥  [1/7] Loaded content from https://www.eia.gov/todayinenergy/ (19365 chars)\n",
      "🧠 Semantica is ingesting: Ingested https://www.eia.gov/todayinenergy/ (200) |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: Ingested https://www.energy.gov/ (200) |███████████████| 100.0% ✅📥  [2/7] Loaded content from https://www.energy.gov/ (5437 chars)\n",
      "🧠 Semantica is ingesting: Ingested https://www.energy.gov/ (200) |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: Ingested https://www.epa.gov/energy (200) |███████████████| 100.0% ✅📥  [3/7] Loaded content from https://www.epa.gov/energy (2743 chars)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "URL https://www.nrel.gov/news/ blocked by robots.txt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is ingesting: Ingested https://www.epa.gov/energy (200) |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: URL blocked by robots.txt: https://www.nrel.gov/news/ ❌📥 (0.4s)  [4/7] Failed to load https://www.nrel.gov/news/: URL blocked by robots.txt: https://www.nrel.gov/ne\n",
      "🧠 Semantica is ingesting: URL blocked by robots.txt: https://www.nrel.gov/news/ ❌📥 (0.4s) | 🧠 Semantica is ingesting: Ingested https://www.energy.gov/office-energy-efficiency-renewable-energy (200) |███████████████| 100.0% ✅📥  [5/7] Loaded content from https://www.energy.gov/office-energy-efficiency-renewable-energy (6564 chars)\n",
      "🧠 Semantica is ingesting: Ingested https://www.energy.gov/office-energy-efficiency-renewable-energy (200) |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: Ingested https://www.energy.gov/oe/office-electricity (200) |███████████████| 100.0% ✅📥  [6/7] Loaded content from https://www.energy.gov/oe/office-electricity (8194 chars)\n",
      "🧠 Semantica is ingesting: Ingested https://www.energy.gov/oe/office-electricity (200) |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: Ingested https://www.energy.gov/ne/office-nuclear-energy (200) |███████████████| 100.0% ✅📥  [7/7] Loaded content from https://www.energy.gov/ne/office-nuclear-energy (7209 chars)\n",
      "\n",
      "Total ingested: 149 documents\n"
     ]
    }
   ],
   "source": [
    "from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
    "import os\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "documents = []\n",
    "\n",
    "# Ingest from energy market RSS feeds\n",
    "energy_feeds = [\n",
    "    \"https://www.energycentral.com/rss\",\n",
    "    \"https://www.renewableenergyworld.com/rss\",\n",
    "    \"https://www.greentechmedia.com/rss\",\n",
    "    \"https://www.utilitydive.com/rss\",\n",
    "    \"https://www.power-eng.com/rss\",\n",
    "    \"https://www.energy-storage.news/rss\",\n",
    "    \"https://www.pv-magazine.com/rss\",\n",
    "    \"https://www.windpowermonthly.com/rss\",\n",
    "    \"https://www.rechargenews.com/rss\",\n",
    "    \"https://www.energystoragejournal.com/rss\",\n",
    "    \"https://feeds.feedburner.com/EnergyBiz\",\n",
    "    \"https://www.energy.gov/rss-feeds\",\n",
    "    \"https://www.iea.org/rss\",\n",
    "    \"https://www.irena.org/rss\",\n",
    "    \"https://www.cleanenergywire.org/rss\"\n",
    "]\n",
    "\n",
    "print(f\"Ingesting from {len(energy_feeds)} RSS feeds...\")\n",
    "feed_ingestor = FeedIngestor()\n",
    "for i, feed_url in enumerate(energy_feeds, 1):\n",
    "    try:\n",
    "        feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_url\n",
    "                item.metadata['source_type'] = 'rss_feed'\n",
    "                documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        print(f\"  [{i}/{len(energy_feeds)}] Loaded {feed_count} documents from {feed_url}\")\n",
    "    except Exception as e:\n",
    "        print(f\"  [{i}/{len(energy_feeds)}] Failed to load {feed_url}: {str(e)[:50]}\")\n",
    "\n",
    "# Web ingestion from energy market websites\n",
    "energy_web_sources = [\n",
    "    \"https://www.eia.gov/todayinenergy/\",\n",
    "    \"https://www.energy.gov/\",\n",
    "    \"https://www.epa.gov/energy\",\n",
    "    \"https://www.nrel.gov/news/\",\n",
    "    \"https://www.energy.gov/office-energy-efficiency-renewable-energy\",\n",
    "    \"https://www.energy.gov/oe/office-electricity\",\n",
    "    \"https://www.energy.gov/ne/office-nuclear-energy\"\n",
    "]\n",
    "\n",
    "print(f\"\\nIngesting from {len(energy_web_sources)} web sources...\")\n",
    "web_ingestor = WebIngestor(respect_robots=True, delay=1.0)\n",
    "for i, web_url in enumerate(energy_web_sources, 1):\n",
    "    try:\n",
    "        web_content = web_ingestor.ingest_url(web_url)\n",
    "        if web_content and web_content.text:\n",
    "            # Add content attribute for compatibility\n",
    "            web_content.content = web_content.text\n",
    "            if not hasattr(web_content, 'metadata'):\n",
    "                web_content.metadata = {}\n",
    "            web_content.metadata['source'] = web_url\n",
    "            web_content.metadata['source_type'] = 'web_page'\n",
    "            documents.append(web_content)\n",
    "            print(f\"  [{i}/{len(energy_web_sources)}] Loaded content from {web_url} ({len(web_content.text)} chars)\")\n",
    "        else:\n",
    "            print(f\"  [{i}/{len(energy_web_sources)}] No content from {web_url}\")\n",
    "    except Exception as e:\n",
    "        print(f\"  [{i}/{len(energy_web_sources)}] Failed to load {web_url}: {str(e)[:50]}\")\n",
    "\n",
    "print(f\"\\nTotal ingested: {len(documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded seed data with 17 entities and 10 relationships\n",
      "Seed entities: 17, Seed relationships: 10\n"
     ]
    }
   ],
   "source": [
    "from semantica.seed import SeedDataManager\n",
    "\n",
    "seed_manager = SeedDataManager()\n",
    "\n",
    "# Load market foundation seed data\n",
    "market_foundation = {\n",
    "    \"markets\": [\"Energy Market\", \"Renewable Energy Market\", \"Electricity Market\"],\n",
    "    \"regions\": [\"North America\", \"Region A\", \"Region B\", \"Europe\", \"Asia\"],\n",
    "    \"energy_types\": [\"Solar\", \"Wind\", \"Hydro\", \"Geothermal\", \"Biomass\"],\n",
    "    \"trends\": [\"increasing\", \"decreasing\", \"stable\", \"volatile\"]\n",
    "}\n",
    "\n",
    "# Convert dictionary to entities and add to seed data\n",
    "seed_entities = []\n",
    "\n",
    "# Add market entities\n",
    "for market in market_foundation[\"markets\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"market_{market.lower().replace(' ', '_')}\",\n",
    "        \"name\": market,\n",
    "        \"type\": \"Market\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add region entities\n",
    "for region in market_foundation[\"regions\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"region_{region.lower().replace(' ', '_')}\",\n",
    "        \"name\": region,\n",
    "        \"type\": \"Region\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add energy type entities\n",
    "for energy_type in market_foundation[\"energy_types\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"energy_type_{energy_type.lower()}\",\n",
    "        \"name\": energy_type,\n",
    "        \"type\": \"EnergyType\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add trend entities\n",
    "for trend in market_foundation[\"trends\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"trend_{trend.lower()}\",\n",
    "        \"name\": trend,\n",
    "        \"type\": \"Trend\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add entities to seed manager\n",
    "seed_manager.seed_data.entities.extend(seed_entities)\n",
    "\n",
    "# Create seed relationships for better graph structure\n",
    "seed_relationships = []\n",
    "\n",
    "# Link renewable energy market to energy types\n",
    "renewable_market_id = \"market_renewable_energy_market\"\n",
    "for energy_type in market_foundation[\"energy_types\"]:\n",
    "    energy_id = f\"energy_type_{energy_type.lower()}\"\n",
    "    seed_relationships.append({\n",
    "        \"source\": renewable_market_id,\n",
    "        \"target\": energy_id,\n",
    "        \"type\": \"includes\",\n",
    "        \"source_name\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Link regions to markets\n",
    "for region in market_foundation[\"regions\"]:\n",
    "    region_id = f\"region_{region.lower().replace(' ', '_')}\"\n",
    "    # Link to main energy market\n",
    "    seed_relationships.append({\n",
    "        \"source\": region_id,\n",
    "        \"target\": \"market_energy_market\",\n",
    "        \"type\": \"located_in\",\n",
    "        \"source_name\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add relationships to seed manager\n",
    "seed_manager.seed_data.relationships.extend(seed_relationships)\n",
    "\n",
    "print(f\"Loaded seed data with {len(seed_entities)} entities and {len(seed_relationships)} relationships\")\n",
    "print(f\"Seed entities: {len(seed_manager.seed_data.entities)}, Seed relationships: {len(seed_manager.seed_data.relationships)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Document Parsing\n",
    "\n",
    "Parse structured energy market data from various formats including JSON, HTML, and XML.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing 149 documents...\n",
      "🧠 Semantica is parsing: Document: Two US companies, GridStor, and CPS Energy are making advancements on Texas energy storage deployments, with GridStor executing a tolling agreement and CPS Energy issuing a new request for proposals. 🔄🔍 (0.0s) | 🧠 Semantica is parsing: Document: 400MWh BESS outage could result in US$1.2 million in monthly losses. 🔄🔍 (0.0s)CPS Energy issuing a new request for proposals. 🔄🔍 (0.0s)  Parsed 50/149 documents...\n",
      "🧠 Semantica is parsing: Document file not found: In 2026, Germany's government under chancellor Friedrich Merz must end the uncertainty prevailing during its first months in office. It can no longer delay major climate and energy policy decisions to put the country on track to climate neutrality by 2045, and has pledged to reset the energy transition by lowering costs and improving resilience. However, it has yet to present crucial reforms in electricity, industry, transport, and buildings. Patience across the economy and society wears thin, while geopolitical tensions, trade disputes and a sluggish economy are increasing pressure to deliver results ahead of state elections where the far right could make significant gains. If Europe's largest economy does not stay the course on climate, the EU will struggle to take ambitious long-term decisions planned for next year. This dossier previews Germany’s and the EU's climate and energy transition in 2026, and features interviews with experts on their expectations for the year ahead. ❌🔍 (0.0s) | 🧠 Semantica is parsing: Document file not found: Journalists! Mark your calendars – or rather let us do it for you. Europe is complicated and even journalists from EU member states sometimes struggle to understand how climate and energy policy is made, and when institutions take key decisions. This timeline flags key European energy and climate reporting events and developments and includes concise background information necessary to jump-start your coverage. ❌🔍 (0.0s)  Parsed 100/149 documents...\n",
      "🧠 Semantica is parsing: Document: Germany covers nearly 56 percent of 2025 electricity use with renewables 🔄🔍 (0.0s) | 🧠 Semantica is parsing: Document file not found:  November saw the EU make progress towards agreeing its 2035 and 2040 climate goals. These are required under the Paris Agreement and a much-needed guiding star for the bloc’s transition towards net-zero emissions by 2050. For the first time, the EU agreed on a targeted range rather than a specific number for its 2035 emissions reduction target. It also submitted this late, having missed two United Nations deadlines. This undermined the EU’s hand at COP30, according to an EU lawmaker. Efforts to cut back on the EU’s green legislation also continue, painting a mixed picture for climate and energy policy in the union. \n",
      "🧠 Semantica is parsing: Document file not found:  November saw the EU make progress towards agreeing its 2035 and 2040 climate goals. These are required under the Paris Agreement and a much-needed guiding star for the bloc’s transition towards net-zero emissions by 2050. For the first time, the EU agreed on a targeted range rather than a specific number for its 2035 emissions reduction target. It also submitted this late, having missed two United Nations deadlines. This undermined the EU’s hand at COP30, according to an EU lawmaker. Efforts to cut back on the EU’s green legislation also continue, painting a mixed picture for climate and energy policy in the union. \n",
      "🧠 Semantica is parsing: Document: German ministry tables ban on new oil and gas projects in protected maritime zones 🔄🔍 (0.0s) | 🧠 Semantica is parsing: Document:  means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. About About Our Agency Our Agency History Our Leadership & Offices The National Nuclear Security Administration The Energy Information Administration National Laboratories Power Marketing Administrations Our Outreach Our Outreach Newsroom Digital Engagement and Media Brand Guides Our Mission Our Mission Genesis Mission Security & Safety Security & Safety Nuclear Security Energy Security Cybersecurity Environmental & Legacy Management Research, Technology, & Economic Security Emergency Response Scientific Excellence Scientific Excellence Scientific Research Database (OSTI) Nobel Prize Laureates National Science Bowl Environment Environment Earth Systems Modeling Extreme Weather Resiliency Energy Access Energy Access 2025 DOE 403 Orders Grid Deployment & Transmission Puerto Rico Grid Resilience & Transitions (PR 100) Tribal Energy Access Economic Growth Economic Growth Energy Workforce American Manufacturing Technology Transitions & Early Investments Commercial Implementation Global Diplomacy & Leadership Global Diplomacy & Leadership Arctic Cooperation International Market Development Topics Topics View All Topics Funding Opportunities Funding Opportunities View Funding Opportunities Funding Opportunities By Office Funding Opportunities By Office Office of the Under Secretary of Energy Advanced Research Projects Agency-Energy Office of Energy Efficiency and Renewable Energy Federal Energy Management Program Office of Fossil Energy Grid Deployment Office Office of Indian Energy Office of Manufacturing and Energy Supply Chains Office of Science Consumer Savings Consumer Savings Tax Credits & Rebates Energy Saving Tips Weatherization Assistance Program Services & Opportunities Services & Opportunities Career Pathways Career Pathways Internships & Fellowships Apprenticeships & Workforce Development Work at DOE Powering today, Transforming tomorrow. Front page carousel Promises Made, Promises Kept DOE is Delivering on President Trump’s Agenda of American Energy Dominance. Read more here Genesis Mission Genesis Mission leverages the Department of Energy's unique scientific datasets-spanning more than 100 petabytes of experimental and simulation data across every major domain of science—to double the pace of American research and innovation within a decade. Visit the Genesis Mission homepage Global Artificial Intelligence Leadership DOE is establishing landmark public-private partnerships to deliver the DOE’s largest AI supercomputer and accelerate scientific discovery. Read more here Fusion Science and Technology Roadmap The Fusion Science and Technology Roadmap is a national strategy to accelerate the development and commercialization of fusion energy on the most rapid, responsible timeline in history. Read more here Trending: LNG Exports Emergency Authority Nuclear 101 Artificial Intelligence Genesis Mission Latest News Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Indiana Coal Plants Remain Open to Prevent Blackouts in Midwest December 23, 2025 Promises Made, Promises Kept December 18, 2025 Artificial Intelligence Energy Department Announces Collaboration Agreements with 24 Organizations to Advance the Genesis Mission December 18, 2025 Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Washington Coal Plant Remains Open to Ensure Affordable, Reliable and Secure Power Heading into Winter December 16, 2025 Learn More Genesis Mission Genesis Mission is building a new instrument for discovery, uniting human curiosity with machine intelligence to double our scientific output and tackle humanity's greatest challenges. Video Url Genesis Mission video US Dept of Energy Visit the Genesis Mission homepage Administrations & National Labs Get to know our semi-independent organizations and see how they’re tackling the nation’s toughest challenges. National Nuclear Security Administration Learn More about National Nuclear Security Administration May 2, 2025 Power Marketing Administrations Learn More about Power Marketing Administrations October 16, 2025 U.S Department of Energy National Laboratories Learn More about U.S Department of Energy National Laboratories October 16, 2025 U.S. Energy Information Administration (EIA) Learn More about U.S. Energy Information Administration (EIA) October 16, 2025 Stay Connected Sign up to receive news and updates from the U.S. Department of Energy straight to your inbox. Email Committed to Restoring America’s Energy Dominance. Follow Us Link to Facebook Link to Instagram Link to Linkedin Link to Youtube Link to X Link to Flickr Quick Links Leadership & Offices Mission Newsroom Contact Us Careers Resources Budget & Performance Directives, Delegations, & Requirements Freedom of Information Act (FOIA) Inspector General Privacy Program Federal Government USA.gov The White House Open Gov Accessibility Privacy Information Quality Web Policies Vulnerability Disclosure Program Whistleblower Protection Equal Employment Opportunity 🔄🔍 (0.0s)  Parsed 149/149 documents...\n",
      "Parsed 149 documents\n"
     ]
    }
   ],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            format=\"auto\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc.content if hasattr(doc, 'content') else str(doc))\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Parsed {len(parsed_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Text Processing\n",
    "\n",
    "Normalize energy market data and split documents using recursive chunking to preserve market context.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalizing 149 documents...\n",
      "🧠 Semantica is parsing: Document:  means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. About About Our Agency Our Agency History Our Leadership & Offices The National Nuclear Security Administration The Energy Information Administration National Laboratories Power Marketing Administrations Our Outreach Our Outreach Newsroom Digital Engagement and Media Brand Guides Our Mission Our Mission Genesis Mission Security & Safety Security & Safety Nuclear Security Energy Security Cybersecurity Environmental & Legacy Management Research, Technology, & Economic Security Emergency Response Scientific Excellence Scientific Excellence Scientific Research Database (OSTI) Nobel Prize Laureates National Science Bowl Environment Environment Earth Systems Modeling Extreme Weather Resiliency Energy Access Energy Access 2025 DOE 403 Orders Grid Deployment & Transmission Puerto Rico Grid Resilience & Transitions (PR 100) Tribal Energy Access Economic Growth Economic Growth Energy Workforce American Manufacturing Technology Transitions & Early Investments Commercial Implementation Global Diplomacy & Leadership Global Diplomacy & Leadership Arctic Cooperation International Market Development Topics Topics View All Topics Funding Opportunities Funding Opportunities View Funding Opportunities Funding Opportunities By Office Funding Opportunities By Office Office of the Under Secretary of Energy Advanced Research Projects Agency-Energy Office of Energy Efficiency and Renewable Energy Federal Energy Management Program Office of Fossil Energy Grid Deployment Office Office of Indian Energy Office of Manufacturing and Energy Supply Chains Office of Science Consumer Savings Consumer Savings Tax Credits & Rebates Energy Saving Tips Weatherization Assistance Program Services & Opportunities Services & Opportunities Career Pathways Career Pathways Internships & Fellowships Apprenticeships & Workforce Development Work at DOE Powering today, Transforming tomorrow. Front page carousel Promises Made, Promises Kept DOE is Delivering on President Trump’s Agenda of American Energy Dominance. Read more here Genesis Mission Genesis Mission leverages the Department of Energy's unique scientific datasets-spanning more than 100 petabytes of experimental and simulation data across every major domain of science—to double the pace of American research and innovation within a decade. Visit the Genesis Mission homepage Global Artificial Intelligence Leadership DOE is establishing landmark public-private partnerships to deliver the DOE’s largest AI supercomputer and accelerate scientific discovery. Read more here Fusion Science and Technology Roadmap The Fusion Science and Technology Roadmap is a national strategy to accelerate the development and commercialization of fusion energy on the most rapid, responsible timeline in history. Read more here Trending: LNG Exports Emergency Authority Nuclear 101 Artificial Intelligence Genesis Mission Latest News Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Indiana Coal Plants Remain Open to Prevent Blackouts in Midwest December 23, 2025 Promises Made, Promises Kept December 18, 2025 Artificial Intelligence Energy Department Announces Collaboration Agreements with 24 Organizations to Advance the Genesis Mission December 18, 2025 Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Washington Coal Plant Remains Open to Ensure Affordable, Reliable and Secure Power Heading into Winter December 16, 2025 Learn More Genesis Mission Genesis Mission is building a new instrument for discovery, uniting human curiosity with machine intelligence to double our scientific output and tackle humanity's greatest challenges. Video Url Genesis Mission video US Dept of Energy Visit the Genesis Mission homepage Administrations & National Labs Get to know our semi-independent organizations and see how they’re tackling the nation’s toughest challenges. National Nuclear Security Administration Learn More about National Nuclear Security Administration May 2, 2025 Power Marketing Administrations Learn More about Power Marketing Administrations October 16, 2025 U.S Department of Energy National Laboratories Learn More about U.S Department of Energy National Laboratories October 16, 2025 U.S. Energy Information Administration (EIA) Learn More about U.S. Energy Information Administration (EIA) October 16, 2025 Stay Connected Sign up to receive news and updates from the U.S. Department of Energy straight to your inbox. Email Committed to Restoring America’s Energy Dominance. Follow Us Link to Facebook Link to Instagram Link to Linkedin Link to Youtube Link to X Link to Flickr Quick Links Leadership & Offices Mission Newsroom Contact Us Careers Resources Budget & Performance Directives, Delegations, & Requirements Freedom of Information Act (FOIA) Inspector General Privacy Program Federal Government USA.gov The White House Open Gov Accessibility Privacy Information Quality Web Policies Vulnerability Disclosure Program Whistleblower Protection Equal Employment Opportunity 🔄🔍 (0.0s) | 🧠 Normalizing text 🔄🔧 (0.0s)  Normalized 50/149 documents...\n",
      "🧠 Semantica is parsing: Document:  means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. About About Our Agency Our Agency History Our Leadership & Offices The National Nuclear Security Administration The Energy Information Administration National Laboratories Power Marketing Administrations Our Outreach Our Outreach Newsroom Digital Engagement and Media Brand Guides Our Mission Our Mission Genesis Mission Security & Safety Security & Safety Nuclear Security Energy Security Cybersecurity Environmental & Legacy Management Research, Technology, & Economic Security Emergency Response Scientific Excellence Scientific Excellence Scientific Research Database (OSTI) Nobel Prize Laureates National Science Bowl Environment Environment Earth Systems Modeling Extreme Weather Resiliency Energy Access Energy Access 2025 DOE 403 Orders Grid Deployment & Transmission Puerto Rico Grid Resilience & Transitions (PR 100) Tribal Energy Access Economic Growth Economic Growth Energy Workforce American Manufacturing Technology Transitions & Early Investments Commercial Implementation Global Diplomacy & Leadership Global Diplomacy & Leadership Arctic Cooperation International Market Development Topics Topics View All Topics Funding Opportunities Funding Opportunities View Funding Opportunities Funding Opportunities By Office Funding Opportunities By Office Office of the Under Secretary of Energy Advanced Research Projects Agency-Energy Office of Energy Efficiency and Renewable Energy Federal Energy Management Program Office of Fossil Energy Grid Deployment Office Office of Indian Energy Office of Manufacturing and Energy Supply Chains Office of Science Consumer Savings Consumer Savings Tax Credits & Rebates Energy Saving Tips Weatherization Assistance Program Services & Opportunities Services & Opportunities Career Pathways Career Pathways Internships & Fellowships Apprenticeships & Workforce Development Work at DOE Powering today, Transforming tomorrow. Front page carousel Promises Made, Promises Kept DOE is Delivering on President Trump’s Agenda of American Energy Dominance. Read more here Genesis Mission Genesis Mission leverages the Department of Energy's unique scientific datasets-spanning more than 100 petabytes of experimental and simulation data across every major domain of science—to double the pace of American research and innovation within a decade. Visit the Genesis Mission homepage Global Artificial Intelligence Leadership DOE is establishing landmark public-private partnerships to deliver the DOE’s largest AI supercomputer and accelerate scientific discovery. Read more here Fusion Science and Technology Roadmap The Fusion Science and Technology Roadmap is a national strategy to accelerate the development and commercialization of fusion energy on the most rapid, responsible timeline in history. Read more here Trending: LNG Exports Emergency Authority Nuclear 101 Artificial Intelligence Genesis Mission Latest News Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Indiana Coal Plants Remain Open to Prevent Blackouts in Midwest December 23, 2025 Promises Made, Promises Kept December 18, 2025 Artificial Intelligence Energy Department Announces Collaboration Agreements with 24 Organizations to Advance the Genesis Mission December 18, 2025 Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Washington Coal Plant Remains Open to Ensure Affordable, Reliable and Secure Power Heading into Winter December 16, 2025 Learn More Genesis Mission Genesis Mission is building a new instrument for discovery, uniting human curiosity with machine intelligence to double our scientific output and tackle humanity's greatest challenges. Video Url Genesis Mission video US Dept of Energy Visit the Genesis Mission homepage Administrations & National Labs Get to know our semi-independent organizations and see how they’re tackling the nation’s toughest challenges. National Nuclear Security Administration Learn More about National Nuclear Security Administration May 2, 2025 Power Marketing Administrations Learn More about Power Marketing Administrations October 16, 2025 U.S Department of Energy National Laboratories Learn More about U.S Department of Energy National Laboratories October 16, 2025 U.S. Energy Information Administration (EIA) Learn More about U.S. Energy Information Administration (EIA) October 16, 2025 Stay Connected Sign up to receive news and updates from the U.S. Department of Energy straight to your inbox. Email Committed to Restoring America’s Energy Dominance. Follow Us Link to Facebook Link to Instagram Link to Linkedin Link to Youtube Link to X Link to Flickr Quick Links Leadership & Offices Mission Newsroom Contact Us Careers Resources Budget & Performance Directives, Delegations, & Requirements Freedom of Information Act (FOIA) Inspector General Privacy Program Federal Government USA.gov The White House Open Gov Accessibility Privacy Information Quality Web Policies Vulnerability Disclosure Program Whistleblower Protection Equal Employment Opportunity 🔄🔍 (0.0s) | 🧠 Semantica is normalizing |███████████████| 100.0% ✅🔧  Normalized 100/149 documents...\n",
      "🧠 Semantica is parsing: Document:  means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites. About About Our Agency Our Agency History Our Leadership & Offices The National Nuclear Security Administration The Energy Information Administration National Laboratories Power Marketing Administrations Our Outreach Our Outreach Newsroom Digital Engagement and Media Brand Guides Our Mission Our Mission Genesis Mission Security & Safety Security & Safety Nuclear Security Energy Security Cybersecurity Environmental & Legacy Management Research, Technology, & Economic Security Emergency Response Scientific Excellence Scientific Excellence Scientific Research Database (OSTI) Nobel Prize Laureates National Science Bowl Environment Environment Earth Systems Modeling Extreme Weather Resiliency Energy Access Energy Access 2025 DOE 403 Orders Grid Deployment & Transmission Puerto Rico Grid Resilience & Transitions (PR 100) Tribal Energy Access Economic Growth Economic Growth Energy Workforce American Manufacturing Technology Transitions & Early Investments Commercial Implementation Global Diplomacy & Leadership Global Diplomacy & Leadership Arctic Cooperation International Market Development Topics Topics View All Topics Funding Opportunities Funding Opportunities View Funding Opportunities Funding Opportunities By Office Funding Opportunities By Office Office of the Under Secretary of Energy Advanced Research Projects Agency-Energy Office of Energy Efficiency and Renewable Energy Federal Energy Management Program Office of Fossil Energy Grid Deployment Office Office of Indian Energy Office of Manufacturing and Energy Supply Chains Office of Science Consumer Savings Consumer Savings Tax Credits & Rebates Energy Saving Tips Weatherization Assistance Program Services & Opportunities Services & Opportunities Career Pathways Career Pathways Internships & Fellowships Apprenticeships & Workforce Development Work at DOE Powering today, Transforming tomorrow. Front page carousel Promises Made, Promises Kept DOE is Delivering on President Trump’s Agenda of American Energy Dominance. Read more here Genesis Mission Genesis Mission leverages the Department of Energy's unique scientific datasets-spanning more than 100 petabytes of experimental and simulation data across every major domain of science—to double the pace of American research and innovation within a decade. Visit the Genesis Mission homepage Global Artificial Intelligence Leadership DOE is establishing landmark public-private partnerships to deliver the DOE’s largest AI supercomputer and accelerate scientific discovery. Read more here Fusion Science and Technology Roadmap The Fusion Science and Technology Roadmap is a national strategy to accelerate the development and commercialization of fusion energy on the most rapid, responsible timeline in history. Read more here Trending: LNG Exports Emergency Authority Nuclear 101 Artificial Intelligence Genesis Mission Latest News Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Indiana Coal Plants Remain Open to Prevent Blackouts in Midwest December 23, 2025 Promises Made, Promises Kept December 18, 2025 Artificial Intelligence Energy Department Announces Collaboration Agreements with 24 Organizations to Advance the Genesis Mission December 18, 2025 Strengthen Grid Reliability and SecurityEmergency Response Energy Secretary Ensures Washington Coal Plant Remains Open to Ensure Affordable, Reliable and Secure Power Heading into Winter December 16, 2025 Learn More Genesis Mission Genesis Mission is building a new instrument for discovery, uniting human curiosity with machine intelligence to double our scientific output and tackle humanity's greatest challenges. Video Url Genesis Mission video US Dept of Energy Visit the Genesis Mission homepage Administrations & National Labs Get to know our semi-independent organizations and see how they’re tackling the nation’s toughest challenges. National Nuclear Security Administration Learn More about National Nuclear Security Administration May 2, 2025 Power Marketing Administrations Learn More about Power Marketing Administrations October 16, 2025 U.S Department of Energy National Laboratories Learn More about U.S Department of Energy National Laboratories October 16, 2025 U.S. Energy Information Administration (EIA) Learn More about U.S. Energy Information Administration (EIA) October 16, 2025 Stay Connected Sign up to receive news and updates from the U.S. Department of Energy straight to your inbox. Email Committed to Restoring America’s Energy Dominance. Follow Us Link to Facebook Link to Instagram Link to Linkedin Link to Youtube Link to X Link to Flickr Quick Links Leadership & Offices Mission Newsroom Contact Us Careers Resources Budget & Performance Directives, Delegations, & Requirements Freedom of Information Act (FOIA) Inspector General Privacy Program Federal Government USA.gov The White House Open Gov Accessibility Privacy Information Quality Web Policies Vulnerability Disclosure Program Whistleblower Protection Equal Employment Opportunity 🔄🔍 (0.0s) | 🧠 Normalizing text 🔄🔧 (0.0s)█████████████| 100.0% ✅🔧  Normalized 149/149 documents...\n",
      "Chunking 149 documents...\n",
      "  Chunked 50/149 documents (50 chunks so far)\n",
      "  Chunked 100/149 documents (100 chunks so far)\n",
      "  Chunked 149/149 documents (215 chunks so far)\n",
      "Created 215 chunks from 149 documents\n"
     ]
    }
   ],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "print(f\"Normalizing {len(parsed_documents)} documents...\")\n",
    "normalized_docs = []\n",
    "\n",
    "for i, doc in enumerate(parsed_documents, 1):\n",
    "    try:\n",
    "        normalized = normalizer.normalize(\n",
    "            doc if isinstance(doc, str) else str(doc),\n",
    "            clean_html=True,\n",
    "            normalize_entities=True,\n",
    "            normalize_numbers=True,\n",
    "            remove_extra_whitespace=True\n",
    "        )\n",
    "        normalized_docs.append(normalized)\n",
    "    except Exception:\n",
    "        normalized_docs.append(doc if isinstance(doc, str) else str(doc))\n",
    "    if i % 50 == 0 or i == len(parsed_documents):\n",
    "        print(f\"  Normalized {i}/{len(parsed_documents)} documents...\")\n",
    "\n",
    "# Use recursive chunking to preserve market context\n",
    "recursive_splitter = TextSplitter(\n",
    "    method=\"recursive\",\n",
    "    chunk_size=CHUNK_SIZE,\n",
    "    chunk_overlap=CHUNK_OVERLAP\n",
    ")\n",
    "\n",
    "print(f\"Chunking {len(normalized_docs)} documents...\")\n",
    "chunked_docs = []\n",
    "for i, doc_text in enumerate(normalized_docs, 1):\n",
    "    try:\n",
    "        chunks = recursive_splitter.split(doc_text)\n",
    "        chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n",
    "    except Exception:\n",
    "        chunked_docs.append(doc_text)\n",
    "    if i % 50 == 0 or i == len(normalized_docs):\n",
    "        print(f\"  Chunked {i}/{len(normalized_docs)} documents ({len(chunked_docs)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_docs)} chunks from {len(normalized_docs)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Normalizing text 🔄🔧 (0.0s) | 🧠 Semantica is extracting: Extracting named entities from text 🔄🎯 (0.0s)            ✅🎯░░░░░░| 0.0% [0/215] 🔄🎯Extracted 4427 entities from 215 chunks\n"
     ]
    }
   ],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "# Use regex-based extraction with custom patterns for energy market entity types\n",
    "extractor = NERExtractor(\n",
    "    method=\"regex\",\n",
    "    patterns={\n",
    "        \"Price\": r\"\\$\\d+(?:\\.\\d+)?/MWh|\\d+(?:\\.\\d+)?\\s*\\$?/MWh|price\\s+of\\s+\\$\\d+\",\n",
    "        \"EnergyType\": r\"\\b(Solar|Wind|Hydro|Geothermal|Biomass|Nuclear|Coal|Gas|renewable\\s+energy|fossil\\s+energy|clean\\s+energy)\\b\",\n",
    "        \"Region\": r\"\\b(Region\\s+[A-Z]|North\\s+America|Europe|Asia|Region\\s+A|Region\\s+B)\\b\",\n",
    "        \"Market\": r\"\\b(Energy\\s+Market|Renewable\\s+Energy\\s+Market|Electricity\\s+Market)\\b\",\n",
    "        \"Trend\": r\"\\b(increasing|decreasing|stable|volatile|rising|falling)\\b\",\n",
    "        \"Forecast\": r\"\\b(forecast|prediction|expected|projected|outlook)\\b\",\n",
    "        \"ORG\": r\"\\b([A-Z][a-zA-Z]+(?:\\s+[A-Z][a-zA-Z]+)*\\s+(?:Inc|Corp|LLC|Ltd|Company|Corporation))\\b\",\n",
    "        \"GPE\": r\"\\b([A-Z][a-z]+(?:\\s+[A-Z][a-z]+)*)\\b\",\n",
    "        \"DATE\": r\"\\b(\\d{4}-\\d{2}-\\d{2}|\\d{1,2}[/-]\\d{1,2}[/-]\\d{2,4}|\\d{4})\\b\",\n",
    "        \"MONEY\": r\"\\b(\\$[\\d,]+(?:\\.\\d{2})?)\\b\",\n",
    "    }\n",
    ")\n",
    "\n",
    "# Semantica handles batch processing automatically - just pass the list of chunks\n",
    "entity_results = extractor.extract(\n",
    "    chunked_docs,\n",
    "    entity_types=[\"Market\", \"Price\", \"Region\", \"Trend\", \"Forecast\", \"EnergyType\", \"ORG\", \"GPE\", \"DATE\", \"MONEY\"]\n",
    ")\n",
    "\n",
    "# Flatten results from batch extraction\n",
    "all_entities = [entity for entities in entity_results for entity in entities]\n",
    "print(f\"Extracted {len(all_entities)} entities from {len(chunked_docs)} chunks\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Relationship Extraction\n",
    "\n",
    "Extract market relationships including price associations, regional locations, trend indicators, and forecasting relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is extracting: Extracting named entities from text 🔄🎯 (0.0s) | 🧠 Semantica is extracting: Extracted 2 relations using dependency |███████████████| 100.0% ✅🎯�| 0.0% [0/215] 🔄🎯Extracted 513 relationships from 215 chunks\n"
     ]
    }
   ],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "# Use dependency parsing for relation extraction\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"dependency\",\n",
    "    model=\"en_core_web_sm\"\n",
    ")\n",
    "\n",
    "# Semantica handles batch processing - pass chunks and corresponding entity lists\n",
    "relation_results = relation_extractor.extract(\n",
    "    chunked_docs,\n",
    "    entities=entity_results,  # Use entity_results (list of lists) for proper batch matching\n",
    "    relation_types=[\"has_price\", \"located_in\", \"shows_trend\", \"predicts\", \"trades_in\"]\n",
    ")\n",
    "\n",
    "# Flatten results from batch extraction\n",
    "all_relationships = [rel for rels in relation_results for rel in rels]\n",
    "print(f\"Extracted {len(all_relationships)} relationships from {len(chunked_docs)} chunks\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conflict Detection and Resolution\n",
    "\n",
    "- Detect conflicts in energy market data from multiple sources using temporal and relationship conflict detection approaches\n",
    "- Resolve conflicts using `most_recent` strategy for time-sensitive market data to prioritize latest information\n",
    "- Combine temporal and relationship conflict detection to ensure data consistency across entities and relationships"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is extracting: Extracted 2 relations using dependency |███████████████| 100.0% ✅🎯 | 🧠 Semantica is resolving: Detected 0 relationship conflicts |███████████████| 100.0% [363/388] ✅⚠️ (455.1/s)63/388] 🔄⚠️ (ETA: 0.1s | 464.2/s))ETA: 0.1s | 609.8/s))Detected 0 temporal conflicts, 0 relationship conflicts\n",
      "No conflicts detected\n"
     ]
    }
   ],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "# Minimal conversion to dict format\n",
    "entities_dict = [\n",
    "    {\"id\": getattr(e, \"text\", str(i)), \"name\": getattr(e, \"text\", \"\"), \"type\": getattr(e, \"label\", \"\"), \"source\": getattr(e, \"metadata\", {}).get(\"source\", \"unknown\") if hasattr(e, \"metadata\") and getattr(e, \"metadata\", {}) else \"unknown\"}\n",
    "    for i, e in enumerate(all_entities)\n",
    "]\n",
    "\n",
    "relationships_dict = [\n",
    "    {\"source_id\": getattr(r.subject, \"text\", \"\") if hasattr(r, \"subject\") else \"\", \"target_id\": getattr(r.object, \"text\", \"\") if hasattr(r, \"object\") else \"\", \"type\": getattr(r, \"predicate\", \"\"), \"source\": getattr(r, \"metadata\", {}).get(\"source\", \"unknown\") if hasattr(r, \"metadata\") and getattr(r, \"metadata\", {}) else \"unknown\"}\n",
    "    for r in all_relationships\n",
    "] if all_relationships else []\n",
    "\n",
    "# Detect and resolve conflicts\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "temporal_conflicts = conflict_detector.detect_temporal_conflicts(entities_dict)\n",
    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(relationships_dict) if relationships_dict else []\n",
    "all_conflicts = temporal_conflicts + relationship_conflicts\n",
    "\n",
    "print(f\"Detected {len(temporal_conflicts)} temporal conflicts, {len(relationship_conflicts)} relationship conflicts\")\n",
    "\n",
    "if all_conflicts:\n",
    "    resolved = conflict_resolver.resolve_conflicts(all_conflicts, strategy=\"most_recent\")\n",
    "    print(f\"Resolved {len([r for r in resolved if r.resolved])} conflicts\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is deduplicating: Merging groups... 1/1 (remaining: 0) |███████████████| 100.0% [1/1] 🔄🔄 (8.2/s) | 🧠 Semantica is deduplicating: Entity merge completed |███████████████| 100.0% [4/4] ✅🔄 (108.6/s))█| 100.0% [4/4] 🔄🔄 (121.8/s) 🔄🔄 (ETA: 0.0s | 117.1/s)248.9/s))4/s) | 1346.3/s)))))Deduplicated 4427 entities to 411 unique entities\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import EntityResolver\n",
    "from semantica.semantic_extract import Entity\n",
    "\n",
    "# Convert Entity objects to dictionaries for EntityResolver\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"name\": getattr(e, \"text\", \"\"),\n",
    "        \"type\": getattr(e, \"label\", \"\"),\n",
    "        \"start_char\": getattr(e, \"start_char\", 0),\n",
    "        \"end_char\": getattr(e, \"end_char\", 0),\n",
    "        \"confidence\": getattr(e, \"confidence\", 1.0)\n",
    "    }\n",
    "    for e in all_entities\n",
    "]\n",
    "\n",
    "# Use EntityResolver to resolve duplicates\n",
    "entity_resolver = EntityResolver(strategy=\"fuzzy\", similarity_threshold=0.85)\n",
    "resolved_entities = entity_resolver.resolve_entities(entity_dicts)\n",
    "\n",
    "# Convert back to Entity objects\n",
    "all_entities = [\n",
    "    Entity(\n",
    "        text=e[\"name\"],\n",
    "        label=e[\"type\"],\n",
    "        start_char=e.get(\"start_char\", 0),\n",
    "        end_char=e.get(\"end_char\", 0),\n",
    "        confidence=e.get(\"confidence\", 1.0)\n",
    "    )\n",
    "    for e in resolved_entities\n",
    "]\n",
    "\n",
    "print(f\"Deduplicated {len(entity_dicts)} entities to {len(all_entities)} unique entities\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Temporal Knowledge Graph Construction\n",
    "\n",
    "Build a temporal knowledge graph with time-aware relationships for tracking energy market trends over time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is deduplicating: Entity merge completed |███████████████| 100.0% [4/4] ✅🔄 (108.6/s) | 🧠 Semantica is building: Knowledge graph from 411 source(s) 🔄🧠 (0.0s)Building graph structure...\n",
      "✅ Graph structure built (0.00s)\n",
      "🧠 Semantica is deduplicating: Entity merge completed |███████████████| 100.0% [4/4] ✅🔄 (108.6/s) | 🧠 Semantica is building: Knowledge graph from 411 source(s) 🔄🧠 (0.0s)\n",
      "============================================================\n",
      "✅ Knowledge Graph Build Complete\n",
      "   Entities: 411\n",
      "   Relationships: 513\n",
      "   Total time: 2.74s\n",
      "============================================================\n",
      "Built temporal KG with 411 entities and 513 relationships\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "# Build temporal knowledge graph with unique entities and relationships\n",
    "builder = GraphBuilder(enable_temporal=True, temporal_granularity=TEMPORAL_GRANULARITY)\n",
    "\n",
    "kg = builder.build(\n",
    "    sources=all_entities,\n",
    "    relationships=all_relationships\n",
    ")\n",
    "\n",
    "print(f\"Built temporal KG with {len(kg.get('entities', []))} entities and {len(kg.get('relationships', []))} relationships\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Embedding Generation & Vector Store\n",
    "\n",
    "Generate embeddings for energy market documents and store them in a vector database for semantic search.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n",
      "fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is building: Knowledge graph from 411 source(s) 🔄🧠 (0.0s) | 🧠 Semantica is indexing: Storing 20 vectors 🔄📊 (0.0s)Generated 20 embeddings and stored in vector database\n"
     ]
    }
   ],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "# Generate embeddings and create vector store\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    model_name=EMBEDDING_MODEL,\n",
    "    dimension=EMBEDDING_DIMENSION\n",
    ")\n",
    "\n",
    "chunks_to_embed = chunked_docs[:20]  # Limit for demo\n",
    "embeddings = embedding_gen.generate_embeddings(chunks_to_embed, data_type=\"text\")\n",
    "\n",
    "# Create and populate vector store\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "metadata = [{\"text\": chunk[:100] if isinstance(chunk, str) else str(chunk)[:100]} for chunk in chunks_to_embed]\n",
    "vector_ids = vector_store.store_vectors(vectors=embeddings, metadata=metadata)\n",
    "\n",
    "print(f\"Generated {len(embeddings)} embeddings and stored in vector database\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Temporal Pattern Detection\n",
    "\n",
    "Detect temporal patterns in energy market data to identify trends. This is unique to this notebook and critical for trend prediction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detected 0 trend patterns\n",
      "Analyzed price evolution over time\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import TemporalPatternDetector\n",
    "\n",
    "pattern_detector = TemporalPatternDetector()\n",
    "\n",
    "# Detect trend patterns\n",
    "trend_patterns = pattern_detector.detect_temporal_patterns(\n",
    "    graph=kg,\n",
    "    pattern_type=\"trend\",\n",
    "    min_frequency=2,\n",
    "    time_window=None\n",
    ")\n",
    "\n",
    "print(f\"Detected {len(trend_patterns)} trend patterns\")\n",
    "\n",
    "# Analyze price evolution over time\n",
    "price_evolution = pattern_detector.analyze_evolution(\n",
    "    graph=kg,\n",
    "    entity_type=\"Price\",\n",
    "    time_window=None\n",
    ") if hasattr(pattern_detector, 'analyze_evolution') else []\n",
    "print(f\"Analyzed price evolution over time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Temporal Graph Queries\n",
    "\n",
    "Query the temporal knowledge graph to analyze market trends over time and identify pricing patterns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved temporal history for price: price of $2\n",
      "Analyzed price evolution over time\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import TemporalGraphQuery\n",
    "\n",
    "temporal_query = TemporalGraphQuery(temporal_granularity=TEMPORAL_GRANULARITY)\n",
    "\n",
    "# Query price trends over time using time range query\n",
    "if all_entities:\n",
    "    price_entities = [e for e in all_entities if getattr(e, \"label\", \"\") == \"Price\"]\n",
    "    if price_entities:\n",
    "        price_id = getattr(price_entities[0], \"text\", \"\")\n",
    "        if price_id:\n",
    "            history = temporal_query.query_time_range(\n",
    "                graph=kg,\n",
    "                query=\"price_history\",\n",
    "                start_time=None,\n",
    "                end_time=None\n",
    "            )\n",
    "            print(f\"Retrieved temporal history for price: {price_id}\")\n",
    "\n",
    "# Analyze evolution of prices over time\n",
    "evolution = temporal_query.analyze_evolution(\n",
    "    graph=kg,\n",
    "    relationship=\"has_price\",\n",
    "    metrics=[\"count\", \"diversity\"]\n",
    ")\n",
    "print(f\"Analyzed price evolution over time\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
