{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/renewable_energy/01_Energy_Market_Analysis.ipynb)\n",
    "\n",
    "# Energy Market Analysis - Temporal KGs & Trend Prediction\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **energy market analysis** using Semantica with focus on **temporal knowledge graphs**, **trend prediction**, and **market entity extraction**. The pipeline analyzes pricing trends and market movements using temporal market knowledge graphs to predict energy market trends and forecast pricing.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **Temporal Knowledge Graphs**: Builds temporal KGs to track energy market trends over time\n",
    "- **Trend Prediction**: Uses temporal analysis and reasoning to predict market movements\n",
    "- **Market Entity Extraction**: Extracts energy market entities (Market, Price, Region, Trend, Forecast, EnergyType)\n",
    "- **Temporal Pattern Detection**: Identifies patterns in energy pricing and market trends\n",
    "- **Seed Data Integration**: Uses market foundation data for entity resolution\n",
    "- **Forecasting**: Emphasizes reasoning-based market forecasting\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Understand how to build temporal knowledge graphs for market analysis\n",
    "- Learn to detect temporal patterns in energy pricing data\n",
    "- Master trend prediction using reasoning and pattern detection\n",
    "- Explore temporal graph queries for market trend analysis\n",
    "- Practice market entity extraction and relationship mapping\n",
    "- Analyze energy market trends and forecasting\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Seed Data Loading]\n",
    "    A --> C[Document Parsing]\n",
    "    B --> D[Text Processing]\n",
    "    C --> D\n",
    "    D --> E[Entity Extraction]\n",
    "    E --> F[Relationship Extraction]\n",
    "    F --> G[Deduplication]\n",
    "    G --> H[Temporal KG Construction]\n",
    "    H --> I[Embedding Generation]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Temporal Pattern Detection]\n",
    "    H --> L[Temporal Queries]\n",
    "```\n",
    "\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Configuration & Setup\n",
    "\n",
    "Configure API keys and set up constants for the energy market analysis pipeline, including temporal granularity for trend tracking.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 200\n",
    "TEMPORAL_GRANULARITY = \"day\"  # For market trend tracking\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Data Ingestion\n",
    "\n",
    "Ingest energy market data from multiple sources including RSS feeds, web APIs, and local files.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FeedIngestor, WebIngestor, FileIngestor\n",
    "import os\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "documents = []\n",
    "\n",
    "# Ingest from energy market RSS feeds\n",
    "energy_feeds = [\n",
    "    \"https://www.energycentral.com/rss\",\n",
    "    \"https://www.renewableenergyworld.com/rss\",\n",
    "    \"https://www.greentechmedia.com/rss\",\n",
    "    \"https://www.utilitydive.com/rss\",\n",
    "    \"https://www.power-eng.com/rss\",\n",
    "    \"https://www.energy-storage.news/rss\",\n",
    "    \"https://www.pv-magazine.com/rss\",\n",
    "    \"https://www.windpowermonthly.com/rss\",\n",
    "    \"https://www.rechargenews.com/rss\",\n",
    "    \"https://www.energystoragejournal.com/rss\",\n",
    "    \"https://feeds.feedburner.com/EnergyBiz\",\n",
    "    \"https://www.energy.gov/rss-feeds\",\n",
    "    \"https://www.iea.org/rss\",\n",
    "    \"https://www.irena.org/rss\",\n",
    "    \"https://www.cleanenergywire.org/rss\"\n",
    "]\n",
    "\n",
    "print(f\"Ingesting from {len(energy_feeds)} RSS feeds...\")\n",
    "feed_ingestor = FeedIngestor()\n",
    "for i, feed_url in enumerate(energy_feeds, 1):\n",
    "    try:\n",
    "        feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_url\n",
    "                item.metadata['source_type'] = 'rss_feed'\n",
    "                documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        print(f\"  [{i}/{len(energy_feeds)}] Loaded {feed_count} documents from {feed_url}\")\n",
    "    except Exception as e:\n",
    "        print(f\"  [{i}/{len(energy_feeds)}] Failed to load {feed_url}: {str(e)[:50]}\")\n",
    "\n",
    "# Web ingestion from energy market websites\n",
    "energy_web_sources = [\n",
    "    \"https://www.eia.gov/todayinenergy/\",\n",
    "    \"https://www.energy.gov/\",\n",
    "    \"https://www.epa.gov/energy\",\n",
    "    \"https://www.nrel.gov/news/\",\n",
    "    \"https://www.energy.gov/office-energy-efficiency-renewable-energy\",\n",
    "    \"https://www.energy.gov/oe/office-electricity\",\n",
    "    \"https://www.energy.gov/ne/office-nuclear-energy\"\n",
    "]\n",
    "\n",
    "print(f\"\\nIngesting from {len(energy_web_sources)} web sources...\")\n",
    "web_ingestor = WebIngestor(respect_robots=True, delay=1.0)\n",
    "for i, web_url in enumerate(energy_web_sources, 1):\n",
    "    try:\n",
    "        web_content = web_ingestor.ingest_url(web_url)\n",
    "        if web_content and web_content.text:\n",
    "            # Add content attribute for compatibility\n",
    "            web_content.content = web_content.text\n",
    "            if not hasattr(web_content, 'metadata'):\n",
    "                web_content.metadata = {}\n",
    "            web_content.metadata['source'] = web_url\n",
    "            web_content.metadata['source_type'] = 'web_page'\n",
    "            documents.append(web_content)\n",
    "            print(f\"  [{i}/{len(energy_web_sources)}] Loaded content from {web_url} ({len(web_content.text)} chars)\")\n",
    "        else:\n",
    "            print(f\"  [{i}/{len(energy_web_sources)}] No content from {web_url}\")\n",
    "    except Exception as e:\n",
    "        print(f\"  [{i}/{len(energy_web_sources)}] Failed to load {web_url}: {str(e)[:50]}\")\n",
    "\n",
    "print(f\"\\nTotal ingested: {len(documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.seed import SeedDataManager\n",
    "\n",
    "seed_manager = SeedDataManager()\n",
    "\n",
    "# Load market foundation seed data\n",
    "market_foundation = {\n",
    "    \"markets\": [\"Energy Market\", \"Renewable Energy Market\", \"Electricity Market\"],\n",
    "    \"regions\": [\"North America\", \"Region A\", \"Region B\", \"Europe\", \"Asia\"],\n",
    "    \"energy_types\": [\"Solar\", \"Wind\", \"Hydro\", \"Geothermal\", \"Biomass\"],\n",
    "    \"trends\": [\"increasing\", \"decreasing\", \"stable\", \"volatile\"]\n",
    "}\n",
    "\n",
    "# Convert dictionary to entities and add to seed data\n",
    "seed_entities = []\n",
    "\n",
    "# Add market entities\n",
    "for market in market_foundation[\"markets\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"market_{market.lower().replace(' ', '_')}\",\n",
    "        \"name\": market,\n",
    "        \"type\": \"Market\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add region entities\n",
    "for region in market_foundation[\"regions\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"region_{region.lower().replace(' ', '_')}\",\n",
    "        \"name\": region,\n",
    "        \"type\": \"Region\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add energy type entities\n",
    "for energy_type in market_foundation[\"energy_types\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"energy_type_{energy_type.lower()}\",\n",
    "        \"name\": energy_type,\n",
    "        \"type\": \"EnergyType\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add trend entities\n",
    "for trend in market_foundation[\"trends\"]:\n",
    "    seed_entities.append({\n",
    "        \"id\": f\"trend_{trend.lower()}\",\n",
    "        \"name\": trend,\n",
    "        \"type\": \"Trend\",\n",
    "        \"source\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add entities to seed manager\n",
    "seed_manager.seed_data.entities.extend(seed_entities)\n",
    "\n",
    "# Create seed relationships for better graph structure\n",
    "seed_relationships = []\n",
    "\n",
    "# Link renewable energy market to energy types\n",
    "renewable_market_id = \"market_renewable_energy_market\"\n",
    "for energy_type in market_foundation[\"energy_types\"]:\n",
    "    energy_id = f\"energy_type_{energy_type.lower()}\"\n",
    "    seed_relationships.append({\n",
    "        \"source\": renewable_market_id,\n",
    "        \"target\": energy_id,\n",
    "        \"type\": \"includes\",\n",
    "        \"source_name\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Link regions to markets\n",
    "for region in market_foundation[\"regions\"]:\n",
    "    region_id = f\"region_{region.lower().replace(' ', '_')}\"\n",
    "    # Link to main energy market\n",
    "    seed_relationships.append({\n",
    "        \"source\": region_id,\n",
    "        \"target\": \"market_energy_market\",\n",
    "        \"type\": \"located_in\",\n",
    "        \"source_name\": \"seed_data\"\n",
    "    })\n",
    "\n",
    "# Add relationships to seed manager\n",
    "seed_manager.seed_data.relationships.extend(seed_relationships)\n",
    "\n",
    "print(f\"Loaded seed data with {len(seed_entities)} entities and {len(seed_relationships)} relationships\")\n",
    "print(f\"Seed entities: {len(seed_manager.seed_data.entities)}, Seed relationships: {len(seed_manager.seed_data.relationships)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Document Parsing\n",
    "\n",
    "Parse structured energy market data from various formats including JSON, HTML, and XML.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            format=\"auto\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc.content if hasattr(doc, 'content') else str(doc))\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Parsed {len(parsed_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Text Processing\n",
    "\n",
    "Normalize energy market data and split documents using recursive chunking to preserve market context.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "print(f\"Normalizing {len(parsed_documents)} documents...\")\n",
    "normalized_docs = []\n",
    "\n",
    "for i, doc in enumerate(parsed_documents, 1):\n",
    "    try:\n",
    "        normalized = normalizer.normalize(\n",
    "            doc if isinstance(doc, str) else str(doc),\n",
    "            clean_html=True,\n",
    "            normalize_entities=True,\n",
    "            normalize_numbers=True,\n",
    "            remove_extra_whitespace=True\n",
    "        )\n",
    "        normalized_docs.append(normalized)\n",
    "    except Exception:\n",
    "        normalized_docs.append(doc if isinstance(doc, str) else str(doc))\n",
    "    if i % 50 == 0 or i == len(parsed_documents):\n",
    "        print(f\"  Normalized {i}/{len(parsed_documents)} documents...\")\n",
    "\n",
    "# Use recursive chunking to preserve market context\n",
    "recursive_splitter = TextSplitter(\n",
    "    method=\"recursive\",\n",
    "    chunk_size=CHUNK_SIZE,\n",
    "    chunk_overlap=CHUNK_OVERLAP\n",
    ")\n",
    "\n",
    "print(f\"Chunking {len(normalized_docs)} documents...\")\n",
    "chunked_docs = []\n",
    "for i, doc_text in enumerate(normalized_docs, 1):\n",
    "    try:\n",
    "        chunks = recursive_splitter.split(doc_text)\n",
    "        chunked_docs.extend([chunk.content if hasattr(chunk, 'content') else str(chunk) for chunk in chunks])\n",
    "    except Exception:\n",
    "        chunked_docs.append(doc_text)\n",
    "    if i % 50 == 0 or i == len(normalized_docs):\n",
    "        print(f\"  Chunked {i}/{len(normalized_docs)} documents ({len(chunked_docs)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_docs)} chunks from {len(normalized_docs)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "# Use regex-based extraction with custom patterns for energy market entity types\n",
    "extractor = NERExtractor(\n",
    "    method=\"regex\",\n",
    "    patterns={\n",
    "        \"Price\": r\"\\$\\d+(?:\\.\\d+)?/MWh|\\d+(?:\\.\\d+)?\\s*\\$?/MWh|price\\s+of\\s+\\$\\d+\",\n",
    "        \"EnergyType\": r\"\\b(Solar|Wind|Hydro|Geothermal|Biomass|Nuclear|Coal|Gas|renewable\\s+energy|fossil\\s+energy|clean\\s+energy)\\b\",\n",
    "        \"Region\": r\"\\b(Region\\s+[A-Z]|North\\s+America|Europe|Asia|Region\\s+A|Region\\s+B)\\b\",\n",
    "        \"Market\": r\"\\b(Energy\\s+Market|Renewable\\s+Energy\\s+Market|Electricity\\s+Market)\\b\",\n",
    "        \"Trend\": r\"\\b(increasing|decreasing|stable|volatile|rising|falling)\\b\",\n",
    "        \"Forecast\": r\"\\b(forecast|prediction|expected|projected|outlook)\\b\",\n",
    "        \"ORG\": r\"\\b([A-Z][a-zA-Z]+(?:\\s+[A-Z][a-zA-Z]+)*\\s+(?:Inc|Corp|LLC|Ltd|Company|Corporation))\\b\",\n",
    "        \"GPE\": r\"\\b([A-Z][a-z]+(?:\\s+[A-Z][a-z]+)*)\\b\",\n",
    "        \"DATE\": r\"\\b(\\d{4}-\\d{2}-\\d{2}|\\d{1,2}[/-]\\d{1,2}[/-]\\d{2,4}|\\d{4})\\b\",\n",
    "        \"MONEY\": r\"\\b(\\$[\\d,]+(?:\\.\\d{2})?)\\b\",\n",
    "    }\n",
    ")\n",
    "\n",
    "# Semantica handles batch processing automatically - just pass the list of chunks\n",
    "entity_results = extractor.extract(\n",
    "    chunked_docs,\n",
    "    entity_types=[\"Market\", \"Price\", \"Region\", \"Trend\", \"Forecast\", \"EnergyType\", \"ORG\", \"GPE\", \"DATE\", \"MONEY\"]\n",
    ")\n",
    "\n",
    "# Flatten results from batch extraction\n",
    "all_entities = [entity for entities in entity_results for entity in entities]\n",
    "print(f\"Extracted {len(all_entities)} entities from {len(chunked_docs)} chunks\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Relationship Extraction\n",
    "\n",
    "Extract market relationships including price associations, regional locations, trend indicators, and forecasting relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "# Use dependency parsing for relation extraction\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"dependency\",\n",
    "    model=\"en_core_web_sm\"\n",
    ")\n",
    "\n",
    "# Semantica handles batch processing - pass chunks and corresponding entity lists\n",
    "relation_results = relation_extractor.extract(\n",
    "    chunked_docs,\n",
    "    entities=entity_results,  # Use entity_results (list of lists) for proper batch matching\n",
    "    relation_types=[\"has_price\", \"located_in\", \"shows_trend\", \"predicts\", \"trades_in\"]\n",
    ")\n",
    "\n",
    "# Flatten results from batch extraction\n",
    "all_relationships = [rel for rels in relation_results for rel in rels]\n",
    "print(f\"Extracted {len(all_relationships)} relationships from {len(chunked_docs)} chunks\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conflict Detection and Resolution\n",
    "\n",
    "- Detect conflicts in energy market data from multiple sources using temporal and relationship conflict detection approaches\n",
    "- Resolve conflicts using `most_recent` strategy for time-sensitive market data to prioritize latest information\n",
    "- Combine temporal and relationship conflict detection to ensure data consistency across entities and relationships"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "# Minimal conversion to dict format\n",
    "entities_dict = [\n",
    "    {\"id\": getattr(e, \"text\", str(i)), \"name\": getattr(e, \"text\", \"\"), \"type\": getattr(e, \"label\", \"\"), \"source\": getattr(e, \"metadata\", {}).get(\"source\", \"unknown\") if hasattr(e, \"metadata\") and getattr(e, \"metadata\", {}) else \"unknown\"}\n",
    "    for i, e in enumerate(all_entities)\n",
    "]\n",
    "\n",
    "relationships_dict = [\n",
    "    {\"source_id\": getattr(r.subject, \"text\", \"\") if hasattr(r, \"subject\") else \"\", \"target_id\": getattr(r.object, \"text\", \"\") if hasattr(r, \"object\") else \"\", \"type\": getattr(r, \"predicate\", \"\"), \"source\": getattr(r, \"metadata\", {}).get(\"source\", \"unknown\") if hasattr(r, \"metadata\") and getattr(r, \"metadata\", {}) else \"unknown\"}\n",
    "    for r in all_relationships\n",
    "] if all_relationships else []\n",
    "\n",
    "# Detect and resolve conflicts\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "temporal_conflicts = conflict_detector.detect_temporal_conflicts(entities_dict)\n",
    "relationship_conflicts = conflict_detector.detect_relationship_conflicts(relationships_dict) if relationships_dict else []\n",
    "all_conflicts = temporal_conflicts + relationship_conflicts\n",
    "\n",
    "print(f\"Detected {len(temporal_conflicts)} temporal conflicts, {len(relationship_conflicts)} relationship conflicts\")\n",
    "\n",
    "if all_conflicts:\n",
    "    resolved = conflict_resolver.resolve_conflicts(all_conflicts, strategy=\"most_recent\")\n",
    "    print(f\"Resolved {len([r for r in resolved if r.resolved])} conflicts\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import EntityResolver\n",
    "from semantica.semantic_extract import Entity\n",
    "\n",
    "# Convert Entity objects to dictionaries for EntityResolver\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"name\": getattr(e, \"text\", \"\"),\n",
    "        \"type\": getattr(e, \"label\", \"\"),\n",
    "        \"start_char\": getattr(e, \"start_char\", 0),\n",
    "        \"end_char\": getattr(e, \"end_char\", 0),\n",
    "        \"confidence\": getattr(e, \"confidence\", 1.0)\n",
    "    }\n",
    "    for e in all_entities\n",
    "]\n",
    "\n",
    "# Use EntityResolver to resolve duplicates\n",
    "entity_resolver = EntityResolver(strategy=\"fuzzy\", similarity_threshold=0.85)\n",
    "resolved_entities = entity_resolver.resolve_entities(entity_dicts)\n",
    "\n",
    "# Convert back to Entity objects\n",
    "all_entities = [\n",
    "    Entity(\n",
    "        text=e[\"name\"],\n",
    "        label=e[\"type\"],\n",
    "        start_char=e.get(\"start_char\", 0),\n",
    "        end_char=e.get(\"end_char\", 0),\n",
    "        confidence=e.get(\"confidence\", 1.0)\n",
    "    )\n",
    "    for e in resolved_entities\n",
    "]\n",
    "\n",
    "print(f\"Deduplicated {len(entity_dicts)} entities to {len(all_entities)} unique entities\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Temporal Knowledge Graph Construction\n",
    "\n",
    "Build a temporal knowledge graph with time-aware relationships for tracking energy market trends over time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "# Build temporal knowledge graph with unique entities and relationships\n",
    "builder = GraphBuilder(enable_temporal=True, temporal_granularity=TEMPORAL_GRANULARITY)\n",
    "\n",
    "kg = builder.build(\n",
    "    sources=all_entities,\n",
    "    relationships=all_relationships\n",
    ")\n",
    "\n",
    "print(f\"Built temporal KG with {len(kg.get('entities', []))} entities and {len(kg.get('relationships', []))} relationships\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Embedding Generation & Vector Store\n",
    "\n",
    "Generate embeddings for energy market documents and store them in a vector database for semantic search.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "# Generate embeddings and create vector store\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    model_name=EMBEDDING_MODEL,\n",
    "    dimension=EMBEDDING_DIMENSION\n",
    ")\n",
    "\n",
    "chunks_to_embed = chunked_docs[:20]  # Limit for demo\n",
    "embeddings = embedding_gen.generate_embeddings(chunks_to_embed, data_type=\"text\")\n",
    "\n",
    "# Create and populate vector store\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "metadata = [{\"text\": chunk[:100] if isinstance(chunk, str) else str(chunk)[:100]} for chunk in chunks_to_embed]\n",
    "vector_ids = vector_store.store_vectors(vectors=embeddings, metadata=metadata)\n",
    "\n",
    "print(f\"Generated {len(embeddings)} embeddings and stored in vector database\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Temporal Pattern Detection\n",
    "\n",
    "Detect temporal patterns in energy market data to identify trends. This is unique to this notebook and critical for trend prediction.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import TemporalPatternDetector\n",
    "\n",
    "pattern_detector = TemporalPatternDetector()\n",
    "\n",
    "# Detect trend patterns\n",
    "trend_patterns = pattern_detector.detect_temporal_patterns(\n",
    "    graph=kg,\n",
    "    pattern_type=\"trend\",\n",
    "    min_frequency=2,\n",
    "    time_window=None\n",
    ")\n",
    "\n",
    "print(f\"Detected {len(trend_patterns)} trend patterns\")\n",
    "\n",
    "# Analyze price evolution over time\n",
    "price_evolution = pattern_detector.analyze_evolution(\n",
    "    graph=kg,\n",
    "    entity_type=\"Price\",\n",
    "    time_window=None\n",
    ") if hasattr(pattern_detector, 'analyze_evolution') else []\n",
    "print(f\"Analyzed price evolution over time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Temporal Graph Queries\n",
    "\n",
    "Query the temporal knowledge graph to analyze market trends over time and identify pricing patterns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import TemporalGraphQuery\n",
    "\n",
    "temporal_query = TemporalGraphQuery(temporal_granularity=TEMPORAL_GRANULARITY)\n",
    "\n",
    "# Query price trends over time using time range query\n",
    "if all_entities:\n",
    "    price_entities = [e for e in all_entities if getattr(e, \"label\", \"\") == \"Price\"]\n",
    "    if price_entities:\n",
    "        price_id = getattr(price_entities[0], \"text\", \"\")\n",
    "        if price_id:\n",
    "            history = temporal_query.query_time_range(\n",
    "                graph=kg,\n",
    "                query=\"price_history\",\n",
    "                start_time=None,\n",
    "                end_time=None\n",
    "            )\n",
    "            print(f\"Retrieved temporal history for price: {price_id}\")\n",
    "\n",
    "# Analyze evolution of prices over time\n",
    "evolution = temporal_query.analyze_evolution(\n",
    "    graph=kg,\n",
    "    relationship=\"has_price\",\n",
    "    metrics=[\"count\", \"diversity\"]\n",
    ")\n",
    "print(f\"Analyzed price evolution over time\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
