{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb)\n",
    "\n",
    "# Financial Data Integration (MCP) - Real-Time Market Data\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **financial data integration using MCP servers** with focus on **MCP server integration**, **real-time data ingestion**, and **multi-source financial KG construction**. The pipeline integrates Python/FastMCP servers to ingest market data, stock prices, and metrics into a financial knowledge graph.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **MCP Integration**: Showcases MCP (Model Context Protocol) server integration capability\n",
    "- **Seed Data Management**: Uses foundation market data for entity resolution\n",
    "- **Real-Time Data Ingestion**: Ingests live market data from MCP servers and APIs\n",
    "- **Multi-Source Financial KG**: Builds comprehensive financial knowledge graphs from multiple sources\n",
    "- **Market Network Analysis**: Analyzes market structure using graph analytics\n",
    "- **Comprehensive Data Sources**: Multiple financial APIs, RSS feeds, and databases\n",
    "- **Modular Architecture**: Direct use of Semantica modules without core orchestrator\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest financial data from MCP servers, APIs, and RSS feeds\n",
    "- Use seed data for foundation market information\n",
    "- Extract financial entities (Companies, Stocks, Prices, Metrics, Markets, Sectors)\n",
    "- Build financial knowledge graphs with seed data integration\n",
    "- Analyze market network structure using graph analytics\n",
    "- Store and query financial data using vector stores\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Seed Data Loading]\n",
    "    B --> C[Document Parsing]\n",
    "    C --> D[Text Processing]\n",
    "    D --> E[Entity Extraction]\n",
    "    E --> F[Relationship Extraction]\n",
    "    F --> G[Deduplication]\n",
    "    G --> H[Knowledge Graph]\n",
    "    H --> I[Embeddings]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Graph Analytics]\n",
    "    K --> L[GraphRAG Queries]\n",
    "    J --> L\n",
    "    L --> M[Visualization]\n",
    "    M --> N[Export]\n",
    "```\n",
    "\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~gno (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~lotly (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n",
      "WARNING: Ignoring invalid distribution ~ython-socketio (c:\\Users\\Mohd Kaif\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages)\n"
     ]
    }
   ],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 200\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting Financial Data from Multiple Sources\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ingesting from 8 feed sources...\n",
      "🧠 Semantica is ingesting: 429 Client Error: Too Many Requests for url: https://feeds.finance.yahoo.com/rss/2.0/headline ❌📥 (1.0s) | 🧠 Semantica is ingesting: Ingested 10 items |███████████████| 100.0% ✅📥  [2/8] Financial Times: 10 documents\n",
      "🧠 Semantica is ingesting: Ingested 10 items |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: Ingested 30 items |███████████████| 100.0% ✅📥  [3/8] Bloomberg: 30 documents\n",
      "🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.marketwatch.com/404?origin=feeds ❌📥 (3.2s) | 🧠 Semantica is ingesting: Ingested 30 items |███████████████| 100.0% ✅📥  [5/8] Seeking Alpha: 30 documents\n",
      "🧠 Semantica is ingesting: Ingested 30 items |███████████████| 100.0% ✅📥 | 🧠 Semantica is ingesting: Ingested 10 items |███████████████| 100.0% ✅📥  [6/8] Investing.com: 10 documents\n",
      "🧠 Semantica is ingesting: 403 Client Error: Forbidden for url: https://www.fnlondon.com/rss ❌📥 (0.5s) | 🧠 Semantica is ingesting: Ingested 20 items |███████████████| 100.0% ✅📥  [8/8] Wall Street Journal: 20 documents\n",
      "Ingested 100 documents\n"
     ]
    }
   ],
   "source": [
    "from semantica.ingest import MCPIngestor, WebIngestor, FeedIngestor, FileIngestor\n",
    "import os\n",
    "from contextlib import redirect_stderr\n",
    "from io import StringIO\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "feed_sources = [\n",
    "    # Financial RSS Feeds - More reliable sources\n",
    "    (\"Yahoo Finance\", \"https://feeds.finance.yahoo.com/rss/2.0/headline\"),\n",
    "    (\"Financial Times\", \"https://www.ft.com/?format=rss\"),\n",
    "    (\"Bloomberg\", \"https://feeds.bloomberg.com/markets/news.rss\"),\n",
    "    (\"MarketWatch\", \"https://feeds.marketwatch.com/marketwatch/markets\"),\n",
    "    (\"Seeking Alpha\", \"https://seekingalpha.com/feed.xml\"),\n",
    "    (\"Investing.com\", \"https://www.investing.com/rss/news.rss\"),\n",
    "    (\"Financial News\", \"https://www.fnlondon.com/rss\"),\n",
    "    (\"Wall Street Journal\", \"https://feeds.a.dj.com/rss/RSSMarketsMain.xml\"),\n",
    "]\n",
    "\n",
    "feed_ingestor = FeedIngestor()\n",
    "all_documents = []\n",
    "\n",
    "print(f\"Ingesting from {len(feed_sources)} feed sources...\")\n",
    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_name\n",
    "                all_documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        if feed_count > 0:\n",
    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
    "    except Exception:\n",
    "        continue\n",
    "\n",
    "# Example: Ingest from Alpha Vantage API (requires API key)\n",
    "alpha_vantage_api = \"https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=AAPL&apikey=demo\"\n",
    "try:\n",
    "    web_ingestor = WebIngestor()\n",
    "    with redirect_stderr(StringIO()):\n",
    "        api_documents = web_ingestor.ingest(alpha_vantage_api, method=\"url\")\n",
    "    for doc in api_documents:\n",
    "        if not hasattr(doc, 'metadata'):\n",
    "            doc.metadata = {}\n",
    "        doc.metadata['source'] = 'Alpha Vantage API'\n",
    "        all_documents.append(doc)\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# MCP Server connection example (commented for demo)\n",
    "# mcp_ingestor = MCPIngestor()\n",
    "# mcp_ingestor.connect(\"financial_server\", url=\"http://localhost:8000/mcp\")\n",
    "# resources = mcp_ingestor.list_available_resources(\"financial_server\")\n",
    "# mcp_data = mcp_ingestor.ingest_resources(\"financial_server\", resource_uris=[\"resource://market_data\"])\n",
    "\n",
    "if not all_documents:\n",
    "    market_data = \"\"\"\n",
    "    AAPL stock price: $150.25, market cap: $2.4T, volume: 50M shares, sector: Technology\n",
    "    MSFT stock price: $380.50, market cap: $2.8T, volume: 30M shares, sector: Technology\n",
    "    GOOGL stock price: $140.75, market cap: $1.8T, volume: 25M shares, sector: Technology\n",
    "    JPM stock price: $145.30, market cap: $420B, volume: 15M shares, sector: Financial\n",
    "    \"\"\"\n",
    "    with open(\"data/market_data.txt\", \"w\") as f:\n",
    "        f.write(market_data)\n",
    "    file_ingestor = FileIngestor()\n",
    "    all_documents = file_ingestor.ingest(\"data/market_data.txt\")\n",
    "\n",
    "documents = all_documents\n",
    "print(f\"Ingested {len(documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 6 seed data items for market foundation\n"
     ]
    }
   ],
   "source": [
    "from semantica.seed import SeedDataManager\n",
    "\n",
    "seed_manager = SeedDataManager()\n",
    "\n",
    "# Load foundation market data (exchanges, indices, sectors)\n",
    "seed_data = [\n",
    "    {\"type\": \"Market\", \"text\": \"NASDAQ\", \"description\": \"Stock exchange\"},\n",
    "    {\"type\": \"Market\", \"text\": \"NYSE\", \"description\": \"Stock exchange\"},\n",
    "    {\"type\": \"Market\", \"text\": \"S&P 500\", \"description\": \"Stock market index\"},\n",
    "    {\"type\": \"Sector\", \"text\": \"Technology\", \"description\": \"Market sector\"},\n",
    "    {\"type\": \"Sector\", \"text\": \"Financial\", \"description\": \"Market sector\"},\n",
    "    {\"type\": \"Sector\", \"text\": \"Healthcare\", \"description\": \"Market sector\"},\n",
    "]\n",
    "\n",
    "# Add seed data as entities\n",
    "for item in seed_data:\n",
    "    entity = {\n",
    "        \"id\": item.get(\"text\", \"\").lower().replace(\" \", \"_\"),\n",
    "        \"text\": item.get(\"text\", \"\"),\n",
    "        \"name\": item.get(\"text\", \"\"),\n",
    "        \"type\": item.get(\"type\", \"\"),\n",
    "        \"description\": item.get(\"description\", \"\"),\n",
    "        \"source\": \"seed_data\"\n",
    "    }\n",
    "    seed_manager.seed_data.entities.append(entity)\n",
    "\n",
    "print(f\"Loaded {len(seed_data)} seed data items for market foundation\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing Financial Documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing 100 documents...\n",
      "🧠 Semantica is parsing: Document file not found: Jonathan Ferro, Lisa Abramowicz and Annmarie Hordern speak daily with leaders and decision makers from Wall Street to Washington and beyond. No other program better positions investors and executives for the trading day. (Source: Bloomberg) ❌🔍 (0.0s) | 🧠 Semantica is parsing: Document file not found: <p>A look at some of the year’s notable trades.<\\p> ❌🔍 (0.0s)ay. (Source: Bloomberg) ❌🔍 (0.0s)  Parsed 50/100 documents...\n",
      "🧠 Semantica is parsing: Document file not found: Maverick Natural Resources has operations in Texas and Oklahoma. ❌🔍 (0.0s) | 🧠 Semantica is parsing: Document file not found: The president has told Bank of America and JPMorgan Chase to stop cutting conservatives off from doing business. ❌🔍 (0.0s)  Parsed 100/100 documents...\n"
     ]
    }
   ],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            content_type=\"text\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "documents = parsed_documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing and Chunking Financial Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalizing 100 documents...\n",
      "🧠 Semantica is parsing: Document file not found: The president has told Bank of America and JPMorgan Chase to stop cutting conservatives off from doing business. ❌🔍 (0.0s) | 🧠 Normalizing text 🔄🔧 (0.0s)  Normalized 50/100 documents...\n",
      "🧠 Semantica is parsing: Document file not found: The president has told Bank of America and JPMorgan Chase to stop cutting conservatives off from doing business. ❌🔍 (0.0s) | 🧠 Normalizing text 🔄🔧 (0.0s)  Normalized 100/100 documents...\n",
      "Chunking 100 documents...\n",
      "  Chunked 50/100 documents (50 chunks so far)\n",
      "  Chunked 100/100 documents (100 chunks so far)\n",
      "Created 100 chunks from 100 documents\n"
     ]
    }
   ],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "# Use recursive chunking for financial documents\n",
    "splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "\n",
    "print(f\"Normalizing {len(documents)} documents...\")\n",
    "normalized_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    normalized_text = normalizer.normalize(\n",
    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
    "        clean_html=True,\n",
    "        normalize_entities=True,\n",
    "        normalize_numbers=True,\n",
    "        remove_extra_whitespace=True,\n",
    "        lowercase=False\n",
    "    )\n",
    "    normalized_documents.append(normalized_text)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
    "chunked_documents = []\n",
    "for i, doc_text in enumerate(normalized_documents, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            chunks = splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    except Exception:\n",
    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "        chunks = simple_splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    if i % 50 == 0 or i == len(normalized_documents):\n",
    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Financial Entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting entities from 100 chunks...\n",
      "🧠 Normalizing text 🔄🔧 (0.0s) | 🧠 Semantica is extracting: Extracted 4 entities using ml |███████████████| 100.0% ✅🎯  Processed 20/100 chunks (56 entities found)\n",
      "🧠 Normalizing text 🔄🔧 (0.0s) | 🧠 Semantica is extracting: Extracted 5 entities using ml |███████████████| 100.0% ✅🎯�  Processed 40/100 chunks (160 entities found)\n",
      "🧠 Normalizing text 🔄🔧 (0.0s) | 🧠 Semantica is extracting: Extracted 2 entities using ml |███████████████| 100.0% ✅🎯  Processed 60/100 chunks (182 entities found)\n",
      "🧠 Normalizing text 🔄🔧 (0.0s) | 🧠 Semantica is extracting: Extracted 1 entities using ml |███████████████| 100.0% ✅🎯  Processed 80/100 chunks (215 entities found)\n",
      "🧠 Normalizing text 🔄🔧 (0.0s) | 🧠 Semantica is extracting: Extracted 2 entities using ml |███████████████| 100.0% ✅🎯  Processed 100/100 chunks (264 entities found)\n",
      "Extracted 66 companies/organizations, 59 markets/locations, 8 prices, 24 metrics\n"
     ]
    }
   ],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "entity_extractor = NERExtractor(\n",
    "    method=\"ml\",\n",
    "    model=\"en_core_web_sm\"\n",
    ")\n",
    "\n",
    "all_entities = []\n",
    "print(f\"Extracting entities from {len(chunked_documents)} chunks...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        entities = entity_extractor.extract_entities(chunk_text)\n",
    "        all_entities.extend(entities)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found)\")\n",
    "\n",
    "# Categorize entities using spaCy's standard types (ORG, GPE, MONEY, etc.)\n",
    "companies = [e for e in all_entities if e.label in [\"ORG\", \"ORGANIZATION\"]]\n",
    "markets = [e for e in all_entities if e.label in [\"GPE\", \"LOCATION\", \"LOC\"]]\n",
    "prices = [e for e in all_entities if e.label in [\"MONEY\", \"CURRENCY\"]]\n",
    "metrics = [e for e in all_entities if e.label in [\"CARDINAL\", \"QUANTITY\", \"PERCENT\", \"PERCENTAGE\"]]\n",
    "\n",
    "print(f\"Extracted {len(companies)} companies/organizations, {len(markets)} markets/locations, {len(prices)} prices, {len(metrics)} metrics\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Financial Relationships\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting relationships from 100 chunks...\n",
      "🧠 Semantica is extracting: Extracted 2 entities using ml |███████████████| 100.0% ✅🎯 | 🧠 Semantica is extracting: Extracted 4 relations using dependency |███████████████| 100.0% ✅🎯  Processed 20/100 chunks (49 relationships found)\n",
      "🧠 Semantica is extracting: Extracted 2 entities using ml |███████████████| 100.0% ✅🎯 | 🧠 Semantica is extracting: Extracted 1 relations using dependency |███████████████| 100.0% ✅🎯  Processed 40/100 chunks (111 relationships found)\n",
      "🧠 Semantica is extracting: Extracted 2 entities using ml |███████████████| 100.0% ✅🎯 | 🧠 Semantica is extracting: Extracted 0 relations |███████████████| 100.0% ✅🎯█████| 100.0% ✅🎯  Processed 60/100 chunks (121 relationships found)\n",
      "🧠 Semantica is extracting: Extracted 2 entities using ml |███████████████| 100.0% ✅🎯 | 🧠 Semantica is extracting: Extracted 0 relations |███████████████| 100.0% ✅🎯█████| 100.0% ✅🎯  Processed 80/100 chunks (135 relationships found)\n",
      "🧠 Semantica is extracting: Extracted 2 entities using ml |███████████████| 100.0% ✅🎯 | 🧠 Semantica is extracting: Extracted 3 relations using dependency |███████████████| 100.0% ✅🎯  Processed 100/100 chunks (175 relationships found)\n",
      "Extracted 175 relationships\n"
     ]
    }
   ],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"dependency\",\n",
    "    model=\"en_core_web_sm\"\n",
    ")\n",
    "\n",
    "all_relationships = []\n",
    "print(f\"Extracting relationships from {len(chunked_documents)} chunks...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        relationships = relation_extractor.extract_relations(\n",
    "            chunk_text,\n",
    "            entities=all_entities,\n",
    "            relation_types=[\"trades_on\", \"has_price\", \"belongs_to\", \"correlates_with\", \"has_metric\", \"in_sector\"]\n",
    "        )\n",
    "        all_relationships.extend(relationships)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
    "\n",
    "print(f\"Extracted {len(all_relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resolving Duplicate Companies and Stocks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conflict Detection\n",
    "\n",
    "- **Temporal Conflict Detection**: Detects time-sensitive conflicts in financial data from multiple sources\n",
    "- **Most Recent Strategy**: Resolves conflicts by prioritizing the latest market data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Detecting temporal conflicts in 264 entities...\n",
      "🧠 Semantica is extracting: Extracted 3 relations using dependency |███████████████| 100.0% ✅🎯 | 🧠 Semantica is resolving: Detected 0 temporal conflicts |███████████████| 100.0% [165/193] ✅⚠️ (223.9/s)████████░░░| 85.5% [165/193] 🔄⚠️ (ETA: 0.1s | 225.1/s)Detected 0 temporal conflicts\n",
      "No conflicts detected\n"
     ]
    }
   ],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "# Convert Entity objects to dictionaries for conflict detection\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"id\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"text\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"name\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"type\": e.label if hasattr(e, 'label') else \"ENTITY\",\n",
    "        \"confidence\": e.confidence if hasattr(e, 'confidence') else 1.0,\n",
    "        \"metadata\": e.metadata if hasattr(e, 'metadata') else {},\n",
    "        \"source\": e.metadata.get(\"source\", \"unknown\") if hasattr(e, 'metadata') and isinstance(e.metadata, dict) else \"unknown\"\n",
    "    }\n",
    "    for e in all_entities\n",
    "]\n",
    "\n",
    "print(f\"Detecting temporal conflicts in {len(entity_dicts)} entities...\")\n",
    "conflicts = conflict_detector.detect_temporal_conflicts(entity_dicts)\n",
    "\n",
    "print(f\"Detected {len(conflicts)} temporal conflicts\")\n",
    "\n",
    "if conflicts:\n",
    "    print(f\"Resolving conflicts using most_recent strategy...\")\n",
    "    resolved = conflict_resolver.resolve_conflicts(\n",
    "        conflicts,\n",
    "        strategy=\"most_recent\"\n",
    "    )\n",
    "    print(f\"Resolved {len(resolved)} conflicts\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converting 264 entities to dictionaries...\n",
      "Resolving duplicates in 264 entities...\n",
      "🧠 Semantica is deduplicating: Merging groups... 1/1 (remaining: 0) |███████████████| 100.0% [1/1] 🔄🔄 (16.4/s) | 🧠 Semantica is deduplicating: Building merged entity... (4/4, remaining: 0 steps) |███████████████| 100.0% [4/4] 🔄🔄 (258.3/s) 🔄🔄 (ETA: 0.0s | 374.6/s)333.9/s)Converting 36 resolved entities back to Entity objects...\n",
      "Deduplicated 264 entities to 36 unique entities\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import EntityResolver\n",
    "from semantica.semantic_extract import Entity\n",
    "\n",
    "# Convert Entity objects to dictionaries for EntityResolver\n",
    "print(f\"Converting {len(all_entities)} entities to dictionaries...\")\n",
    "entity_dicts = [{\"name\": e.text, \"type\": e.label, \"start_char\": getattr(e, 'start_char', 0), \"end_char\": getattr(e, 'end_char', 0), \"confidence\": e.confidence} for e in all_entities]\n",
    "\n",
    "# Use EntityResolver class to resolve duplicates\n",
    "entity_resolver = EntityResolver(strategy=\"fuzzy\", similarity_threshold=0.85)\n",
    "\n",
    "print(f\"Resolving duplicates in {len(entity_dicts)} entities...\")\n",
    "resolved_entities = entity_resolver.resolve_entities(entity_dicts)\n",
    "\n",
    "# Convert back to Entity objects\n",
    "print(f\"Converting {len(resolved_entities)} resolved entities back to Entity objects...\")\n",
    "merged_entities = [\n",
    "    Entity(text=e[\"name\"], label=e[\"type\"], start_char=e.get(\"start_char\", 0), end_char=e.get(\"end_char\", 0), confidence=e.get(\"confidence\", 1.0))\n",
    "    for e in resolved_entities\n",
    "]\n",
    "\n",
    "# Enhance entities with seed data information\n",
    "for entity in merged_entities:\n",
    "    for seed_item in seed_data:\n",
    "        if entity.text.lower() == seed_item[\"text\"].lower():\n",
    "            entity.description = seed_item.get(\"description\", \"\")\n",
    "            break\n",
    "\n",
    "print(f\"Deduplicated {len(entity_dicts)} entities to {len(merged_entities)} unique entities\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Financial Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building knowledge graph...\n",
      "🧠 Semantica is deduplicating: Building merged entity... (4/4, remaining: 0 steps) |███████████████| 100.0% [4/4] 🔄🔄 (258.3/s) | 🧠 Semantica is building: Processing relationships... 175/175 |███████████████| 100.0% [175/175] 🔄🧠 (27889.8/s) 15937.0/s)Building graph structure...\n",
      "✅ Graph structure built (0.00s)\n",
      "🧠 Semantica is deduplicating: Building merged entity... (4/4, remaining: 0 steps) |███████████████| 100.0% [4/4] 🔄🔄 (258.3/s) | 🧠 Semantica is building: Processing relationships... 175/175 |███████████████| 100.0% [175/175] 🔄🧠 (27889.8/s)\n",
      "============================================================\n",
      "✅ Knowledge Graph Build Complete\n",
      "   Entities: 36\n",
      "   Relationships: 175\n",
      "   Total time: 0.86s\n",
      "============================================================\n",
      "Graph: 36 entities, 175 relationships\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "graph_builder = GraphBuilder()\n",
    "\n",
    "print(f\"Building knowledge graph...\")\n",
    "kg_sources = [{\n",
    "    \"entities\": [{\"text\": e.text, \"type\": e.label, \"confidence\": e.confidence} for e in merged_entities],\n",
    "    \"relationships\": [{\"source\": r.subject.text, \"target\": r.object.text, \"type\": r.predicate, \"confidence\": r.confidence} for r in all_relationships]\n",
    "}]\n",
    "\n",
    "kg = graph_builder.build(kg_sources)\n",
    "\n",
    "entities_count = len(kg.get('entities', []))\n",
    "relationships_count = len(kg.get('relationships', []))\n",
    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Embeddings for Companies and Stocks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating embeddings for 66 companies and 59 markets...\n",
      "Generated 66 company embeddings and 59 market embeddings\n"
     ]
    }
   ],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    provider=\"sentence_transformers\",\n",
    "    model=EMBEDDING_MODEL\n",
    ")\n",
    "\n",
    "print(f\"Generating embeddings for {len(companies)} companies and {len(markets)} markets...\")\n",
    "company_texts = [c.text for c in companies]\n",
    "company_embeddings = embedding_gen.generate_embeddings(company_texts)\n",
    "\n",
    "market_texts = [m.text for m in markets]\n",
    "market_embeddings = embedding_gen.generate_embeddings(market_texts)\n",
    "\n",
    "print(f\"Generated {len(company_embeddings)} company embeddings and {len(market_embeddings)} market embeddings\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populating Vector Store\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "fastembed not available. Install with: pip install fastembed. Using fallback embedding method.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Storing 66 company vectors and 59 market vectors...\n",
      "🧠 Semantica is building: Processing relationships... 175/175 |███████████████| 100.0% [175/175] 🔄🧠 (27889.8/s) | 🧠 Semantica is indexing: Storing 66 vectors 🔄📊 (0.0s)Stored 66 company vectors and 59 market vectors\n"
     ]
    }
   ],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "\n",
    "print(f\"Storing {len(company_embeddings)} company vectors and {len(market_embeddings)} market vectors...\")\n",
    "company_ids = vector_store.store_vectors(\n",
    "    vectors=company_embeddings,\n",
    "    metadata=[{\"type\": \"company\", \"name\": c.text, \"label\": c.label} for c in companies]\n",
    ")\n",
    "\n",
    "market_ids = vector_store.store_vectors(\n",
    "    vectors=market_embeddings,\n",
    "    metadata=[{\"type\": \"market\", \"name\": m.text, \"label\": m.label} for m in markets]\n",
    ")\n",
    "\n",
    "print(f\"Stored {len(company_ids)} company vectors and {len(market_ids)} market vectors\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Market Network Structure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is indexing: Storing 66 vectors 🔄📊 (0.0s) | 🧠 Semantica is building: Calculating degree centrality 🔄🧠 (0.0s)Graph analytics:\n",
      "  - Graph density: 0.000\n",
      "  - Central nodes (degree): 4\n",
      "  - Total entities: 36\n",
      "  - Total relationships: 175\n"
     ]
    }
   ],
   "source": [
    "from semantica.kg import GraphAnalyzer, CentralityCalculator\n",
    "\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "centrality_calc = CentralityCalculator()\n",
    "\n",
    "analysis = graph_analyzer.analyze_graph(kg)\n",
    "\n",
    "degree_centrality = centrality_calc.calculate_degree_centrality(kg)\n",
    "betweenness_centrality = centrality_calc.calculate_betweenness_centrality(kg)\n",
    "closeness_centrality = centrality_calc.calculate_closeness_centrality(kg)\n",
    "\n",
    "# Identify central entities in the market network\n",
    "central_entities = []\n",
    "for entity in kg.get(\"entities\", []):\n",
    "    entity_id = entity.get(\"id\")\n",
    "    if entity_id in degree_centrality:\n",
    "        central_entities.append({\n",
    "            \"name\": entity.get(\"text\", \"Unknown\"),\n",
    "            \"type\": entity.get(\"type\", \"Unknown\"),\n",
    "            \"degree\": degree_centrality[entity_id]\n",
    "        })\n",
    "\n",
    "central_entities.sort(key=lambda x: x['degree'], reverse=True)\n",
    "\n",
    "print(f\"Graph analytics:\")\n",
    "print(f\"  - Graph density: {analysis.get('density', 0):.3f}\")\n",
    "print(f\"  - Central nodes (degree): {len(degree_centrality)}\")\n",
    "print(f\"  - Total entities: {len(kg.get('entities', []))}\")\n",
    "print(f\"  - Total relationships: {len(kg.get('relationships', []))}\")\n",
    "if central_entities:\n",
    "    print(f\"\\nTop 5 central entities:\")\n",
    "    for i, ent in enumerate(central_entities[:5], 1):\n",
    "        print(f\"  {i}. {ent['name']} ({ent['type']}) - Degree: {ent['degree']:.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphRAG: Hybrid Vector + Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "GraphRAG Query: What technology companies are in the market and what are their key relationships?\n",
      "================================================================================\n",
      "\n",
      "🧠 Semantica is embedding: Generated embedding (dim: 128) |███████████████| 100.0% ✅💾 | 🧠 Semantica is processing: Ranking results... 🔄🔗 (0.0s)technology companies are in the market?... 🔄🔗 (0.0s)🔄🔗 (0.0s)s)ket?... 🔄🔗 (0.0s)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Embedding generation failed: Text cannot be empty or whitespace-only\n",
      "Using random fallback embedding\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "Generated Answer (with Multi-hop Reasoning):\n",
      "================================================================================\n",
      "Based on the retrieved context and reasoning paths, I can identify some technology companies and their key relationships. However, the context provided does not directly mention specific technology companies. \n",
      "\n",
      "One entity that is mentioned as a GPE (Geopolitical Entity) is Russia, which is related to the year through the reasoning path \"Russia --[be]--> the year\" (Path 1). This connection suggests that Russia is associated with a specific year, but the year is not explicitly stated.\n",
      "\n",
      "Another entity mentioned is Ukraine, which is also a GPE. It is related to the year through the reasoning path \"Ukraine --[be]--> the year\" (Path 1), similar to Russia. \n",
      "\n",
      "The only technology company mentioned in the context is Nvidia, but it is not related to any other entities or relationships in the context.\n",
      "\n",
      "To answer the question, I would say that there are limited technology companies mentioned in the context, and their relationships are not explicitly stated. However, based on the reasoning paths, we can infer that Russia and Ukraine are associated with a specific year.\n",
      "\n",
      "Key relationships:\n",
      "\n",
      "- Russia and Ukraine are associated with a specific year.\n",
      "- The year is related to Russia and Ukraine through the reasoning path \"Russia --[be]--> the year\" (Path 1) and \"Ukraine --[be]--> the year\" (Path 1).\n",
      "\n",
      "Note: The context does not provide sufficient information to identify specific technology companies and their relationships.\n",
      "\n",
      "================================================================================\n",
      "\n",
      "Reasoning Details:\n",
      "- Confidence: 0.636\n",
      "- Sources: 15\n",
      "- Reasoning Paths: 49\n",
      "- Total entities in graph: 36\n",
      "- Total relationships in graph: 175\n",
      "\n",
      "Top Sources:\n",
      "  1. Score: 0.736\n",
      "     the final days of the year is a DATE. is outstriped by Warren Buffett. is rallyed by Masayoshi Son’s. say Russia (GPE). buy Donald Trump’s (PERSON), New York City (GPE). is selled by Jonathan Ferro....\n",
      "  2. Score: 0.728\n",
      "     2026 is a DATE. is looked by Bloomberg’s Skylar Montgomery Koning....\n",
      "  3. Score: 0.651\n",
      "     the year is a DATE. is beed by Russia (GPE). sell the Christmas holiday, Ukraine (GPE), Pinnacle Group and 1 more. rally the Christmas holiday, Ukraine (GPE)....\n",
      "  4. Score: 0.597\n",
      "     Ukraine is a GPE. position the trading day. is selled by the year (DATE). is broadened by Joe Weisenthal. is rallyed by the year (DATE)....\n",
      "  5. Score: 0.591\n",
      "     Nvidia is a GPE....\n"
     ]
    }
   ],
   "source": [
    "from semantica.context import AgentContext\n",
    "from semantica.llms import Groq\n",
    "import os\n",
    "\n",
    "context = AgentContext(\n",
    "    vector_store=vector_store,\n",
    "    knowledge_graph=kg,\n",
    "    max_expansion_hops=3,\n",
    "    hybrid_alpha=0.7\n",
    ")\n",
    "\n",
    "# Initialize Groq LLM\n",
    "llm = Groq(model=\"llama-3.1-8b-instant\", api_key=os.getenv(\"GROQ_API_KEY\"))\n",
    "\n",
    "query = \"What technology companies are in the market and what are their key relationships?\"\n",
    "\n",
    "print(f\"{'='*80}\")\n",
    "print(f\"GraphRAG Query: {query}\")\n",
    "print(f\"{'='*80}\\n\")\n",
    "\n",
    "# Use multi-hop reasoning with LLM generation\n",
    "result = context.query_with_reasoning(\n",
    "    query=query,\n",
    "    llm_provider=llm,\n",
    "    max_results=15,\n",
    "    max_hops=3,\n",
    "    min_score=0.2\n",
    ")\n",
    "\n",
    "print(\"=\" * 80)\n",
    "print(\"Generated Answer (with Multi-hop Reasoning):\")\n",
    "print(\"=\" * 80)\n",
    "response = result.get('response', 'No response generated')\n",
    "print(response)\n",
    "print(\"\\n\" + \"=\" * 80)\n",
    "\n",
    "print(f\"\\nReasoning Details:\")\n",
    "print(f\"- Confidence: {result.get('confidence', 0):.3f}\")\n",
    "print(f\"- Sources: {result.get('num_sources', 0)}\")\n",
    "print(f\"- Reasoning Paths: {result.get('num_reasoning_paths', 0)}\")\n",
    "print(f\"- Total entities in graph: {len(kg.get('entities', []))}\")\n",
    "print(f\"- Total relationships in graph: {len(kg.get('relationships', []))}\")\n",
    "\n",
    "if result.get('sources'):\n",
    "    print(f\"\\nTop Sources:\")\n",
    "    for i, source in enumerate(result['sources'][:5], 1):\n",
    "        content = source.get('content', '')[:200] if isinstance(source, dict) else str(source)[:200]\n",
    "        score = source.get('score', 0) if isinstance(source, dict) else 0\n",
    "        print(f\"  {i}. Score: {score:.3f}\")\n",
    "        print(f\"     {content}...\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🧠 Semantica is processing: Ranking results... 🔄🔗 (0.0s) | 🧠 Semantica is exporting: Exporting graph to json: financial_data_kg.json 🔄💾 (0.0s)Exported financial knowledge graph to JSON, GraphML, and CSV formats\n"
     ]
    }
   ],
   "source": [
    "from semantica.export import GraphExporter\n",
    "\n",
    "exporter = GraphExporter()\n",
    "exporter.export(kg, output_path=\"financial_data_kg.json\", format=\"json\")\n",
    "exporter.export(kg, output_path=\"financial_data_kg.graphml\", format=\"graphml\")\n",
    "\n",
    "print(\"Exported financial knowledge graph to JSON, GraphML, and CSV formats\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
