{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration_MCP.ipynb)\n",
    "\n",
    "# Financial Data Integration (MCP) - Real-Time Market Data\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates **financial data integration using MCP servers** with focus on **MCP server integration**, **real-time data ingestion**, and **multi-source financial KG construction**. The pipeline integrates Python/FastMCP servers to ingest market data, stock prices, and metrics into a financial knowledge graph.\n",
    "\n",
    "### Key Features\n",
    "\n",
    "- **MCP Integration**: Showcases MCP (Model Context Protocol) server integration capability\n",
    "- **Seed Data Management**: Uses foundation market data for entity resolution\n",
    "- **Real-Time Data Ingestion**: Ingests live market data from MCP servers and APIs\n",
    "- **Multi-Source Financial KG**: Builds comprehensive financial knowledge graphs from multiple sources\n",
    "- **Market Network Analysis**: Analyzes market structure using graph analytics\n",
    "- **Comprehensive Data Sources**: Multiple financial APIs, RSS feeds, and databases\n",
    "- **Modular Architecture**: Direct use of Semantica modules without core orchestrator\n",
    "\n",
    "### Learning Objectives\n",
    "\n",
    "- Ingest financial data from MCP servers, APIs, and RSS feeds\n",
    "- Use seed data for foundation market information\n",
    "- Extract financial entities (Companies, Stocks, Prices, Metrics, Markets, Sectors)\n",
    "- Build financial knowledge graphs with seed data integration\n",
    "- Analyze market network structure using graph analytics\n",
    "- Store and query financial data using vector stores\n",
    "\n",
    "### Pipeline Flow\n",
    "\n",
    "```mermaid\n",
    "graph TD\n",
    "    A[Data Ingestion] --> B[Seed Data Loading]\n",
    "    B --> C[Document Parsing]\n",
    "    C --> D[Text Processing]\n",
    "    D --> E[Entity Extraction]\n",
    "    E --> F[Relationship Extraction]\n",
    "    F --> G[Deduplication]\n",
    "    G --> H[Knowledge Graph]\n",
    "    H --> I[Embeddings]\n",
    "    I --> J[Vector Store]\n",
    "    H --> K[Graph Analytics]\n",
    "    K --> L[GraphRAG Queries]\n",
    "    J --> L\n",
    "    L --> M[Visualization]\n",
    "    M --> N[Export]\n",
    "```\n",
    "\n",
    "\n",
    "## Installation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qU semantica networkx matplotlib plotly pandas faiss-cpu beautifulsoup4 groq sentence-transformers scikit-learn\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration & Setup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"GROQ_API_KEY\"] = os.getenv(\"GROQ_API_KEY\", \"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\")\n",
    "\n",
    "# Configuration constants\n",
    "EMBEDDING_DIMENSION = 384\n",
    "EMBEDDING_MODEL = \"all-MiniLM-L6-v2\"\n",
    "CHUNK_SIZE = 1000\n",
    "CHUNK_OVERLAP = 200\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingesting Financial Data from Multiple Sources\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import MCPIngestor, WebIngestor, FeedIngestor, FileIngestor\n",
    "import os\n",
    "from contextlib import redirect_stderr\n",
    "from io import StringIO\n",
    "\n",
    "os.makedirs(\"data\", exist_ok=True)\n",
    "\n",
    "feed_sources = [\n",
    "    # Financial RSS Feeds - More reliable sources\n",
    "    (\"Yahoo Finance\", \"https://feeds.finance.yahoo.com/rss/2.0/headline\"),\n",
    "    (\"Financial Times\", \"https://www.ft.com/?format=rss\"),\n",
    "    (\"Bloomberg\", \"https://feeds.bloomberg.com/markets/news.rss\"),\n",
    "    (\"MarketWatch\", \"https://feeds.marketwatch.com/marketwatch/markets\"),\n",
    "    (\"Seeking Alpha\", \"https://seekingalpha.com/feed.xml\"),\n",
    "    (\"Investing.com\", \"https://www.investing.com/rss/news.rss\"),\n",
    "    (\"Financial News\", \"https://www.fnlondon.com/rss\"),\n",
    "    (\"Wall Street Journal\", \"https://feeds.a.dj.com/rss/RSSMarketsMain.xml\"),\n",
    "]\n",
    "\n",
    "feed_ingestor = FeedIngestor()\n",
    "all_documents = []\n",
    "\n",
    "print(f\"Ingesting from {len(feed_sources)} feed sources...\")\n",
    "for i, (feed_name, feed_url) in enumerate(feed_sources, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            feed_data = feed_ingestor.ingest_feed(feed_url, validate=False)\n",
    "        \n",
    "        feed_count = 0\n",
    "        for item in feed_data.items:\n",
    "            if not item.content:\n",
    "                item.content = item.description or item.title or \"\"\n",
    "            if item.content:\n",
    "                if not hasattr(item, 'metadata'):\n",
    "                    item.metadata = {}\n",
    "                item.metadata['source'] = feed_name\n",
    "                all_documents.append(item)\n",
    "                feed_count += 1\n",
    "        \n",
    "        if feed_count > 0:\n",
    "            print(f\"  [{i}/{len(feed_sources)}] {feed_name}: {feed_count} documents\")\n",
    "    except Exception:\n",
    "        continue\n",
    "\n",
    "# Example: Ingest from Alpha Vantage API (requires API key)\n",
    "alpha_vantage_api = \"https://www.alphavantage.co/query?function=GLOBAL_QUOTE&symbol=AAPL&apikey=demo\"\n",
    "try:\n",
    "    web_ingestor = WebIngestor()\n",
    "    with redirect_stderr(StringIO()):\n",
    "        api_documents = web_ingestor.ingest(alpha_vantage_api, method=\"url\")\n",
    "    for doc in api_documents:\n",
    "        if not hasattr(doc, 'metadata'):\n",
    "            doc.metadata = {}\n",
    "        doc.metadata['source'] = 'Alpha Vantage API'\n",
    "        all_documents.append(doc)\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# MCP Server connection example (commented for demo)\n",
    "# mcp_ingestor = MCPIngestor()\n",
    "# mcp_ingestor.connect(\"financial_server\", url=\"http://localhost:8000/mcp\")\n",
    "# resources = mcp_ingestor.list_available_resources(\"financial_server\")\n",
    "# mcp_data = mcp_ingestor.ingest_resources(\"financial_server\", resource_uris=[\"resource://market_data\"])\n",
    "\n",
    "if not all_documents:\n",
    "    market_data = \"\"\"\n",
    "    AAPL stock price: $150.25, market cap: $2.4T, volume: 50M shares, sector: Technology\n",
    "    MSFT stock price: $380.50, market cap: $2.8T, volume: 30M shares, sector: Technology\n",
    "    GOOGL stock price: $140.75, market cap: $1.8T, volume: 25M shares, sector: Technology\n",
    "    JPM stock price: $145.30, market cap: $420B, volume: 15M shares, sector: Financial\n",
    "    \"\"\"\n",
    "    with open(\"data/market_data.txt\", \"w\") as f:\n",
    "        f.write(market_data)\n",
    "    file_ingestor = FileIngestor()\n",
    "    all_documents = file_ingestor.ingest(\"data/market_data.txt\")\n",
    "\n",
    "documents = all_documents\n",
    "print(f\"Ingested {len(documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.seed import SeedDataManager\n",
    "\n",
    "seed_manager = SeedDataManager()\n",
    "\n",
    "# Load foundation market data (exchanges, indices, sectors)\n",
    "seed_data = [\n",
    "    {\"type\": \"Market\", \"text\": \"NASDAQ\", \"description\": \"Stock exchange\"},\n",
    "    {\"type\": \"Market\", \"text\": \"NYSE\", \"description\": \"Stock exchange\"},\n",
    "    {\"type\": \"Market\", \"text\": \"S&P 500\", \"description\": \"Stock market index\"},\n",
    "    {\"type\": \"Sector\", \"text\": \"Technology\", \"description\": \"Market sector\"},\n",
    "    {\"type\": \"Sector\", \"text\": \"Financial\", \"description\": \"Market sector\"},\n",
    "    {\"type\": \"Sector\", \"text\": \"Healthcare\", \"description\": \"Market sector\"},\n",
    "]\n",
    "\n",
    "# Add seed data as entities\n",
    "for item in seed_data:\n",
    "    entity = {\n",
    "        \"id\": item.get(\"text\", \"\").lower().replace(\" \", \"_\"),\n",
    "        \"text\": item.get(\"text\", \"\"),\n",
    "        \"name\": item.get(\"text\", \"\"),\n",
    "        \"type\": item.get(\"type\", \"\"),\n",
    "        \"description\": item.get(\"description\", \"\"),\n",
    "        \"source\": \"seed_data\"\n",
    "    }\n",
    "    seed_manager.seed_data.entities.append(entity)\n",
    "\n",
    "print(f\"Loaded {len(seed_data)} seed data items for market foundation\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parsing Financial Documents\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.parse import DocumentParser\n",
    "\n",
    "parser = DocumentParser()\n",
    "\n",
    "print(f\"Parsing {len(documents)} documents...\")\n",
    "parsed_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    try:\n",
    "        parsed = parser.parse(\n",
    "            doc.content if hasattr(doc, 'content') else str(doc),\n",
    "            content_type=\"text\"\n",
    "        )\n",
    "        parsed_documents.append(parsed)\n",
    "    except Exception:\n",
    "        parsed_documents.append(doc)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Parsed {i}/{len(documents)} documents...\")\n",
    "\n",
    "documents = parsed_documents\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalizing and Chunking Financial Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.normalize import TextNormalizer\n",
    "from semantica.split import TextSplitter\n",
    "\n",
    "normalizer = TextNormalizer()\n",
    "# Use recursive chunking for financial documents\n",
    "splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "\n",
    "print(f\"Normalizing {len(documents)} documents...\")\n",
    "normalized_documents = []\n",
    "for i, doc in enumerate(documents, 1):\n",
    "    normalized_text = normalizer.normalize(\n",
    "        doc.content if hasattr(doc, 'content') else str(doc),\n",
    "        clean_html=True,\n",
    "        normalize_entities=True,\n",
    "        normalize_numbers=True,\n",
    "        remove_extra_whitespace=True,\n",
    "        lowercase=False\n",
    "    )\n",
    "    normalized_documents.append(normalized_text)\n",
    "    if i % 50 == 0 or i == len(documents):\n",
    "        print(f\"  Normalized {i}/{len(documents)} documents...\")\n",
    "\n",
    "print(f\"Chunking {len(normalized_documents)} documents...\")\n",
    "chunked_documents = []\n",
    "for i, doc_text in enumerate(normalized_documents, 1):\n",
    "    try:\n",
    "        with redirect_stderr(StringIO()):\n",
    "            chunks = splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    except Exception:\n",
    "        simple_splitter = TextSplitter(method=\"recursive\", chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
    "        chunks = simple_splitter.split(doc_text)\n",
    "        chunked_documents.extend(chunks)\n",
    "    if i % 50 == 0 or i == len(normalized_documents):\n",
    "        print(f\"  Chunked {i}/{len(normalized_documents)} documents ({len(chunked_documents)} chunks so far)\")\n",
    "\n",
    "print(f\"Created {len(chunked_documents)} chunks from {len(normalized_documents)} documents\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Financial Entities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import NERExtractor\n",
    "\n",
    "entity_extractor = NERExtractor(\n",
    "    method=\"ml\",\n",
    "    model=\"en_core_web_sm\"\n",
    ")\n",
    "\n",
    "all_entities = []\n",
    "print(f\"Extracting entities from {len(chunked_documents)} chunks...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        entities = entity_extractor.extract_entities(chunk_text)\n",
    "        all_entities.extend(entities)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_entities)} entities found)\")\n",
    "\n",
    "# Categorize entities using spaCy's standard types (ORG, GPE, MONEY, etc.)\n",
    "companies = [e for e in all_entities if e.label in [\"ORG\", \"ORGANIZATION\"]]\n",
    "markets = [e for e in all_entities if e.label in [\"GPE\", \"LOCATION\", \"LOC\"]]\n",
    "prices = [e for e in all_entities if e.label in [\"MONEY\", \"CURRENCY\"]]\n",
    "metrics = [e for e in all_entities if e.label in [\"CARDINAL\", \"QUANTITY\", \"PERCENT\", \"PERCENTAGE\"]]\n",
    "\n",
    "print(f\"Extracted {len(companies)} companies/organizations, {len(markets)} markets/locations, {len(prices)} prices, {len(metrics)} metrics\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Financial Relationships\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.semantic_extract import RelationExtractor\n",
    "\n",
    "relation_extractor = RelationExtractor(\n",
    "    method=\"dependency\",\n",
    "    model=\"en_core_web_sm\"\n",
    ")\n",
    "\n",
    "all_relationships = []\n",
    "print(f\"Extracting relationships from {len(chunked_documents)} chunks...\")\n",
    "for i, chunk in enumerate(chunked_documents, 1):\n",
    "    chunk_text = chunk.text if hasattr(chunk, 'text') else str(chunk)\n",
    "    try:\n",
    "        relationships = relation_extractor.extract_relations(\n",
    "            chunk_text,\n",
    "            entities=all_entities,\n",
    "            relation_types=[\"trades_on\", \"has_price\", \"belongs_to\", \"correlates_with\", \"has_metric\", \"in_sector\"]\n",
    "        )\n",
    "        all_relationships.extend(relationships)\n",
    "    except Exception:\n",
    "        continue\n",
    "    \n",
    "    if i % 20 == 0 or i == len(chunked_documents):\n",
    "        print(f\"  Processed {i}/{len(chunked_documents)} chunks ({len(all_relationships)} relationships found)\")\n",
    "\n",
    "print(f\"Extracted {len(all_relationships)} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resolving Duplicate Companies and Stocks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conflict Detection\n",
    "\n",
    "- **Temporal Conflict Detection**: Detects time-sensitive conflicts in financial data from multiple sources\n",
    "- **Most Recent Strategy**: Resolves conflicts by prioritizing the latest market data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.conflicts import ConflictDetector, ConflictResolver\n",
    "\n",
    "conflict_detector = ConflictDetector()\n",
    "conflict_resolver = ConflictResolver()\n",
    "\n",
    "# Convert Entity objects to dictionaries for conflict detection\n",
    "entity_dicts = [\n",
    "    {\n",
    "        \"id\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"text\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"name\": e.text if hasattr(e, 'text') else str(e),\n",
    "        \"type\": e.label if hasattr(e, 'label') else \"ENTITY\",\n",
    "        \"confidence\": e.confidence if hasattr(e, 'confidence') else 1.0,\n",
    "        \"metadata\": e.metadata if hasattr(e, 'metadata') else {},\n",
    "        \"source\": e.metadata.get(\"source\", \"unknown\") if hasattr(e, 'metadata') and isinstance(e.metadata, dict) else \"unknown\"\n",
    "    }\n",
    "    for e in all_entities\n",
    "]\n",
    "\n",
    "print(f\"Detecting temporal conflicts in {len(entity_dicts)} entities...\")\n",
    "conflicts = conflict_detector.detect_temporal_conflicts(entity_dicts)\n",
    "\n",
    "print(f\"Detected {len(conflicts)} temporal conflicts\")\n",
    "\n",
    "if conflicts:\n",
    "    print(f\"Resolving conflicts using most_recent strategy...\")\n",
    "    resolved = conflict_resolver.resolve_conflicts(\n",
    "        conflicts,\n",
    "        strategy=\"most_recent\"\n",
    "    )\n",
    "    print(f\"Resolved {len(resolved)} conflicts\")\n",
    "else:\n",
    "    print(\"No conflicts detected\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import EntityResolver\n",
    "from semantica.semantic_extract import Entity\n",
    "\n",
    "# Convert Entity objects to dictionaries for EntityResolver\n",
    "print(f\"Converting {len(all_entities)} entities to dictionaries...\")\n",
    "entity_dicts = [{\"name\": e.text, \"type\": e.label, \"start_char\": getattr(e, 'start_char', 0), \"end_char\": getattr(e, 'end_char', 0), \"confidence\": e.confidence} for e in all_entities]\n",
    "\n",
    "# Use EntityResolver class to resolve duplicates\n",
    "entity_resolver = EntityResolver(strategy=\"fuzzy\", similarity_threshold=0.85)\n",
    "\n",
    "print(f\"Resolving duplicates in {len(entity_dicts)} entities...\")\n",
    "resolved_entities = entity_resolver.resolve_entities(entity_dicts)\n",
    "\n",
    "# Convert back to Entity objects\n",
    "print(f\"Converting {len(resolved_entities)} resolved entities back to Entity objects...\")\n",
    "merged_entities = [\n",
    "    Entity(text=e[\"name\"], label=e[\"type\"], start_char=e.get(\"start_char\", 0), end_char=e.get(\"end_char\", 0), confidence=e.get(\"confidence\", 1.0))\n",
    "    for e in resolved_entities\n",
    "]\n",
    "\n",
    "# Enhance entities with seed data information\n",
    "for entity in merged_entities:\n",
    "    for seed_item in seed_data:\n",
    "        if entity.text.lower() == seed_item[\"text\"].lower():\n",
    "            entity.description = seed_item.get(\"description\", \"\")\n",
    "            break\n",
    "\n",
    "print(f\"Deduplicated {len(entity_dicts)} entities to {len(merged_entities)} unique entities\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Financial Knowledge Graph\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphBuilder\n",
    "\n",
    "graph_builder = GraphBuilder()\n",
    "\n",
    "print(f\"Building knowledge graph...\")\n",
    "kg_sources = [{\n",
    "    \"entities\": [{\"text\": e.text, \"type\": e.label, \"confidence\": e.confidence} for e in merged_entities],\n",
    "    \"relationships\": [{\"source\": r.subject.text, \"target\": r.object.text, \"type\": r.predicate, \"confidence\": r.confidence} for r in all_relationships]\n",
    "}]\n",
    "\n",
    "kg = graph_builder.build(kg_sources)\n",
    "\n",
    "entities_count = len(kg.get('entities', []))\n",
    "relationships_count = len(kg.get('relationships', []))\n",
    "print(f\"Graph: {entities_count} entities, {relationships_count} relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Embeddings for Companies and Stocks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.embeddings import EmbeddingGenerator\n",
    "\n",
    "embedding_gen = EmbeddingGenerator(\n",
    "    provider=\"sentence_transformers\",\n",
    "    model=EMBEDDING_MODEL\n",
    ")\n",
    "\n",
    "print(f\"Generating embeddings for {len(companies)} companies and {len(markets)} markets...\")\n",
    "company_texts = [c.text for c in companies]\n",
    "company_embeddings = embedding_gen.generate_embeddings(company_texts)\n",
    "\n",
    "market_texts = [m.text for m in markets]\n",
    "market_embeddings = embedding_gen.generate_embeddings(market_texts)\n",
    "\n",
    "print(f\"Generated {len(company_embeddings)} company embeddings and {len(market_embeddings)} market embeddings\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populating Vector Store\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.vector_store import VectorStore\n",
    "\n",
    "vector_store = VectorStore(backend=\"faiss\", dimension=EMBEDDING_DIMENSION)\n",
    "\n",
    "print(f\"Storing {len(company_embeddings)} company vectors and {len(market_embeddings)} market vectors...\")\n",
    "company_ids = vector_store.store_vectors(\n",
    "    vectors=company_embeddings,\n",
    "    metadata=[{\"type\": \"company\", \"name\": c.text, \"label\": c.label} for c in companies]\n",
    ")\n",
    "\n",
    "market_ids = vector_store.store_vectors(\n",
    "    vectors=market_embeddings,\n",
    "    metadata=[{\"type\": \"market\", \"name\": m.text, \"label\": m.label} for m in markets]\n",
    ")\n",
    "\n",
    "print(f\"Stored {len(company_ids)} company vectors and {len(market_ids)} market vectors\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing Market Network Structure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.kg import GraphAnalyzer, CentralityCalculator\n",
    "\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "centrality_calc = CentralityCalculator()\n",
    "\n",
    "analysis = graph_analyzer.analyze_graph(kg)\n",
    "\n",
    "degree_centrality = centrality_calc.calculate_degree_centrality(kg)\n",
    "betweenness_centrality = centrality_calc.calculate_betweenness_centrality(kg)\n",
    "closeness_centrality = centrality_calc.calculate_closeness_centrality(kg)\n",
    "\n",
    "# Identify central entities in the market network\n",
    "central_entities = []\n",
    "for entity in kg.get(\"entities\", []):\n",
    "    entity_id = entity.get(\"id\")\n",
    "    if entity_id in degree_centrality:\n",
    "        central_entities.append({\n",
    "            \"name\": entity.get(\"text\", \"Unknown\"),\n",
    "            \"type\": entity.get(\"type\", \"Unknown\"),\n",
    "            \"degree\": degree_centrality[entity_id]\n",
    "        })\n",
    "\n",
    "central_entities.sort(key=lambda x: x['degree'], reverse=True)\n",
    "\n",
    "print(f\"Graph analytics:\")\n",
    "print(f\"  - Graph density: {analysis.get('density', 0):.3f}\")\n",
    "print(f\"  - Central nodes (degree): {len(degree_centrality)}\")\n",
    "print(f\"  - Total entities: {len(kg.get('entities', []))}\")\n",
    "print(f\"  - Total relationships: {len(kg.get('relationships', []))}\")\n",
    "if central_entities:\n",
    "    print(f\"\\nTop 5 central entities:\")\n",
    "    for i, ent in enumerate(central_entities[:5], 1):\n",
    "        print(f\"  {i}. {ent['name']} ({ent['type']}) - Degree: {ent['degree']:.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphRAG: Hybrid Vector + Graph Queries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.context import AgentContext\n",
    "from semantica.llms import Groq\n",
    "import os\n",
    "\n",
    "context = AgentContext(\n",
    "    vector_store=vector_store,\n",
    "    knowledge_graph=kg,\n",
    "    max_expansion_hops=3,\n",
    "    hybrid_alpha=0.7\n",
    ")\n",
    "\n",
    "# Initialize Groq LLM\n",
    "llm = Groq(model=\"llama-3.1-8b-instant\", api_key=os.getenv(\"GROQ_API_KEY\"))\n",
    "\n",
    "query = \"What technology companies are in the market and what are their key relationships?\"\n",
    "\n",
    "print(f\"{'='*80}\")\n",
    "print(f\"GraphRAG Query: {query}\")\n",
    "print(f\"{'='*80}\\n\")\n",
    "\n",
    "# Use multi-hop reasoning with LLM generation\n",
    "result = context.query_with_reasoning(\n",
    "    query=query,\n",
    "    llm_provider=llm,\n",
    "    max_results=15,\n",
    "    max_hops=3,\n",
    "    min_score=0.2\n",
    ")\n",
    "\n",
    "print(\"=\" * 80)\n",
    "print(\"Generated Answer (with Multi-hop Reasoning):\")\n",
    "print(\"=\" * 80)\n",
    "response = result.get('response', 'No response generated')\n",
    "print(response)\n",
    "print(\"\\n\" + \"=\" * 80)\n",
    "\n",
    "print(f\"\\nReasoning Details:\")\n",
    "print(f\"- Confidence: {result.get('confidence', 0):.3f}\")\n",
    "print(f\"- Sources: {result.get('num_sources', 0)}\")\n",
    "print(f\"- Reasoning Paths: {result.get('num_reasoning_paths', 0)}\")\n",
    "print(f\"- Total entities in graph: {len(kg.get('entities', []))}\")\n",
    "print(f\"- Total relationships in graph: {len(kg.get('relationships', []))}\")\n",
    "\n",
    "if result.get('sources'):\n",
    "    print(f\"\\nTop Sources:\")\n",
    "    for i, source in enumerate(result['sources'][:5], 1):\n",
    "        content = source.get('content', '')[:200] if isinstance(source, dict) else str(source)[:200]\n",
    "        score = source.get('score', 0) if isinstance(source, dict) else 0\n",
    "        print(f\"  {i}. Score: {score:.3f}\")\n",
    "        print(f\"     {content}...\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting Results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.export import GraphExporter\n",
    "\n",
    "exporter = GraphExporter()\n",
    "exporter.export(kg, output_path=\"financial_data_kg.json\", format=\"json\")\n",
    "exporter.export(kg, output_path=\"financial_data_kg.graphml\", format=\"graphml\")\n",
    "\n",
    "print(\"Exported financial knowledge graph to JSON, GraphML, and CSV formats\")\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
