{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/01_Financial_Data_Integration.ipynb)\n",
    "\n",
    "# 📈 Financial Data Integration Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates how to integrate Python/FastMCP MCP servers as data sources for financial data ingestion. Connect to financial data MCP servers via URL, ingest market data, stock prices, and financial metrics, then build a knowledge graph for financial analysis.\n",
    "\n",
    "> [!IMPORTANT]\n",
    "> This implementation supports ONLY Python-based MCP servers and FastMCP servers. Users can bring their own Python/FastMCP MCP servers via URL connections.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### 🧩 Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: `MCPParser`, `JSONParser`, `StructuredDataParser`\n",
    "- **Extraction**: `NERExtractor`, `RelationExtractor`, `EventDetector`, `SemanticAnalyzer`\n",
    "- **KG**: `GraphBuilder`, `TemporalGraphQuery`, `GraphAnalyzer`\n",
    "- **Analytics**: `CentralityCalculator`, `CommunityDetector`, `ConnectivityAnalyzer`\n",
    "- **Reasoning**: `InferenceEngine`, `RuleManager`, `ExplanationGenerator`\n",
    "- **Export**: `JSONExporter`, `CSVExporter`, `RDFExporter`, `ReportGenerator`\n",
    "- **Visualization**: `KGVisualizer`, `TemporalVisualizer`, `AnalyticsVisualizer`\n",
    "\n",
    "### 🔄 Pipeline\n",
    "\n",
    "**Connect to Financial MCP Server → Ingest Market Data via MCP → Parse MCP Responses → Extract Financial Entities → Build Financial KG → Analyze Trends → Generate Reports → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## 🔌 Step 1: Connect to Financial Data MCP Server\n",
    "\n",
    "Connect to a Python/FastMCP MCP server that provides financial data via URL. The MCP server can expose resources (datasets, market data) and tools (queries, calculations).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import MCPIngestor, ingest_mcp\n",
    "from semantica.parse import MCPParser, JSONParser, StructuredDataParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, GraphAnalyzer\n",
    "from semantica.kg import CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.export import JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "# Initialize MCP ingestor\n",
    "mcp_ingestor = MCPIngestor()\n",
    "\n",
    "# Connect to financial data MCP server via URL\n",
    "# Replace with your actual MCP server URL\n",
    "# Example: http://localhost:8000/mcp or https://api.example.com/financial-mcp\n",
    "financial_mcp_url = \"http://localhost:8000/mcp\"\n",
    "\n",
    "# Connect to MCP server\n",
    "mcp_ingestor.connect(\n",
    "    \"financial_server\",\n",
    "    url=financial_mcp_url,\n",
    "    headers={\"Authorization\": \"Bearer your_token\"} if \"api.example.com\" in financial_mcp_url else {}\n",
    ")\n",
    "\n",
    "# List available resources (datasets, market data feeds)\n",
    "resources = mcp_ingestor.list_available_resources(\"financial_server\")\n",
    "print(f\"\\n📊 Available Resources ({len(resources)}):\")\n",
    "for resource in resources[:5]:  # Show first 5\n",
    "    print(f\"  - {resource.uri}: {resource.name}\")\n",
    "    if resource.description:\n",
    "        print(f\"    {resource.description[:80]}...\")\n",
    "\n",
    "# List available tools (queries, calculations)\n",
    "tools = mcp_ingestor.list_available_tools(\"financial_server\")\n",
    "print(f\"\\n🔧 Available Tools ({len(tools)}):\")\n",
    "for tool in tools[:5]:  # Show first 5\n",
    "    print(f\"  - {tool.name}: {tool.description or 'No description'}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📥 Step 2: Ingest Financial Data from MCP Server\n",
    "\n",
    "Ingest financial data using both resource-based and tool-based methods from the MCP server.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize parsers\n",
    "mcp_parser = MCPParser()\n",
    "json_parser = JSONParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "\n",
    "financial_data = []\n",
    "\n",
    "# Method 1: Resource-based ingestion\n",
    "# Ingest from MCP resources (pre-defined datasets)\n",
    "# Example: Ingest market data resource\n",
    "resource_data = mcp_ingestor.ingest_resources(\n",
    "    \"financial_server\",\n",
    "    resource_uris=[\"resource://market_data/daily\", \"resource://market_data/stocks\"]\n",
    ")\n",
    "\n",
    "for item in resource_data:\n",
    "    financial_data.append(item)\n",
    "    print(f\"  Ingested resource: {item}\")\n",
    "\n",
    "# Method 2: Tool-based ingestion\n",
    "# Call MCP tools to retrieve data dynamically\n",
    "# Example: Get stock prices for specific symbols\n",
    "stock_prices = mcp_ingestor.ingest_tool_output(\n",
    "    \"financial_server\",\n",
    "    tool_name=\"get_stock_prices\",\n",
    "    arguments={\n",
    "        \"symbols\": [\"AAPL\", \"MSFT\", \"GOOGL\", \"TSLA\"],\n",
    "        \"date\": datetime.now().isoformat()\n",
    "    }\n",
    ")\n",
    "\n",
    "if stock_prices:\n",
    "    financial_data.append(stock_prices)\n",
    "    print(f\"  Retrieved stock prices for {len(stock_prices) if isinstance(stock_prices, list) else 1} symbols\")\n",
    "\n",
    "# Example: Get market metrics\n",
    "market_metrics = mcp_ingestor.ingest_tool_output(\n",
    "    \"financial_server\",\n",
    "    tool_name=\"get_market_metrics\",\n",
    "    arguments={\"sector\": \"Technology\"}\n",
    ")\n",
    "\n",
    "if market_metrics:\n",
    "    financial_data.append(market_metrics)\n",
    "    print(f\"  Retrieved market metrics\")\n",
    "\n",
    "# Sample financial data (if MCP server is not available)\n",
    "if not financial_data:\n",
    "    sample_data = {\n",
    "        \"stock_prices\": [\n",
    "            {\n",
    "                \"symbol\": \"AAPL\",\n",
    "                \"company\": \"Apple Inc.\",\n",
    "                \"price\": 175.50,\n",
    "                \"change\": 2.30,\n",
    "                \"change_percent\": 1.33,\n",
    "                \"volume\": 45000000,\n",
    "                \"timestamp\": (datetime.now() - timedelta(hours=1)).isoformat(),\n",
    "                \"sector\": \"Technology\"\n",
    "            },\n",
    "            {\n",
    "                \"symbol\": \"MSFT\",\n",
    "                \"company\": \"Microsoft Corporation\",\n",
    "                \"price\": 380.25,\n",
    "                \"change\": -1.50,\n",
    "                \"change_percent\": -0.39,\n",
    "                \"volume\": 28000000,\n",
    "                \"timestamp\": (datetime.now() - timedelta(hours=1)).isoformat(),\n",
    "                \"sector\": \"Technology\"\n",
    "            },\n",
    "            {\n",
    "                \"symbol\": \"GOOGL\",\n",
    "                \"company\": \"Alphabet Inc.\",\n",
    "                \"price\": 142.80,\n",
    "                \"change\": 3.20,\n",
    "                \"change_percent\": 2.29,\n",
    "                \"volume\": 32000000,\n",
    "                \"timestamp\": (datetime.now() - timedelta(minutes=30)).isoformat(),\n",
    "                \"sector\": \"Technology\"\n",
    "            },\n",
    "            {\n",
    "                \"symbol\": \"TSLA\",\n",
    "                \"company\": \"Tesla Inc.\",\n",
    "                \"price\": 245.60,\n",
    "                \"change\": 5.40,\n",
    "                \"change_percent\": 2.25,\n",
    "                \"volume\": 55000000,\n",
    "                \"timestamp\": datetime.now().isoformat(),\n",
    "                \"sector\": \"Automotive\"\n",
    "            }\n",
    "        ],\n",
    "        \"market_metrics\": {\n",
    "            \"total_volume\": 150000000,\n",
    "            \"market_cap\": 15000000000000,\n",
    "            \"sectors\": [\"Technology\", \"Automotive\"]\n",
    "        }\n",
    "    }\n",
    "    financial_data.append(sample_data)\n",
    "    print(f\"  Loaded {len(sample_data['stock_prices'])} stock prices\")\n",
    "\n",
    "print(f\"\\n📊 Total financial data items ingested: {len(financial_data)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📄 Step 3: Parse MCP Data\n",
    "\n",
    "Parse the data received from MCP server responses (JSON, structured data).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed_financial_data = []\n",
    "\n",
    "# Parse MCP responses\n",
    "for data_item in financial_data:\n",
    "    # Parse MCP response (handles JSON, text, binary)\n",
    "    if isinstance(data_item, dict):\n",
    "        # If it's already structured, use it directly\n",
    "        parsed_item = data_item\n",
    "    else:\n",
    "        # Parse using MCP parser\n",
    "        parsed_item = mcp_parser.parse_response(data_item, response_type=\"json\")\n",
    "    \n",
    "    parsed_financial_data.append(parsed_item)\n",
    "    print(f\"  Parsed data item\")\n",
    "\n",
    "# Extract stock prices from parsed data\n",
    "stock_prices = []\n",
    "for item in parsed_financial_data:\n",
    "    if isinstance(item, dict):\n",
    "        if \"stock_prices\" in item:\n",
    "            stock_prices.extend(item[\"stock_prices\"])\n",
    "        elif \"symbol\" in item:\n",
    "            stock_prices.append(item)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ⛏️ Step 4: Extract Financial Entities and Relationships\n",
    "\n",
    "Extract financial entities (companies, stocks, sectors) and relationships from MCP data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "semantic_analyzer = SemanticAnalyzer()\n",
    "\n",
    "financial_entities = []\n",
    "financial_relationships = []\n",
    "\n",
    "# Extract entities and relationships from stock prices\n",
    "for stock in stock_prices:\n",
    "    if isinstance(stock, dict):\n",
    "        symbol = stock.get(\"symbol\", \"\")\n",
    "        company = stock.get(\"company\", \"\")\n",
    "        sector = stock.get(\"sector\", \"\")\n",
    "        \n",
    "        # Stock entity\n",
    "        financial_entities.append({\n",
    "            \"id\": symbol,\n",
    "            \"type\": \"Stock\",\n",
    "            \"name\": symbol,\n",
    "            \"properties\": {\n",
    "                \"price\": stock.get(\"price\", 0),\n",
    "                \"change\": stock.get(\"change\", 0),\n",
    "                \"change_percent\": stock.get(\"change_percent\", 0),\n",
    "                \"volume\": stock.get(\"volume\", 0),\n",
    "                \"timestamp\": stock.get(\"timestamp\", \"\")\n",
    "            }\n",
    "        })\n",
    "        \n",
    "        # Company entity\n",
    "        if company:\n",
    "            financial_entities.append({\n",
    "                \"id\": company,\n",
    "                \"type\": \"Company\",\n",
    "                \"name\": company,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            \n",
    "            # Stock-Company relationship\n",
    "            financial_relationships.append({\n",
    "                \"source\": symbol,\n",
    "                \"target\": company,\n",
    "                \"type\": \"ticker_for\",\n",
    "                \"properties\": {\"timestamp\": stock.get(\"timestamp\", \"\")}\n",
    "            })\n",
    "        \n",
    "        # Sector entity\n",
    "        if sector:\n",
    "            financial_entities.append({\n",
    "                \"id\": sector,\n",
    "                \"type\": \"Sector\",\n",
    "                \"name\": sector,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            \n",
    "            # Company-Sector relationship\n",
    "            if company:\n",
    "                financial_relationships.append({\n",
    "                    \"source\": company,\n",
    "                    \"target\": sector,\n",
    "                    \"type\": \"belongs_to\",\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "\n",
    "# Remove duplicates\n",
    "seen_entities = set()\n",
    "unique_entities = []\n",
    "for entity in financial_entities:\n",
    "    entity_key = (entity[\"id\"], entity[\"type\"])\n",
    "    if entity_key not in seen_entities:\n",
    "        seen_entities.add(entity_key)\n",
    "        unique_entities.append(entity)\n",
    "\n",
    "financial_entities = unique_entities\n",
    "\n",
    "print(f\"  - Stocks: {len([e for e in financial_entities if e['type'] == 'Stock'])}\")\n",
    "print(f\"  - Companies: {len([e for e in financial_entities if e['type'] == 'Company'])}\")\n",
    "print(f\"  - Sectors: {len([e for e in financial_entities if e['type'] == 'Sector'])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🕸️ Step 5: Build Financial Knowledge Graph\n",
    "\n",
    "Build a knowledge graph from the extracted financial entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "temporal_query = TemporalGraphQuery()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "\n",
    "# Build knowledge graph\n",
    "financial_kg = builder.build(financial_entities, financial_relationships)\n",
    "\n",
    "# Analyze graph structure\n",
    "metrics = graph_analyzer.compute_metrics(financial_kg)\n",
    "centrality_calculator = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "# Calculate graph metrics\n",
    "centrality_result = centrality_calculator.calculate_degree_centrality(financial_kg)\n",
    "centrality_scores = centrality_result.get('centrality', {})\n",
    "communities = community_detector.detect_communities(financial_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(financial_kg)\n",
    "\n",
    "print(f\"  Entities: {len(financial_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(financial_kg.get('relationships', []))}\")\n",
    "print(f\"  Graph density: {metrics.get('density', 0):.3f}\")\n",
    "print(f\"  Communities detected: {len(communities)}\")\n",
    "print(f\"  Central entities: {len([e for e, score in centrality_scores.items() if score > 0])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📊 Step 6: Analyze Financial Trends\n",
    "\n",
    "Analyze financial trends using temporal queries and pattern detection.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Temporal analysis\n",
    "start_time = (datetime.now() - timedelta(days=7)).isoformat()\n",
    "end_time = datetime.now().isoformat()\n",
    "\n",
    "temporal_results = temporal_query.query_time_range(\n",
    "    graph=financial_kg,\n",
    "    query=\"Find stock price movements\",\n",
    "    start_time=start_time,\n",
    "    end_time=end_time\n",
    ")\n",
    "\n",
    "# Inference engine for financial rules\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Financial analysis rules\n",
    "inference_engine.add_rule(\"IF change_percent > 2 AND volume > 40000000 THEN strong_momentum\")\n",
    "inference_engine.add_rule(\"IF change_percent < -1 AND volume > 50000000 THEN selling_pressure\")\n",
    "inference_engine.add_rule(\"IF change_percent > 0 AND sector == 'Technology' THEN tech_growth\")\n",
    "\n",
    "# Add facts from stock data\n",
    "for stock in stock_prices:\n",
    "    if isinstance(stock, dict):\n",
    "        inference_engine.add_fact({\n",
    "            \"symbol\": stock.get(\"symbol\", \"\"),\n",
    "            \"change_percent\": stock.get(\"change_percent\", 0),\n",
    "            \"volume\": stock.get(\"volume\", 0),\n",
    "            \"sector\": stock.get(\"sector\", \"\")\n",
    "        })\n",
    "\n",
    "# Generate insights\n",
    "financial_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"  Temporal entities: {len(temporal_results.get('entities', []))}\")\n",
    "print(f\"  Financial insights: {len(financial_insights)}\")\n",
    "\n",
    "# Display insights\n",
    "for insight in financial_insights[:3]:\n",
    "    print(f\"  - {insight}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 📤 Step 7: Export and Visualize\n",
    "\n",
    "Export the financial knowledge graph and generate visualizations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tempfile\n",
    "import os\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "# Export knowledge graph\n",
    "json_exporter.export_knowledge_graph(financial_kg, os.path.join(temp_dir, \"financial_kg.json\"))\n",
    "csv_exporter.export_entities(financial_entities, os.path.join(temp_dir, \"financial_entities.csv\"))\n",
    "rdf_exporter.export_knowledge_graph(financial_kg, os.path.join(temp_dir, \"financial_kg.rdf\"))\n",
    "\n",
    "# Generate report\n",
    "report_data = {\n",
    "    \"summary\": f\"Financial data integration from MCP server identified {len(financial_insights)} insights\",\n",
    "    \"stocks_analyzed\": len([e for e in financial_entities if e['type'] == 'Stock']),\n",
    "    \"companies\": len([e for e in financial_entities if e['type'] == 'Company']),\n",
    "    \"sectors\": len([e for e in financial_entities if e['type'] == 'Sector']),\n",
    "    \"insights\": len(financial_insights)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"  JSON: {os.path.join(temp_dir, 'financial_kg.json')}\")\n",
    "print(f\"  CSV: {os.path.join(temp_dir, 'financial_entities.csv')}\")\n",
    "print(f\"  RDF: {os.path.join(temp_dir, 'financial_kg.rdf')}\")\n",
    "\n",
    "# Visualize\n",
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(financial_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(financial_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(financial_kg, output=\"interactive\")\n",
    "\n",
    "\n",
    "# Cleanup: Disconnect from MCP server\n",
    "mcp_ingestor.disconnect(\"financial_server\")\n",
    "print(\"  Disconnected from MCP server\")\n",
    "\n",
    "print(f\"📊 Total modules used: 20+\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}