{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/trading/06_Strategy_Backtesting.ipynb)\n",
    "\n",
    "# Strategy Backtesting Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete strategy backtesting pipeline: ingest historical market data from multiple sources (databases, market data APIs, historical feeds), build temporal knowledge graph, test trading strategies on historical data, and analyze performance metrics.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: JSONParser, CSVParser, StructuredDataParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
    "- **KG**: GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer\n",
    "- **Analytics**: CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor\n",
    "- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Historical Data → Parse → Extract Entities → Build Temporal KG → Test Strategies → Analyze Performance → Generate Reports → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Historical Market Data\n",
    "\n",
    "Ingest historical market data from databases, market data APIs, and historical feeds.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import DBIngestor, FileIngestor, WebIngestor, FeedIngestor\n",
    "from semantica.parse import JSONParser, CSVParser, StructuredDataParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, TemporalPatternDetector, GraphAnalyzer\n",
    "from semantica.kg import CentralityCalculator, CommunityDetector, ConnectivityAnalyzer\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.export import JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "db_ingestor = DBIngestor()\n",
    "file_ingestor = FileIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "json_parser = JSONParser()\n",
    "csv_parser = CSVParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "\n",
    "# Real historical market data APIs\n",
    "historical_market_apis = [\n",
    "    \"https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/day/2023-01-01/2024-01-01\",  # Polygon.io historical\n",
    "    \"https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=AAPL&apikey=demo\",  # Alpha Vantage historical\n",
    "    \"https://api.github.com/repos/ranaroussi/yfinance\"  # Yahoo Finance historical data\n",
    "]\n",
    "\n",
    "# Real financial news feeds for historical context\n",
    "historical_feeds = [\n",
    "    \"https://feeds.reuters.com/reuters/businessNews\",\n",
    "    \"https://rss.cnn.com/rss/money_latest.rss\",\n",
    "    \"https://feeds.bloomberg.com/markets/news.rss\"\n",
    "]\n",
    "\n",
    "# Real database connection for historical market data\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/historical_market_db\"\n",
    "db_query = \"SELECT symbol, date, open, high, low, close, volume FROM historical_prices WHERE date >= '2023-01-01' AND date <= '2024-01-01' ORDER BY date DESC\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample historical market data (simulating real historical data structure)\n",
    "historical_data_file = os.path.join(temp_dir, \"historical_data.json\")\n",
    "historical_data = [\n",
    "    {\"symbol\": \"AAPL\", \"date\": \"2023-01-15\", \"open\": 150.00, \"high\": 152.00, \"low\": 149.50, \"close\": 151.50, \"volume\": 50000000},\n",
    "    {\"symbol\": \"AAPL\", \"date\": \"2023-01-16\", \"open\": 151.50, \"high\": 153.00, \"low\": 151.00, \"close\": 152.75, \"volume\": 52000000},\n",
    "    {\"symbol\": \"MSFT\", \"date\": \"2023-01-15\", \"open\": 350.00, \"high\": 352.00, \"low\": 349.50, \"close\": 351.25, \"volume\": 30000000},\n",
    "    {\"symbol\": \"MSFT\", \"date\": \"2023-01-16\", \"open\": 351.25, \"high\": 353.50, \"low\": 350.75, \"close\": 352.50, \"volume\": 31000000}\n",
    "]\n",
    "\n",
    "with open(historical_data_file, 'w') as f,\n",
    "    json.dump(historical_data, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(historical_data_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_data(historical_data_file, data_format=\"json\")\n",
    "\n",
    "# Ingest from historical market APIs\n",
    "historical_api_list = []\n",
    "for api_url in historical_market_apis[:1]:\n",
    "    api_content = web_ingestor.ingest_url(api_url)\n",
    "    if api_content:\n",
    "        historical_api_list.append(api_content)\n",
    "        print(f\"  Ingested historical API: {api_url}\")\n",
    "\n",
    "# Ingest from historical news feeds\n",
    "historical_feed_list = []\n",
    "for feed_url in historical_feeds:\n",
    "    feed_data = feed_ingestor.ingest_feed(feed_url)\n",
    "    if feed_data:\n",
    "        historical_feed_list.append(feed_data)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "\n",
    "print(f\"\\n📊 Historical Data Ingestion Summary:\")\n",
    "print(f\"  Historical data files: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  Historical market APIs: {len(historical_api_list)}\")\n",
    "print(f\"  Historical feeds: {len(historical_feed_list)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Market Entities and Build Temporal Knowledge Graph\n",
    "\n",
    "Extract market entities from historical data and build temporal knowledge graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "semantic_analyzer = SemanticAnalyzer()\n",
    "\n",
    "historical_entities = []\n",
    "historical_relationships = []\n",
    "\n",
    "# Extract from historical data\n",
    "if parsed_data and parsed_data.get(\"data\"):\n",
    "    for entry in parsed_data.get(\"data\") if isinstance(parsed_data.get(\"data\"), list) else [parsed_data.get(\"data\")]:\n",
    "        if isinstance(entry, dict):\n",
    "            symbol = entry.get(\"symbol\", \"\")\n",
    "            date = entry.get(\"date\", \"\")\n",
    "            \n",
    "            historical_entities.append({\n",
    "                \"id\": f\"{symbol}_{date}\",\n",
    "                \"type\": \"Historical_Price\",\n",
    "                \"name\": f\"{symbol} on {date}\",\n",
    "                \"properties\": {\n",
    "                    \"symbol\": symbol,\n",
    "                    \"date\": date,\n",
    "                    \"open\": entry.get(\"open\", 0),\n",
    "                    \"high\": entry.get(\"high\", 0),\n",
    "                    \"low\": entry.get(\"low\", 0),\n",
    "                    \"close\": entry.get(\"close\", 0),\n",
    "                    \"volume\": entry.get(\"volume\", 0)\n",
    "                }\n",
    "            })\n",
    "            \n",
    "            historical_entities.append({\n",
    "                \"id\": symbol,\n",
    "                \"type\": \"Stock\",\n",
    "                \"name\": symbol,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            \n",
    "            historical_relationships.append({\n",
    "                \"source\": symbol,\n",
    "                \"target\": f\"{symbol}_{date}\",\n",
    "                \"type\": \"has_price_on\",\n",
    "                \"properties\": {\"date\": date}\n",
    "            })\n",
    "\n",
    "builder = GraphBuilder()\n",
    "temporal_query = TemporalGraphQuery()\n",
    "temporal_pattern_detector = TemporalPatternDetector()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "\n",
    "historical_kg = builder.build(historical_entities, historical_relationships)\n",
    "\n",
    "metrics = graph_analyzer.compute_metrics(historical_kg)\n",
    "\n",
    "print(f\"Extracted {len(historical_entities)} historical entities\")\n",
    "print(f\"Extracted {len(historical_relationships)} relationships\")\n",
    "print(f\"Built temporal knowledge graph with {len(historical_kg.get('entities', []))} entities\")\n",
    "print(f\"Graph density: {metrics.get('density', 0):.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Test Trading Strategies\n",
    "\n",
    "Test trading strategies on historical data using temporal analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define trading strategies\n",
    "strategies = [\n",
    "    {\n",
    "        \"name\": \"Moving Average Crossover\",\n",
    "        \"entry_rule\": \"IF close > moving_average_20 THEN buy\",\n",
    "        \"exit_rule\": \"IF close < moving_average_20 THEN sell\"\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"Momentum Strategy\",\n",
    "        \"entry_rule\": \"IF price_change > 2% AND volume > average_volume THEN buy\",\n",
    "        \"exit_rule\": \"IF price_change < -1% THEN sell\"\n",
    "    }\n",
    "]\n",
    "\n",
    "# Backtest strategies\n",
    "backtest_results = []\n",
    "for strategy in strategies:\n",
    "    trades = []\n",
    "    positions = {}\n",
    "    \n",
    "    if parsed_data and parsed_data.get(\"data\"):\n",
    "        sorted_data = sorted(parsed_data.get(\"data\") if isinstance(parsed_data.get(\"data\"), list) else [parsed_data.get(\"data\")], \n",
    "                           key=lambda x: x.get(\"date\", \"\"))\n",
    "        \n",
    "        for entry in sorted_data:\n",
    "            if isinstance(entry, dict):\n",
    "                symbol = entry.get(\"symbol\", \"\")\n",
    "                close_price = entry.get(\"close\", 0)\n",
    "                date = entry.get(\"date\", \"\")\n",
    "                \n",
    "                # Simple strategy logic (moving average simulation)\n",
    "                if symbol not in positions:\n",
    "                    # Entry signal\n",
    "                    if close_price > 150:  # Simplified entry condition\n",
    "                        positions[symbol] = {\n",
    "                            \"entry_price\": close_price,\n",
    "                            \"entry_date\": date,\n",
    "                            \"quantity\": 100\n",
    "                        }\n",
    "                else:\n",
    "                    # Exit signal\n",
    "                    if close_price > positions[symbol][\"entry_price\"] * 1.02:  # 2% profit target\n",
    "                        trades.append({\n",
    "                            \"symbol\": symbol,\n",
    "                            \"entry_price\": positions[symbol][\"entry_price\"],\n",
    "                            \"exit_price\": close_price,\n",
    "                            \"entry_date\": positions[symbol][\"entry_date\"],\n",
    "                            \"exit_date\": date,\n",
    "                            \"profit\": (close_price - positions[symbol][\"entry_price\"]) * positions[symbol][\"quantity\"],\n",
    "                            \"return_pct\": ((close_price - positions[symbol][\"entry_price\"]) / positions[symbol][\"entry_price\"]) * 100\n",
    "                        })\n",
    "                        del positions[symbol]\n",
    "    \n",
    "    total_profit = sum(t[\"profit\"] for t in trades)\n",
    "    total_return = sum(t[\"return_pct\"] for t in trades) / len(trades) if trades else 0\n",
    "    \n",
    "    backtest_results.append({\n",
    "        \"strategy\": strategy[\"name\"],\n",
    "        \"trades\": len(trades),\n",
    "        \"total_profit\": total_profit,\n",
    "        \"average_return\": total_return,\n",
    "        \"win_rate\": len([t for t in trades if t[\"profit\"] > 0]) / len(trades) if trades else 0\n",
    "    })\n",
    "\n",
    "print(f\"Backtested {len(strategies)} trading strategies\")\n",
    "for result in backtest_results:\n",
    "    print(f\"  Strategy: {result['strategy']} - Trades: {result['trades']}, Profit: ${result['total_profit']:.2f}, Avg Return: {result['average_return']:.2f}%\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Analyze Performance Metrics\n",
    "\n",
    "Analyze strategy performance using graph analytics and inference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "centrality_calculator = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Analyze graph structure\n",
    "centrality_result = centrality_calculator.calculate_degree_centrality(historical_kg)\n",
    "centrality_scores = centrality_result.get('centrality', {})\n",
    "communities = community_detector.detect_communities(historical_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(historical_kg)\n",
    "\n",
    "# Temporal pattern detection\n",
    "start_date = \"2023-01-01\"\n",
    "end_date = \"2024-01-01\"\n",
    "\n",
    "temporal_results = temporal_query.query_time_range(\n",
    "    graph=historical_kg,\n",
    "    query=\"Find price movements in backtest period\",\n",
    "    start_time=start_date,\n",
    "    end_time=end_date\n",
    ")\n",
    "\n",
    "temporal_patterns = temporal_pattern_detector.detect_temporal_patterns(\n",
    "    historical_kg,\n",
    "    pattern_type=\"trend\",\n",
    "    min_frequency=1\n",
    ")\n",
    "\n",
    "# Performance inference rules\n",
    "inference_engine.add_rule(\"IF average_return > 5% AND win_rate > 0.6 THEN profitable_strategy\")\n",
    "inference_engine.add_rule(\"IF total_profit > 1000 AND trades > 10 THEN successful_backtest\")\n",
    "\n",
    "for result in backtest_results:\n",
    "    inference_engine.add_fact({\n",
    "        \"strategy\": result[\"strategy\"],\n",
    "        \"average_return\": result[\"average_return\"],\n",
    "        \"win_rate\": result[\"win_rate\"],\n",
    "        \"total_profit\": result[\"total_profit\"],\n",
    "        \"trades\": result[\"trades\"]\n",
    "    })\n",
    "\n",
    "performance_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Performance analysis complete\")\n",
    "print(f\"  Temporal patterns: {len(temporal_patterns)}\")\n",
    "print(f\"  Central stocks: {len([e for e, score in centrality_scores.items() if score > 0])}\")\n",
    "print(f\"  Communities: {len(communities)}\")\n",
    "print(f\"  Performance insights: {len(performance_insights)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Backtest Reports and Visualize\n",
    "\n",
    "Generate comprehensive backtest reports and visualize results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(historical_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(historical_kg, os.path.join(temp_dir, \"backtest_kg.json\"))\n",
    "csv_exporter.export_entities(historical_entities, os.path.join(temp_dir, \"historical_entities.csv\"))\n",
    "rdf_exporter.export_knowledge_graph(historical_kg, os.path.join(temp_dir, \"backtest_kg.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Strategy backtesting analyzed {len(backtest_results)} strategies on {len(historical_entities)} historical data points\",\n",
    "    \"strategies_tested\": len(backtest_results),\n",
    "    \"total_trades\": sum(r[\"trades\"] for r in backtest_results),\n",
    "    \"best_strategy\": max(backtest_results, key=lambda x: x[\"total_profit\"])[\"strategy\"] if backtest_results else \"N/A\",\n",
    "    \"patterns_detected\": len(temporal_patterns),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(historical_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(historical_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(historical_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Historical Data → Parse → Extract → Build Temporal KG → Test Strategies → Analyze Performance → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
