{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/trading/02_News_Sentiment_Analysis.ipynb)\n",
    "\n",
    "# News Sentiment Analysis Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete news sentiment analysis pipeline for trading: ingest financial news from multiple sources (RSS feeds, news APIs, web sources), extract entities, build news knowledge graph, analyze sentiment using embeddings, and generate trading signals.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: HTMLParser, JSONParser, StructuredDataParser, DocumentParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
    "- **KG**: GraphBuilder, GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "- **Embeddings**: EmbeddingGenerator, TextEmbedder\n",
    "- **Analytics**: ConnectivityAnalyzer, TemporalGraphQuery, TemporalPatternDetector\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor\n",
    "- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Ingest News → Parse → Extract Entities → Build News KG → Generate Embeddings → Analyze Sentiment → Generate Trading Signals → Export → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Financial News from Multiple Sources\n",
    "\n",
    "Ingest financial news from RSS feeds, news APIs, and web sources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import WebIngestor, FeedIngestor, DBIngestor, FileIngestor\n",
    "from semantica.parse import HTMLParser, JSONParser, StructuredDataParser, DocumentParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "from semantica.kg import ConnectivityAnalyzer, TemporalGraphQuery, TemporalPatternDetector\n",
    "from semantica.embeddings import EmbeddingGenerator, TextEmbedder\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    
    "from semantica.export import JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "web_ingestor = WebIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "file_ingestor = FileIngestor()\n",
    "\n",
    "html_parser = HTMLParser()\n",
    "json_parser = JSONParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "document_parser = DocumentParser()\n",
    "\n",
    "# Real financial news feed URLs\n",
    "financial_feeds = [\n",
    "    \"https://feeds.reuters.com/reuters/businessNews\",  # Reuters Business\n",
    "    \"https://feeds.reuters.com/reuters/topNews\",  # Reuters Top News\n",
    "    \"https://rss.cnn.com/rss/money_latest.rss\",  # CNN Money\n",
    "    \"https://feeds.bloomberg.com/markets/news.rss\",  # Bloomberg Markets\n",
    "    \"https://www.ft.com/?format=rss\"  # Financial Times\n",
    "]\n",
    "\n",
    "# Real news API endpoints\n",
    "news_apis = [\n",
    "    \"https://newsapi.org/v2/everything?q=finance&apiKey=demo\",  # NewsAPI (requires API key)\n",
    "    \"https://api.github.com/repos/financial-news/aggregator\"  # Financial news aggregator\n",
    "]\n",
    "\n",
    "# Real database connection for news data\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/news_db\"\n",
    "db_query = \"SELECT article_id, title, content, sentiment, published_date FROM financial_news WHERE published_date > NOW() - INTERVAL '24 hours' ORDER BY published_date DESC\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample financial news data\n",
    "news_file = os.path.join(temp_dir, \"financial_news.json\")\n",
    "news_data = {\n",
    "    \"articles\": [\n",
    "        {\n",
    "            \"title\": \"Apple Reports Strong Q4 Earnings\",\n",
    "            \"content\": \"Apple Inc. reported strong fourth quarter earnings, beating analyst expectations with record revenue.\",\n",
    "            \"sentiment\": \"positive\",\n",
    "            \"published_date\": (datetime.now() - timedelta(hours=2)).isoformat(),\n",
    "            \"symbols\": [\"AAPL\"]\n",
    "        },\n",
    "        {\n",
    "            \"title\": \"Market Volatility Concerns Rise\",\n",
    "            \"content\": \"Financial markets show increased volatility amid economic uncertainty and geopolitical tensions.\",\n",
    "            \"sentiment\": \"negative\",\n",
    "            \"published_date\": (datetime.now() - timedelta(hours=1)).isoformat(),\n",
    "            \"symbols\": [\"SPY\", \"QQQ\"]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "with open(news_file, 'w') as f:\n",
    "    json.dump(news_data, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(news_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_data(news_file, data_format=\"json\")\n",
    "\n",
    "# Ingest from financial feeds\n",
    "financial_feed_list = []\n",
    "for feed_url in financial_feeds[:3]:  # Process first 3 feeds\n",
    "    feed_data = feed_ingestor.ingest_feed(feed_url)\n",
    "    if feed_data:\n",
    "        financial_feed_list.append(feed_data)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "        print(f\"  Items: {len(feed_data.items) if hasattr(feed_data, 'items') else 0}\")\n",
    "\n",
    "print(f\"\\n📊 Ingestion Summary:\")\n",
    "print(f\"  News files: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  Financial feeds: {len(financial_feed_list)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract News Entities and Build Knowledge Graph\n",
    "\n",
    "Extract entities from news articles and build knowledge graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "semantic_analyzer = SemanticAnalyzer()\n",
    "\n",
    "news_entities = []\n",
    "news_relationships = []\n",
    "all_news_texts = []\n",
    "\n",
    "# Extract from news data\n",
    "if parsed_data and parsed_data.get(\"data\"):\n",
    "    articles = parsed_data.get(\"data\").get(\"articles\", []) if isinstance(parsed_data.get(\"data\"), dict) else parsed_data.get(\"data\") if isinstance(parsed_data.get(\"data\"), list) else []\n",
    "    \n",
    "    for article in articles:\n",
    "        if isinstance(article, dict):\n",
    "            article_text = f\"{article.get('title', '')} {article.get('content', '')}\"\n",
    "            all_news_texts.append(article_text)\n",
    "            \n",
    "            news_entities.append({\n",
    "                \"id\": article.get(\"title\", \"\"),\n",
    "                \"type\": \"News_Article\",\n",
    "                \"name\": article.get(\"title\", \"\"),\n",
    "                \"properties\": {\n",
    "                    \"sentiment\": article.get(\"sentiment\", \"\"),\n",
    "                    \"published_date\": article.get(\"published_date\", \"\")\n",
    "                }\n",
    "            })\n",
    "            \n",
    "            # Symbols mentioned\n",
    "            for symbol in article.get(\"symbols\", []):\n",
    "                news_entities.append({\n",
    "                    \"id\": symbol,\n",
    "                    \"type\": \"Stock\",\n",
    "                    \"name\": symbol,\n",
    "                    \"properties\": {}\n",
    "                })\n",
    "                news_relationships.append({\n",
    "                    \"source\": article.get(\"title\", \"\"),\n",
    "                    \"target\": symbol,\n",
    "                    \"type\": \"mentions\",\n",
    "                    \"properties\": {\n",
    "                        \"sentiment\": article.get(\"sentiment\", \"\")\n",
    "                    }\n",
    "                })\n",
    "\n",
    "builder = GraphBuilder()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "centrality_calculator = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "\n",
    "news_kg = builder.build(news_entities, news_relationships)\n",
    "\n",
    "metrics = graph_analyzer.compute_metrics(news_kg)\n",
    "centrality_result = centrality_calculator.calculate_degree_centrality(news_kg)\n",
    "centrality_scores = centrality_result.get('centrality', {})\n",
    "communities = community_detector.detect_communities(news_kg)\n",
    "\n",
    "print(f\"Extracted {len(news_entities)} news entities\")\n",
    "print(f\"Extracted {len(news_relationships)} relationships\")\n",
    "print(f\"Collected {len(all_news_texts)} news articles\")\n",
    "print(f\"Built news knowledge graph with {len(news_kg.get('entities', []))} entities\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Generate Embeddings and Analyze Sentiment\n",
    "\n",
    "Generate embeddings from news articles and analyze sentiment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_generator = EmbeddingGenerator()\n",
    "text_embedder = TextEmbedder()\n",
    "\n",
    "embeddings = embedding_generator.generate_embeddings(all_news_texts, data_type=\"text\")\n",
    "\n",
    "# Analyze sentiment from embeddings and article properties\n",
    "sentiment_scores = []\n",
    "for i, article in enumerate(parsed_data.get(\"data\").get(\"articles\", []) if parsed_data and parsed_data.get(\"data\") and isinstance(parsed_data.get(\"data\"), dict) else []):\n",
    "    if isinstance(article, dict):\n",
    "        sentiment = article.get(\"sentiment\", \"neutral\")\n",
    "        sentiment_value = 1.0 if sentiment == \"positive\" else -1.0 if sentiment == \"negative\" else 0.0\n",
    "        \n",
    "        sentiment_scores.append({\n",
    "            \"article\": article.get(\"title\", \"\"),\n",
    "            \"sentiment\": sentiment,\n",
    "            \"score\": sentiment_value,\n",
    "            \"symbols\": article.get(\"symbols\", [])\n",
    "        })\n",
    "\n",
    "print(f\"Generated embeddings for {len(all_news_texts)} news articles\")\n",
    "print(f\"Analyzed sentiment for {len(sentiment_scores)} articles\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Generate Trading Signals\n",
    "\n",
    "Generate trading signals based on sentiment analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "temporal_query = TemporalGraphQuery()\n",
    "temporal_pattern_detector = TemporalPatternDetector()\n",
    "\n",
    "# Trading signal generation rules\n",
    "inference_engine.add_rule(\"IF sentiment is positive AND multiple articles mention symbol THEN buy_signal\")\n",
    "inference_engine.add_rule(\"IF sentiment is negative AND multiple articles mention symbol THEN sell_signal\")\n",
    "inference_engine.add_rule(\"IF sentiment score > 0.5 THEN strong_positive_signal\")\n",
    "\n",
    "# Generate trading signals\n",
    "trading_signals = []\n",
    "for sentiment_data in sentiment_scores:\n",
    "    symbol_sentiment = {}\n",
    "    for symbol in sentiment_data.get(\"symbols\", []):\n",
    "        if symbol not in symbol_sentiment:\n",
    "            symbol_sentiment[symbol] = []\n",
    "        symbol_sentiment[symbol].append(sentiment_data.get(\"score\", 0))\n",
    "    \n",
    "    for symbol, scores in symbol_sentiment.items():\n",
    "        avg_sentiment = sum(scores) / len(scores) if scores else 0\n",
    "        signal_type = \"buy\" if avg_sentiment > 0.3 else \"sell\" if avg_sentiment < -0.3 else \"hold\"\n",
    "        \n",
    "        trading_signals.append({\n",
    "            \"symbol\": symbol,\n",
    "            \"signal\": signal_type,\n",
    "            \"sentiment_score\": avg_sentiment,\n",
    "            \"confidence\": abs(avg_sentiment),\n",
    "            \"timestamp\": datetime.now().isoformat()\n",
    "        })\n",
    "        \n",
    "        inference_engine.add_fact({\n",
    "            \"symbol\": symbol,\n",
    "            \"sentiment\": sentiment_data.get(\"sentiment\", \"\"),\n",
    "            \"score\": avg_sentiment\n",
    "        })\n",
    "\n",
    "signal_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Generated {len(trading_signals)} trading signals\")\n",
    "print(f\"Inferred {len(signal_insights)} signal patterns\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Reports and Visualize\n",
    "\n",
    "Generate sentiment analysis reports and visualize results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(news_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(news_kg, os.path.join(temp_dir, \"news_kg.json\"))\n",
    "csv_exporter.export_entities(news_entities, os.path.join(temp_dir, \"news_entities.csv\"))\n",
    "rdf_exporter.export_knowledge_graph(news_kg, os.path.join(temp_dir, \"news_kg.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"News sentiment analysis identified {len(trading_signals)} trading signals from {len(news_entities)} entities\",\n",
    "    \"articles_analyzed\": len([e for e in news_entities if e.get(\"type\") == \"News_Article\"]),\n",
    "    \"signals\": len(trading_signals),\n",
    "    \"buy_signals\": len([s for s in trading_signals if s.get(\"signal\") == \"buy\"]),\n",
    "    \"sell_signals\": len([s for s in trading_signals if s.get(\"signal\") == \"sell\"]),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(news_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(news_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(news_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Ingest News → Parse → Extract → Build KG → Embeddings → Sentiment Analysis → Trading Signals → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
