{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/04_Investment_Analysis_Hybrid_RAG.ipynb)\n",
    "\n",
    "# Investment Analysis Hybrid RAG Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete investment analysis hybrid RAG pipeline: ingest investment data from multiple sources (market data APIs, financial feeds, databases), extract investment entities, build knowledge graph, generate embeddings, set up hybrid search (vector + temporal KG), and query investment insights using advanced RAG.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: JSONParser, CSVParser, StructuredDataParser, HTMLParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
    "- **KG**: GraphBuilder, TemporalGraphQuery, GraphAnalyzer, ConnectivityAnalyzer\n",
    "- **Embeddings**: EmbeddingGenerator, TextEmbedder\n",
    "- **Vector Store**: VectorStore, HybridSearch\n",
    "- **Context**: ContextRetriever, ContextGraphBuilder\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Multi-Source Investment Data → Parse → Extract Entities → Build KG → Generate Embeddings → Vector Store → Hybrid RAG Setup → Query Insights → Generate Reports → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Multi-Source Investment Data Ingestion\n",
    "\n",
    "Ingest investment data from market APIs, financial feeds, and databases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, FeedIngestor\n",
    "from semantica.parse import JSONParser, CSVParser, StructuredDataParser, HTMLParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, EventDetector, SemanticAnalyzer\n",
    "from semantica.kg import GraphBuilder, TemporalGraphQuery, GraphAnalyzer, ConnectivityAnalyzer\n",
    "from semantica.embeddings import EmbeddingGenerator, TextEmbedder\n",
    "from semantica.vector_store import VectorStore, HybridSearch\n",
    "from semantica.context import ContextRetriever, ContextGraphBuilder\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.export import JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, TemporalVisualizer, AnalyticsVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "json_parser = JSONParser()\n",
    "csv_parser = CSVParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "html_parser = HTMLParser()\n",
    "\n",
    "# Real investment data sources\n",
    "investment_apis = [\n",
    "    \"https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/day/2024-01-01/2024-01-31\",  # Polygon.io\n",
    "    \"https://www.alphavantage.co/query?function=OVERVIEW&symbol=AAPL&apikey=demo\",  # Alpha Vantage\n",
    "    \"https://api.github.com/repos/ranaroussi/yfinance\"  # Yahoo Finance API\n",
    "]\n",
    "\n",
    "financial_feeds = [\n",
    "    \"https://feeds.reuters.com/reuters/businessNews\",\n",
    "    \"https://feeds.reuters.com/reuters/topNews\",\n",
    "    \"https://rss.cnn.com/rss/money_latest.rss\",\n",
    "    \"https://feeds.bloomberg.com/markets/news.rss\"\n",
    "]\n",
    "\n",
    "# Real database connection for investment data\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/investment_db\"\n",
    "db_query = \"SELECT symbol, company_name, sector, market_cap, pe_ratio, dividend_yield FROM investments WHERE last_updated > NOW() - INTERVAL '7 days' ORDER BY market_cap DESC\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample investment data\n",
    "investment_data_file = os.path.join(temp_dir, \"investment_data.json\")\n",
    "investment_data = [\n",
    "    {\n",
    "        \"symbol\": \"AAPL\",\n",
    "        \"company\": \"Apple Inc.\",\n",
    "        \"sector\": \"Technology\",\n",
    "        \"market_cap\": 2800000000000,\n",
    "        \"pe_ratio\": 28.5,\n",
    "        \"dividend_yield\": 0.5,\n",
    "        \"price\": 175.50,\n",
    "        \"timestamp\": (datetime.now() - timedelta(days=1)).isoformat()\n",
    "    },\n",
    "    {\n",
    "        \"symbol\": \"MSFT\",\n",
    "        \"company\": \"Microsoft Corporation\",\n",
    "        \"sector\": \"Technology\",\n",
    "        \"market_cap\": 2800000000000,\n",
    "        \"pe_ratio\": 32.1,\n",
    "        \"dividend_yield\": 0.7,\n",
    "        \"price\": 380.25,\n",
    "        \"timestamp\": (datetime.now() - timedelta(days=1)).isoformat()\n",
    "    }\n",
    "]\n",
    "\n",
    "with open(investment_data_file, 'w') as f:\n",
    "    json.dump(investment_data, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(investment_data_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_json(investment_data_file)\n",
    "\n",
    "# Ingest from financial feeds\n",
    "financial_feed_list = []\n",
    "for feed_url in financial_feeds[:2]:\n",
    "    feed_data = feed_ingestor.ingest_feed(feed_url)\n",
    "    if feed_data:\n",
    "        financial_feed_list.append(feed_data)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "\n",
    "# Ingest from investment APIs\n",
    "api_content_list = []\n",
    "for api_url in investment_apis[:1]:\n",
    "    api_content = web_ingestor.ingest_url(api_url)\n",
    "    if api_content:\n",
    "        api_content_list.append(api_content)\n",
    "        print(f\"  Ingested API: {api_url}\")\n",
    "\n",
    "print(f\"\\n📊 Ingestion Summary:\")\n",
    "print(f\"  Investment data files: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  Financial feeds: {len(financial_feed_list)}\")\n",
    "print(f\"  Investment APIs: {len(api_content_list)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Investment Entities and Build Knowledge Graph\n",
    "\n",
    "Extract investment entities and build knowledge graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "event_detector = EventDetector()\n",
    "semantic_analyzer = SemanticAnalyzer()\n",
    "\n",
    "investment_entities = []\n",
    "investment_relationships = []\n",
    "all_documents = []\n",
    "\n",
    "# Extract from investment data\n",
    "if parsed_data and parsed_data.data:\n",
    "    for investment in parsed_data.data if isinstance(parsed_data.data, list) else [parsed_data.data]:\n",
    "        if isinstance(investment, dict):\n",
    "            investment_text = f\"{investment.get('company', '')} ({investment.get('symbol', '')}) in {investment.get('sector', '')} sector\"\n",
    "            all_documents.append(investment_text)\n",
    "            \n",
    "            investment_entities.append({\n",
    "                \"id\": investment.get(\"symbol\", \"\"),\n",
    "                \"type\": \"Stock\",\n",
    "                \"name\": investment.get(\"company\", \"\"),\n",
    "                \"properties\": {\n",
    "                    \"symbol\": investment.get(\"symbol\", \"\"),\n",
    "                    \"sector\": investment.get(\"sector\", \"\"),\n",
    "                    \"market_cap\": investment.get(\"market_cap\", 0),\n",
    "                    \"pe_ratio\": investment.get(\"pe_ratio\", 0),\n",
    "                    \"dividend_yield\": investment.get(\"dividend_yield\", 0),\n",
    "                    \"price\": investment.get(\"price\", 0),\n",
    "                    \"timestamp\": investment.get(\"timestamp\", \"\")\n",
    "                }\n",
    "            })\n",
    "            \n",
    "            investment_entities.append({\n",
    "                \"id\": investment.get(\"sector\", \"\"),\n",
    "                \"type\": \"Sector\",\n",
    "                \"name\": investment.get(\"sector\", \"\"),\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            \n",
    "            investment_relationships.append({\n",
    "                \"source\": investment.get(\"symbol\", \"\"),\n",
    "                \"target\": investment.get(\"sector\", \"\"),\n",
    "                \"type\": \"belongs_to\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "\n",
    "builder = GraphBuilder()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "investment_kg = builder.build(investment_entities, investment_relationships)\n",
    "\n",
    "metrics = graph_analyzer.compute_metrics(investment_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(investment_kg)\n",
    "\n",
    "print(f\"Extracted {len(investment_entities)} investment entities\")\n",
    "print(f\"Extracted {len(investment_relationships)} relationships\")\n",
    "print(f\"Collected {len(all_documents)} investment documents\")\n",
    "print(f\"Built investment knowledge graph with {len(investment_kg.get('entities', []))} entities\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Generate Embeddings and Setup Vector Store\n",
    "\n",
    "Generate embeddings and setup vector store for hybrid RAG.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_generator = EmbeddingGenerator()\n",
    "text_embedder = TextEmbedder()\n",
    "vector_store = VectorStore()\n",
    "hybrid_search = HybridSearch()\n",
    "\n",
    "embeddings = embedding_generator.generate_embeddings(all_documents, data_type=\"text\")\n",
    "\n",
    "metadata = []\n",
    "for i, doc in enumerate(all_documents):\n",
    "    metadata.append({\n",
    "        \"id\": f\"doc_{i}\",\n",
    "        \"text\": doc,\n",
    "        \"source\": \"investment_data\"\n",
    "    })\n",
    "\n",
    "vector_ids = vector_store.store_vectors(embeddings, metadata)\n",
    "\n",
    "print(f\"Generated embeddings for {len(all_documents)} documents\")\n",
    "print(f\"Stored {len(vector_ids)} vectors in vector store\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Setup Hybrid RAG and Query Investment Insights\n",
    "\n",
    "Setup hybrid search and query investment insights.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "context_retriever = ContextRetriever(\n",
    "    knowledge_graph=investment_kg,\n",
    "    vector_store=vector_store\n",
    ")\n",
    "\n",
    "temporal_query = TemporalGraphQuery()\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Query examples\n",
    "queries = [\n",
    "    \"What are the best performing sectors?\",\n",
    "    \"Find technology stocks with high market cap\",\n",
    "    \"What investments have good dividend yields?\"\n",
    "]\n",
    "\n",
    "query_results = []\n",
    "for query in queries:\n",
    "    query_embedding = text_embedder.embed_text(query)\n",
    "    vector_results = vector_store.search_vectors(query_embedding, k=3)\n",
    "    \n",
    "    start_time = (datetime.now() - timedelta(days=30)).isoformat()\n",
    "    end_time = datetime.now().isoformat()\n",
    "    \n",
    "    temporal_results = temporal_query.query_time_range(\n",
    "        graph=investment_kg,\n",
    "        query=query,\n",
    "        start_time=start_time,\n",
    "        end_time=end_time\n",
    "    )\n",
    "    \n",
    "    context_results = context_retriever.retrieve(\n",
    "        query=query,\n",
    "        top_k=3,\n",
    "        use_graph_expansion=True\n",
    "    )\n",
    "    \n",
    "    query_results.append({\n",
    "        \"query\": query,\n",
    "        \"vector_results\": len(vector_results),\n",
    "        \"temporal_results\": len(temporal_results.get('entities', [])),\n",
    "        \"context_results\": len(context_results) if context_results else 0\n",
    "    })\n",
    "\n",
    "# Investment analysis rules\n",
    "inference_engine.add_rule(\"IF pe_ratio < 20 AND dividend_yield > 0.5 THEN value_stock\")\n",
    "inference_engine.add_rule(\"IF market_cap > 1000000000000 AND sector is Technology THEN mega_cap_tech\")\n",
    "\n",
    "for investment in parsed_data.data if parsed_data and parsed_data.data else []:\n",
    "    if isinstance(investment, dict):\n",
    "        inference_engine.add_fact({\n",
    "            \"symbol\": investment.get(\"symbol\", \"\"),\n",
    "            \"pe_ratio\": investment.get(\"pe_ratio\", 0),\n",
    "            \"dividend_yield\": investment.get(\"dividend_yield\", 0),\n",
    "            \"market_cap\": investment.get(\"market_cap\", 0),\n",
    "            \"sector\": investment.get(\"sector\", \"\")\n",
    "        })\n",
    "\n",
    "investment_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Processed {len(queries)} investment queries\")\n",
    "for result in query_results:\n",
    "    print(f\"  Query: '{result['query']}' - Vector: {result['vector_results']}, Temporal: {result['temporal_results']}, Context: {result['context_results']}\")\n",
    "print(f\"Generated {len(investment_insights)} investment insights\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Reports and Visualize\n",
    "\n",
    "Generate investment analysis reports and visualize results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(investment_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(investment_kg, os.path.join(temp_dir, \"investment_kg.json\"))\n",
    "csv_exporter.export_entities(investment_entities, os.path.join(temp_dir, \"investment_entities.csv\"))\n",
    "rdf_exporter.export_knowledge_graph(investment_kg, os.path.join(temp_dir, \"investment_kg.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Investment analysis identified {len(investment_insights)} insights from {len(investment_entities)} entities\",\n",
    "    \"investments_analyzed\": len(parsed_data.data) if parsed_data and parsed_data.data else 0,\n",
    "    \"insights\": len(investment_insights),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "kg_visualizer = KGVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(investment_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(investment_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(investment_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Multi-Source Investment Data → Parse → Extract → Build KG → Embeddings → Vector Store → Hybrid RAG → Query → Reports → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
