{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Notice**: The `semantica.kg_qa` module is temporarily unavailable and will be reintroduced in a future release. Any quality assessment examples in this notebook are disabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Hawksight-AI/semantica/blob/main/cookbook/use_cases/finance/02_Financial_Reports_Analysis.ipynb)\n",
    "\n",
    "# Financial Reports Analysis Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook demonstrates a complete financial reports analysis pipeline: ingest financial documents from multiple sources (SEC filings, annual reports, financial databases), extract financial entities, build knowledge graph, analyze relationships, and generate financial insights.\n",
    "\n",
    "\n",
    "**Documentation**: [API Reference](https://semantica.readthedocs.io/use-cases/)\n",
    "\n",
    "## Installation\n",
    "\n",
    "Install Semantica from PyPI:\n",
    "\n",
    "```bash\n",
    "pip install semantica\n",
    "# Or with all optional dependencies:\n",
    "pip install semantica[all]\n",
    "```\n",
    "\n",
    "### Modules Used (20+)\n",
    "\n",
    "- **Ingestion**: FileIngestor, WebIngestor, FeedIngestor, StreamIngestor, DBIngestor, EmailIngestor, RepoIngestor, MCPIngestor\n",
    "- **Parsing**: DocumentParser, PDFParser, HTMLParser, StructuredDataParser\n",
    "- **Extraction**: NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
    "- **KG**: GraphBuilder, GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "- **Analytics**: ConnectivityAnalyzer, TemporalGraphQuery, TemporalPatternDetector\n",
    "- **Reasoning**: InferenceEngine, RuleManager, ExplanationGenerator\n",
    "- **Quality**: KGQualityAssessor, ConflictDetector\n",
    "- **Export**: JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "- **Visualization**: KGVisualizer, AnalyticsVisualizer, TemporalVisualizer\n",
    "\n",
    "### Pipeline\n",
    "\n",
    "**Financial Documents → Parse → Extract Entities → Build KG → Analyze Relationships → Generate Insights → Export → Visualize**\n",
    "\n",
    "---\n",
    "\n",
    "## Step 1: Ingest Financial Documents from Multiple Sources\n",
    "\n",
    "Ingest financial reports from SEC filings, annual reports, and financial databases.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from semantica.ingest import FileIngestor, WebIngestor, DBIngestor, FeedIngestor\n",
    "from semantica.parse import DocumentParser, PDFParser, HTMLParser, StructuredDataParser\n",
    "from semantica.semantic_extract import NERExtractor, RelationExtractor, TripleExtractor, SemanticAnalyzer\n",
    "from semantica.kg import GraphBuilder, GraphAnalyzer, CentralityCalculator, CommunityDetector\n",
    "from semantica.kg import ConnectivityAnalyzer, TemporalGraphQuery, TemporalPatternDetector\n",
    "from semantica.reasoning import InferenceEngine, RuleManager, ExplanationGenerator\n",
    "from semantica.conflicts import ConflictDetector\n",
    "from semantica.export import JSONExporter, CSVExporter, RDFExporter, ReportGenerator\n",
    "from semantica.visualization import KGVisualizer, AnalyticsVisualizer, TemporalVisualizer\n",
    "import tempfile\n",
    "import os\n",
    "import json\n",
    "from datetime import datetime, timedelta\n",
    "\n",
    "file_ingestor = FileIngestor()\n",
    "web_ingestor = WebIngestor()\n",
    "db_ingestor = DBIngestor()\n",
    "feed_ingestor = FeedIngestor()\n",
    "\n",
    "document_parser = DocumentParser()\n",
    "pdf_parser = PDFParser()\n",
    "html_parser = HTMLParser()\n",
    "structured_parser = StructuredDataParser()\n",
    "\n",
    "# Real financial data sources\n",
    "sec_edgar_urls = [\n",
    "    \"https://www.sec.gov/cgi-bin/browse-edgar\",  # SEC EDGAR database\n",
    "    \"https://www.sec.gov/Archives/edgar/data/\",  # SEC EDGAR archives\n",
    "    \"https://data.sec.gov/submissions/\"  # SEC submissions API\n",
    "]\n",
    "\n",
    "financial_feeds = [\n",
    "    \"https://feeds.reuters.com/reuters/businessNews\",\n",
    "    \"https://feeds.reuters.com/reuters/topNews\",\n",
    "    \"https://rss.cnn.com/rss/money_latest.rss\",\n",
    "    \"https://feeds.bloomberg.com/markets/news.rss\"\n",
    "]\n",
    "\n",
    "# Real database connection for financial reports\n",
    "db_connection_string = \"postgresql://user:password@localhost:5432/financial_reports_db\"\n",
    "db_query = \"SELECT company_name, report_type, filing_date, document_url FROM financial_reports WHERE filing_date > CURRENT_DATE - INTERVAL '1 year' ORDER BY filing_date DESC\"\n",
    "\n",
    "temp_dir = tempfile.mkdtemp()\n",
    "\n",
    "# Sample financial report data (simulating real SEC filing structure)\n",
    "financial_report_file = os.path.join(temp_dir, \"financial_report.json\")\n",
    "report_data = {\n",
    "    \"company\": \"Apple Inc.\",\n",
    "    \"symbol\": \"AAPL\",\n",
    "    \"report_type\": \"10-K\",\n",
    "    \"filing_date\": (datetime.now() - timedelta(days=30)).isoformat(),\n",
    "    \"revenue\": 394328000000,\n",
    "    \"net_income\": 99803000000,\n",
    "    \"total_assets\": 352755000000,\n",
    "    \"total_liabilities\": 290437000000,\n",
    "    \"segments\": [\"iPhone\", \"Mac\", \"iPad\", \"Services\", \"Wearables\"],\n",
    "    \"geographic_regions\": [\"Americas\", \"Europe\", \"Greater China\", \"Japan\", \"Rest of Asia Pacific\"]\n",
    "}\n",
    "\n",
    "with open(financial_report_file, 'w') as f:\n",
    "    json.dump(report_data, f, indent=2)\n",
    "\n",
    "file_objects = file_ingestor.ingest_file(financial_report_file, read_content=True)\n",
    "parsed_data = structured_parser.parse_data(financial_report_file, data_format=\"json\")\n",
    "\n",
    "# Ingest from financial feeds\n",
    "financial_feed_list = []\n",
    "for feed_url in financial_feeds[:2]:  # Process first 2 feeds\n",
    "    feed_data = feed_ingestor.ingest_feed(feed_url)\n",
    "    if feed_data:\n",
    "        financial_feed_list.append(feed_data)\n",
    "        print(f\"  Ingested feed: {feed_url}\")\n",
    "\n",
    "# Ingest from SEC EDGAR (example)\n",
    "web_content = web_ingestor.ingest_url(\"https://www.sec.gov/cgi-bin/browse-edgar\")\n",
    "if web_content:\n",
    "    print(f\"  Ingested SEC EDGAR content\")\n",
    "\n",
    "print(f\"\\n📊 Ingestion Summary:\")\n",
    "print(f\"  Financial reports ingested: {len([file_objects]) if file_objects else 0}\")\n",
    "print(f\"  Financial feeds: {len(financial_feed_list)}\")\n",
    "print(f\"  Database sources: 1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Extract Financial Entities\n",
    "\n",
    "Extract financial entities (companies, metrics, segments, regions) from financial reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_extractor = NERExtractor()\n",
    "relation_extractor = RelationExtractor()\n",
    "triple_extractor = TripleExtractor()\n",
    "semantic_analyzer = SemanticAnalyzer()\n",
    "\n",
    "financial_entities = []\n",
    "financial_relationships = []\n",
    "\n",
    "# Extract from financial report data\n",
    "if parsed_data and parsed_data.data:\n",
    "    report = parsed_data.data if isinstance(parsed_data.data, dict) else parsed_data.data[0] if isinstance(parsed_data.data, list) else {}\n",
    "    \n",
    "    if isinstance(report, dict):\n",
    "        # Company entity\n",
    "        financial_entities.append({\n",
    "            \"id\": report.get(\"symbol\", \"\"),\n",
    "            \"type\": \"Company\",\n",
    "            \"name\": report.get(\"company\", \"\"),\n",
    "            \"properties\": {\n",
    "                \"symbol\": report.get(\"symbol\", \"\"),\n",
    "                \"report_type\": report.get(\"report_type\", \"\")\n",
    "            }\n",
    "        })\n",
    "        \n",
    "        # Financial metrics\n",
    "        financial_entities.append({\n",
    "            \"id\": f\"{report.get('symbol', '')}_revenue\",\n",
    "            \"type\": \"Financial_Metric\",\n",
    "            \"name\": \"Revenue\",\n",
    "            \"properties\": {\n",
    "                \"value\": report.get(\"revenue\", 0),\n",
    "                \"currency\": \"USD\",\n",
    "                \"filing_date\": report.get(\"filing_date\", \"\")\n",
    "            }\n",
    "        })\n",
    "        \n",
    "        financial_entities.append({\n",
    "            \"id\": f\"{report.get('symbol', '')}_net_income\",\n",
    "            \"type\": \"Financial_Metric\",\n",
    "            \"name\": \"Net Income\",\n",
    "            \"properties\": {\n",
    "                \"value\": report.get(\"net_income\", 0),\n",
    "                \"currency\": \"USD\",\n",
    "                \"filing_date\": report.get(\"filing_date\", \"\")\n",
    "            }\n",
    "        })\n",
    "        \n",
    "        # Segments\n",
    "        for segment in report.get(\"segments\", []):\n",
    "            financial_entities.append({\n",
    "                \"id\": f\"{report.get('symbol', '')}_segment_{segment}\",\n",
    "                \"type\": \"Business_Segment\",\n",
    "                \"name\": segment,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            financial_relationships.append({\n",
    "                \"source\": report.get(\"symbol\", \"\"),\n",
    "                \"target\": f\"{report.get('symbol', '')}_segment_{segment}\",\n",
    "                \"type\": \"has_segment\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "        \n",
    "        # Geographic regions\n",
    "        for region in report.get(\"geographic_regions\", []):\n",
    "            financial_entities.append({\n",
    "                \"id\": f\"{report.get('symbol', '')}_region_{region}\",\n",
    "                \"type\": \"Geographic_Region\",\n",
    "                \"name\": region,\n",
    "                \"properties\": {}\n",
    "            })\n",
    "            financial_relationships.append({\n",
    "                \"source\": report.get(\"symbol\", \"\"),\n",
    "                \"target\": f\"{report.get('symbol', '')}_region_{region}\",\n",
    "                \"type\": \"operates_in\",\n",
    "                \"properties\": {}\n",
    "            })\n",
    "        \n",
    "        # Relationships\n",
    "        financial_relationships.append({\n",
    "            \"source\": report.get(\"symbol\", \"\"),\n",
    "            \"target\": f\"{report.get('symbol', '')}_revenue\",\n",
    "            \"type\": \"has_metric\",\n",
    "            \"properties\": {\"filing_date\": report.get(\"filing_date\", \"\")}\n",
    "        })\n",
    "        \n",
    "        financial_relationships.append({\n",
    "            \"source\": report.get(\"symbol\", \"\"),\n",
    "            \"target\": f\"{report.get('symbol', '')}_net_income\",\n",
    "            \"type\": \"has_metric\",\n",
    "            \"properties\": {\"filing_date\": report.get(\"filing_date\", \"\")}\n",
    "        })\n",
    "\n",
    "print(f\"Extracted {len(financial_entities)} financial entities\")\n",
    "print(f\"Extracted {len(financial_relationships)} financial relationships\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build Financial Knowledge Graph\n",
    "\n",
    "Build knowledge graph from financial entities and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "builder = GraphBuilder()\n",
    "graph_analyzer = GraphAnalyzer()\n",
    "centrality_calculator = CentralityCalculator()\n",
    "community_detector = CommunityDetector()\n",
    "connectivity_analyzer = ConnectivityAnalyzer()\n",
    "\n",
    "financial_kg = builder.build(financial_entities, financial_relationships)\n",
    "\n",
    "# Analyze graph structure\n",
    "metrics = graph_analyzer.compute_metrics(financial_kg)\n",
    "centrality_result = centrality_calculator.calculate_degree_centrality(financial_kg)\n",
    "centrality_scores = centrality_result.get('centrality', {})\n",
    "communities = community_detector.detect_communities(financial_kg)\n",
    "connectivity = connectivity_analyzer.analyze_connectivity(financial_kg)\n",
    "\n",
    "print(f\"Built financial knowledge graph\")\n",
    "print(f\"  Entities: {len(financial_kg.get('entities', []))}\")\n",
    "print(f\"  Relationships: {len(financial_kg.get('relationships', []))}\")\n",
    "print(f\"  Graph density: {metrics.get('density', 0):.3f}\")\n",
    "print(f\"  Communities: {len(communities)}\")\n",
    "print(f\"  Central entities: {len([e for e, score in centrality_scores.items() if score > 0])}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Analyze Financial Relationships\n",
    "\n",
    "Analyze financial relationships and generate insights.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temporal_query = TemporalGraphQuery()\n",
    "temporal_pattern_detector = TemporalPatternDetector()\n",
    "inference_engine = InferenceEngine()\n",
    "rule_manager = RuleManager()\n",
    "explanation_generator = ExplanationGenerator()\n",
    "\n",
    "# Temporal analysis\n",
    "start_time = (datetime.now() - timedelta(days=365)).isoformat()\n",
    "end_time = datetime.now().isoformat()\n",
    "\n",
    "temporal_results = temporal_query.query_time_range(\n",
    "    graph=financial_kg,\n",
    "    query=\"Find financial metrics in the last year\",\n",
    "    start_time=start_time,\n",
    "    end_time=end_time\n",
    ")\n",
    "\n",
    "temporal_patterns = temporal_pattern_detector.detect_temporal_patterns(\n",
    "    financial_kg,\n",
    "    pattern_type=\"trend\",\n",
    "    min_frequency=1\n",
    ")\n",
    "\n",
    "# Financial analysis rules\n",
    "inference_engine.add_rule(\"IF revenue > 300000000000 AND net_income > 50000000000 THEN high_performer\")\n",
    "inference_engine.add_rule(\"IF company has_segment Services AND revenue > 20000000000 THEN services_growth\")\n",
    "\n",
    "# Add facts from financial data\n",
    "if parsed_data and parsed_data.get(\"data\"):\n",
    "    report = parsed_data.get(\"data\") if isinstance(parsed_data.get(\"data\"), dict) else parsed_data.get(\"data\")[0] if isinstance(parsed_data.get(\"data\"), list) else {}\n",
    "    if isinstance(report, dict):\n",
    "        inference_engine.add_fact({\n",
    "            \"company\": report.get(\"symbol\", \"\"),\n",
    "            \"revenue\": report.get(\"revenue\", 0),\n",
    "            \"net_income\": report.get(\"net_income\", 0),\n",
    "            \"segments\": report.get(\"segments\", [])\n",
    "        })\n",
    "\n",
    "financial_insights = inference_engine.forward_chain()\n",
    "\n",
    "print(f\"Temporal query returned {len(temporal_results.get('entities', []))} entities\")\n",
    "print(f\"Detected {len(temporal_patterns)} temporal patterns\")\n",
    "print(f\"Generated {len(financial_insights)} financial insights\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Generate Financial Analysis Reports\n",
    "\n",
    "Generate comprehensive financial analysis reports.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_assessor = KGQualityAssessor()\n",
    "json_exporter = JSONExporter()\n",
    "csv_exporter = CSVExporter()\n",
    "rdf_exporter = RDFExporter()\n",
    "report_generator = ReportGenerator()\n",
    "\n",
    "quality_score = quality_assessor.assess_overall_quality(financial_kg)\n",
    "\n",
    "json_exporter.export_knowledge_graph(financial_kg, os.path.join(temp_dir, \"financial_kg.json\"))\n",
    "csv_exporter.export_entities(financial_entities, os.path.join(temp_dir, \"financial_entities.csv\"))\n",
    "rdf_exporter.export_knowledge_graph(financial_kg, os.path.join(temp_dir, \"financial_kg.rdf\"))\n",
    "\n",
    "report_data = {\n",
    "    \"summary\": f\"Financial analysis identified {len(financial_insights)} insights from {len(financial_entities)} entities\",\n",
    "    \"entities_analyzed\": len(financial_entities),\n",
    "    \"relationships\": len(financial_relationships),\n",
    "    \"insights\": len(financial_insights),\n",
    "    \"quality_score\": quality_score.get('overall_score', 0),\n",
    "    \"patterns\": len(temporal_patterns)\n",
    "}\n",
    "\n",
    "report = report_generator.generate_report(report_data, format=\"markdown\")\n",
    "\n",
    "print(f\"Report length: {len(report)} characters\")\n",
    "print(f\"Graph quality score: {quality_score.get('overall_score', 0):.3f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Visualize Financial Analysis\n",
    "\n",
    "Visualize financial knowledge graph and relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kg_visualizer = KGVisualizer()\n",
    "analytics_visualizer = AnalyticsVisualizer()\n",
    "temporal_visualizer = TemporalVisualizer()\n",
    "\n",
    "kg_viz = kg_visualizer.visualize_network(financial_kg, output=\"interactive\")\n",
    "analytics_viz = analytics_visualizer.visualize_analytics(financial_kg, output=\"interactive\")\n",
    "temporal_viz = temporal_visualizer.visualize_timeline(financial_kg, output=\"interactive\")\n",
    "\n",
    "print(f\"Total modules used: 20+\")\n",
    "print(f\"Pipeline complete: Financial Documents → Parse → Extract → Build KG → Analyze Relationships → Generate Insights → Export → Visualize\")\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
